Subject: RE: [nfsv4] pNFS - file commit

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 3 Oct 2005 17:29:52 -0400

> Machines/devices can disappear and be 
> replaced by otters and we should have more elaborate recovery than just 
> wait for them to come back.

We should have more functional recovery than just waiting for them to
come back, but I hope it would not be very elaborate.

When we talk about the async write case, there a couple of important
points to note:

    The only one who has the information necessary to recover is the
    client since he has the dirty data and probably nobody else does.

    If the client redoes his async writes when he many not have to,
    this is only a possible performance issue so that any solution 
    would tend to err on the side of writing these a little bit more
    than necessary.

So I would think that what the spec should say is something like:

When writes to a data server are async and thus require a COMMIT to
be done to that data server, the client is not obligated to wait 
indefinitely for that data server to become available again in order
to do the COMMIT.  The client, rather than hold the asynchronously
written data indefinitely is encouraged to and can make sure that
the data is written by using other paths to that data.  The client 
may write the data to the metadata server, either synchronously or
asynchronously with a subsequent COMMIT.  Once it does this, there 
is no need to wait for the original data server.  In the event that
the data range to be committed is transferred to a different data 
server, as indicated in a new layout, the client may write to
that data server and COMMIT there if necessary, and should consider
the transfer of responsibility for the data to the new server as
strong evidence that that is more efficacious way to get the data 
written.  In either case, once the COMMIT (or synchronous WRITE) is 
validly done (to either data server or metadata server), there is no 
need to continue attempting to COMMIT to the original data server 
or wait for it to become available.  That data server may never be 
visible to the client again. 

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]] 
Sent: Monday, October 03, 2005 4:20 PM
To: Trond Myklebust
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] pNFS - file commit

[email-masked] wrote on 10/03/2005 12:43:42 PM:

> må den 03.10.2005 Klokka 14:49 (-0400) skreiv Jim Zelenka:
> 
> > I agree that COMMIT to the MDS should result in appropriate COMMITs
> > to the data storage devices, just like a regular NFSv4 write to the 
MDS
> > should result in the correct set of writes being directed to the
> > data servers. I don't think that we should mandate that COMMIT always
> > go through the MDS, however.
> 
> Once again. I'd very much like to see a proper discussion/proposal on
> how to implement this by the people advocating it.
> 
First let me say that I don't think the reboot verifier that was use with 
a single server is an appropriate solution for the more complex system 
that we will have with pNFS. Machines/devices can disappear and be 
replaced by otters and we should have more elaborate recovery than just 
wait for them to come back. We now have the MDS that can help in the case 
of a DS failure. One way to do it is discover at LAYOUTCOMMIT that one or 
more DSs are down. It can than provide a different layout or request that 
all writes be directed to the MDS. I know this not a complete proposal we 
all should think some more about this multi device environment recovery 
than just assume that they will just reappear. I know for example that if 
in a cluster file system environment a machine will crash I will recall 
all layout that included that machine and try to replace it with a 
different machine or work without it.
Marc.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 3 Oct 2005 17:30:45 -0500

While not having thought about it very deeply, it seems that 
allowing the client to change the target for a data write may
open a series of either data inconsistencies or build assumptions
into the meta-data and data server interactions.  Is this desired?

For example, if the data server may not be responding to the NFSv4
client because the data server has a network partition between
itself and some other component of the NFSv4(pNFS) server.  If the
client switches to the meta-data server for data writes after
having instantiated a WRITE to the data server, then the modified
data at the data server may still be "in-flight" because of its
own network partition.  As mentioned, the client may see its own
data over-written by the initial WRITE to the data server.  If
that is to be protected against, then there is an assumption
that the meta-data server has control in such a way to determine
that modified data received by the data server in question is
stale in some fashion.

Obviously this has not been a problem in the standard NFSv4 world
since there is only one funnel in which to stuff data but this
ability to redirect seems complicated.  Unnecessarily so?  I am not sure.

Spencer

On Mon, Noveck, Dave wrote:
> > m? den 03.10.2005 Klokka 14:49 (-0400) skreiv Jim Zelenka:
> >