Subject: [nfsv4] pNFS - file commit

From: Marc Eshel <[email-masked]>
Date: Sun, 2 Oct 2005 13:45:32 -0700

What does regular file commit to the metadata server mean for a file with 
a layout? I would like it to mean the same as it means for a file without 
layouts. So the client has the option of calling layout-commit to all the 
data server or have one call to the metadata server that will accomplish 
the same. The benefit of the one call to the MDS is that the metadata will 
not be written multiple times to disk. The MDS must write metadata to disk 
for each DS layout-commit but when it is driven from the MDS with file 
commit it can request all DS to commit their data to disk and when they 
are all done make one update to the metadata on disk.
Marc. 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Sun, 02 Oct 2005 23:38:24 -0700

> are all done make one update to the metadata on disk.

LAYOUTCOMMIT is an operation to the metadata server, only.  The backend 
takes care of any synchronization.  COMMIT to the metadata server makes 
sure data written asynchronously through the metadata server is visible 
at the data servers (upon completion).  COMMIT to the data servers does 
what COMMIT normally does.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 09:13:18 -0700

Yes, I mixed up commit and layout commit. My point was the commit on the 
data server needs to communicate with the metadata server using the 
backend protocol and MDS would have to commit metadata to disk. There 
should be a call to the MDS that commits the entire file by calling commit 
file on all data servers. Is it commit, commit layout or both are needed? 
I don't see the difference between commit layout and commit file for the 
file protocol.
Marc.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 03 Oct 2005 09:31:36 -0700

> I don't see the difference between commit layout and commit file for the 
Commit need not propagate to the MDS (thus, we have layoutcommit). 
COMMIT on the data server just makes asynchronously written data visible 
at that data server, it does not propagate attributes, that is what 
LAYOUTCOMMIT is for (and hence the difference).  LAYOUTCOMMIT called at 
the MDS is responsible for syncing the metadata between the MDS and all 
DSs.  (At least this is how I have viewed it).  As I stated earlier, 
COMMIT at the metadata server is responsible for pushing out and 
committing data written through the metadata server to the data servers 
(I don't think it needs to COMMIT asynchronously written data that was 
written to the data servers -- that is what COMMIT at the data servers 
is for).

-Garth

>>>What does regular file commit to the metadata server mean for a file 
> 
> with 
> 
>>>a layout? I would like it to mean the same as it means for a file 
> 
> without 
> 
>>>layouts. So the client has the option of calling layout-commit to all 
> 
> the 
> 
>>>data server or have one call to the metadata server that will 
> 
> accomplish 
> 
>>>the same. The benefit of the one call to the MDS is that the metadata 
> 
> will 
> 
>>>not be written multiple times to disk. The MDS must write metadata to 
> 
> disk 
> 
>>>commit it can request all DS to commit their data to disk and when 
> 
> they 
>

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 3 Oct 2005 12:33:56 -0400 (EDT)

I think both are needed.  COMMIT syncs the data to disk.  LAYOUTCOMMIT
commits the layout and updates the server with the last byte written.
For the file layout, the length of the file still need to be updated, so a
LAYOUTCOMMIT is still necessary.  I would assume they can be sent in the
same compound.

Dean

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 3 Oct 2005 12:40:30 -0400 (EDT)

Just to clarify, I agree with Marc about full file commit to the MDS
commits all data on all data servers.  I think this could be extended to a
byte-range too.  Am I right to assume it is implementation dependent
whether a client sends his commits through the MDS or directly to the data
servers?

Dean

---

From: Garth Goodson <[email-masked]>
Date: Mon, 03 Oct 2005 09:45:54 -0700

Dean Hildebrand wrote:
> commits all data on all data servers.  

I don't agree.  We want to limit the operations that dictate how the 
control protocol is to be used.  In this case, it is just as simple to 
insert a COMMIT into some compound going to the data servers.

See my previous post (pasted below) for how I believe it should work.

-Garth

(I don't think the MDS needs to COMMIT asynchronously written data that 
was written to the data servers -- that is what COMMIT at the data 
servers is for).

I think this could be extended to a

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 09:48:10 -0700

> at that data server, it does not propagate attributes, that is what 

I assume you mean it makes it visible from all data servers ? I am not 
sure how you do that without communicating with the MDS.

> written to the data servers -- that is what COMMIT at the data servers 

So what is the state of a file after writing to a DS and committing it on 
that DS and than that DS crashes before committing on the MDS. Is the data 
there or not ?

Marc.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 03 Oct 2005 09:55:40 -0700

Marc Eshel wrote:
>>>Yes, I mixed up commit and layout commit. My point was the commit on 
> 
> the 
> 
>>>should be a call to the MDS that commits the entire file by calling 
> 
> commit 
> 
>>>file on all data servers. Is it commit, commit layout or both are 
> 
> needed? 
> 
>>>I don't see the difference between commit layout and commit file for 
> 
> the 
> 
>>>file protocol.
>>>Marc. 
>>>
>>

I don't know what you mean by 'visible from all data servers'.  There is 
a data server hosting a stripe of data.  A COMMIT to that data server 
make asynchronously written data stable on THAT data server.
> that DS and than that DS crashes before committing on the MDS. Is the data 
In the worst case, the new data is on the data server, but the 
attributes may not have been updated on the MDS.  Thus if it was an 
overwrite, the data is visible, if it was a file extension, the MDS 
still has the old file length.  However, it is quite possible, and 
highly likely that implementations will provide stronger guarantees. 
For example, if the MDS knows a DS has crashed, and which files were 
open on it, it can scan the attributes of the local DS files when that 
DS comes back online and update its attributes.  There are many many 
options for providing stronger consistency guarantees...  But these 
aren't being dictated by the specification.  As well, if the client sees 
that the DS has crashed, it can either attempt to write through the 
metadata server, or it can do a LAYOUTCOMMIT with the new size through 
the metadata server.

-Garth

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 03 Oct 2005 13:04:22 -0400

> whether a client sends his commits through the MDS or directly to the data

Currently, the NFS server will return a verifier with each unstable
WRITE and COMMIT operation that allows the client to detect if the
server has rebooted (in which case it needs to resend all those
uncommitted writes).
How do you propose to extend this scheme to allow the client to detect
that one of the data servers has rebooted?

Cheers,
  Trond

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 3 Oct 2005 13:13:47 -0400 (EDT)

> See my previous post (pasted below) for how I believe it should work.

What worries me here is that we are pushing too much responsibility into
the client layout drivers.  How do the layout drivers that work with non
NFSv4 files, e.g., OBD, commit data to disk if not through the NFSv4
COMMIT call to the NFSv4 server?  These layout drivers could write their
own COMMIT/sync operation, but are we mandating that they do so?  This
could greatly increase the size of a layout driver and discourage porting
them from platform to platform.  From all my conversations with people
about the functionality of the layout driver, I get the impression that we
want to make it as simple and lightweight as possible.  Having a layout
driver that simply supports reading and writing data seems advantageous in
certain situations.  My reading of the NFSv4 spec is that COMMIT syncs
data to disk, not just data written through the NFSv4 server.  If an
implementation chooses not to send COMMITs to the MDS, I think that should
be implementation dependent.

Dean

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 10:29:15 -0700

Trond Myklebust <[email-masked]> wrote on 10/03/2005 10:04:22 

> that one of the data servers has rebooted?
How would the client recover from this situation and whatever the recovery 
scheme is why can't it be triggered by and error from the commit to the 
MDS.
Marc.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 03 Oct 2005 13:38:43 -0400

> scheme is why can't it be triggered by and error from the commit to the 

If the client is sending COMMIT operations to the data servers, then the
recovery would presumably be the same as for standard NFSv3/v4 unstable
writes: upon COMMIT, the client detects that the verifier has changed,
and knows that it has to resend the writes.

If the client only sends COMMIT to the metadata server and expects that
to be propagated to the data servers, then you need some replacement
scheme that can tell the client that it needs to resend unstable writes.
What are you and Dean proposing?

Cheers,
  Trond

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 10:49:59 -0700

Trond Myklebust <[email-masked]> wrote on 10/03/2005 10:38:43 

> What are you and Dean proposing?
There can be few alternative because it is possible that the data server 
that crashed will not come back. One simple approach is that if the commit 
to the MDS fails the client rewrites all the dirty data to the MDS.
Marc.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 03 Oct 2005 14:06:22 -0400

> to the MDS fails the client rewrites all the dirty data to the MDS.

That is irrelevant as far as the COMMIT operation is concerned. The
function of COMMIT is twofold:

 1) Flush any dirty data on the server to permanent storage

 2) Allow the client to detect if the NFS server rebooted at any time
before the dirty data from an unstable WRITE operation was committed to
disk (see the discussion on Page 146 of RFC3530).

So I repeat my question: if you are sending WRITEs to the data servers,
and the data servers are allowed to cache those writes, how would you
implement point (2) above if the client is supposed to send COMMIT only
to the metadata server?

Cheers,
  Trond

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 11:12:02 -0700

> implement point (2) above if the client is supposed to send COMMIT only
The MDS should be able to tell if a DS that is a part of a layout it gave 
has crashed/rebooted
and than fail the commit.
Marc.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 03 Oct 2005 14:36:35 -0400

> and than fail the commit.

Firstly, that would be a change in behaviour w.r.t. NFSv4. COMMIT does
not fail just because the server has rebooted. Rather the client
determines that the reboot occurred by examining the verifier.

Secondly, nowhere does it say in the pNFS draft that I have to do a
COMMIT before returning the layout. I might decide to WRITE unstable
data, return the layout, grab the layout again, then write some more
data before committing. The current scheme (COMMIT sent to the data
server) enables me to easily detect if the server rebooted while I
wasn't holding the layout.

Cheers,
  Trond

---

From: Jim Zelenka <[email-masked]>
Date: Mon, 03 Oct 2005 14:49:48 -0400

Comments in-line

> own COMMIT/sync operation, but are we mandating that they do so?

Yes, I believe that we are.

> This
> them from platform to platform.

I think that this is a good concern to have. IMO, the thing to worry about
is not the number of operations that a layout driver must provide, but the
complexity of these operations. For example, consider the difference between
keeping the aggregation schema specific to the layout driver versus
generalizing it across drivers. In the former case, the WRITE operation
that a layout driver must provide has to map a "logical" write to a
virtual file into a set of WRITEs to specific storage devices, along with
all of the logic to actually perform these WRITEs.

> From all my conversations with people

I agree that COMMIT to the MDS should result in appropriate COMMITs
to the data storage devices, just like a regular NFSv4 write to the MDS
should result in the correct set of writes being directed to the
data servers. I don't think that we should mandate that COMMIT always
go through the MDS, however.

[ rest of the message snipped here ]

-jim

-- 
Jim Zelenka
Software Engineer, Panasas, Inc.
Accelerating Time to Results(TM) with Clustered Storage
www.panasas.com
412.323-3500

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 03 Oct 2005 15:43:42 -0400

> go through the MDS, however.

Once again. I'd very much like to see a proper discussion/proposal on
how to implement this by the people advocating it.

Cheers,
  Trond

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 13:19:53 -0700

> how to implement this by the people advocating it.
First let me say that I don't think the reboot verifier that was use with 
a single server is an appropriate solution for the more complex system 
that we will have with pNFS. Machines/devices can disappear and be 
replaced by otters and we should have more elaborate recovery than just 
wait for them to come back. We now have the MDS that can help in the case 
of a DS failure. One way to do it is discover at LAYOUTCOMMIT that one or 
more DSs are down. It can than provide a different layout or request that 
all writes be directed to the MDS. I know this not a complete proposal we 
all should think some more about this multi device environment recovery 
than just assume that they will just reappear. I know for example that if 
in a cluster file system environment a machine will crash I will recall 
all layout that included that machine and try to replace it with a 
different machine or work without it.
Marc.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 03 Oct 2005 13:30:52 -0700

Marc Eshel wrote:
>>>to the data storage devices, just like a regular NFSv4 write to the 
> 
> MDS
> 
> all writes be directed to the MDS.

Such as: recall layout, client sees recall, client writes dirty data 
through metadata server, client obtains new layout.  My point is, let's 
use existing mechanisms before inventing new ones.  I agree we probably 
need more text in this regard, and I'm open to accepting text.

And there is no reason to limit this discovery to LAYOUTCOMMIT, the MDS 
is free to perform the above at any time.  Let's not tie together things 
that needn't be...

-Garth

  I know this not a complete proposal we

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 13:54:35 -0700

I don't see a section on Data Server Recovery is that an overlook or there 
is some text about it in some other section of the document?

---

From: Brent Welch <[email-masked]>
Date: Wed, 05 Oct 2005 17:54:27 -0700

Gee - I'm glad I didn't read this thread until today.  I will
try to summarize where I see things.  I'm not really looking
for more debate :-)

The problems I see with COMMIT to the MDS triggering a commit
on the data servers are:
(a) it doesn't work well with the NFS commit verifier protocol
(b) it probably doesn't save any network traffic or disk I/O
(c) it makes assumptions about the back-end protocol between the
    MDS and the DS that I don't think we want to mandate.
(d) ultimately I think it'll be more efficient for the client
    to send COMMIT to the DS - it can always piggyback those
    with the last write.

In my view, the system as a whole is simplest if the client
does a series of WRITES to the data server.  If it chooses to
do async writes, then it follows with a COMMIT.  That's exactly
how it behaves with respect to regular NFS, so it doesn't
seem like a big burden to place on the client.  It is simple.
Dean indicates this is what PVFS does. 

Dave Noveck described well how the client is in
the best position to know what data has been COMMITed, and
is the one in the best position to deal with failures by
retrying things.

I think this discussion does highlight the fact that we haven't
tried to specify the behavior of the client layoutdriver
if much detail.  Marc raised the issue about "where is the
section on data server failures", for example.  Again, we
we do the WRITE followed by COMMIT to the DS, then we just
handle this case by saying it is just like NFSv4.

I really, really don't want to add the semantic that Marc
asked for (sorry, Marc) that LAYOUTCOMMIT to the MDS
serves as a COMMIT to the data servers.  The intent of that
operation is to commit attributes, and it assumes (and it is
noted in the text) that the data is written (through) to
the data servers before the LAYOUTCOMMIT occurs.

Finally, the issue that Spencer raised is interesting.
Again, I think Dave has a good analysis.  My own simple reasoning
is that if the client replays the writes to the MDS, then
ultimately the MDS will be writing the data through to the
same data server as the client was originally trying to write to.
I can't think of a realistic failure scenario that would cause
that data server (or the network) to buffer the write in such a
way that, after it crashed and recovered, it was able to accept
the write from the MDS first, and then process the "in limbo" write.

In the case where the back end can write the same data to different
data servers, then it had dang well protect itself from having
an async write linger in a failing data server, only to re-emerge
at an arbitrary point in time later when the rest of the clients
have been using some alternate server.  That would be bad, but is
out of scope for the pNFS spec, I think.  Again, I like Dave's
chain of reasoning (not repeated here) that you really need to
be able to fall back to writing data to the MDS.

One final note - Dave had a question about how error status is
conveyed from the clients to the MDS.  This is the role of the
parameters to LAYOUTCOMMIT, which allow for a layout-specific
type that can convey this kind of information.
   The "layoutupdate" argument to LAYOUTCOMMIT provides a mechanism for
   a client to provide layout specific updates to the metadata server.
(You may recall that my original proposal had a "report I/O error"
operation, but we folded it into LAYOUTCOMMIT by now.)

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Marc Eshel <[email-masked]>
Date: Thu, 6 Oct 2005 20:36:51 -0700

> (b) it probably doesn't save any network traffic or disk I/O

I am in the process of prototyping pNFS with a cluster file system, so 
usually the problems that a I raise are problems that I see and believe me 
it does save disk I/O. There is more metadata than the few attributes that 
we discuss (mtime, atime, file size, ...) the indirect blocks are also 
metadata and if it is not updated in a coordinated way by one node you 
will do more I/Os to disk. I don't want to force everyone to something 
that might be unique to cluster file system (unless it is critical :) so 
for now I will just send the commit from the DS to the MDS to do a 
coordinated commit.
 
>     MDS and the DS that I don't think we want to mandate.

We do make many assumption about the back-end, we just have different 
models of the back-end.

> handle this case by saying it is just like NFSv4.
 
One good thing that came out from this discussion is that we are starting 
to address DS failure. I like Dave proposal.

[snip]