Subject: RE: [nfsv4] pNFS - file commit

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 3 Oct 2005 14:20:14 -0400

I agree with Garth.

With regard to Marc's desire that COMMIT on the MDS propagate
to the the DS's, I really don't see how this could work.

One basic job of COMMIT is to return the current verifier
for server, so that the client can determine whether if the
server has rebooted.  When it has, the client has to reissue
the async writes.  If the verifier matches, then the client
can retire the buffers devoted to the async write incormation
since it knows they have been securely written to disk on the
server.

If you do a COMMIT to the MDS, he can return the verifier
for himself but what about the verifier for the DS's?  There
is no place for them to be in the response.  If may be that
this issue could be worked around in the presence of sessions
since you have more to work with than you did in v3 as far as
distinguishing different server instances.  But that would add
complexity and I don't really see where the benefit of this
would be.

First note that this would increase latency since the client 
asks the MDS to tell the DS to do something that it can just 
as well ask the DS to do itself: one extra hop.

There is extra network traffic since the MDS has no good way
of knowing which DS's the client has done asynchronous writes
to, or at least he doesn't with additional complexity in the
MDS-DS interaction.  The client knows to what DS's he has
written asynchronously and whether those writes have indicated
that the data was safe after the write or not.  So you have
extra newtwork traffic, either when the MDS does a COMMIT to
DS's that don't have any async writes, or when the DS tells
the MDS when it has async data that needs to be committed.

I think we have an existing COMMIT mechanism which is designed
for the environment of a single cleint and a single server,
and that we should maintain this design.  When that single 
server is a DS, it works just as it does in v3 and v4.0. 
Similarly when the single server is an MDS.  I don't see
the problem with that and changing that model will add extra
complexity.

-----Original Message-----
From: Goodson, Garth 
Sent: Monday, October 03, 2005 12:32 PM
To: Marc Eshel
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] pNFS - file commit

Marc Eshel wrote:
> Yes, I mixed up commit and layout commit. My point was the commit on
the 
> data server needs to communicate with the metadata server using the 
> backend protocol and MDS would have to commit metadata to disk. There 
> should be a call to the MDS that commits the entire file by calling
commit 
> file on all data servers. Is it commit, commit layout or both are
needed? 
> I don't see the difference between commit layout and commit file for
the 
> file protocol.
> Marc. 
Commit need not propagate to the MDS (thus, we have layoutcommit). 
COMMIT on the data server just makes asynchronously written data visible

at that data server, it does not propagate attributes, that is what 
LAYOUTCOMMIT is for (and hence the difference).  LAYOUTCOMMIT called at 
the MDS is responsible for syncing the metadata between the MDS and all 
DSs.  (At least this is how I have viewed it).  As I stated earlier, 
COMMIT at the metadata server is responsible for pushing out and 
committing data written through the metadata server to the data servers 
(I don't think it needs to COMMIT asynchronously written data that was 
written to the data servers -- that is what COMMIT at the data servers 
is for).

-Garth

> Garth Goodson wrote on 10/02/2005 11:38:24 PM:
> 
> 
>>Marc Eshel wrote:
>>
>>>What does regular file commit to the metadata server mean for a file 
> 
> with 
> 
>>>a layout? I would like it to mean the same as it means for a file 
> 
> without 
> 
>>>layouts. So the client has the option of calling layout-commit to all

> 
> the 
> 
>>>data server or have one call to the metadata server that will 
> 
> accomplish 
> 
>>>the same. The benefit of the one call to the MDS is that the metadata

> 
> will 
> 
>>>not be written multiple times to disk. The MDS must write metadata to

> 
> disk 
> 
>>>for each DS layout-commit but when it is driven from the MDS with
file 
> 
> 
>>>commit it can request all DS to commit their data to disk and when 
> 
> they 
> 
>>>are all done make one update to the metadata on disk.
>>>Marc. 
>>>
>>
>>LAYOUTCOMMIT is an operation to the metadata server, only.  The
backend 
>>takes care of any synchronization.  COMMIT to the metadata server
makes 
>>sure data written asynchronously through the metadata server is
visible 
>>at the data servers (upon completion).  COMMIT to the data servers
does 
>>what COMMIT normally does.
>>
>>-Garth
>>
>>
>>
>>>_______________________________________________
>>>nfsv4 mailing list
>>>[email-masked]
>>>https://www1.ietf.org/mailman/listinfo/nfsv4
>>
>>_______________________________________________
>>nfsv4 mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 3 Oct 2005 14:25:54 -0400

From: Marc Eshel [mailto:[email-masked]] 
> [email-masked] wrote on 10/03/2005 11:06:22 AM:
> 
> > ..... The 
> > function of COMMIT is twofold:
> > 
> >  1) Flush any dirty data on the server to permanent storage
> > 
> >  2) Allow the client to detect if the NFS server rebooted 
> at any time 
> > before the dirty data from an unstable WRITE operation was 
> committed 
> > to disk (see the discussion on Page 146 of RFC3530).
> > 
> > So I repeat my question: if you are sending WRITEs to the data 
> > servers, and the data servers are allowed to cache those 
> writes, how 
> > would you implement point (2) above if the client is 
> supposed to send 
> > COMMIT only to the metadata server?
> > 
> The MDS should be able to tell if a DS that is a part of a 
> layout it gave has crashed/rebooted and than fail the commit.
> Marc.

I disagree.  If the client has been sending data (via WRITE) to a DS,
why can't it send the COMMIT to that same DS?  If it's sending it to an
OBS or a block device, they presumably have their own COMMIT-like
functionality that allows the client to manage how it cleans pages.
(Yes, yes, in the case of block devices, that functionality doesn't have
much to do with the storage and more to do with client threading etc.)

It's clear that if the different clients are all front-ending the same
cluster FS, then it could be more convenient for the server if the MDS
did all the disk-flushing rather than lots of separate clients.  But
that's certainly not the way the WRITE/COMMIT pair work today with WRITE
returning a verifier and COMMIT using it.  The COMMIT sent to the MDS
would have no meaningful verifier to work with.

		Craig

---

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 3 Oct 2005 14:39:13 -0400

> 
> My motivation as I said in the first note is to save I/Os to 
> disk. I am not against commit to data server it should be 
> allowed and some times it is required before client can free 
> memory of written data. All I want is an operation on the MDS 
> that will coordinate and make sure that all dirty data on the 
> data servers is committed to disk and than update the 
> metadata on the MDS once. It works better for cluster file systems.
> Marc.

I don't think that it saves I/Os to disk to do it this way.  The DSs
know what dirty pages they have and won't store the clean ones in
processing their COMMITs--and those dirty pages will just be wherever
the clients have been writing, to which they've been directed by the
MDS, so there shouldn't be contention for any given region of the file:
there should be at most one dirty DS page for any given page of the
file, right?

It seems to work really well (in principle, of course) to me: send a lot
of COMMIT messages to the various DSs that have been receiving your
data, then process the replies before thinking that an entire file is
stored back.  It wouldn't matter whether the DSs are part of a cluster
file system or not.  Now maybe you have available a cluster file system
operation that can get all the DSs to store their dirty pages back for a
given range of a given file, but that isn't tied to the COMMIT verifier
scheme in the same way that NFS expects.  Would you prefer an FSYNC
operation that was to be issued to the MDS?  How would a client know
that it should use that rather than COMMITs?  And why wouldn't the
client want to use COMMIT to the various DSs, since it can get better
throughput that way--pages wouldn't have to be COMMITted on more than
one server, and they can generally be reused as soon as the COMMIT reply
arrives.  In the big-FSYNC case, the client would have to resynch
everything if the big FSYNC fails.  So using COMMITs to the DSs gets you
better failure isolation.

		Craig

---

From: "Halevy, Benny" <[email-masked]>
Date: Mon, 3 Oct 2005 18:03:20 -0400

Garth Goodson wrote:
> Marc Eshel wrote:
> > Yes, I mixed up commit and layout commit. My point was the 
> commit on the 
> > should be a call to the MDS that commits the entire file by 
> calling commit 
> > file on all data servers. Is it commit, commit layout or 
> both are needed? 
> > I don't see the difference between commit layout and commit 
> file for the 
> > file protocol.
> > Marc. 
> > 

That COMMITted data may have not been written over pnfs at all.
How would a regular nfsv4 client commit the data at the data servers
if it doesn't support pnfs and have no access to them?

Benny

> >>>for each DS layout-commit but when it is driven from the 
> MDS with file 
> > 
> > 
> >>LAYOUTCOMMIT is an operation to the metadata server, only.  
> The backend 
> >>takes care of any synchronization.  COMMIT to the metadata 
> server makes 
> >>sure data written asynchronously through the metadata 
> server is visible 
> >>at the data servers (upon completion).  COMMIT to the data 
> servers does

---

From: "Halevy, Benny" <[email-masked]>
Date: Mon, 3 Oct 2005 18:13:11 -0400

Dean Hildebrand wrote:
> > Dean Hildebrand wrote:
> > > Just to clarify, I agree with Marc about full file commit 
> to the MDS
> > > commits all data on all data servers.
> >
> > I don't agree.  We want to limit the operations that dictate how the
> > control protocol is to be used.  In this case, it is just 
> as simple to
> > insert a COMMIT into some compound going to the data servers.
> >
> > See my previous post (pasted below) for how I believe it 
> should work.
> 
> What worries me here is that we are pushing too much 
> responsibility into
> the client layout drivers.  How do the layout drivers that 
> work with non
> NFSv4 files, e.g., OBD, commit data to disk if not through the NFSv4
> COMMIT call to the NFSv4 server?  These layout drivers could 
> write their
> own COMMIT/sync operation, but are we mandating that they do so?  This
> could greatly increase the size of a layout driver and 
> discourage porting
> them from platform to platform.  From all my conversations with people
> about the functionality of the layout driver, I get the 
> impression that we
> want to make it as simple and lightweight as possible.  
> Having a layout
> driver that simply supports reading and writing data seems 
> advantageous in
> certain situations.  My reading of the NFSv4 spec is that COMMIT syncs
> data to disk, not just data written through the NFSv4 server.  If an
> implementation chooses not to send COMMITs to the MDS, I 
> think that should
> be implementation dependent.

The OSD protocol has a FLUSH operation which is the equivalent of NFS
COMMIT.  IMO, it isn't a big deal for the client to commit data to
stable storage on the data servers.

Benny

> Dean
> > (I don't think the MDS needs to COMMIT asynchronously 
> written data that
> > was written to the data servers -- that is what COMMIT at the data
> > servers is for).
> >
> >
> >
> >
> > I think this could be extended to a
> > > byte-range too.  Am I right to assume it is 
> implementation dependent
> > > whether a client sends his commits through the MDS or 
> directly to the data
> > > servers?
> > >
> > > Dean
> > >
> > > On Mon, 3 Oct 2005, Dean Hildebrand wrote:
> > >
> > >
> > >>I think both are needed.  COMMIT syncs the data to disk.  
> LAYOUTCOMMIT
> > >>commits the layout and updates the server with the last 
> byte written.
> > >>For the file layout, the length of the file still need to 
> be updated, so a
> > >>LAYOUTCOMMIT is still necessary.  I would assume they can 
> be sent in the
> > >>same compound.
> > >>
> > >>Dean
> > >>
> > >>On Mon, 3 Oct 2005, Marc Eshel wrote:
> > >>
> > >>
> > >>>Yes, I mixed up commit and layout commit. My point was 
> the commit on the
> > >>>should be a call to the MDS that commits the entire file 
> by calling commit
> > >>>I don't see the difference between commit layout and 
> commit file for the
> > >>>file protocol.
> > >>>Marc.
> > >>>
> > >>>>>for each DS layout-commit but when it is driven from 
> the MDS with file
> > >>>
> > >>>>LAYOUTCOMMIT is an operation to the metadata server, 
> only.  The backend
> > >>>>takes care of any synchronization.  COMMIT to the 
> metadata server makes
> > >>>>at the data servers (upon completion).  COMMIT to the 
> data servers does

---

From: "Halevy, Benny" <[email-masked]>
Date: Tue, 4 Oct 2005 11:00:17 -0400

Trond Myklebust wrote:
> må den 03.10.2005 Klokka 19:23 (-0400) skreiv Halevy, Benny:
> > Garth Goodson wrote:
> > > Halevy, Benny wrote:
> > > > Dave, I think we're mixing two distinct cases, both
> > > > involve a COMMIT sent to the MDS.
> > > > 
> > > > Case 1 is where the MDS itself accepted asynchronous writes
> > > > from either pnfs or nfsv4 clients.  In this case the MDS
> > > > MUST flush the dirty data to stable storage, i.e. it should
> > > > either write the data to the DSs synchronously or write
> > > > asynchronously and follow with a COMMIT to the DSs to
> > > > ensure the written data reached stable storage.
> > > > 
> > > Case 1 is covered in the text.  COMMITs to the metadata 
> > > server are for 
> > > writes that went to the metadata server.
> > > 
> > 
> > Not exactly.  The current text does not require the MDS
> > to ensure data reached stable storage on the DSs which
> > I think we both agree it should.  It only requires it
> > to write the data to the DSs so it would be visible
> > there which is not quite the same thing. Quoting
> > from draft-welch-pnfs-ops-04.txt:
> 
> If the client is writing to the metadata server, then it is using
> standard NFSv4, not the pNFS extensions. The behaviour of 
> COMMIT in this
> situation is therefore already covered by RFC3530.

Agreed, and it need not change.  However, I think this issue
deserves a note in the pnfs operations RFC that will clarify
what is expected from the metadata server in the pnfs architecture
where both the MDS and pnfs clients access data servers and they
depend on each other to achieve a consistent view of the data.

Benny