Subject: RE: [nfsv4] pNFS - file commit

From: "Halevy, Benny" <[email-masked]>
Date: Mon, 3 Oct 2005 18:24:34 -0400

Dave, I think we're mixing two distinct cases, both
involve a COMMIT sent to the MDS.

Case 1 is where the MDS itself accepted asynchronous writes
from either pnfs or nfsv4 clients.  In this case the MDS
MUST flush the dirty data to stable storage, i.e. it should
either write the data to the DSs synchronously or write
asynchronously and follow with a COMMIT to the DSs to
ensure the written data reached stable storage.

Case 2 is where some pnfs clients have written data to the DSs
and the MDS received a NFS COMMIT.  In this case I see no reason
to recall layouts to cause pnfs clients to flush data and/or
send COMMITs to DSs to ensure data written directly by the
pnfs clients to the DSs reached stable storage.

For case 2, each pnfs client should be responsible to commit
the dirty data it has written to stable storage in response
to a local trigger like cache buffer replacement, fsync(), or
close() and issue COMMIT or equivalent commands to wherever
it wrote the data to asynchronously (this will typically
by DSs or the MDS in some cases).

Benny

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]]On Behalf Of
> Noveck, Dave
> Sent: Monday, October 03, 2005 9:20 PM
> To: Goodson, Garth; Marc Eshel
> Cc: [email-masked]; [email-masked]
> Subject: RE: [nfsv4] pNFS - file commit
> 
> 
> I agree with Garth.
> 
> With regard to Marc's desire that COMMIT on the MDS propagate
> to the the DS's, I really don't see how this could work.
> 
> One basic job of COMMIT is to return the current verifier
> for server, so that the client can determine whether if the
> server has rebooted.  When it has, the client has to reissue
> the async writes.  If the verifier matches, then the client
> can retire the buffers devoted to the async write incormation
> since it knows they have been securely written to disk on the
> server.
> 
> If you do a COMMIT to the MDS, he can return the verifier
> for himself but what about the verifier for the DS's?  There
> is no place for them to be in the response.  If may be that
> this issue could be worked around in the presence of sessions
> since you have more to work with than you did in v3 as far as
> distinguishing different server instances.  But that would add
> complexity and I don't really see where the benefit of this
> would be.
> 
> First note that this would increase latency since the client 
> asks the MDS to tell the DS to do something that it can just 
> as well ask the DS to do itself: one extra hop.
> 
> There is extra network traffic since the MDS has no good way
> of knowing which DS's the client has done asynchronous writes
> to, or at least he doesn't with additional complexity in the
> MDS-DS interaction.  The client knows to what DS's he has
> written asynchronously and whether those writes have indicated
> that the data was safe after the write or not.  So you have
> extra newtwork traffic, either when the MDS does a COMMIT to
> DS's that don't have any async writes, or when the DS tells
> the MDS when it has async data that needs to be committed.
> 
> I think we have an existing COMMIT mechanism which is designed
> for the environment of a single cleint and a single server,
> and that we should maintain this design.  When that single 
> server is a DS, it works just as it does in v3 and v4.0. 
> Similarly when the single server is an MDS.  I don't see
> the problem with that and changing that model will add extra
> complexity.
> 
> -----Original Message-----
> From: Goodson, Garth 
> Sent: Monday, October 03, 2005 12:32 PM
> To: Marc Eshel
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nfsv4] pNFS - file commit
> 
> 
> Marc Eshel wrote:
> > Yes, I mixed up commit and layout commit. My point was the commit on
> the 
> > data server needs to communicate with the metadata server using the 
> > backend protocol and MDS would have to commit metadata to 
> disk. There 
> > should be a call to the MDS that commits the entire file by calling
> commit 
> > file on all data servers. Is it commit, commit layout or both are
> needed? 
> > I don't see the difference between commit layout and commit file for
> the 
> > file protocol.
> > Marc. 
> > 
> Commit need not propagate to the MDS (thus, we have layoutcommit). 
> COMMIT on the data server just makes asynchronously written 
> data visible
> 
> at that data server, it does not propagate attributes, that is what 
> LAYOUTCOMMIT is for (and hence the difference).  LAYOUTCOMMIT 
> called at 
> the MDS is responsible for syncing the metadata between the 
> MDS and all 
> DSs.  (At least this is how I have viewed it).  As I stated earlier, 
> COMMIT at the metadata server is responsible for pushing out and 
> committing data written through the metadata server to the 
> data servers 
> (I don't think it needs to COMMIT asynchronously written data 
> that was 
> written to the data servers -- that is what COMMIT at the 
> data servers 
> is for).
> 
> -Garth
> 
> 
> > Garth Goodson wrote on 10/02/2005 11:38:24 PM:
> > 
> > 
> >>Marc Eshel wrote:
> >>
> >>>What does regular file commit to the metadata server mean 
> for a file 
> > 
> > with 
> > 
> >>>a layout? I would like it to mean the same as it means for a file 
> > 
> > without 
> > 
> >>>layouts. So the client has the option of calling 
> layout-commit to all
> 
> > 
> > the 
> > 
> >>>data server or have one call to the metadata server that will 
> > 
> > accomplish 
> > 
> >>>the same. The benefit of the one call to the MDS is that 
> the metadata
> 
> > 
> > will 
> > 
> >>>not be written multiple times to disk. The MDS must write 
> metadata to
> 
> > 
> > disk 
> > 
> >>>for each DS layout-commit but when it is driven from the MDS with
> file 
> > 
> > 
> >>>commit it can request all DS to commit their data to disk and when 
> > 
> > they 
> > 
> >>>are all done make one update to the metadata on disk.
> >>>Marc. 
> >>>
> >>
> >>LAYOUTCOMMIT is an operation to the metadata server, only.  The
> backend 
> >>takes care of any synchronization.  COMMIT to the metadata server
> makes 
> >>sure data written asynchronously through the metadata server is
> visible 
> >>at the data servers (upon completion).  COMMIT to the data servers
> does 
> >>what COMMIT normally does.
> >>
> >>-Garth
> >>
> >>
> >>
> >>>_______________________________________________
> >>>nfsv4 mailing list
> >>>[email-masked]
> >>>https://www1.ietf.org/mailman/listinfo/nfsv4
> >>
> >>_______________________________________________
> >>nfsv4 mailing list
> >>[email-masked]
> >>https://www1.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Mon, 03 Oct 2005 15:36:48 -0700

> ensure the written data reached stable storage.
Case 1 is covered in the text.  COMMITs to the metadata server are for 
writes that went to the metadata server.

> pnfs clients to the DSs reached stable storage.
I'm arguing that sending COMMITs to the data servers (to which the 
writes were sent) is no harder and is how NFSv4 works, so why not use it 
(still haven't heard a good argument)?  No need to recall layouts, that 
was in the context of failed DSs.

> by DSs or the MDS in some cases).
Exactly, and it can do this by sending COMMIT to the DS.

-Garth