Subject: RE: [nfsv4] pNFS commits with cluster file systems

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 26 Mar 2006 11:51:29 -0500

This is about what we agreed to.  Some comments below.

> A new flag in the file layout (COMMIT_THROUGH_MDS) will allow the MDS to 
> tell the client of a preferred way of doing COMMIT. If this flag is set, 
> the client SHOULD send COMMIT to the MDS instead of sending it to the 
> same DS to which the associated WRITEs were sent. 

If it is not set the client must send COMMIT's to the DS and a COMMIT
to the MDS will be rejected (with what error I wonder).

> In order to maintain 
> the current NFSv4 commit and recovery model, all the DS'es MUST return 
> the same verifier for all WRITEs in a given file layout. 

Since in the next sentence you talk about the verifier changing, i.e 
not being "the same", I think you want to talk about returning a "common"
(i.e. globally maintained among the MDS and DS's) verifier.

There is an issue here about the scope of verifiers.  There is some text
in the current spec which implies they are uniform for a server as a
whole.  When you add in your uniformity requirement these together 
could force them to be of much wider scope than you want.

For example, MDS A uses DS's P, Q, R, S and MDS B uses DS's S, T, U, V
and MDS C uses DS's V, W, X, Y, Z.  Do A, B, C, P, Q, R, S, T, U, V, W,
X, Y, Z all have to share a verifier?  Help!

The verifier scope issue also turns up in migration in that if I 
migrate an fs to a server, I might want to move verifier information,
making server uniformity untenable.  Anyway I think we want to 
either say that the client should not compare verifiers from different
fh's, or say that the client should not compare verifiers from 
different fs's.  In what I'm currently drafting, I'm going to assume
the latter but I'll create an issue to track this and come to a
working group consensus about verifiers.

> Whenever any DS 
> that is referenced in the layout reboots, the verifier MUST be changed.

I think a better way to express this is to build on what COMMIT says:

   The server must vary the value of the write
   verifier at each server event or instantiation that may lead to a
   loss of uncommitted data. 

and say,

   The value of the write verifier must be changed at each server event
   (including events on the MDS and any of the DS's) and server
   instantiations of whatever kind that can possibly lead to a loss 
   of uncommitted data.

> The single COMMIT to the MDS will return a verifier and the client must 
> compare it to all the verifiers from the WRITEs and fail the COMMIT if
> there is any mismatched verifiers. If COMMIT to the MDS fails, the client 
> MUST reissue WRITEs for all the dirty data in the file as follows. 

Your assumption here is that a server will accumulate writes with multiple
different verifier values for the same file.  My understanding of this,
although the spec isn't that clear is that once you saw a different verifier
you would note that previous writes with a different verifier were already
lost and so you would only have a single outstanding verifier associated
with unstable writes to be committed.
 
> If the 
> layout has not been recalled, all WRITEs that have a different verifier 
> than the one returned by COMMIT MUST be rewritten through the DS. 

I think you can always write using the MDS, although it would be a poor 
choice if you have a layout.

> If the 
> layout has been recalled, the client MUST write the data through the MDS.

The client could try to get a new layout and use that.  The client doesn't
really have to write the data at all.  We are not saying "You MUST write 
the data or we will report you to the IESG", but rather "You know that 
data you asked us to write.  Somehow, we can't find it and fear it is 
lost.  Deal with it.  How is your choice.".

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]]
Sent: Saturday, March 25, 2006 8:05 PM
To: Goodson, Garth
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS commits with cluster file systems

This note is to summarize the agreement we had at the IETF regarding an 
alternative way to commit data that should improve performance for 
cluster file systems. This applies to file layouts only.

A new flag in the file layout (COMMIT_THROUGH_MDS) will allow the MDS to 
tell the client of a preferred way of doing COMMIT. If this flag is set, 
the client SHOULD send COMMIT to the MDS instead of sending it to the 
same DS to which the associated WRITEs were sent. In order to maintain 
the current NFSv4 commit and recovery model, all the DS'es MUST return 
the same verifier for all WRITEs in a given file layout. Whenever any DS 
that is referenced in the layout reboots, the verifier MUST be changed. 
The single COMMIT to the MDS will return a verifier and the client must 
compare it to all the verifiers from the WRITEs and fail the COMMIT if
there is any mismatched verifiers. If COMMIT to the MDS fails, the client 
MUST reissue WRITEs for all the dirty data in the file as follows. If the 
layout has not been recalled, all WRITEs that have a different verifier 
than the one returned by COMMIT MUST be rewritten through the DS. If the 
layout has been recalled, the client MUST write the data through the MDS.

If there are no new objections than I will put in a format more 
appropriate for the draft.

Marc.

Garth Goodson <[email-masked]> wrote on 03/22/2006 07:38:03 PM:

> One addendum to my last post on this subject.
> 
> In order to properly recover from server failures, nfsv4 relies on 
> COMMITs validating a verifier passed into COMMIT which was generated by 
> the WRITE.
> 
> The solution used by a cluster file system, must fail a MDS commit if it 

> detects that a DS holding an unstable write has failed (and rebooted). 
> To do this, the implementation is most likely going to require some 
> state associated with the write (for NFS this is the write verifier) to 
> be passed into the commit for verification.  A proposal for a solution 
> to the cluster FS commit must address the problem solved by verifiers.
> 
> -Garth
> 
> Garth Goodson wrote:
> > A discussion arose at the IETF meeting today regarding commits and 
> > cluster file systems.  I'll try to summarize:
> > 
> > NOTE: this is only relavent to the file-layout type.
> > 
> > The Problem
> > -----------
> > Some background first... In a cluster file system metadata must be 
> > committed through a single 'metanode'.  Thus, each NFSv4 COMMIT 
> > operation to a data server (DS) triggers a commit message to this 
> > metanode to commit the data and the portion of the metadata that 
pertain 
> > to that DS.  The current pNFS draft requires that COMMITs go to each 
> > data server that holds dirty data and that they maintain NFSv4 COMMIT 
> > semantics (i.e., they commit both data and metadata).
> > 
> > Marc Eshel has pointed out that this has a dire performance impact for 

> > cluster file systems: the DS COMMITs trigger commits to the metanode 
and 
> > may result in 2 I/Os being performed (one to write out dirty data, one 

> > for the metanode to write out the metadata describing that data).  The 

> > multiple I/Os for the metadata (one from each DS) can be avoided 
through 
> > a single commit to the metanode once all the data has been synced. 
> > Ultimately, these systems would like a single commit to the metanode 
to 
> > commit all dirty data.  (The current model is slightly backwards in 
that 
> > the DSs tell the metanode when to commit rather than the other way 
around).
> > 
> > Possible Solutions
> > ------------------
> > 
> > 1)
> > Marc initially proposed a 'data only' commit be sent to the DSs and 
> > another commit be sent to the metadata server (MDS), to sync the 
> > metadata, once the DS commits have completed.  This would batch the, 
> > possibly many, I/Os that would otherwise be necessary to commit the 
> > metadata into a single I/O.  He proposed using LAYOUTCOMMIT as the 
> > 'metadata commit'.  This alters pNFS's current behavior, since the 
> > client would have to keep the data around until the MDS commit (i.e., 
> > LAYOUTCOMMIT) completed.  Marc proposed that these special commits 
only 
> > be done when the client knows that it is going to be following data 
> > commits with a LAYOUTCOMMIT (e.g., at a fsync call), thus not 
> > introducing more LAYOUTCOMMIT operations.  However, this introduces 
more 
> > complexity into the client.  As well, the separate metadata and data 
> > commits complicate recover.
> > 
> > 2)
> > Marc, Manoj, Andy, and I talked today and came up with a solution that 
I 
> > prefer.  First, we must acknowledge that two types of file layouts 
> > exist: 1) those that require/prefer commits to each DS; 2) those that 
> > require/prefer a single commit to the MDS.  The cluster file system 
> > described above fits into (2).  This information must be made 
available 
> > to the client, probably through a flag in the file-layout structure.
> > 
> > Upon receiving a layout with this flag set (COMMIT_THROUGH_MDS), the 
> > client should not issue COMMITs to the DSs.  Instead it must issue a 
> > special commit (we can talk about what exactly to use, but lets think 
of 
> > it as a new op for now), to the MDS when it is ready to commit the 
dirty 
> > data.  Once the data is committed (i.e., the new MDS commit completes) 

> > the client can discard the data.
> > 
> > If this new MDS commit fails, the client must rewrite the dirty data 
it 
> > holds.  Most likely the layout pertaining to the file (region) will be 

> > recalled.  The client can either write the data through the MDS or it 
> > can obtain a new layout and use that to commit the data.
> > 
> > This solution avoids some of the recovery issues that occur with 
> > solution 1, it avoids the client sending different types of commits to 

> > DSs, and it feels more like the current v4 commit model.
> > 
> > -Garth
> > 
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Marc Eshel <[email-masked]>
Date: Sun, 26 Mar 2006 11:44:17 -0800

> to the MDS will be rejected (with what error I wonder).

Maybe we should have one return code that means that the client sent an 
operation to the wrong server node (MDS or DS) It is probably more useful 
for the DS that should reject most RPC types. How about NFS4ERR_INVAL. 

> working group consensus about verifiers.

I don't see the benefit of making the verifier scope bigger than one file. 
I say file and not FH because the FH given for the same file for each DS 
might be different. For pNFS the scope should be the layout. The server 
implementation can make it more global if it wants to but that can cause 
extra recovery like in the above example. If a node that is no a part of a 
given layout rebooted it should not cause extra recovery.
 
>    instantiations of whatever kind that can possibly lead to a loss 
OK.
 
> with unstable writes to be committed.

It will complicate the recovery on the client if it needs to keep track of 
the common verifier and compare on each write with verifiers from other 
writes that where done for a file with other DSs. It would be much simpler 
to just record the verifier for each write and at commit time check once 
for all committed data and trigger recovery if needed.

> choice if you have a layout.
Right.

> lost.  Deal with it.  How is your choice.".
Yes.

Marc.