Subject: RE: [nfsv4] pNFS commits with cluster file systems

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 27 Mar 2006 11:45:07 -0500

In reply to message from: Marc Eshel [mailto:[email-masked]] 
> "Noveck, Dave" <[email-masked]> wrote on 03/26/2006 
> 08:51:29 AM:
> 
> > This is about what we agreed to.  Some comments below.
> > 
> > > A new flag in the file layout (COMMIT_THROUGH_MDS) will allow the 
> > > MDS
> to 
> > > tell the client of a preferred way of doing COMMIT. If 
> this flag is
> set, 
> > > the client SHOULD send COMMIT to the MDS instead of sending it to 
> > > the same DS to which the associated WRITEs were sent.
> > 
> > If it is not set the client must send COMMIT's to the DS 
> and a COMMIT 
> > to the MDS will be rejected (with what error I wonder).

OK, it's only SHOULD for clients-do-MDS-commits.  And I assume the
implication is for the server that if it offers a layout with
COMMIT_THROUGH_MDS, then write verifiers under that layout MUST
share a value.  And if the client chooses to commit to the MDS,
then it MUST compare the single verifier value returned from the 
DSs with the value returned from the MDS commit and retransmit
everything that had returned a different value, right?

> Maybe we should have one return code that means that the 
> client sent an operation to the wrong server node (MDS or DS) 
> It is probably more useful for the DS that should reject most 
> RPC types. How about NFS4ERR_INVAL. 
>  
> > > In order to maintain
> > > the current NFSv4 commit and recovery model, all the DS'es MUST 
> > > return
> 
> > > the same verifier for all WRITEs in a given file layout. 
> > 
> > Since in the next sentence you talk about the verifier 
> changing, i.e 
> > not being "the same", I think you want to talk about returning a
> "common"
> > (i.e. globally maintained among the MDS and DS's) verifier.
> > 
> > There is an issue here about the scope of verifiers.  There is some 
> > text in the current spec which implies they are uniform for 
> a server 
> > as a whole.  When you add in your uniformity requirement these 
> > together could force them to be of much wider scope than you want.
> > 
> > For example, MDS A uses DS's P, Q, R, S and MDS B uses DS's 
> S, T, U, V 
> > and MDS C uses DS's V, W, X, Y, Z.  Do A, B, C, P, Q, R, S, 
> T, U, V, 
> > W, X, Y, Z all have to share a verifier?  Help!
> > 
> > The verifier scope issue also turns up in migration in that if I 
> > migrate an fs to a server, I might want to move verifier 
> information, 
> > making server uniformity untenable.  Anyway I think we want 
> to either 
> > say that the client should not compare verifiers from 
> different fh's, 
> > or say that the client should not compare verifiers from different 
> > fs's.  In what I'm currently drafting, I'm going to assume 
> the latter 
> > but I'll create an issue to track this and come to a working group 
> > consensus about verifiers.
> 
> I don't see the benefit of making the verifier scope bigger 
> than one file. 
> I say file and not FH because the FH given for the same file 
> for each DS might be different. For pNFS the scope should be 
> the layout. The server implementation can make it more global 
> if it wants to but that can cause extra recovery like in the 
> above example. If a node that is no a part of a given layout 
> rebooted it should not cause extra recovery.

Without pNFS, the scope is no bigger than a single server, right?
With pNFS, the scope needs to be as large as the layout.
Whether the values are shared with other layouts is immaterial.

> > > Whenever any DS
> > > that is referenced in the layout reboots, the verifier MUST be
> changed.
> > 
> > I think a better way to express this is to build on what 
> COMMIT says:
> > 
> >    The server must vary the value of the write
> >    verifier at each server event or instantiation that may lead to a
> >    loss of uncommitted data. 
> > 
> > and say,
> > 
> >    The value of the write verifier must be changed at each 
> server event
> >    (including events on the MDS and any of the DS's) and server
> >    instantiations of whatever kind that can possibly lead to a loss 
> >    of uncommitted data.
> >
> OK.
>  
> > > The single COMMIT to the MDS will return a verifier and the client
> must 
> > > compare it to all the verifiers from the WRITEs and fail 
> the COMMIT 
> > > if there is any mismatched verifiers. If COMMIT to the MDS fails, 
> > > the
> client 
> > > MUST reissue WRITEs for all the dirty data in the file as 
> follows. 
> > 
> > Your assumption here is that a server will accumulate writes with
> multiple
> > different verifier values for the same file.  My understanding of 
> > this, although the spec isn't that clear is that once you saw a 
> > different
> verifier
> > you would note that previous writes with a different verifier were
> already
> > lost and so you would only have a single outstanding verifier 
> > associated with unstable writes to be committed.
> 
> It will complicate the recovery on the client if it needs to 
> keep track of the common verifier and compare on each write 
> with verifiers from other writes that where done for a file 
> with other DSs. It would be much simpler to just record the 
> verifier for each write and at commit time check once for all 
> committed data and trigger recovery if needed.
> 
> > 
> > > If the
> > > layout has not been recalled, all WRITEs that have a different
> verifier 
> > > than the one returned by COMMIT MUST be rewritten through the DS. 
> > 
> > I think you can always write using the MDS, although it would be a 
> > poor choice if you have a layout.
> >
> Right.
> 
> > 
> > > If the
> > > layout has been recalled, the client MUST write the data 
> through the
> MDS.
> > 
> > The client could try to get a new layout and use that.  The client
> doesn't
> > really have to write the data at all.  We are not saying "You MUST 
> > write
> 
> > the data or we will report you to the IESG", but rather 
> "You know that 
> > data you asked us to write.  Somehow, we can't find it and 
> fear it is 
> > lost.  Deal with it.  How is your choice.".
> > 
> Yes.
> 
> Marc.

This sounds reasonable.  Sounds like it's time for Marc to write it up.

		Craig

> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 27 Mar 2006 11:48:44 -0500

If it's only a SHOULD that the client should commit thru MDS, then the
DSs can't be failing incoming data commit calls.

		Craig 

> -----Original Message-----
> From: Marc Eshel [mailto:[email-masked]] 
> Sent: Sunday, March 26, 2006 2:44 PM
> To: Noveck, Dave
> Cc: Goodson, Garth; [email-masked]
> Subject: RE: [nfsv4] pNFS commits with cluster file systems
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 27 Mar 2006 12:00:18 -0500

> If it's only a SHOULD that the client should commit thru 
> MDS, then the DSs can't be failing incoming data commit calls.

Yes.  However, I was worried about the the reverse case in 
which this bit is not set.  I was thinking that if the client
sent the COMMIT's to the MDS anyway (despite the fact that they
are supposed to go to the DS), you would get an error.

But I now realize that you won't.  Sending COMMIT's to the MDS
is perfectly valid since the MDS is a v4.1 server.  The COMMIT
is OK but ... it only commits data written to the MDS directly,
i.e. in many cases it is a no-op, and the data at the DS's would
not be committed.  I think some there-be-dragons text might be
appropriate to make sure poeple are awfully careful about this 
since there will be no error to tell you you did something (very)
wrong.

-----Original Message-----
From: Everhart, Craig 
Sent: Monday, March 27, 2006 11:49 AM
To: Marc Eshel; Noveck, Dave
Cc: Goodson, Garth; [email-masked]
Subject: RE: [nfsv4] pNFS commits with cluster file systems

		Craig

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 27 Mar 2006 12:15:45 -0500

> I don't see the benefit of making the verifier scope bigger than one file. 

The only possible one would be that people might complain about having to 
hold on to too much verifier state.  We'll see if that happens.

> I say file and not FH because the FH given for the same file for each DS 
> might be different. 

Good point.

> For pNFS the scope should be the layout.

When your bit is in effect then it's scope has to reflect the layout but
note, that if the verifier is V (not for vendetta) with no layout, and 
then we have a layout granted then there is common scope that includes 
the non-layout portion f the file's existence as well as the MDS and all
the DS's within the layout.

> The server 
> implementation can make it more global if it wants to but that can cause 
> extra recovery like in the above example. If a node that is no a part of a 
> given layout rebooted it should not cause extra recovery.

Whatever we decide the scope is, the server is free to maintain it in a 
more coarse-grained fashion.

> It will complicate the recovery on the client if it needs to keep track of 
> the common verifier and compare on each write with verifiers from other 
> writes that where done for a file with other DSs. It would be much simpler 
> to just record the verifier for each write and at commit time check once 
> for all committed data and trigger recovery if needed.

My view of what is complicated is different from yours, but OK.

I think the spec has to be written for either case.  You can note a verifier
change within the scope of the verifier (whatever we decide that is) when 
you can defer this decision to COMMIT time.

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]]
Sent: Sunday, March 26, 2006 2:44 PM
To: Noveck, Dave
Cc: Goodson, Garth; [email-masked]
Subject: RE: [nfsv4] pNFS commits with cluster file systems

"Noveck, Dave" <[email-masked]> wrote on 03/26/2006 08:51:29 AM:

> > A new flag in the file layout (COMMIT_THROUGH_MDS) will allow the MDS 
to 
> > tell the client of a preferred way of doing COMMIT. If this flag is 
set, 
> > the client SHOULD send COMMIT to the MDS instead of sending it to the 
> > same DS to which the associated WRITEs were sent. 
> 
> If it is not set the client must send COMMIT's to the DS and a COMMIT

Maybe we should have one return code that means that the client sent an 
operation to the wrong server node (MDS or DS) It is probably more useful 
for the DS that should reject most RPC types. How about NFS4ERR_INVAL. 
 
> > In order to maintain 
> > the current NFSv4 commit and recovery model, all the DS'es MUST return 

> Since in the next sentence you talk about the verifier changing, i.e 
"common"
> There is an issue here about the scope of verifiers.  There is some text
> in the current spec which implies they are uniform for a server as a
> whole.  When you add in your uniformity requirement these together 
> could force them to be of much wider scope than you want.
> 
> For example, MDS A uses DS's P, Q, R, S and MDS B uses DS's S, T, U, V
> and MDS C uses DS's V, W, X, Y, Z.  Do A, B, C, P, Q, R, S, T, U, V, W,
> X, Y, Z all have to share a verifier?  Help!
> 
> migrate an fs to a server, I might want to move verifier information,
> making server uniformity untenable.  Anyway I think we want to 
> either say that the client should not compare verifiers from different
> fh's, or say that the client should not compare verifiers from 
> different fs's.  In what I'm currently drafting, I'm going to assume
> the latter but I'll create an issue to track this and come to a
> working group consensus about verifiers.

I don't see the benefit of making the verifier scope bigger than one file. 
I say file and not FH because the FH given for the same file for each DS 
might be different. For pNFS the scope should be the layout. The server 
implementation can make it more global if it wants to but that can cause 
extra recovery like in the above example. If a node that is no a part of a 
given layout rebooted it should not cause extra recovery.
 
changed.
> 
> I think a better way to express this is to build on what COMMIT says:
> 
>    The value of the write verifier must be changed at each server event
OK.
 
must 
> > compare it to all the verifiers from the WRITEs and fail the COMMIT if
> > there is any mismatched verifiers. If COMMIT to the MDS fails, the 
client 
> > MUST reissue WRITEs for all the dirty data in the file as follows. 
> 
multiple
> different verifier values for the same file.  My understanding of this,
> although the spec isn't that clear is that once you saw a different 
verifier
already
> lost and so you would only have a single outstanding verifier associated
> with unstable writes to be committed.

It will complicate the recovery on the client if it needs to keep track of 
the common verifier and compare on each write with verifiers from other 
writes that where done for a file with other DSs. It would be much simpler 
to just record the verifier for each write and at commit time check once 
for all committed data and trigger recovery if needed.

verifier 
> I think you can always write using the MDS, although it would be a poor 
> choice if you have a layout.
>
Right.

> 
> > If the 
> > layout has been recalled, the client MUST write the data through the 
MDS.
doesn't
> really have to write the data at all.  We are not saying "You MUST write 

> the data or we will report you to the IESG", but rather "You know that 
> data you asked us to write.  Somehow, we can't find it and fear it is 
Yes.

Marc.