Subject: RE: [nfsv4] pNFS commits with cluster file systems

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 27 Mar 2006 12:20:27 -0500

> Without pNFS, the scope is no bigger than a single server, right?

But it also needs to be smaller in some case to accommodate migration.

> With pNFS, the scope needs to be as large as the layout.
> Whether the values are shared with other layouts is immaterial.

It can be made immaterial by clearly saying that the scope is what 
you want it to be: one file, and across any MDS/DS comination if 
this bit is set.

The thing is we have to decide on that and get the appropriate text 
into WRITE and COMMIT, rather than just the pNFS section of the spec.
Given that, I'd like to avoid statements of the form "It's X for pNFS
and Y for everything else" if we can.

-----Original Message-----
From: Everhart, Craig 
Sent: Monday, March 27, 2006 11:45 AM
To: Marc Eshel; Noveck, Dave
Cc: Goodson, Garth; [email-masked]
Subject: RE: [nfsv4] pNFS commits with cluster file systems

In reply to message from: Marc Eshel [mailto:[email-masked]] 
> "Noveck, Dave" <[email-masked]> wrote on 03/26/2006 
> 08:51:29 AM:
> 
> > This is about what we agreed to.  Some comments below.
> > 
> > > A new flag in the file layout (COMMIT_THROUGH_MDS) will allow the 
> > > MDS
> to 
> > > tell the client of a preferred way of doing COMMIT. If 
> this flag is
> set, 
> > > the client SHOULD send COMMIT to the MDS instead of sending it to 
> > > the same DS to which the associated WRITEs were sent.
> > 
> > If it is not set the client must send COMMIT's to the DS 
> and a COMMIT 
> > to the MDS will be rejected (with what error I wonder).

OK, it's only SHOULD for clients-do-MDS-commits.  And I assume the
implication is for the server that if it offers a layout with
COMMIT_THROUGH_MDS, then write verifiers under that layout MUST
share a value.  And if the client chooses to commit to the MDS,
then it MUST compare the single verifier value returned from the 
DSs with the value returned from the MDS commit and retransmit
everything that had returned a different value, right?

> Maybe we should have one return code that means that the 
> client sent an operation to the wrong server node (MDS or DS) 
> It is probably more useful for the DS that should reject most 
> RPC types. How about NFS4ERR_INVAL. 
>  
> > > In order to maintain
> > > the current NFSv4 commit and recovery model, all the DS'es MUST 
> > > return
> 
> > > the same verifier for all WRITEs in a given file layout. 
> > 
> > Since in the next sentence you talk about the verifier 
> changing, i.e 
> > not being "the same", I think you want to talk about returning a
> "common"
> > (i.e. globally maintained among the MDS and DS's) verifier.
> > 
> > There is an issue here about the scope of verifiers.  There is some 
> > text in the current spec which implies they are uniform for 
> a server 
> > as a whole.  When you add in your uniformity requirement these 
> > together could force them to be of much wider scope than you want.
> > 
> > For example, MDS A uses DS's P, Q, R, S and MDS B uses DS's 
> S, T, U, V 
> > and MDS C uses DS's V, W, X, Y, Z.  Do A, B, C, P, Q, R, S, 
> T, U, V, 
> > W, X, Y, Z all have to share a verifier?  Help!
> > 
> > The verifier scope issue also turns up in migration in that if I 
> > migrate an fs to a server, I might want to move verifier 
> information, 
> > making server uniformity untenable.  Anyway I think we want 
> to either 
> > say that the client should not compare verifiers from 
> different fh's, 
> > or say that the client should not compare verifiers from different 
> > fs's.  In what I'm currently drafting, I'm going to assume 
> the latter 
> > but I'll create an issue to track this and come to a working group 
> > consensus about verifiers.
> 
> I don't see the benefit of making the verifier scope bigger 
> than one file. 
> I say file and not FH because the FH given for the same file 
> for each DS might be different. For pNFS the scope should be 
> the layout. The server implementation can make it more global 
> if it wants to but that can cause extra recovery like in the 
> above example. If a node that is no a part of a given layout 
> rebooted it should not cause extra recovery.

Without pNFS, the scope is no bigger than a single server, right?
With pNFS, the scope needs to be as large as the layout.
Whether the values are shared with other layouts is immaterial.

> > > Whenever any DS
> > > that is referenced in the layout reboots, the verifier MUST be
> changed.
> > 
> > I think a better way to express this is to build on what 
> COMMIT says:
> > 
> >    The server must vary the value of the write
> >    verifier at each server event or instantiation that may lead to a
> >    loss of uncommitted data. 
> > 
> > and say,
> > 
> >    The value of the write verifier must be changed at each 
> server event
> >    (including events on the MDS and any of the DS's) and server
> >    instantiations of whatever kind that can possibly lead to a loss 
> >    of uncommitted data.
> >
> OK.
>  
> > > The single COMMIT to the MDS will return a verifier and the client
> must 
> > > compare it to all the verifiers from the WRITEs and fail 
> the COMMIT 
> > > if there is any mismatched verifiers. If COMMIT to the MDS fails, 
> > > the
> client 
> > > MUST reissue WRITEs for all the dirty data in the file as 
> follows. 
> > 
> > Your assumption here is that a server will accumulate writes with
> multiple
> > different verifier values for the same file.  My understanding of 
> > this, although the spec isn't that clear is that once you saw a 
> > different
> verifier
> > you would note that previous writes with a different verifier were
> already
> > lost and so you would only have a single outstanding verifier 
> > associated with unstable writes to be committed.
> 
> It will complicate the recovery on the client if it needs to 
> keep track of the common verifier and compare on each write 
> with verifiers from other writes that where done for a file 
> with other DSs. It would be much simpler to just record the 
> verifier for each write and at commit time check once for all 
> committed data and trigger recovery if needed.
> 
> > 
> > > If the
> > > layout has not been recalled, all WRITEs that have a different
> verifier 
> > > than the one returned by COMMIT MUST be rewritten through the DS. 
> > 
> > I think you can always write using the MDS, although it would be a 
> > poor choice if you have a layout.
> >
> Right.
> 
> > 
> > > If the
> > > layout has been recalled, the client MUST write the data 
> through the
> MDS.
> > 
> > The client could try to get a new layout and use that.  The client
> doesn't
> > really have to write the data at all.  We are not saying "You MUST 
> > write
> 
> > the data or we will report you to the IESG", but rather 
> "You know that 
> > data you asked us to write.  Somehow, we can't find it and 
> fear it is 
> > lost.  Deal with it.  How is your choice.".
> > 
> Yes.
> 
> Marc.

This sounds reasonable.  Sounds like it's time for Marc to write it up.

		Craig

> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Mon, 27 Mar 2006 09:29:21 -0800

> and Y for everything else" if we can.

We already have some pNFS specific COMMIT text about what a COMMIT means 
at the metadata server in regard to data written through the MDS and 
when it is visible at the DSs.  Any other text we add on COMMIT should 
not contradict this text (and if we are going to merge COMMIT text, we 
must not forget about it).

-Garth