Subject: RE: [nfsv4] Issue 21: pnfs EOF handling

From: [email-masked]
Date: Wed, 5 Jul 2006 15:50:37 -0400

Craig,

> So, David, you're proposing, in the name of pragmatism, that:
> 	client processing of SIZECHANGED should be quick;

Specifically, update internal state and respond.  Writes and commits
by the client should not be necessary, and it would be ok to specify
them as MUST NOT be done before responding to the callback RPC.

> 	and servers should not reply to the SETATTR request
> 		until all clients have replied to SIZECHANGED;
> 	servers that need special resources [read: threads] in order not
> to reply should allocate those resources up front, possibly returning
> DELAY before issuing any SIZECHANGED callbacks to clients;
> 	servers should regard truncations as committed before issuing
> any SIZECHANGED callbacks.
> 
> Does that about cover it?

Yes.

> One "new" state is then the client that acts on a long EOF by writing
> data and committing layouts at the same time that the server has
> committed (whatever that means) its shorter EOF and just hasn't gotten
> the news to that client yet.  How confused will that client get?
>
> Are there any other weirdnesses that are caused by clients thinking they
> know EOF while the server in fact disagrees with them?

pNFS already does this in the extend case - the server's notion of EOF is
not updated until the LAYOUTCOMMIT beyond the current end of file occurs.

There are 3 cases for uncommitted written data beyond EOF:
a) Beyond old EOF, inside new EOF.  Write and commit in normal fashion.
b) Inside old EOF, beyond new EOF.  Abandon write of cached data and
	commit of previously written/currently cached data.
c) Beyond old EOF, beyond new EOF.  Problem Area.

The problem in the 3rd case is that the pNFS client needs to know *why*
the EOF update occurred.  If the new EOF was set explicitly, we're in
case b) where data beyond the new EOF is gone (even if this was an
extension).
If the new EOF was set implicitly by a write, I think the write and commit
(and implicit extension) still need to occur.  To understand why, consider
two pNFS clients both writing beyond the EOF; it would be wrong if the
sizechanged callback from the first client's commit caused the second
client to abandon its writes and commits because they're beyond the new
EOF set by the first client.

The simplest fix may be to only issue CB_SIZECHANGED for an explicit
SETATTR,
and never for any implicit EOF change (write or commit).  A more
comprehensive
approach would be to add a "why" flag to CB_SIZECHANGED to distinguish
implicit extension from explicit length set, although I can't immediately
figure out why a pNFS client needs to see an implicit EOF change for file
extension.  This will need to be addressed by the changes to close Issue 21,
unless someone wants a new Issue for this.
 
> I agree that it is reasonable to delay the SETATTR caller...

Great.

Thanks,
--David

> -----Original Message-----
> From: Everhart, Craig [mailto:[email-masked]] 
> Sent: Wednesday, July 05, 2006 2:25 PM
> To: Black, David
> Cc: [email-masked]
> Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> 
> So, David, you're proposing, in the name of pragmatism, that:
> 	client processing of SIZECHANGED should be quick;
> 	and servers should not reply to the SETATTR request
> 		until all clients have replied to SIZECHANGED;
> 	servers that need special resources [read: threads] in order not
> to reply should allocate those resources up front, possibly returning
> DELAY before issuing any SIZECHANGED callbacks to clients;
> 	servers should regard truncations as committed before issuing
> any SIZECHANGED callbacks.
> 
> Does that about cover it?
> 
> One "new" state is then the client that acts on a long EOF by writing
> data and committing layouts at the same time that the server has
> committed (whatever that means) its shorter EOF and just hasn't gotten
> the news to that client yet.  How confused will that client get?
> 
> Are there any other weirdnesses that are caused by clients 
> thinking they
> know EOF while the server in fact disagrees with them?
> 
> I agree that it is reasonable to delay the SETATTR caller...
> 
> 		Craig
> 
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]] 
> > Sent: Monday, July 03, 2006 8:59 PM
> > To: Everhart, Craig
> > Cc: [email-masked]
> > Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> > 
> > The concern in the block/volume layout, is that unlike file 
> > or object where the servers track the file sizes, the 
> > block/volume clients have to track sizes.  The result is that 
> > moving the EOF across a block layout range held by another 
> > client can require that other client to change the state of 
> > its layout.  The crucial case is that for truncate, blocks 
> > that previously did not require a commit in response to a 
> > write have to be changed to a state where they now require 
> > one (and if there was pending delayed write, it's abandoned). 
> >  If one gets too lazy about EOF propagation, one can lose 
> > writes that shouldn't have been lost in a truncate case 
> > (client write has an  application level dependency on an EOF 
> > change that propagated to the application via some means 
> > faster than pNFS).
> > I don't think anything goes wrong in the extend case.  The 
> > original proposal to not complete the SETATTR until the 
> > callbacks are dealt with in some fashion was intended to 
> > prevent this race.
> > 
> > A potential problem with recalling all affected layouts is a 
> > case in which a scratch file is truncated to zero between 
> > rounds of a parallel computation - having the server tell 
> > each of the clients with write permission that their layout 
> > just went from valid data to invalid data is significantly 
> > more effective than recalling everything followed by an 
> > influx of requests to get the layouts back.
> > 
> > Picking up Craig's crucial comment:
> > 
> > > > > I agree with Dave.  I think the most the server can tell the 
> > > > > clients before committing the truncate is that they no 
> > longer know 
> > > > > what EOF is, presumably by recalling layouts, and this 
> > can return 
> > > > > DELAY to the original client if needed.
> > 
> > In contrast to layout recalls, EOF changes don't absolutely 
> > require clients to do anything in response to the EOF change 
> > (all the state updates are internal); the EOF callback can be 
> > responded to even if there's a return for the block 
> > containing the new EOF in the same callback compound.  If the 
> > server bails out and returns DELAY to a client after issuing 
> > EOF change callbacks to other clients, the result can be just 
> > plain wrong for the block layout because the EOF change 
> > callback can have irreversible effects at clients (dirty 
> > cached blocks beyond new EOF may get discarded instead of 
> > written to storage).  The HighRoad design approach was that 
> > once the server committed to the EOF change, it saw it 
> > through, period.
> > I think there may be an approach that comes close to this 
> > which is to be careful about operation admission on the 
> > server side, and only take a SETATTR that truncates if 
> > there's plenty of time to get to the clients before a DELAY 
> > might be needed (i.e., aggressively issue a DELAY if there's 
> > been a delay in the inbound operation for some reason, but 
> > never issue one if an EOF change callback has been issued, 
> > rely on clients to respond to them quickly).
> > 
> > I think the idea of stalling the pNFS clients so that one can 
> > respond to the SETATTR quickly is probably backwards - the 
> > client that issued the SETATTR is causing the problem, so it 
> > should be stalled for as long as it takes, and have the pNFS 
> > clients implemented to deal with the EOF callback quickly - 
> > the immediate reply to the EOF callback RPC should get the job done.
> > 
> > Thanks,
> > --David
> > 
> > > -----Original Message-----
> > > From: Everhart, Craig [mailto:[email-masked]]
> > > Sent: Thursday, June 29, 2006 5:32 PM
> > > To: fridella, stephen; Noveck, Dave; Goodson, Garth
> > > Cc: [email-masked]
> > > Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> > > 
> > > OK, so you're saying
> > > 	client A sends a SETATTR to truncate a file downward
> > > 	server truncates it locally and replies OK
> > > 	clients B, C, D, ... et al. that had layouts beyond the 
> > new EOF (and 
> > > its new last block) are in a *new* state with respect to 
> > the server, 
> > > "I'm ignoring all your LAYOUTGET and LAYOUTRETURN 
> > operations until I 
> > > like you better?"
> > > 
> > > Don't we have to think about this new state a little harder?  
> > > What else
> > > is allowed and what else is disallowed?  Does the server 
> > drop requests 
> > > from B, C, D on the floor, or does it reply with a new 
> > > OPTIMISTIC_OUTOFSYNCH code?  How do they know they're 
> being ignored 
> > > and that they have to wade through their input queues to 
> get to the
> > CB_SIZECHANGED op?
> > > 
> > > Presumably B, C, D could use a SETATTR themselves, right?  What 
> > > happens to their layouts then?
> > > 
> > > Is this an optimization really worth preserving, 
> > specifically for the 
> > > truncate case?  To support performance optimizations for dueling 
> > > applications, one truncating and the other extending?  Am I 
> > crazy or 
> > > what?
> > > 
> > > 		Craig
> > > 
> > > > -----Original Message-----
> > > > From: [email-masked] [mailto:[email-masked]]
> > > > Sent: Thursday, June 29, 2006 5:10 PM
> > > > To: Everhart, Craig; Noveck, Dave; Goodson, Garth
> > > > Cc: [email-masked]
> > > > Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> > > > 
> > > > Craig---
> > > > Your view that layouts must be recalled creates problems for a 
> > > > block/volume layout.  With this philosophy each SETATTR 
> > will force 
> > > > the deallocation of all blocks beyond the new end of file.
> > > > Clients that continue to write beyond the end of file (because 
> > > > applications still have the file open) must ask the server to 
> > > > re-allocate blocks for them to write into and provide 
> > them with the 
> > > > new layout.  The original motivation behind the CB_SIZECHANGED 
> > > > (suggested by David B, I believe) was to find a way to 
> > optimize this 
> > > > cost associated with SETATTR.
> > > > 
> > > > The basic idea was to inform clients of the change in 
> > size, but to 
> > > > allow layouts beyond the end of file to be preserved.
> > > > Of course, the layouts would transition to an "uncommitted"
> > > > state, and would need to be re-committed by the client.  
> > For clients 
> > > > with applications that continue to write beyond the eof 
> after the 
> > > > SETATTR, this reduces the impact of the SETATTR from a 
> > > > RECALL/RETURN/GETLAYOUT/COMMIT to a SIZECHANGED/COMMIT.
> > > > I'd like to find a way to keep this optimization.
> > > > 
> > > > 
> > > > I should also note that my new proposed text below does not
> > > > *require* the server to wait for the CB_SIZECHANGED 
> callbacks to 
> > > > complete before processing the SETATTR.  It says only that some 
> > > > layout types may choose this method of operation.  So 
> the problem 
> > > > that Dave N and Craig describe only applies to specific 
> > layout types 
> > > > (like block/volume).
> > > > 
> > > > But to address the problem in the block/volume layout 
> > type, I think 
> > > > we can use the approach the Dave N alludes to in his
> > > > note:
> > > > 
> > > > Process the SETATTR immediately without recalling the layouts 
> > > > (except for layouts that cover the new last file block--- 
> > these we 
> > > > MUST recall).  At the server, preserve the layouts that 
> > lie beyond 
> > > > the EOF but mark them as uncommitted.  Inform the clients 
> > after the 
> > > > SETATTR is complete using CB_SIZECHANGED, and while 
> > waiting for the 
> > > > response from the callback, reject all LAYOUTGETS and 
> > LAYOUTRETURNS 
> > > > from the client.  Once the client acknowledges the 
> > callback, and the 
> > > > server and client are in sync, normal operation can continue.
> > > > 
> > > > I'll re-write the text again to remove the reference to 
> > waiting for 
> > > > the callbacks.
> > > > 
> > > > -Steve
> > > > 
> > > > 
> > > > > -----Original Message-----
> > > > > From: Everhart, Craig [mailto:[email-masked]]
> > > > > Sent: Thursday, June 29, 2006 4:22 PM
> > > > > To: Noveck, Dave; Goodson, Garth; fridella, stephen
> > > > > Cc: [email-masked]
> > > > > Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> > > > > 
> > > > > I agree with Dave.  I think the most the server can tell the 
> > > > > clients before committing the truncate is that they no 
> > longer know 
> > > > > what EOF is, presumably by recalling layouts, and this 
> > can return 
> > > > > DELAY to the original client if needed.  So the only time the 
> > > > > server can commit to a new size is while clients don't 
> > share this 
> > > > > knowledge, but they know they need to check things 
> out with the 
> > > > > server.  If you're growing sizes, that could work OK, 
> > but I think 
> > > > > shrinking can't be using SIZECHANGED.
> > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Noveck, Dave
> > > > > > Sent: Thursday, June 29, 2006 3:24 PM
> > > > > > To: Goodson, Garth; [email-masked]
> > > > > > Cc: [email-masked]
> > > > > > Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> > > > > > 
> > > > > > At this point I'm not sure who, but somebody (Steven?) said:
> > > > > > > The idea was that the server should be sure that 
> > the clients 
> > > > > > > have
> > a
> > > > > > > consistent view of the file size before allowing 
> > the truncate 
> > > > > > > to succeed.
> > > > > > 
> > > > > > Because of the DELAY, I'm not at all sure you can do that. 
> > > > > > If you accept the new size to the point of telling 
> > other clients 
> > > > > > that it is indeed the new size (and they depend on 
> > this), then 
> > > > > > you have allowed the truncate to proceed pretty far, 
> > so it isn't 
> > > > > > true that everybody has the same view, before you 
> > proceed to do 
> > > > > > the truncate.  Then if return DELAY, you are really 
> > in trouble 
> > > > > > because you have committed yourself to the other 
> > clients but the 
> > > > > > initiating client has been told it hasn't happened.
> > > > > > 
> > > > > > I don't think you can make the server's acceptance 
> > the truncate 
> > > > > > conditional on completion of the notification.  I think the 
> > > > > > model has to be that you accept the truncate and then 
> > notify everybody.
> > > > > > You are free to grievously restrict clients can do until 
> > > > > > everybody gets the notification and agrees but I 
> > don't think you 
> > > > > > can consistently both refuse to commit yourself to 
> > the truncate 
> > > > > > while propagating notifications predicated on what is 
> > > > > > essentially a commitment to actually performing the 
> truncate.
> > > > > > 
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Goodson, Garth
> > > > > > Sent: Thursday, June 29, 2006 12:18 PM
> > > > > > To: [email-masked]
> > > > > > Cc: Goodson, Garth; [email-masked]
> > > > > > Subject: Re: [nfsv4] Issue 21: pnfs EOF handling
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > [email-masked] wrote:
> > > > > > >> -----Original Message-----
> > > > > > >> From: Garth Goodson [mailto:[email-masked]]
> > > > > > >> Sent: Tuesday, June 27, 2006 4:51 PM
> > > > > > >> To: fridella, stephen
> > > > > > >> Cc: [email-masked]
> > > > > > >> Subject: Re: [nfsv4] Issue 21: pnfs EOF handling
> > > > > > >>
> > > > > > > 
> > > > > > > <snip>
> > > > > > > 
> > > > > > >> Is this too restrictive for some implementations (i.e., 
> > > > > > >> impose unnecessary delays when the backend system 
> > can handle 
> > > > > > >> the changes)?  My concern arises over the MUST send 
> > > > > > >> CB_SIZECHANGED and SHOULD NOT respond to the 
> SETATTR until 
> > > > > > >> all responses have been received (I guess it is a 
> > SHOULD NOT 
> > > > > > >> rather than a MUST NOT).  Remind me why we need to 
> > wait for the CB replies.
> > > > > > >>
> > > > > > > 
> > > > > > > The idea was that the server should be sure that 
> > the clients 
> > > > > > > have a consistent view of the file size before 
> allowing the 
> > > > > > > truncate to
> > 
> > > > > > > succeed. I suppose the problem is that different 
> > layout types 
> > > > > > > have
> > 
> > > > > > > different requirements for the consistency of file 
> > size, with 
> > > > > > > the file layout type having the loosest 
> > requirements of all.  
> > > > > > > I don't mind making this a layout-type specific 
> > requirement.  
> > > > > > > The question
> > 
> > > > > > > is, what do we say in the generic spec?  Perhaps we 
> > could say 
> > > > > > > something like:
> > > > > > > 
> > > > > > > 
> > > > > > > Depending on the requirements of the storage 
> protocol, the 
> > > > > > > server MAY need to ensure that all clients have a 
> > consistent 
> > > > > > > view of the file size before allowing the SETATTR 
> > to succeed.
> > > > > > > If so, the server MAY need to delay processing 
> the SETATTR 
> > > > > > > until receiving responses to the associated 
> > CB_SIZECHANGED and 
> > > > > > > CB_LAYOUTRECALL messages that it has sent to clients.
> > > > > > > The precise details of when the server can process 
> > the SETATTR 
> > > > > > > are
> > 
> > > > > > > described in the layout-specific specifications.
> > > > > > > 
> > > > > > > 
> > > > > > I do prefer something like this.  It gives a bit more 
> > latitude 
> > > > > > and allows for layout-specific details to be handled 
> > separately.
> > > > > > 
> > > > > > > 
> > > > > > >> I also want to bring up something else that 
> > Spencer brought 
> > > > > > >> up earlier.
> > > > > > >>   It went something like this:
> > > > > > >>
> > > > > > >> 1) SETATTR that truncates file arrives
> > > > > > >> 2) Server sends out CB_SIZECHANGED, and DELAY error to 
> > > > > > >> setattr
> > caller
> > > > > > >> 3) Setattr caller never retries SETATTR, but CB 
> > has been sent 
> > > > > > >> out
> > > > > > >>
> > > > > > >> I'm not sure how to work around this.  Is this a 
> > problem -- 
> > > > > > >> or is
> > 
> > > > > > >> it ok that the client can confuse other clients?
> > > > > > >>
> > > > > > >> -Garth
> > > > > > >>
> > > > > > > 
> > > > > > > It does seem like a problem.  I have to think some 
> > more about it.
> > 
> > > > > > > The danger is that a client will believe that its 
> LAYOUT is 
> > > > > > > beyond
> > 
> > > > > > > the end of file, and hence not committed, when in 
> > fact, it is 
> > > > > > > actually fine (since the truncate never 
> occurred).  In this 
> > > > > > > case the cost is the client re-committing a LAYOUT 
> > unnecessarily.
> > > > > > > Hopefully the server is tolerant of this case.  Is 
> > it possible 
> > > > > > > to require that the server not return DELAY?
> > > > > > > 
> > > > > > I'm not an expert on the DELAY rules.  However, it 
> > seems that if 
> > > > > > the server needs to block for a possibly long 
> period of time, 
> > > > > > waiting
> > for
> > > > > > all the callback responses, then it must send out a 
> > DELAY message.  
> > > > > > The spec says: 'The server initiated the request, 
> but was not 
> > > > > > able to complete it in a timely fashion...'  Maybe 
> others can 
> > > > > > chime in here on how to handle this.
> > > > > > 
> > > > > > -Garth
> > > > > > 
> > > > > > > --Steve
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > _______________________________________________
> > > > > > > nfsv4 mailing list
> > > > > > > [email-masked]
> > > > > > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > > > > > 
> > > > > > _______________________________________________
> > > > > > nfsv4 mailing list
> > > > > > [email-masked]
> > > > > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > > > > > 
> > > > > > _______________________________________________
> > > > > > nfsv4 mailing list
> > > > > > [email-masked]
> > > > > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > > > > > 
> > > > > 
> > > > 
> > > 
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [email-masked]
> > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > > 
> > 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Mon, 10 Jul 2006 12:16:31 -0400

[email-masked] wrote:

> extension. 

[snip]

David, In the objects case, when the file is striped over multiple 
objects and the
client gets a read beyond its notion of the EOF it has to go read from 
storage
to determine the current file length or issue a GETATTR to the MDS.
If the file is holey and was sparsely written by another client beyond 
this client's
notion of the EOF it is possible that the client will get a short read 
from one
of the objects but there's more data in higher offsets on other objects.
In order to make this data visible the client needs to know the new EOF.

Benny

---

From: Garth Goodson <[email-masked]>
Date: Mon, 10 Jul 2006 09:28:35 -0700

> In order to make this data visible the client needs to know the new EOF.

Same for files.  Alternatively, extending writes could extend the files 
on the data servers.

-Garth

---

From: Benny Halevy <[email-masked]>
Date: Mon, 10 Jul 2006 14:05:19 -0400

Garth Goodson wrote:

>> client gets a read beyond its notion of the EOF it has to go read 
>> from storage
>> If the file is holey and was sparsely written by another client 
>> beyond this client's
>> notion of the EOF it is possible that the client will get a short 
>> read from one
> Same for files.  Alternatively, extending writes could extend the 
> files on the data servers.

This has a similar race between writers since the SETATTR operation 
would need an "extend only"
flag to make sure nobody else's data is being truncated.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 10 Jul 2006 12:48:24 -0700

> flag to make sure nobody else's data is being truncated.

I was thinking it could be implemented by the backend protocol, which as 
you say, would have to have some way of serializing writes and truncates...

-Garth