Subject: RE: [nfsv4] Issue 21: pnfs EOF handling

From: [email-masked]
Date: Mon, 3 Jul 2006 20:58:31 -0400

The concern in the block/volume layout, is that unlike file or
object where the servers track the file sizes, the block/volume
clients have to track sizes.  The result is that moving the EOF
across a block layout range held by another client can require
that other client to change the state of its layout.  The crucial
case is that for truncate, blocks that previously did not require
a commit in response to a write have to be changed to a state
where they now require one (and if there was pending delayed write,
it's abandoned).  If one gets too lazy about EOF propagation, one
can lose writes that shouldn't have been lost in a truncate case
(client write has an  application level dependency on an EOF change
that propagated to the application via some means faster than pNFS).
I don't think anything goes wrong in the extend case.  The original
proposal to not complete the SETATTR until the callbacks are dealt
with in some fashion was intended to prevent this race.

A potential problem with recalling all affected layouts is a case
in which a scratch file is truncated to zero between rounds of a
parallel computation - having the server tell each of the clients
with write permission that their layout just went from valid data
to invalid data is significantly more effective than recalling
everything followed by an influx of requests to get the layouts back.

Picking up Craig's crucial comment:

> > > I agree with Dave.  I think the most the server can tell the clients 
> > > before committing the truncate is that they no longer know what EOF 
> > > is, presumably by recalling layouts, and this can return DELAY to the 
> > > original client if needed.

In contrast to layout recalls, EOF changes don't absolutely require clients
to do anything in response to the EOF change (all the state updates are
internal); the EOF callback can be responded to even if there's a return
for the block containing the new EOF in the same callback compound.  If
the server bails out and returns DELAY to a client after issuing EOF
change callbacks to other clients, the result can be just plain wrong
for the block layout because the EOF change callback can have irreversible
effects at clients (dirty cached blocks beyond new EOF may get discarded
instead of written to storage).  The HighRoad design approach was that
once the server committed to the EOF change, it saw it through, period.
I think there may be an approach that comes close to this which is to
be careful about operation admission on the server side, and only take
a SETATTR that truncates if there's plenty of time to get to the clients
before a DELAY might be needed (i.e., aggressively issue a DELAY if there's
been a delay in the inbound operation for some reason, but never issue
one if an EOF change callback has been issued, rely on clients to respond
to them quickly).

I think the idea of stalling the pNFS clients so that one can respond to
the SETATTR quickly is probably backwards - the client that issued the
SETATTR is causing the problem, so it should be stalled for as long as
it takes, and have the pNFS clients implemented to deal with the EOF
callback quickly - the immediate reply to the EOF callback RPC should
get the job done.

Thanks,
--David

> -----Original Message-----
> From: Everhart, Craig [mailto:[email-masked]] 
> Sent: Thursday, June 29, 2006 5:32 PM
> To: fridella, stephen; Noveck, Dave; Goodson, Garth
> Cc: [email-masked]
> Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> 
> OK, so you're saying
> 	client A sends a SETATTR to truncate a file downward
> 	server truncates it locally and replies OK
> 	clients B, C, D, ... et al. that had layouts beyond the new EOF
> (and its new last block) are in a *new* state with respect to the
> server, "I'm ignoring all your LAYOUTGET and LAYOUTRETURN operations
> until I like you better?"
> 
> Don't we have to think about this new state a little harder?  
> What else
> is allowed and what else is disallowed?  Does the server drop requests
> from B, C, D on the floor, or does it reply with a new
> OPTIMISTIC_OUTOFSYNCH code?  How do they know they're being ignored and
> that they have to wade through their input queues to get to the
CB_SIZECHANGED op?
> 
> Presumably B, C, D could use a SETATTR themselves, right?  What happens
> to their layouts then?
> 
> Is this an optimization really worth preserving, specifically for the
> truncate case?  To support performance optimizations for dueling
> applications, one truncating and the other extending?  Am I crazy or
> what?
> 
> 		Craig
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]] 
> > Sent: Thursday, June 29, 2006 5:10 PM
> > To: Everhart, Craig; Noveck, Dave; Goodson, Garth
> > Cc: [email-masked]
> > Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> > 
> > Craig---
> > Your view that layouts must be recalled creates problems for 
> > a block/volume layout.  With this philosophy each SETATTR 
> > will force the deallocation of all blocks beyond the new end of file.
> > Clients that continue to write beyond the end of file 
> > (because applications still have the file open) must ask the 
> > server to re-allocate blocks for them to write into and 
> > provide them with the new layout.  The original motivation 
> > behind the CB_SIZECHANGED (suggested by David B, I believe) 
> > was to find a way to optimize this cost associated with SETATTR.
> > 
> > The basic idea was to inform clients of the change in size, 
> > but to allow layouts beyond the end of file to be preserved.
> > Of course, the layouts would transition to an "uncommitted"
> > state, and would need to be re-committed by the client.  For 
> > clients with applications that continue to write beyond the 
> > eof after the SETATTR, this reduces the impact of the SETATTR 
> > from a RECALL/RETURN/GETLAYOUT/COMMIT to a SIZECHANGED/COMMIT.
> > I'd like to find a way to keep this optimization.
> > 
> > 
> > I should also note that my new proposed text below does not
> > *require* the server to wait for the CB_SIZECHANGED callbacks 
> > to complete before processing the SETATTR.  It says only that 
> > some layout types may choose this method of operation.  So 
> > the problem that Dave N and Craig describe only applies to 
> > specific layout types (like block/volume).
> > 
> > But to address the problem in the block/volume layout type, I 
> > think we can use the approach the Dave N alludes to in his
> > note:
> > 
> > Process the SETATTR immediately without recalling the layouts 
> > (except for layouts that cover the new last file block--- 
> > these we MUST recall).  At the server, preserve the layouts 
> > that lie beyond the EOF but mark them as uncommitted.  Inform 
> > the clients after the SETATTR is complete using 
> > CB_SIZECHANGED, and while waiting for the response from the 
> > callback, reject all LAYOUTGETS and LAYOUTRETURNS from the 
> > client.  Once the client acknowledges the callback, and the 
> > server and client are in sync, normal operation can continue.
> > 
> > I'll re-write the text again to remove the reference to 
> > waiting for the callbacks.
> > 
> > -Steve
> > 
> > 
> > > -----Original Message-----
> > > From: Everhart, Craig [mailto:[email-masked]]
> > > Sent: Thursday, June 29, 2006 4:22 PM
> > > To: Noveck, Dave; Goodson, Garth; fridella, stephen
> > > Cc: [email-masked]
> > > Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> > > 
> > > I agree with Dave.  I think the most the server can tell the clients 
> > > before committing the truncate is that they no longer know what EOF 
> > > is, presumably by recalling layouts, and this can return DELAY to the 
> > > original client if needed.  So the only time the server can commit to 
> > > a new size is while clients don't share this knowledge, but they know 
> > > they need to check things out with the server.  If you're growing 
> > > sizes, that could work OK, but I think shrinking can't be using 
> > > SIZECHANGED.
> > > 
> > > > -----Original Message-----
> > > > From: Noveck, Dave
> > > > Sent: Thursday, June 29, 2006 3:24 PM
> > > > To: Goodson, Garth; [email-masked]
> > > > Cc: [email-masked]
> > > > Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> > > > 
> > > > At this point I'm not sure who, but somebody (Steven?) said:
> > > > > The idea was that the server should be sure that the clients have
a
> > > > > consistent view of the file size before allowing the truncate to 
> > > > > succeed.
> > > > 
> > > > Because of the DELAY, I'm not at all sure you can do that. 
> > > > If you accept the new size to the point of telling other clients 
> > > > that it is indeed the new size (and they depend on this), then you 
> > > > have allowed the truncate to proceed pretty far, so it isn't true 
> > > > that everybody has the same view, before you proceed to do the 
> > > > truncate.  Then if return DELAY, you are really in trouble because 
> > > > you have committed yourself to the other clients but the initiating 
> > > > client has been told it hasn't happened.
> > > > 
> > > > I don't think you can make the server's acceptance the truncate 
> > > > conditional on completion of the notification.  I think the model 
> > > > has to be that you accept the truncate and then notify everybody.
> > > > You are free to grievously restrict clients can do until everybody 
> > > > gets the notification and agrees but I don't think you can 
> > > > consistently both refuse to commit yourself to the truncate while 
> > > > propagating notifications predicated on what is essentially a 
> > > > commitment to actually performing the truncate.
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: Goodson, Garth
> > > > Sent: Thursday, June 29, 2006 12:18 PM
> > > > To: [email-masked]
> > > > Cc: Goodson, Garth; [email-masked]
> > > > Subject: Re: [nfsv4] Issue 21: pnfs EOF handling
> > > > 
> > > > 
> > > > 
> > > > 
> > > > [email-masked] wrote:
> > > > >> -----Original Message-----
> > > > >> From: Garth Goodson [mailto:[email-masked]]
> > > > >> Sent: Tuesday, June 27, 2006 4:51 PM
> > > > >> To: fridella, stephen
> > > > >> Cc: [email-masked]
> > > > >> Subject: Re: [nfsv4] Issue 21: pnfs EOF handling
> > > > >>
> > > > > 
> > > > > <snip>
> > > > > 
> > > > >> Is this too restrictive for some implementations (i.e., impose 
> > > > >> unnecessary delays when the backend system can handle the 
> > > > >> changes)?  My concern arises over the MUST send CB_SIZECHANGED 
> > > > >> and SHOULD NOT respond to the SETATTR until all responses have 
> > > > >> been received (I guess it is a SHOULD NOT rather than a MUST 
> > > > >> NOT).  Remind me why we need to wait for the CB replies.
> > > > >>
> > > > > 
> > > > > The idea was that the server should be sure that the clients have 
> > > > > a consistent view of the file size before allowing the truncate to

> > > > > succeed. I suppose the problem is that different layout types have

> > > > > different requirements for the consistency of file size, with the 
> > > > > file layout type having the loosest requirements of all.  I don't 
> > > > > mind making this a layout-type specific requirement.  The question

> > > > > is, what do we say in the generic spec?  Perhaps we could say 
> > > > > something like:
> > > > > 
> > > > > 
> > > > > Depending on the requirements of the storage protocol, the server 
> > > > > MAY need to ensure that all clients have a consistent view of the 
> > > > > file size before allowing the SETATTR to succeed.
> > > > > If so, the server MAY need to delay processing the SETATTR until 
> > > > > receiving responses to the associated CB_SIZECHANGED and 
> > > > > CB_LAYOUTRECALL messages that it has sent to clients.
> > > > > The precise details of when the server can process the SETATTR are

> > > > > described in the layout-specific specifications.
> > > > > 
> > > > > 
> > > > I do prefer something like this.  It gives a bit more latitude and 
> > > > allows for layout-specific details to be handled separately.
> > > > 
> > > > > 
> > > > >> I also want to bring up something else that Spencer brought up 
> > > > >> earlier.
> > > > >>   It went something like this:
> > > > >>
> > > > >> 1) SETATTR that truncates file arrives
> > > > >> 2) Server sends out CB_SIZECHANGED, and DELAY error to setattr
caller
> > > > >> 3) Setattr caller never retries SETATTR, but CB has been sent out
> > > > >>
> > > > >> I'm not sure how to work around this.  Is this a problem -- or is

> > > > >> it ok that the client can confuse other clients?
> > > > >>
> > > > >> -Garth
> > > > >>
> > > > > 
> > > > > It does seem like a problem.  I have to think some more about it.

> > > > > The danger is that a client will believe that its LAYOUT is beyond

> > > > > the end of file, and hence not committed, when in fact, it is 
> > > > > actually fine (since the truncate never occurred).  In this case 
> > > > > the cost is the client re-committing a LAYOUT unnecessarily.  
> > > > > Hopefully the server is tolerant of this case.  Is it possible to 
> > > > > require that the server not return DELAY?
> > > > > 
> > > > I'm not an expert on the DELAY rules.  However, it seems that if the
> > > > server needs to block for a possibly long period of time, waiting
for
> > > > all the callback responses, then it must send out a DELAY message.  
> > > > The spec says: 'The server initiated the request, but was not able 
> > > > to complete it in a timely fashion...'  Maybe others can chime in 
> > > > here on how to handle this.
> > > > 
> > > > -Garth
> > > > 
> > > > > --Steve
> > > > > 
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > nfsv4 mailing list
> > > > > [email-masked]
> > > > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > > > 
> > > > _______________________________________________
> > > > nfsv4 mailing list
> > > > [email-masked]
> > > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > > > 
> > > > _______________________________________________
> > > > nfsv4 mailing list
> > > > [email-masked]
> > > > https://www1.ietf.org/mailman/listinfo/nfsv4
> > > > 
> > > 
> > 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Everhart, Craig" <[email-masked]>
Date: Wed, 5 Jul 2006 14:25:07 -0400

So, David, you're proposing, in the name of pragmatism, that:
	client processing of SIZECHANGED should be quick;
	and servers should not reply to the SETATTR request
		until all clients have replied to SIZECHANGED;
	servers that need special resources [read: threads] in order not
to reply should allocate those resources up front, possibly returning
DELAY before issuing any SIZECHANGED callbacks to clients;
	servers should regard truncations as committed before issuing
any SIZECHANGED callbacks.

Does that about cover it?

One "new" state is then the client that acts on a long EOF by writing
data and committing layouts at the same time that the server has
committed (whatever that means) its shorter EOF and just hasn't gotten
the news to that client yet.  How confused will that client get?

Are there any other weirdnesses that are caused by clients thinking they
know EOF while the server in fact disagrees with them?

I agree that it is reasonable to delay the SETATTR caller...

		Craig

> From: [email-masked] [mailto:[email-masked]] 
> Sent: Monday, July 03, 2006 8:59 PM
> To: Everhart, Craig
> Cc: [email-masked]
> The concern in the block/volume layout, is that unlike file 
> or object where the servers track the file sizes, the 
> block/volume clients have to track sizes.  The result is that 
> moving the EOF across a block layout range held by another 
> client can require that other client to change the state of 
> its layout.  The crucial case is that for truncate, blocks 
> that previously did not require a commit in response to a 
> write have to be changed to a state where they now require 
> one (and if there was pending delayed write, it's abandoned). 
>  If one gets too lazy about EOF propagation, one can lose 
> writes that shouldn't have been lost in a truncate case 
> (client write has an  application level dependency on an EOF 
> change that propagated to the application via some means 
> faster than pNFS).
> I don't think anything goes wrong in the extend case.  The 
> original proposal to not complete the SETATTR until the 
> callbacks are dealt with in some fashion was intended to 
> prevent this race.
> 
> A potential problem with recalling all affected layouts is a 
> case in which a scratch file is truncated to zero between 
> rounds of a parallel computation - having the server tell 
> each of the clients with write permission that their layout 
> just went from valid data to invalid data is significantly 
> more effective than recalling everything followed by an 
> influx of requests to get the layouts back.
> 
> > > > I agree with Dave.  I think the most the server can tell the 
> > > > clients before committing the truncate is that they no 
> longer know 
> > > > what EOF is, presumably by recalling layouts, and this 
> can return 
> > > > DELAY to the original client if needed.
> 
> In contrast to layout recalls, EOF changes don't absolutely 
> require clients to do anything in response to the EOF change 
> (all the state updates are internal); the EOF callback can be 
> responded to even if there's a return for the block 
> containing the new EOF in the same callback compound.  If the 
> server bails out and returns DELAY to a client after issuing 
> EOF change callbacks to other clients, the result can be just 
> plain wrong for the block layout because the EOF change 
> callback can have irreversible effects at clients (dirty 
> cached blocks beyond new EOF may get discarded instead of 
> written to storage).  The HighRoad design approach was that 
> I think there may be an approach that comes close to this 
> which is to be careful about operation admission on the 
> server side, and only take a SETATTR that truncates if 
> there's plenty of time to get to the clients before a DELAY 
> might be needed (i.e., aggressively issue a DELAY if there's 
> been a delay in the inbound operation for some reason, but 
> never issue one if an EOF change callback has been issued, 
> rely on clients to respond to them quickly).
> 
> I think the idea of stalling the pNFS clients so that one can 
> respond to the SETATTR quickly is probably backwards - the 
> client that issued the SETATTR is causing the problem, so it 
> should be stalled for as long as it takes, and have the pNFS 
> clients implemented to deal with the EOF callback quickly - 
> the immediate reply to the EOF callback RPC should get the job done.
> 
> Thanks,
> --David
> 
> > 	clients B, C, D, ... et al. that had layouts beyond the 
> new EOF (and 
> > its new last block) are in a *new* state with respect to 
> the server, 
> > "I'm ignoring all your LAYOUTGET and LAYOUTRETURN 
> operations until I 
> > like you better?"
> > 
> > OPTIMISTIC_OUTOFSYNCH code?  How do they know they're being ignored 
> > and that they have to wade through their input queues to get to the
> CB_SIZECHANGED op?
> > 
> > Presumably B, C, D could use a SETATTR themselves, right?  What 
> > happens to their layouts then?
> > 
> > > Your view that layouts must be recalled creates problems for a 
> > > block/volume layout.  With this philosophy each SETATTR 
> will force 
> > > the deallocation of all blocks beyond the new end of file.
> > > Clients that continue to write beyond the end of file (because 
> > > applications still have the file open) must ask the server to 
> > > re-allocate blocks for them to write into and provide 
> them with the 
> > > new layout.  The original motivation behind the CB_SIZECHANGED 
> > > (suggested by David B, I believe) was to find a way to 
> optimize this 
> > > cost associated with SETATTR.
> > > 
> > > The basic idea was to inform clients of the change in 
> size, but to 
> > > allow layouts beyond the end of file to be preserved.
> > > state, and would need to be re-committed by the client.  
> For clients 
> > > with applications that continue to write beyond the eof after the 
> > > SETATTR, this reduces the impact of the SETATTR from a 
> > > RECALL/RETURN/GETLAYOUT/COMMIT to a SIZECHANGED/COMMIT.
> > > *require* the server to wait for the CB_SIZECHANGED callbacks to 
> > > complete before processing the SETATTR.  It says only that some 
> > > layout types may choose this method of operation.  So the problem 
> > > that Dave N and Craig describe only applies to specific 
> layout types 
> > > (like block/volume).
> > > 
> > > But to address the problem in the block/volume layout 
> type, I think 
> > > we can use the approach the Dave N alludes to in his
> > > note:
> > > 
> these we 
> > > MUST recall).  At the server, preserve the layouts that 
> lie beyond 
> > > the EOF but mark them as uncommitted.  Inform the clients 
> after the 
> > > SETATTR is complete using CB_SIZECHANGED, and while 
> waiting for the 
> > > response from the callback, reject all LAYOUTGETS and 
> LAYOUTRETURNS 
> > > from the client.  Once the client acknowledges the 
> callback, and the 
> > > > I agree with Dave.  I think the most the server can tell the 
> > > > clients before committing the truncate is that they no 
> longer know 
> > > > what EOF is, presumably by recalling layouts, and this 
> can return 
> > > > DELAY to the original client if needed.  So the only time the 
> > > > server can commit to a new size is while clients don't 
> share this 
> > > > knowledge, but they know they need to check things out with the 
> > > > server.  If you're growing sizes, that could work OK, 
> but I think 
> > > > shrinking can't be using SIZECHANGED.
> > > > 
> > > > > > The idea was that the server should be sure that 
> the clients 
> > > > > > have
> a
> > > > > > consistent view of the file size before allowing 
> the truncate 
> > > > > > to succeed.
> > > > > 
> > > > > that it is indeed the new size (and they depend on 
> this), then 
> > > > > you have allowed the truncate to proceed pretty far, 
> so it isn't 
> > > > > true that everybody has the same view, before you 
> proceed to do 
> > > > > the truncate.  Then if return DELAY, you are really 
> in trouble 
> > > > > because you have committed yourself to the other 
> clients but the 
> > > > > initiating client has been told it hasn't happened.
> > > > > 
> > > > > conditional on completion of the notification.  I think the 
> > > > > model has to be that you accept the truncate and then 
> notify everybody.
> > > > > You are free to grievously restrict clients can do until 
> > > > > everybody gets the notification and agrees but I 
> don't think you 
> > > > > can consistently both refuse to commit yourself to 
> the truncate 
> > > > > while propagating notifications predicated on what is 
> > > > > essentially a commitment to actually performing the truncate.
> > > > > 
> > > > > 
> > > > > >> Is this too restrictive for some implementations (i.e., 
> > > > > >> impose unnecessary delays when the backend system 
> can handle 
> > > > > >> the changes)?  My concern arises over the MUST send 
> > > > > >> CB_SIZECHANGED and SHOULD NOT respond to the SETATTR until 
> > > > > >> all responses have been received (I guess it is a 
> SHOULD NOT 
> > > > > >> rather than a MUST NOT).  Remind me why we need to 
> wait for the CB replies.
> > > > > >>
> > > > > > 
> > > > > > The idea was that the server should be sure that 
> the clients 
> > > > > > have a consistent view of the file size before allowing the 
> > > > > > truncate to
> 
> > > > > > succeed. I suppose the problem is that different 
> layout types 
> > > > > > have
> 
> > > > > > different requirements for the consistency of file 
> size, with 
> > > > > > the file layout type having the loosest 
> requirements of all.  
> > > > > > I don't mind making this a layout-type specific 
> requirement.  
> > > > > > The question
> 
> > > > > > Depending on the requirements of the storage protocol, the 
> > > > > > server MAY need to ensure that all clients have a 
> consistent 
> > > > > > view of the file size before allowing the SETATTR 
> to succeed.
> > > > > > If so, the server MAY need to delay processing the SETATTR 
> > > > > > until receiving responses to the associated 
> CB_SIZECHANGED and 
> > > > > > The precise details of when the server can process 
> the SETATTR 
> > > > > > are
> 
> > > > > I do prefer something like this.  It gives a bit more 
> latitude 
> > > > > and allows for layout-specific details to be handled 
> separately.
> > > > > 
> > > > > > 
> > > > > >> I also want to bring up something else that 
> Spencer brought 
> > > > > >> up earlier.
> > > > > >> 3) Setattr caller never retries SETATTR, but CB 
> has been sent 
> > > > > >> out
> > > > > >>
> > > > > >> I'm not sure how to work around this.  Is this a 
> problem -- 
> > > > > >> or is
> 
> > > > > > actually fine (since the truncate never occurred).  In this 
> > > > > > case the cost is the client re-committing a LAYOUT 
> unnecessarily.
> > > > > > Hopefully the server is tolerant of this case.  Is 
> it possible 
> > > > > > to require that the server not return DELAY?
> > > > > > 
> > > > > I'm not an expert on the DELAY rules.  However, it 
> seems that if 
> > > > > the server needs to block for a possibly long period of time, 
> > > > > waiting
> for
> > > > > The spec says: 'The server initiated the request, but was not 
> > > > > able to complete it in a timely fashion...'  Maybe others can 
> > > > > chime in here on how to handle this.
> > > > > 
> > > > > -Garth
> > > > > 
> > > > > > --Steve
> > > > > > 
> > > > > > 
> > > > > >