Subject: RE: [nfsv4] Issue 21: pnfs EOF handling

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Jun 2006 15:24:25 -0400

At this point I'm not sure who, but somebody (Steven?) said:
> The idea was that the server should be sure that the clients
> have a consistent view of the file size before allowing the
> truncate to succeed.

Because of the DELAY, I'm not at all sure you can do that.  If
you accept the new size to the point of telling other clients
that it is indeed the new size (and they depend on this), then
you have allowed the truncate to proceed pretty far, so it 
isn't true that everybody has the same view, before you proceed
to do the truncate.  Then if return DELAY, you are really in
trouble because you have committed yourself to the other clients
but the initiating client has been told it hasn't happened.

I don't think you can make the server's acceptance the truncate 
conditional on completion of the notification.  I think the model 
has to be that you accept the truncate and then notify everybody.  
You are free to grievously restrict clients can do until everybody 
gets the notification and agrees but I don't think you can 
consistently both refuse to commit yourself to the truncate 
while propagating notifications predicated on what is essentially 
a commitment to actually performing the truncate.

-----Original Message-----
From: Goodson, Garth 
Sent: Thursday, June 29, 2006 12:18 PM
To: [email-masked]
Cc: Goodson, Garth; [email-masked]
Subject: Re: [nfsv4] Issue 21: pnfs EOF handling

[email-masked] wrote:
>> -----Original Message-----
>> From: Garth Goodson [mailto:[email-masked]] 
>> Sent: Tuesday, June 27, 2006 4:51 PM
>> To: fridella, stephen
>> Cc: [email-masked]
>> Subject: Re: [nfsv4] Issue 21: pnfs EOF handling
>>
> 
> <snip> 
> 
>> Is this too restrictive for some implementations (i.e., impose 
>> unnecessary delays when the backend system can handle the 
>> changes)?  My 
>> concern arises over the MUST send CB_SIZECHANGED and SHOULD 
>> NOT respond 
>> to the SETATTR until all responses have been received (I 
>> guess it is a 
>> SHOULD NOT rather than a MUST NOT).  Remind me why we need to 
>> wait for 
>> the CB replies.
>>
> 
> The idea was that the server should be sure that the clients
> have a consistent view of the file size before allowing the
> truncate to succeed. I suppose the problem is that different
> layout types have different requirements for the consistency
> of file size, with the file layout type having the loosest
> requirements of all.  I don't mind making this a layout-type
> specific requirement.  The question is, what do we say in the
> generic spec?  Perhaps we could say something like:
> 
> 
> Depending on the requirements of the storage protocol, the
> server MAY need to ensure that all clients have a consistent
> view of the file size before allowing the SETATTR to succeed.
> If so, the server MAY need to delay processing the SETATTR
> until receiving responses to the associated CB_SIZECHANGED
> and CB_LAYOUTRECALL messages that it has sent to clients.
> The precise details of when the server can process the
> SETATTR are described in the layout-specific specifications.
> 
> 
I do prefer something like this.  It gives a bit more latitude and 
allows for layout-specific details to be handled separately.

> 
>> I also want to bring up something else that Spencer brought 
>> up earlier. 
>>   It went something like this:
>>
>> 1) SETATTR that truncates file arrives
>> 2) Server sends out CB_SIZECHANGED, and DELAY error to setattr caller
>> 3) Setattr caller never retries SETATTR, but CB has been sent out
>>
>> I'm not sure how to work around this.  Is this a problem -- 
>> or is it ok 
>> that the client can confuse other clients?
>>
>> -Garth
>>
> 
> It does seem like a problem.  I have to think some more
> about it.  The danger is that a client will believe that
> its LAYOUT is beyond the end of file, and hence not committed,
> when in fact, it is actually fine (since the truncate never
> occurred).  In this case the cost is the client re-committing
> a LAYOUT unnecessarily.  Hopefully the server is tolerant
> of this case.  Is it possible to require that the server
> not return DELAY?
> 
I'm not an expert on the DELAY rules.  However, it seems that if the 
server needs to block for a possibly long period of time, waiting for 
all the callback responses, then it must send out a DELAY message.  The 
spec says: 'The server initiated the request, but was not able to 
complete it in a timely fashion...'  Maybe others can chime in here on 
how to handle this.

-Garth

> --Steve
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 29 Jun 2006 16:21:50 -0400

I agree with Dave.  I think the most the server can tell the clients
before committing the truncate is that they no longer know what EOF is,
presumably by recalling layouts, and this can return DELAY to the
original client if needed.  So the only time the server can commit to a
new size is while clients don't share this knowledge, but they know they
need to check things out with the server.  If you're growing sizes, that
could work OK, but I think shrinking can't be using SIZECHANGED.

> From: Noveck, Dave 
> Sent: Thursday, June 29, 2006 3:24 PM
> To: Goodson, Garth; [email-masked]
> Cc: [email-masked]
> Subject: RE: [nfsv4] Issue 21: pnfs EOF handling
> 
> > The idea was that the server should be sure that the clients have a 
> > consistent view of the file size before allowing the truncate to 
> > succeed.
> 
> Because of the DELAY, I'm not at all sure you can do that.  
> If you accept the new size to the point of telling other 
> clients that it is indeed the new size (and they depend on 
> this), then you have allowed the truncate to proceed pretty 
> far, so it isn't true that everybody has the same view, 
> before you proceed to do the truncate.  Then if return DELAY, 
> you are really in trouble because you have committed yourself 
> to the other clients but the initiating client has been told 
> it hasn't happened.
> 
> I don't think you can make the server's acceptance the 
> truncate conditional on completion of the notification.  I 
> think the model has to be that you accept the truncate and 
> then notify everybody.  
> You are free to grievously restrict clients can do until 
> everybody gets the notification and agrees but I don't think 
> you can consistently both refuse to commit yourself to the 
> truncate while propagating notifications predicated on what 
> is essentially a commitment to actually performing the truncate.
> 
>

---

From: [email-masked]
Date: Thu, 29 Jun 2006 17:10:01 -0400

Craig---
Your view that layouts must be recalled creates problems for
a block/volume layout.  With this philosophy each SETATTR will
force the deallocation of all blocks beyond the new end of file.
Clients that continue to write beyond the end of file (because
applications still have the file open) must ask the server to
re-allocate blocks for them to write into and provide them with
the new layout.  The original motivation behind the CB_SIZECHANGED
(suggested by David B, I believe) was to find a way to optimize
this cost associated with SETATTR.

The basic idea was to inform clients of the change in size,
but to allow layouts beyond the end of file to be preserved.
Of course, the layouts would transition to an "uncommitted"
state, and would need to be re-committed by the client.  For
clients with applications that continue to write beyond the
eof after the SETATTR, this reduces the impact of the SETATTR
from a RECALL/RETURN/GETLAYOUT/COMMIT to a SIZECHANGED/COMMIT.
I'd like to find a way to keep this optimization.

I should also note that my new proposed text below does not
*require* the server to wait for the CB_SIZECHANGED callbacks to
complete before processing the SETATTR.  It says only that
some layout types may choose this method of operation.  So
the problem that Dave N and Craig describe only applies to
specific layout types (like block/volume).

But to address the problem in the block/volume layout type,
I think we can use the approach the Dave N alludes to in his
note:

Process the SETATTR immediately without recalling the layouts
(except for layouts that cover the new last file block---
these we MUST recall).  At the server, preserve the layouts
that lie beyond the EOF but mark them as uncommitted.  Inform
the clients after the SETATTR is complete using CB_SIZECHANGED,
and while waiting for the response from the callback, reject all
LAYOUTGETS and LAYOUTRETURNS from the client.  Once the client
acknowledges the callback, and the server and client are in sync,
normal operation can continue.

I'll re-write the text again to remove the reference to waiting
for the callbacks.

-Steve

> From: Everhart, Craig [mailto:[email-masked]] 
> Sent: Thursday, June 29, 2006 4:22 PM
> To: Noveck, Dave; Goodson, Garth; fridella, stephen
> Cc: [email-masked]

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 29 Jun 2006 17:31:46 -0400

OK, so you're saying
	client A sends a SETATTR to truncate a file downward
	server truncates it locally and replies OK
	clients B, C, D, ... et al. that had layouts beyond the new EOF
(and its new last block) are in a *new* state with respect to the
server, "I'm ignoring all your LAYOUTGET and LAYOUTRETURN operations
until I like you better?"

Don't we have to think about this new state a little harder?  What else
is allowed and what else is disallowed?  Does the server drop requests
from B, C, D on the floor, or does it reply with a new
OPTIMISTIC_OUTOFSYNCH code?  How do they know they're being ignored and
that they have to wade through their input queues to get to the
CB_SIZECHANGED op?

Presumably B, C, D could use a SETATTR themselves, right?  What happens
to their layouts then?

Is this an optimization really worth preserving, specifically for the
truncate case?  To support performance optimizations for dueling
applications, one truncating and the other extending?  Am I crazy or
what?

		Craig

> From: [email-masked] [mailto:[email-masked]] 
> Sent: Thursday, June 29, 2006 5:10 PM
> To: Everhart, Craig; Noveck, Dave; Goodson, Garth
> Cc: [email-masked]
> a block/volume layout.  With this philosophy each SETATTR 
> will force the deallocation of all blocks beyond the new end of file.
> Clients that continue to write beyond the end of file 
> (because applications still have the file open) must ask the 
> server to re-allocate blocks for them to write into and 
> provide them with the new layout.  The original motivation 
> behind the CB_SIZECHANGED (suggested by David B, I believe) 
> was to find a way to optimize this cost associated with SETATTR.
> 
> *require* the server to wait for the CB_SIZECHANGED callbacks 
> to complete before processing the SETATTR.  It says only that 
> But to address the problem in the block/volume layout type, I 
> think we can use the approach the Dave N alludes to in his
> note:
> 
> the clients after the SETATTR is complete using 
> CB_SIZECHANGED, and while waiting for the response from the 
> callback, reject all LAYOUTGETS and LAYOUTRETURNS from the 
> client.  Once the client acknowledges the callback, and the 
> server and client are in sync, normal operation can continue.
> 
> I'll re-write the text again to remove the reference to 
> waiting for the callbacks.
> 
> -Steve
> 
> 
> > before committing the truncate is that they no longer know what EOF 
> > is, presumably by recalling layouts, and this can return 
> DELAY to the 
> > original client if needed.  So the only time the server can 
> commit to 
> > a new size is while clients don't share this knowledge, but 
> they know 
> > they need to check things out with the server.  If you're growing 
> > sizes, that could work OK, but I think shrinking can't be using 
> > SIZECHANGED.
> > 
> > > If you accept the new size to the point of telling other clients 
> > > that it is indeed the new size (and they depend on this), 
> then you 
> > > have allowed the truncate to proceed pretty far, so it isn't true 
> > > that everybody has the same view, before you proceed to do the 
> > > truncate.  Then if return DELAY, you are really in 
> trouble because 
> > > you have committed yourself to the other clients but the 
> initiating 
> > > client has been told it hasn't happened.
> > > 
> everybody 
> > > consistently both refuse to commit yourself to the truncate while 
> > > propagating notifications predicated on what is essentially a 
> > > commitment to actually performing the truncate.
> > > 
> > > 
> > > >> changes)?  My concern arises over the MUST send CB_SIZECHANGED 
> > > >> and SHOULD NOT respond to the SETATTR until all responses have 
> > > >> been received (I guess it is a SHOULD NOT rather than a MUST 
> > > >> NOT).  Remind me why we need to wait for the CB replies.
> > > >>
> > > > 
> > > > The idea was that the server should be sure that the 
> clients have 
> > > > a consistent view of the file size before allowing the 
> truncate to 
> > > > succeed. I suppose the problem is that different layout 
> types have 
> > > > different requirements for the consistency of file 
> size, with the 
> > > > file layout type having the loosest requirements of 
> all.  I don't 
> > > > mind making this a layout-type specific requirement.  
> The question 
> > > > is, what do we say in the generic spec?  Perhaps we could say 
> > > > something like:
> > > > 
> > > > 
> > > > Depending on the requirements of the storage protocol, 
> the server 
> > > > MAY need to ensure that all clients have a consistent 
> view of the 
> > > > file size before allowing the SETATTR to succeed.
> > > > If so, the server MAY need to delay processing the 
> SETATTR until 
> > > > receiving responses to the associated CB_SIZECHANGED and 
> > > > CB_LAYOUTRECALL messages that it has sent to clients.
> SETATTR are 
> > > > described in the layout-specific specifications.
> > > > 
> > > > 
> > > >> I also want to bring up something else that Spencer brought up 
> > > >> earlier.
> > > >> I'm not sure how to work around this.  Is this a 
> problem -- or is 
> > > >> it ok that the client can confuse other clients?
> > > >>
> > > >> -Garth
> > > >>
> > > > 
> about it.  
> > > > The danger is that a client will believe that its 
> LAYOUT is beyond 
> > > > the end of file, and hence not committed, when in fact, it is 
> > > > actually fine (since the truncate never occurred).  In 
> this case 
> > > > the cost is the client re-committing a LAYOUT unnecessarily.  
> > > > Hopefully the server is tolerant of this case.  Is it 
> possible to 
> > > > require that the server not return DELAY?
> > > > 
> > > all the callback responses, then it must send out a DELAY 
> message.  
> > > The spec says: 'The server initiated the request, but was 
> not able 
> > > to complete it in a timely fashion...'  Maybe others can chime in 
> > > here on how to handle this.
> > > 
> > > -Garth
> > > 
> > > > --Steve
> > > > 
> > > > 
> > > >