Subject: RE: [nfsv4] setattr request and CB_SIZECHANGED

From: [email-masked]
Date: Wed, 8 Mar 2006 16:50:35 -0500

Ok, so nobody (but me) seems to like the idea of layout
based side-effects of CB_SIZECHANGED.

Well, it should be possible for us to correctly handle
the SETATTR case for block/volume by using LAYOUTRECALL
for any conflicting layouts beyond the new end-of-file, and
to let the server handle any on-disk details (partial block
zeroing, etc).  This approach pretty much means we don't
need to send the CB_SIZECHANGED at all.

So it sounds like there will be a semantic split between
the sub-protocols around how they process SETATTR.  Object
and file servers may simply let the SETATTR complete
and send out CB_SIZECHANGED after the fact, while block-
volume servers will most likely delay the SETATTR while
yanking layouts from clients.  My question is:  does this
semantic split require explanatory text in the generic
doc, or should the differences only be explained in the
layout-specific documents?

-Steve 

> -----Original Message-----
> From: Garth Goodson [mailto:[email-masked]] 
> Sent: Wednesday, March 08, 2006 12:35 PM
> To: Noveck, Dave
> Cc: Benny Halevy; [email-masked]; [email-masked]
> Subject: Re: [nfsv4] setattr request and CB_SIZECHANGED
> 
> 
> Noveck, Dave wrote:
> >>If the layout becomes invalid beyond the new EOF then I 
> >>think it should be implicitly recalled rather than implying 
> that as a 
> >>side effect of CB_SIZECHANGED.
> > 
> > 
> > I'm assuming you mean "explicitly recalled", in which case I
> > agree.
> > 
> 
> If for any reason any layout that is being held becomes 
> invalid in any 
> part of its range, I believe the layout MUST be recalled.  The files 
> case doesn't have this issue, so I think we can give the server the 
> option of recalling layouts (although, it always has this 
> option at any 
> time), but I wouldn't make it mandatory to recall a layout due to EOF 
> changes.
> 
> Also, I'm all for adding clarifying text about SETATTR and 
> CB_SIZECHANGED synchronization.  But remember that only generic pnfs 
> text should go into the v4.1 draft, layout specific text 
> should go into 
> the respective layout doc.
> 
> -Garth
> 
> > -----Original Message-----
> > From: Benny Halevy [mailto:[email-masked]]
> > Sent: Wednesday, March 08, 2006 10:21 AM
> > To: [email-masked]
> > Cc: [email-masked]
> > Subject: Re: [nfsv4] setattr request and CB_SIZECHANGED
> > 
> > 
> > [email-masked] wrote:
> > 
> > 
> >>Hi All--
> >>
> >>Just to follow up some discussion that we had on the
> >>con-call a couple of weeks back... I want to talk about
> >>the CB_SIZECHANGED callback and SETATTR.  The current
> >>i-d is fairly vague about how to synchronize the processing
> >>of the SETATTR and the CB_SIZECHANGED callback.  David
> >>Black and I discussed it, and we believe that the right
> >>way to go (at least from the block/volume perspective) is
> >>to delay the actual processing of the SETATTR until all
> >>clients with outstanding layouts have been notified about
> >>the pending size change, and have had a chance to take
> >>appropriate actions.  For block/volume clients, these
> >>actions may include partial zeroing of the new end-of-file
> >>block if the client holds a layout on it, and then committing
> >>the updated layout back to the server.  
> >>
> > 
> > Writing to storage and committing layouts (without returning them?) 
> > seems to me well beyond the intended scope of 
> CB_SIZECHANGED.  What's 
> > the state of any complete block in the layout beyond the 
> new EOF after 
> > this call?  If the layout becomes invalid beyond the new EOF then I 
> > think it should be implicitly recalled rather than implying 
> that as a 
> > side effect of CB_SIZECHANGED.
> > As an optimization, we may want to consider adding an optional 
> > "new_size" field to CB_LAYOUTRECALL that can provide a hint to the 
> > client not to bother flushing any dirty data beyond the new 
> size but 
> > rather just drop it from their cache, and in addition, for 
> block based 
> > layouts, the client must zero out the bytes from the new 
> size up to the 
> > end of the corresponding block, if it has a layout covering 
> that block 
> > (what happens if multiple clients hold a reference to this 
> offset?  Is 
> > it always true they will point to different blocks on disk 
> so they will 
> > never step on each other's toes?)
> > 
> > 
> >>Object and file
> >>clients may have different actions that need to be taken.
> >>In general however, it seems reasonable that the server
> >>should only process the SETATTR if it is sure that the
> >>new state information has been propagated to all the clients
> >>that need it.
> >>
> >>Our thinking is that we need some text in the i-d which
> >>specifically states how SETATTR and CB_SIZECHANGED should
> >>be synchronized.  Other thoughts?
> >>
> >>-Steve
> >>
> >>------------------------------------------------------------
> >>Stephen Fridella                            ph: 508-305-8512
> >>Principal Software Engineer                fax: 508-305-4803
> >>EMC NAS Advanced Development         sfridell AT emc DOT com
> >>
> >>
> >>
> >>_______________________________________________
> >>nfsv4 mailing list
> >>[email-masked]
> >>https://www1.ietf.org/mailman/listinfo/nfsv4
> >> 
> >>
> > 
> > 
> > 
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/nfsv4
> > 
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: [email-masked]
Date: Wed, 8 Mar 2006 16:09:53 -0500

From: Spencer Shepler [mailto:[email-masked]] 
> 
> On Tue, [email-masked] wrote:
> > Hi All--
> > 
> > the updated layout back to the server.  Object and file
> Not other thoughts but clarification of behavior.
> 
> How sure should the server be that the SETATTR is likely to
> succeed before doing the CB_SIZECHANGED call?  Obviously
> passing all checks up to the final change in size?  Seems
> reasonable.
> 
> I assume that the server should return DELAY to the setattr
> while it awaits the clients to sort out their details.
> In that case, the client that instantiated the SETATTR may
> not send the request again.  Does that change the behavior
> you describe?
> 
> Spencer
>

I think that we should be committed to doing the size change
if we are going to ask clients to perform actions that change
the contents of the file.  So yes, if we return DELAY and the
client doesn't come back that creates a problem.

This discussion may be moot, however, since the general reaction
to my suggested side-effects of CB_SIZECHANGED was:  "GAG!!!"
More on that in another email... 
-Steve