Subject: Re: [nfsv4] More layout stuff: layout return on close

From: Benny Halevy <[email-masked]>
Date: Wed, 01 Dec 2010 17:49:54 +0200

On 2010-12-01 00:14, Trond Myklebust wrote:
> On Tue, 2010-11-30 at 18:18 +0200, Benny Halevy wrote:
>> On 2010-11-29 20:53, Trond Myklebust wrote:
>>> On Mon, 2010-11-29 at 13:27 -0500, [email-masked] wrote:
>>>> There are three possibilities that occur to me.  If there are more, I'd
>>>> prefer not to know :-)
>>>>
>>>> Stateid-that-created the layout:  I take your point about usefulness,
>>>> and the text doesn't say anything specifically that says this is right.
>>>> On the other hand, if it is not used for this (determining when to
>>>> return return-on-close segments), what is it used for?  It seems like it
>>>> should be used for something, if it there.
>>>>
>>>> Any-close: Not sure this is very useful, but if you were just trying to
>>>> read what the spec says, it seems like that is what the spec says.
>>>> There are lots of places where you would expect it to say "last close"
>>>> if that was what was meant, but I don't see them.  Does it say this
>>>> anywhere?
>>>>
>>>> Last-close: This seems the most useful choice but it isn't what the spec
>>>> says, so if we can all agree that this is what it should say, maybe the
>>>> errata mechanism would be used to fix this as well.
>>>
>>> Yes, it does appear that a spec clarification is needed here too.
>>>
>>>> Also, it occurs to me that delegation adds another level
>>>> complexity/nightmare and a delegation stateid can be passed just as a
>>>> open or lock stateid can.  One purpose of the delegation is that
>>>> opens/closes on the client, as long as there is a delegation can be
>>>> hidden from the server, so I guess this means that the responsibility
>>>> for effecting a last-close return, when a delegation is present is for
>>>> the client to do by means of LAYOUTRETURN to the server.  Given that the
>>>> client is in control, he knows the ordering and doesn't have to be
>>>> conservative on open/close ordering.
>>>
>>> Doesn't have to be, but should be allowed to be.
>>>
>>> I'm gunning for a client position where we don't have to add any extra
>>> serialisation between LAYOUTGET and LAYOUTRETURN, but can rely on the
>>> forgetful model to let us drop the layout whenever we please.
>>>
>>
>> We need to separate between the layout and the layout stateid.
>> The server doesn't care weather the client uses the layout it was given or not.
>> That's up to the client.  However, the server and client must agree on
>> the layout stateid.
> 
> Yes. Why do they not?
> 

One corner issue that I'm concerned about is what stateid results in
CB_LAYOUTRECALL that does not cover the whole layout (either byte range or
iomode) that returns CB_NOMATCHING_LAYOUT.  We want this case to be equivalent
to a LAYOUTRETURN for the recalled range, however, unlike LAYOUTRETURN
the server does not provide the client with the resulting stateid
and if the client dropped the layout segments in the remainder of the recalled
range it may believe it has no layout and reset its layout stateid, while
the server thinks it still holds a part of the layout and will just update
the layout seqid.

>>> In such a scenario, the automatic last-close return is actually quite
>>> useful, since it allows the server to be forgetful too...
>>>
>>
>> What do you mean by that?
> 
> I mean that the server does not have to wait for and/or recall those
> layouts that are close-on-return if the client fails to send
> LAYOUTRETURNs before CLOSE. It just forgets them.
> 

OK. I just don't consider this "forgetting" but rather responding to
an implicit return...

Benny

---

From: Benny Halevy <[email-masked]>
Date: Wed, 01 Dec 2010 18:00:40 +0200

On 2010-11-30 20:17, Trond Myklebust wrote:
> On Tue, 2010-11-30 at 17:56 +0200, Benny Halevy wrote:
>> On 2010-11-30 17:26, Trond Myklebust wrote:
>>> On Tue, 2010-11-30 at 10:20 -0500, Trond Myklebust wrote:
>>>> On Tue, 2010-11-30 at 17:14 +0200, Benny Halevy wrote:
>>>>> On 2010-11-29 01:23, [email-masked] wrote:
>>>>>> I think these two case are different.
>>>>>>
>>>>>> I think the CB_LAYOUTRECALL case does have sequencing issues but they
>>>>>> are dealt with, at least in the case of LAYOUTRECALL4_FILE, by the fact
>>>>>> that that the seqid in lor_stateid tells the client what revision of
>>>>>> layout state is being referred to, and thus implicitly tells the server
>>>>>> what layout changes in flight are or are not covered by the
>>>>>> NFS4ERR_NOMATCHING_LAYOUT.
>>>>>>
>>>>>> As Benny points out, the CLOSE doesn't have that, but I don't see an
>>>>>> ordering problem.  The seqid ordering mechanism is to make sure that the
>>>>>> client and the server agree on the corpus of layout-segments with
>>>>>> return-on-close best thought of as attribute for each segment part of
>>>>>> that layout segment corpus.  The close doesn't change that corpus in a
>>>>>> way that interferes with ongoing LAYOUT operations.  It merely
>>>>>> eliminates from active use those bearing the return-on-close attribute
>>>>>> and that shift can be done independently on the client and the server.
>>>>>>
>>>>>> Let's get more concrete here.  I don't see how there should be ordering
>>>>>> issues between close and LAYOUTRETURNs.  If the segment is returned
>>>>>> before the close or after the close, it is still gone.  I don't see any
>>>>>> perceivable difference.  Now let's take the case of LAYOUTGET and close.
>>>>>> Suppose you get a LAYOUTGET response, and it overlaps (for simplicity
>>>>>> let use say it is identical but I don't think the overlap case is really
>>>>>> any different).  What determines whether the client should treat this as
>>>>>> a segment he still has (post-close), is whether it has the
>>>>>> return-on-close flag set.  If it does, he should assume it is gone (and
>>>>>> it must have been sent before the close was processed).  If it doesn't
>>>>>> have that set he should assume it still exists, apart from ordering.  If
>>>>>> it was sent before the close, then the layout segment was upgraded to
>>>>>> non-open-specific, so it doesn't go away at close.  OTOH, if it was sent
>>>>>> after the close, then the segment went away at the close and was added
>>>>>> afterwards.  However, in either case, he (the client) should assume he
>>>>>> has still has it and the that server matches in that regard.
>>>>>
>>>>> How about the resulting layout stateid?
>>>>> What should the client use on the next layout operation?
>>>>
>>>> The same layout stateid he would have used if the server had recalled
>>>> that particular layout segment and he'd replied NOMATCHING_LAYOUT.
>>>>
>>
>> I think that gets ugly with parallel LAYOUTGETs that has return_on_close
>> set, since the client does not know if the server processed them
>> before the CLOSE or after it.  Even if they are ignored the layout stateid
>> can get out of sync.
> 
> That depends entirely on how we clarify the meaning of return-on-close,
> and is one of several reasons why I'm pushing for return-on-close ==
> return-on-last-close (i.e. when the client holds no more open state on
> the file).
> Please see the other discussion.
> 

Yup.  I'm with you on that.

>> I'm very nervous about the forgetful model's fuzziness.
>> We need a detailed algorithm and a simulator that verifies it to make
>> sure the client and server so not go out of sync and in no case
>> the client ends up holding a layout the server is not aware of.
> 
> The spec says that we support it, and that's the model I want to see
> implemented in Linux.

I'm still not buying that it really makes the client much simpler BTW.
Personally, I'm inclined towards the explicit layoutreturn model.

> If there is remaining fuzziness, then please point
> out where.
> 

Sure.  The simulator does layout gets and returns at the moment
and I'm working on verification and then adding layout recall, forgetful
client, and return-on-close.  I sure hope that it will help find holes
in our algorithm, if any.

>>>> IOW: the most recent stateid returned by LAYOUTGET, unless the client
>>>> has forgotten all layout segments for that file (i.e. the whole layout),
>>>> in which case it needs to use the open stateid.
>>>
>>> So, the thing to remember here is that the layout stateid is only useful
>>> for ordering LAYOUTGET w.r.t. LAYOUTRETURN. It has no other purpose.
>>
>> Wrong.  It's also critical to handling of CB_LAYOUTRECALL and in particular
>> draining outstanding LAYOUTGETs and LAYOUTRETURNs if the stateid presented by
>> CB_LAYOUTRECALL is greater than the one the client holds plus one.
>> If you want this semantic for CLOSE then the safe thing to do is to serialize
>> the last CLOSE with LAYOUTGET and LAYOUTRETURN and then the stateid will be either
>> the current layout stateid plus one if there are other layout segments w/o
>> return_on_close, or reset to the non-layout stateid in case the client never
>> got such segments or it _explicitly_ returned them with LAYOUTRETURN or CB_LAYOUTRECALL+
>> NFS4ERR_NOMATCHINGLAYOUT.
> 
> No LAYOUTRETURN is required by the protocol if you are forgetful, and I
> refuse to add a load more serialisation primitives just in order to
> support the crazy serialisation crap that
> LAYOUTGET/LAYOUTRECALL/LAYOUTRETURN requires. The above paragraph is
> exactly the reason why I hate the non-forgetful model.
> 

Before removing serialization primitive just because you hate them we need
more than hand waving to make sure we're not shooting ourselves in the foot.

>> If the client just forgot them, it can't just reset the stateid on its own.
> 
> Sure it can. If the client doesn't know the current value of the stateid
> (because it has forgotten it), then it signals that by using the open
> stateid. How else can the model work?
> 

Well, that's not the way the spec is currently defined.
If we change it to say that, then it will restrict the client
to serialize LAYOUTGETs carrying the non-layout stateid.
That's quite a burden. 

>>> If the client is using the forgetful model, then there is no
>>> LAYOUTRETURN, and so there is no ordering problem. Instead, the client
>>> uses the principle that it must be conservative w.r.t. ordering
>>> assumptions if it launches an OPEN+LAYOUTGET in parallel with a last
>>> close...
>>
>> Also outstanding LAYOUTGETs sent before the open that has return_on_close set.
>> This is why the safe thing is to implement a barrier and drain them before CLOSEing.
> 
> Why would you have outstanding LAYOUTGETs if you are CLOSEing the file?
> Either there is more to read and write, and you may have outstanding
> LAYOUTGETs, or there is nothing more to read/write and so you can close
> the file.

That's fine, as long as the client makes sure there's no race.

Benny

> Trond

---

From: Benny Halevy <[email-masked]>
Date: Wed, 01 Dec 2010 18:11:00 +0200

On 2010-12-01 00:48, [email-masked] wrote:
> Let me just say I have to say with no quoting.  Once you have five sets
> of greater-than's, it just gets impossible.
> 
> We're talking about scrapping RFC5661?!  What I mean is Trond is talking
> about that, and we do feel his pain :-)  But still, it seems a bit,
> well, extreme.
> 
> My interpretation was (and maybe there is something in the spec which
> contradicts this) or it is in some way wrong is:
> 
>      The purpose of the seqid in the CB_LAYOUTRECALL
>      stateid is to define what version of the layout
>      state corpus being referred to, since, if 
>      you the client don't know, you can't know what
>      I (the server) am talking about.
> 
>      So if you have a LAYOUTGET or return in flight
>      and the CB_LAYOUTRECALL is beyond the one I have
>      received (i.e. it is more than one more than the 
>      last one I have received), I cannot process the 
>      recall and should wait until I can.  

Agreed.  A second case which I'm not sure is explicitly called
is when the client holds a non-layout stateid.  In this case,
if the CB_LAYOUTRECALL's seqid equals 1 then it is a superfluous
recall and the client can return NOMATCHING_LAYOUT, otherwise
it will have a seqid greater than one and the client should follow
the case you mentioned above.

> 
>      If I don't do that, my no-matching-layout is 
>      of no value because it may mean that I just haven't
>      seen the layout that was recalled yet and so the 
>      server could not deduce anything from my 
>      no-matching-layout response.  Me being forgetful
>      is different from me lacking precognition :-)
> 

True.

>      As has been noted, the CB_LAYOUTRECALL doesn't 
>      change state, so it doesn't really make sense for
>      it to change the seqid.  I know this seems wrong to
>      Fred, and maybe it is wrong, but if it is, I think 
>      we need a stronger argument, of the form "If you 
>      don't do this, then this case breaks".

We must define what the stateid ends up being after NFS4ERR_NOMATCHING_LAYOUT.
Off the top of my head I propose using the CB_LAYOUTRECALL stateid if it is
a partial recall or, if it is a whole file recall or a bulk recall with
iomode==IOMODE_ANY, reset the stateid the the initial state (an open/deleg/lock stateid).

> 
>      The seqid sequencing applies to state transitions
>      that are made on the server (and provides for ordering
>      those successive versions of *server* layout state).
>      So, having somebody other than the server change the
>      seqid, seems wrong to me.  Maybe I'm wrong.

The issue is that NOMATCHING_LAYOUT implies a state change as it is equivalent
in some sense to a LAYOUTRETURN of the recalled range.

> 
>      If the server and the client are both changing the 
>      seqid, I need good strong drink, before somebody
>      explains to me how that can result in anything but
>      chaos.  (Just one drink;  I really will listen).
> 

Count me in :-)

Benny

> It seems to me from the stuff I have heard recently is that many of the
> problems are not in the protocol per se (i.e. it is not full of
> mistakes), but rather in the explanation, which does need work to be
> clearer.  I certainly could be proven wrong.
> 
>  
> 
> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Trond Myklebust
> Sent: Tuesday, November 30, 2010 5:11 PM
> To: Fred Isaman
> Cc: Benny Halevy; [email-masked]
> Subject: Re: [nfsv4] More layout stuff: layout return on close
> 
> On Tue, 2010-11-30 at 10:39 -0500, Fred Isaman wrote:
>> On Tue, Nov 30, 2010 at 10:26 AM, Trond Myklebust
>> <[email-masked]> wrote:
>> The CB_LAYOUTRECALL supplies a stateid.  Are you suggesting that the
>> client should not update its seqid to that of the recall when it
>> replies NOMATCHING_LAYOUT?  Because that seems wrong.
> 
> Oh crap! You're kidding me....
> 
> We invented a whole new damned convention for bloody stateid seqids ONLY
> for use in the damned CB_LAYOUTRECALL?
> 
> I've got a better idea. Let's scrap RFC5661 and start over again. This
> time, we pay attention to the existing conventions for stateids. One of
> those conventions is that if the seqid changes occur _only_ when a state
> change occurs.
> 
> No wonder we're having problems...
> 
>> It is also for ordering of LAYOUTGET wrt CB_LAYOUTRECALL.  For all
>> intents and purposes, a return of NOMATCHING_LAYOUT to a
>> CB_LAYOUTRECALL is a LAYOUTRETURN, and ordering needs to be paid
>> attention to.
> 
> Right, but there is no state transition for a layoutrecall. Just like a
> delegation recall, it leaves the state unchanged as far as the client is
> concerned...
> 
> IOW: if we had made the damned LAYOUTRECALL take a stateid with the
> _same_ seqid as the most recent LAYOUTGET that is being recalled, then
> we wouldn't be in this ridiculous debate.
> 
> 
> The point is: as far as the client is concerned, forgetting a layout
> segment is not a seqid-changing state transition. Apparently we have to
> do the seqid change on LAYOUTRECALL, but that is an aberration due to
> apparent blindness on the part of the spec reviewers (of which I was
> one, yes). There is no reason to introduce the same insanity everywhere
> else.
> 
> Trond