Subject: [nfsv4] More layout stuff: layout return on close

From: Trond Myklebust <[email-masked]>
Date: Tue, 23 Nov 2010 17:47:39 -0500

Another conflict between the forgetful pNFS client model and RFC5661.

Currently, section 18.43.3 states that:

        "The logr_return_on_close result field is a directive to return
        the layout before closing the file.  When the metadata server
        sets this return value to TRUE, it MUST be prepared to recall
        the layout in the case in which the client fails to return the
        layout before close."

The point is that the client is fully entitled by section 12.5.5.1 to
forget that it holds a layout for this file altogether. With this in
mind, the requirement in 18.43.3 appears excessively bureaucratic:
      * the server _knows_ that the client is required to return layouts
        on close,
      * it _knows_ that the client may have forgotten it holds layouts
yet it is required to go through the motions of issuing a layout recall?

Can anyone think of a reason why we should not just allow the server to
assume the client has forgotten all layouts when it issues the close in
apparent violation of the 'return on close' here?

Cheers
  Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Benny Halevy <[email-masked]>
Date: Wed, 24 Nov 2010 09:49:22 +0200

> apparent violation of the 'return on close' here?

Let me rephrase that to see if I understood your suggestion correctly.
You're basically suggesting additional semantics for CLOSE that would imply
return of the whole layout for the file, when any layout segments gotten
with a stateid derived from the respective open stateid that stated
return_on_close==true.

I like that though I think that the semantics can be tuned down a notch to just
those layout segments marked with return_on_close.  This can be useful in
case the server, for example, requests return_on_close only for RW layouts
while the client can keep the READ layout across CLOSE.

What I really don't want is to define this non-definitively so to leave
it up to the client to forget and up to the server to assume the client
forgot the layout.

Benny

---

From: Tigran Mkrtchyan <[email-masked]>
Date: Wed, 24 Nov 2010 10:55:16 +0100

> it up to the client to forget and up to the server to assume the client

As a main reason why we have return_on_close in the spec I will allow me
to comment.
 
We are in a quite weird situation here. From one side servers asks  a
client to return
layout on close, but from other side you have to keep track of the
layout and recall it to
ensure that the client will not use it any more.  I would love to see
this part of the spec to
be more deterministic. But it we change it not to be a more strong
requirements, then
some of the current clients will become not complaint with the spec. Are
we ok with that?
And of course I have to change layout handling in our server we rely on
layoutreturn. Which
is not a big change (I think). BTW, is there any other server uses
return_on_close ?

Tigran.
> Benny
>
>   
>> Cheers
>>   Trond
>>
>>     
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Benny Halevy <[email-masked]>
Date: Wed, 24 Nov 2010 12:15:02 +0200

> is not a big change (I think). BTW, is there any other server uses

Yes. Panasas uses return_on_close.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 24 Nov 2010 08:56:17 -0500

> while the client can keep the READ layout across CLOSE.

I agree. It should only apply to the return-on-close segments. However
if the server has labeled all segments with return-on-close, then that
would imply the entire layout has been returned.

Note that CLOSE in this case may also mean DELEGRETURN.

> it up to the client to forget and up to the server to assume the client

ACK.

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 24 Nov 2010 09:56:31 -0500

> some of the current clients will become not complaint with the spec. Are

I fail to see your point. Which clients would become spec incompliant if
we relax the requirement that they must send a layoutreturn before close
when the server sets the logr_return_on_close flag? Relaxing a
requirement is not equivalent to prohibiting the behaviour.

> is not a big change (I think). BTW, is there any other server uses

You don't have to change the server, but if you don't then it will
perform less than optimally with forgetful clients.

Trond

NetApp

---

From: Tigran Mkrtchyan <[email-masked]>
Date: Thu, 25 Nov 2010 10:11:25 +0100

> requirement is not equivalent to prohibiting the behaviour.
If I read it correctly, then spec change will allow servers to forget 
client layout on close.
The new clients will simply forget layout on close. Nevertheless the old 
( existing ) clients
are allowed to reuse it as return_on_close was ( are ) only a hint.

May be I see it wrong.

Tigran.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 25 Nov 2010 08:49:08 -0500

> If I read it correctly, then spec change will allow servers to forget 

Only for layout segments that have return on close set.

> The new clients will simply forget layout on close. Nevertheless the old 

The forgetful client model is already part of the spec. This is not new
client behaviour. Please see RFC5661 section 12.5.5.1.

> are allowed to reuse it as return_on_close was ( are ) only a hint.

That is incorrect: layout return_on_close is not a hint. In RFC5661
section 18.43.3, the server is required to recall the layout if the
client doesn't return it.

The point is that if that is _mandatory_ behaviour (i.e. the layout
segment MUST be returned) then why not just say that if the client
doesn't return it, then that layout segment is automatically revoked? No
recall should be necessary, since both parties know that it is being
revoked.

Trond

NetApp

---

From: Tigran Mkrtchyan <[email-masked]>
Date: Thu, 25 Nov 2010 17:11:21 +0100

> recall should be necessary, since both parties know that it is being
      it MUST be prepared to recall the layout in the
     case in which the client fails to return the layout before close.

As there is no 'client MUST return' I read it as a hint.
Anyway, I think we are talking the same thing. So, yes, I am happy with 
that.

Tigran.

---

From: <[email-masked]>
Date: Thu, 25 Nov 2010 13:56:19 -0500

The text says it is a "directive".  Unfortunately, it does not clarify
that with a "MAY", "SHOULD", or "MUST".

Still a "directive" seems quite a way from a "hint".

The problem here is that the "MUST" which follows this is about the
server, not the client.  Saying the server "MUST" recall, does suggest
that the "directive" for the client is not all that forceful, and you
could go all the way down to "hint" I suppose.  It's more a hint of the
form "Why don't you return that segment (hint, hint), since if you don't
I'm going to have recall it anyway."  If it is a hint, it is not a hint
suggesting that you could somehow keep the layout segment.

The question I think we have is whether we can justifiably address this
via the errata mechanism.  Are we fixing a mistake, rather than just
improving the protocol, which would be a nice thing for v4.2, and not
something for v4.1.

I would argue that that the "MUST" is inconsistent with the forgetful
client model which the spec allows and so is we can treat this as a
mistake.  Also, we are not making any existing client or server
non-compliant.

My understanding of the correction:

     Flag still described as a "directive".

     Follow-up is that client MUST forget (i.e. no longer use) the
segments 
     marked that way.  Client MAY advise server of their non-use, by
LAYOUTRETURN
     but server has enough information to infer layout segment non-use
without 
     that.

     Server MUST assume such layout segments are not to be further used
when
     CLOSE occurs.
 
     Server MAY recall such layout segments if they have not been
returned, but 
     there is no requirement that he do so.  He SHOULD ensure that if
they are
     used subsequently by the client, this is treated as an error.

Comments?

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Tigran Mkrtchyan
Sent: Thursday, November 25, 2010 11:11 AM
To: [email-masked]
Cc: Trond Myklebust
Subject: Re: [nfsv4] More layout stuff: layout return on close

Tigran.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 25 Nov 2010 14:08:16 -0500

>      used subsequently by the client, this is treated as an error.

That interpretation works for me, and yes, I do agree it should be
considered a v4.1 errata rather than a v4.2 protocol.

Cheers
  Trond

NetApp

---

From: "Sorin Faibish" <[email-masked]>
Date: Thu, 25 Nov 2010 15:08:36 -0500

>      used subsequently by the client, this is treated as an error.
I am not confortable with the posibility of an "error" such as the
client will use the segments following a CLOSE. How would the server know
that the clients used the segments. You do not imply that the server has
to check anything.

-- 
Best Regards
Sorin Faibish
Corporate Distinguished Engineer
Unified Storage Division

        EMC²
where information lives

Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 25 Nov 2010 15:15:30 -0500

> that the clients used the segments. You do not imply that the server has

How is this any different from the client using the segments following a
LAYOUTRETURN?

Trond

NetApp

---

From: "Sorin Faibish" <[email-masked]>
Date: Thu, 25 Nov 2010 15:33:32 -0500

> How is this any different from the client using the segments following a
The file is still OPEN so the use a returned layout can be an error of
the client that the server can detect while access of a CLOSED file is more
difficult to detect as the server doesn't keep track of any activity of
the client on that file. There sre no layouts that the server sent to the
client. Otherwise the server has to keep track of layouts of closed files?
Ough..

/Sorin

        EMC²
where information lives

---

From: <[email-masked]>
Date: Thu, 25 Nov 2010 15:40:35 -0500

First of all, this is stated as "SHOULD".  If you can't do it, you can't do it.  But clearly it is desirable for the server to be aware.  Whether the server keeps track of activity of closed files is up to it.

If the server makes all of the layouts return-on-close, then it doesn't have to keep track of information about closed files.  It only keeps track of information about layouts on open files.  Then, if a request is made that doesn't match something in that set of layout (all for open files), it is rejected.

If the server makes no layouts return-on-close, then it does keep track of such layouts and allows the client to retain them, but there is no change there.

If it makes some layouts or some segments, then it has some layouts that exist beyond the open file, but then that is a choice it has made.

I don't see that we are making anything more difficult here. 

-----Original Message-----
From: Sorin Faibish [mailto:[email-masked]] 
Sent: Thursday, November 25, 2010 3:34 PM
To: Trond Myklebust
Cc: Noveck, David; [email-masked]; [email-masked]
Subject: Re: [nfsv4] More layout stuff: layout return on close

/Sorin

        EMC²
where information lives

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 25 Nov 2010 15:42:26 -0500

> client. Otherwise the server has to keep track of layouts of closed files?

Again, how is this any different from having the client first do a
LAYOUTRETURN, CLOSE and then continuing to use the layout segments that
it just returned?
My point is that if you have no mechanism for detecting and fencing off
rogue clients, then this errata changes nothing: a client that wants to
reuse a layout that is no longer valid is going to be able do so
irrespectively of whether or not it actually needs to send a
layoutreturn before close.

Trond

NetApp

---

From: <[email-masked]>
Date: Thu, 25 Nov 2010 16:11:20 -0500

The difference is that I made that explicit in my text.  I'm willing to
change "SHOULD" to "should" if that matches what the spec says if the
LAYOUTRETURN actually were done.  But I think Trond is right about the
basic point that these are fundamentally the same issue as regards the
difficulty of server checking, and that that is true without regard to
the explicitness (or not) of the return. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Trond Myklebust
Sent: Thursday, November 25, 2010 3:42 PM
To: faibish, sorin
Cc: [email-masked]
Subject: Re: [nfsv4] More layout stuff: layout return on close

On Thu, 2010-11-25 at 15:33 -0500, Sorin Faibish wrote:
> >> > could go all the way down to "hint" I suppose.  It's more a hint
of  
> >> the
> >> > form "Why don't you return that segment (hint, hint), since if
you  
> >> don't
> >> > I'm going to have recall it anyway."  If it is a hint, it is not
a  
> >> hint
> >> > The question I think we have is whether we can justifiably
address  
> >> this
> >> >      Server MUST assume such layout segments are not to be
further  
> >> used
> >> client will use the segments following a CLOSE. How would the
server  
> >> know

Trond

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Sun, 28 Nov 2010 15:00:32 +0200

>      used subsequently by the client, this is treated as an error.

Re-thinking the resolution of this implicit return of the layout
and its interaction with other layout state-changing operations;
since CLOSE does not return a layout stateid, like LAYOUTRETURN,
it would be useful only to return the whole file layout, and not just
the segments marked with return_on_close.  Otherwise, what would be
the layout stateid post CLOSE, and how will the client order this
state change with other in-flight LAYOUTGETs and/or LAYOUTRETURNs.

So I suggest we make the implicit layout return an equivalent of:

lora_reclaim == false (let's say that during the grace period the
client must used LAYOUTRETURN with lora_reclaim == true)
lora_layout_type == good question.  I'm inclined towards defining it
as a wildcard, i.e. all layout_types from which the client got a layout
for this file.
lora_iomode == LAYOUTIOMODE4_ANY
lr_returntype == LAYOUTRETURN4_FILE
lrf_offset == 0
lrf_length == NFS4_MAXFILELEN
lrf_stateid == the most recent layout stateid provided by the server
empty lrf_body

The implicit successful result is equivalent to
lrs_present == false
and any outstanding LAYOUTGETs and LAYOUTRETURNs, sent before CLOSE
MUST be drained and ignored by the client.
Any new layout operation MUST be issued after re-opening the file
and obtaining a new open, lock, or delegation stateid.

In addition, the following error codes that are currently applicable
for LAYOUTRETURN and not for CLOSE need to be added to CLOSE:
NFS4ERR_DELEG_REVOKED
NFS4ERR_GRACE

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Sun, 28 Nov 2010 13:27:27 -0500

> state change with other in-flight LAYOUTGETs and/or LAYOUTRETURNs.

Err.... In what way does this differ from the client returning
NFS4ERR_NOMATCHING_LAYOUT to CB_LAYOUTRECALL? The spec allows the client
to do so in the case where it holds no layout segments that overlap with
the layout recall argument. In that case the server just updates its
idea of what the layout looks like without changing the stateid.

> for LAYOUTRETURN and not for CLOSE need to be added to CLOSE:

That seems unnecessary.

If your client cares about updating the stateid (why would it?) then
send the LAYOUTRETURN before the CLOSE, if not, dont, and just forget
the segments that were marked as return-on-close.

Trond

NetApp

---

From: <[email-masked]>
Date: Sun, 28 Nov 2010 18:23:56 -0500

I think these two case are different.

I think the CB_LAYOUTRECALL case does have sequencing issues but they
are dealt with, at least in the case of LAYOUTRECALL4_FILE, by the fact
that that the seqid in lor_stateid tells the client what revision of
layout state is being referred to, and thus implicitly tells the server
what layout changes in flight are or are not covered by the
NFS4ERR_NOMATCHING_LAYOUT.

As Benny points out, the CLOSE doesn't have that, but I don't see an
ordering problem.  The seqid ordering mechanism is to make sure that the
client and the server agree on the corpus of layout-segments with
return-on-close best thought of as attribute for each segment part of
that layout segment corpus.  The close doesn't change that corpus in a
way that interferes with ongoing LAYOUT operations.  It merely
eliminates from active use those bearing the return-on-close attribute
and that shift can be done independently on the client and the server.

Let's get more concrete here.  I don't see how there should be ordering
issues between close and LAYOUTRETURNs.  If the segment is returned
before the close or after the close, it is still gone.  I don't see any
perceivable difference.  Now let's take the case of LAYOUTGET and close.
Suppose you get a LAYOUTGET response, and it overlaps (for simplicity
let use say it is identical but I don't think the overlap case is really
any different).  What determines whether the client should treat this as
a segment he still has (post-close), is whether it has the
return-on-close flag set.  If it does, he should assume it is gone (and
it must have been sent before the close was processed).  If it doesn't
have that set he should assume it still exists, apart from ordering.  If
it was sent before the close, then the layout segment was upgraded to
non-open-specific, so it doesn't go away at close.  OTOH, if it was sent
after the close, then the segment went away at the close and was added
afterwards.  However, in either case, he (the client) should assume he
has still has it and the that server matches in that regard.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Trond Myklebust
Sent: Sunday, November 28, 2010 1:27 PM
To: Benny Halevy
Cc: [email-masked]
Subject: Re: [nfsv4] More layout stuff: layout return on close

That seems unnecessary.

Trond

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 29 Nov 2010 10:43:07 -0500

> has still has it and the that server matches in that regard.

The client knows if this is the last close, and so even if it sends out
a parallel OPEN + LAYOUTGET, then it knows that the server may have
received the close before the parallel LAYOUTGET. As long as the client
is conservative, and forgets about the segments that may have been
invalidated by the close, then the forgetful model works fine.

Note that you cannot use the OPEN stateid to figure out the ordering
here, since the close and OPEN may have different associated open owners
(and/or the close may really have been a DELEGRETURN), so you really do
have to rely on the client being conservative.

NetApp

---

From: <[email-masked]>
Date: Mon, 29 Nov 2010 12:43:34 -0500

So I think we have held different positions (I held many different
positions in the last five minutes) about what close(s) return-on-close
is referring to.  When I wrote my last post I was assuming that it
referred to only the open associated with the stateid from which the
layout stateid was derived.

I can see that I was jumping to a conclusion, but the alternatives that
allow the close and thus the open to which return-on-close refers to be
uncertain.  I'm not clear how this can work in the general case.

So you are referring to the "last close" and I assume that that means
you think that only "last close" (last close among all open of the fh
for the same client, for the same openowner?).  Is that said somewhere
in the spec?  Am I missing some explanation of this in the spec?  

I take your general point that the conservative approach on the client
can resolve the synchronization issues, at least without layout value
conflict, but what I'm having trouble with is understanding how this
feature might be useful to the server.  That is a layout segment, that
is associated with (some set of) open(s), and will go away under some
set of closes, with it going away even more frequently on the client due
to client conservatism.  Is this really useful, when all the complexity
associated with it, is fully recognized?  (Nor a rhetorical question). 

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Monday, November 29, 2010 10:43 AM
To: Noveck, David
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] More layout stuff: layout return on close

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 29 Nov 2010 12:57:40 -0500

> associated with it, is fully recognized?  (Nor a rhetorical question). 

I've always understood from the spec that layouts are supposed to be
independent of the OPEN stateid/delegation stateid that was used to
obtain them. That is why I have interpreted return-on-close to mean
return before last close.

If you go away from this interpretation, and say that the
return-on-close is tied to the particular stateid that was used in the
original LAYOUTGET, then the client is going to need a way to obtain
per-stateid layouts in order to be able to handle multiple processes
reading and writing to the same file using different file descriptors.

Cheers
  Trond

NetApp

---

From: <[email-masked]>
Date: Mon, 29 Nov 2010 13:27:55 -0500

There are three possibilities that occur to me.  If there are more, I'd
prefer not to know :-)

Stateid-that-created the layout:  I take your point about usefulness,
and the text doesn't say anything specifically that says this is right.
On the other hand, if it is not used for this (determining when to
return return-on-close segments), what is it used for?  It seems like it
should be used for something, if it there.

Any-close: Not sure this is very useful, but if you were just trying to
read what the spec says, it seems like that is what the spec says.
There are lots of places where you would expect it to say "last close"
if that was what was meant, but I don't see them.  Does it say this
anywhere?

Last-close: This seems the most useful choice but it isn't what the spec
says, so if we can all agree that this is what it should say, maybe the
errata mechanism would be used to fix this as well.

Also, it occurs to me that delegation adds another level
complexity/nightmare and a delegation stateid can be passed just as a
open or lock stateid can.  One purpose of the delegation is that
opens/closes on the client, as long as there is a delegation can be
hidden from the server, so I guess this means that the responsibility
for effecting a last-close return, when a delegation is present is for
the client to do by means of LAYOUTRETURN to the server.  Given that the
client is in control, he knows the ordering and doesn't have to be
conservative on open/close ordering.

Finally, let me ask this question.  You pass a stateid, which if it is
not layout stateid, is either a lock stateid, and open stateid, or a
delegation stateid.  What I seem to be hearing (and correct me if I'm
wrong), is that the only information conveyed by any of these three to
the server is the fh of the file.  That is any lock stateid is
equivalent to the associated open stated; any open stateid is equivalent
to any other open-stateid (given the last-close interpretation, anyway),
and that is equivalent of any delegation stateid for that same file.  If
not, what are the semantic differences among these three?  

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Monday, November 29, 2010 12:58 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]
Subject: RE: [nfsv4] More layout stuff: layout return on close

Cheers
  Trond

> > any different).  What determines whether the client should treat
this
> as

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 29 Nov 2010 13:53:39 -0500

> errata mechanism would be used to fix this as well.

Yes, it does appear that a spec clarification is needed here too.

> conservative on open/close ordering.

Doesn't have to be, but should be allowed to be.

I'm gunning for a client position where we don't have to add any extra
serialisation between LAYOUTGET and LAYOUTRETURN, but can rely on the
forgetful model to let us drop the layout whenever we please.

In such a scenario, the automatic last-close return is actually quite
useful, since it allows the server to be forgetful too...

> not, what are the semantic differences among these three?  

I seem to recollect that the main purpose of passing the open
stateid/delegation stateid (not sure if we ever discussed lock stateid)
was to ensure that the client does indeed have the file open.
IOW: it is to ensure that the client doesn't issue a grab-bag of
LAYOUTGETs before it even knows whether or not it will need them.

The resulting layout's lifespan is, however, not tied to the lifespan of
the stateid that was used. The only possible exception to this rule is
the return-on-close that we are discussing in this thread.

Cheers
  Trond

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Tue, 30 Nov 2010 17:14:02 +0200

> has still has it and the that server matches in that regard.

How about the resulting layout stateid?
What should the client use on the next layout operation?

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 30 Nov 2010 10:20:18 -0500

> What should the client use on the next layout operation?

The same layout stateid he would have used if the server had recalled
that particular layout segment and he'd replied NOMATCHING_LAYOUT.

IOW: the most recent stateid returned by LAYOUTGET, unless the client
has forgotten all layout segments for that file (i.e. the whole layout),
in which case it needs to use the open stateid.

Trond

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 30 Nov 2010 10:26:21 -0500

> in which case it needs to use the open stateid.

So, the thing to remember here is that the layout stateid is only useful
for ordering LAYOUTGET w.r.t. LAYOUTRETURN. It has no other purpose.

If the client is using the forgetful model, then there is no
LAYOUTRETURN, and so there is no ordering problem. Instead, the client
uses the principle that it must be conservative w.r.t. ordering
assumptions if it launches an OPEN+LAYOUTGET in parallel with a last
close...

Trond

Trond

---

From: Fred Isaman <[email-masked]>
Date: Tue, 30 Nov 2010 10:39:12 -0500

>> that particular layout segment and he'd replied NOMATCHING_LAYOUT.

The CB_LAYOUTRECALL supplies a stateid.  Are you suggesting that the
client should not update its seqid to that of the recall when it
replies NOMATCHING_LAYOUT?  Because that seems wrong.

> for ordering LAYOUTGET w.r.t. LAYOUTRETURN. It has no other purpose.

It is also for ordering of LAYOUTGET wrt CB_LAYOUTRECALL.  For all
intents and purposes, a return of NOMATCHING_LAYOUT to a
CB_LAYOUTRECALL is a LAYOUTRETURN, and ordering needs to be paid
attention to.

Fred

---

From: Benny Halevy <[email-masked]>
Date: Tue, 30 Nov 2010 17:56:33 +0200

>> that particular layout segment and he'd replied NOMATCHING_LAYOUT.

I think that gets ugly with parallel LAYOUTGETs that has return_on_close
set, since the client does not know if the server processed them
before the CLOSE or after it.  Even if they are ignored the layout stateid
can get out of sync.

I'm very nervous about the forgetful model's fuzziness.
We need a detailed algorithm and a simulator that verifies it to make
sure the client and server so not go out of sync and in no case
the client ends up holding a layout the server is not aware of.

> for ordering LAYOUTGET w.r.t. LAYOUTRETURN. It has no other purpose.

Wrong.  It's also critical to handling of CB_LAYOUTRECALL and in particular
draining outstanding LAYOUTGETs and LAYOUTRETURNs if the stateid presented by
CB_LAYOUTRECALL is greater than the one the client holds plus one.
If you want this semantic for CLOSE then the safe thing to do is to serialize
the last CLOSE with LAYOUTGET and LAYOUTRETURN and then the stateid will be either
the current layout stateid plus one if there are other layout segments w/o
return_on_close, or reset to the non-layout stateid in case the client never
got such segments or it _explicitly_ returned them with LAYOUTRETURN or CB_LAYOUTRECALL+
NFS4ERR_NOMATCHINGLAYOUT.

If the client just forgot them, it can't just reset the stateid on its own.

> assumptions if it launches an OPEN+LAYOUTGET in parallel with a last

Also outstanding LAYOUTGETs sent before the open that has return_on_close set.
This is why the safe thing is to implement a barrier and drain them before CLOSEing.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Tue, 30 Nov 2010 18:18:32 +0200

> forgetful model to let us drop the layout whenever we please.

We need to separate between the layout and the layout stateid.
The server doesn't care weather the client uses the layout it was given or not.
That's up to the client.  However, the server and client must agree on
the layout stateid.

> useful, since it allows the server to be forgetful too...

What do you mean by that?

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 30 Nov 2010 13:17:41 -0500

> before the CLOSE or after it.  Even if they are ignored the layout stateid

That depends entirely on how we clarify the meaning of return-on-close,
and is one of several reasons why I'm pushing for return-on-close ==
return-on-last-close (i.e. when the client holds no more open state on
the file).
Please see the other discussion.

> the client ends up holding a layout the server is not aware of.

The spec says that we support it, and that's the model I want to see
implemented in Linux. If there is remaining fuzziness, then please point
out where.

> NFS4ERR_NOMATCHINGLAYOUT.

No LAYOUTRETURN is required by the protocol if you are forgetful, and I
refuse to add a load more serialisation primitives just in order to
support the crazy serialisation crap that
LAYOUTGET/LAYOUTRECALL/LAYOUTRETURN requires. The above paragraph is
exactly the reason why I hate the non-forgetful model.

> If the client just forgot them, it can't just reset the stateid on its own.

Sure it can. If the client doesn't know the current value of the stateid
(because it has forgotten it), then it signals that by using the open
stateid. How else can the model work?

> This is why the safe thing is to implement a barrier and drain them before CLOSEing.

Why would you have outstanding LAYOUTGETs if you are CLOSEing the file?
Either there is more to read and write, and you may have outstanding
LAYOUTGETs, or there is nothing more to read/write and so you can close
the file.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 30 Nov 2010 17:10:32 -0500

> replies NOMATCHING_LAYOUT?  Because that seems wrong.

Oh crap! You're kidding me....

We invented a whole new damned convention for bloody stateid seqids ONLY
for use in the damned CB_LAYOUTRECALL?

I've got a better idea. Let's scrap RFC5661 and start over again. This
time, we pay attention to the existing conventions for stateids. One of
those conventions is that if the seqid changes occur _only_ when a state
change occurs.

No wonder we're having problems...

> CB_LAYOUTRECALL is a LAYOUTRETURN, and ordering needs to be paid

Right, but there is no state transition for a layoutrecall. Just like a
delegation recall, it leaves the state unchanged as far as the client is
concerned...

IOW: if we had made the damned LAYOUTRECALL take a stateid with the
_same_ seqid as the most recent LAYOUTGET that is being recalled, then
we wouldn't be in this ridiculous debate.

The point is: as far as the client is concerned, forgetting a layout
segment is not a seqid-changing state transition. Apparently we have to
do the seqid change on LAYOUTRECALL, but that is an aberration due to
apparent blindness on the part of the spec reviewers (of which I was
one, yes). There is no reason to introduce the same insanity everywhere
else.

Trond

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 30 Nov 2010 17:14:53 -0500

> That's up to the client.  However, the server and client must agree on

Yes. Why do they not?

> What do you mean by that?

I mean that the server does not have to wait for and/or recall those
layouts that are close-on-return if the client fails to send
LAYOUTRETURNs before CLOSE. It just forgets them.

NetApp

---

From: <[email-masked]>
Date: Tue, 30 Nov 2010 17:48:17 -0500

Let me just say I have to say with no quoting.  Once you have five sets
of greater-than's, it just gets impossible.

We're talking about scrapping RFC5661?!  What I mean is Trond is talking
about that, and we do feel his pain :-)  But still, it seems a bit,
well, extreme.

My interpretation was (and maybe there is something in the spec which
contradicts this) or it is in some way wrong is:

     The purpose of the seqid in the CB_LAYOUTRECALL
     stateid is to define what version of the layout
     state corpus being referred to, since, if 
     you the client don't know, you can't know what
     I (the server) am talking about.

     So if you have a LAYOUTGET or return in flight
     and the CB_LAYOUTRECALL is beyond the one I have
     received (i.e. it is more than one more than the 
     last one I have received), I cannot process the 
     recall and should wait until I can.  

     If I don't do that, my no-matching-layout is 
     of no value because it may mean that I just haven't
     seen the layout that was recalled yet and so the 
     server could not deduce anything from my 
     no-matching-layout response.  Me being forgetful
     is different from me lacking precognition :-)

     As has been noted, the CB_LAYOUTRECALL doesn't 
     change state, so it doesn't really make sense for
     it to change the seqid.  I know this seems wrong to
     Fred, and maybe it is wrong, but if it is, I think 
     we need a stronger argument, of the form "If you 
     don't do this, then this case breaks".

     The seqid sequencing applies to state transitions
     that are made on the server (and provides for ordering
     those successive versions of *server* layout state).
     So, having somebody other than the server change the
     seqid, seems wrong to me.  Maybe I'm wrong.

     If the server and the client are both changing the 
     seqid, I need good strong drink, before somebody
     explains to me how that can result in anything but
     chaos.  (Just one drink;  I really will listen).

It seems to me from the stuff I have heard recently is that many of the
problems are not in the protocol per se (i.e. it is not full of
mistakes), but rather in the explanation, which does need work to be
clearer.  I certainly could be proven wrong.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Trond Myklebust
Sent: Tuesday, November 30, 2010 5:11 PM
To: Fred Isaman
Cc: Benny Halevy; [email-masked]
Subject: Re: [nfsv4] More layout stuff: layout return on close

Trond

NetApp

---

From: "Spencer Shepler" <[email-masked]>
Date: Tue, 30 Nov 2010 15:01:13 -0800

Agreed.  The server is responsible for the state transitions and
as you point out, the client is given access to the content so the
ordering of its processing can be made to match the ordering the
server has provided.  Given the various latencies in network and processing
this is needed.

Spencer

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Tuesday, November 30, 2010 2:48 PM
> To: [email-masked]; [email-masked]
> Let me just say I have to say with no quoting.  Once you have five sets of
> greater-than's, it just gets impossible.
> 
> about that, and we do feel his pain :-)  But still, it seems a bit, well,
> extreme.
> 
> problems are not in the protocol per se (i.e. it is not full of mistakes),
> but rather in the explanation, which does need work to be clearer.  I
> certainly could be proven wrong.
> 
> 
> 
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Trond Myklebust
> IOW: if we had made the damned LAYOUTRECALL take a stateid with the _same_
> seqid as the most recent LAYOUTGET that is being recalled, then we
> wouldn't be in this ridiculous debate.
> 
> 
> segment is not a seqid-changing state transition. Apparently we have to do
> the seqid change on LAYOUTRECALL, but that is an aberration due to
> apparent blindness on the part of the spec reviewers (of which I was one,
> yes). There is no reason to introduce the same insanity everywhere else.
> 
> Trond
> --
> Trond Myklebust

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 30 Nov 2010 18:16:23 -0500

> about that, and we do feel his pain :-)  But still, it seems a bit,

These nits are extremely frustrating since they appear without any
accompanying justification.

>      is different from me lacking precognition :-)

Note that we also have the csa_referring_call_lists array in the
CB_SEQUENCE that precedes the CB_LAYOUTRECALL operation, and that allows
the server to notify the client of any outstanding LAYOUTGET calls that
might be affected by this CB_LAYOUTRECALL.

>      don't do this, then this case breaks".

Agreed.

>      chaos.  (Just one drink;  I really will listen).

Mine's the same...

> clearer.  I certainly could be proven wrong.

In this case, the protocol is obfuscating the argument because it is
imposing apparently arbitrary new rules w.r.t. layout stateids. A
clarification should seek to either justify the new rules, (using your
"if we don't do this, then that breaks" guideline) or it should revert
layouts to the established stateid rules, or (if that is too late) it
should explain how we can apply the new rules in the presence of the
forgetful model.

Trond

NetApp