Subject: Re: [nfsv4] [Errata Rejected] RFC5661 (3901)

From: Trond Myklebust <[email-masked]>
Date: Sat, 23 Jan 2016 11:28:31 -0500

Hi Dave,

Apologies for dropping the ball on this question. Please see inlined comments.

On Wed, Dec 16, 2015 at 9:48 PM, David Noveck <[email-masked]> wrote:
> I have a question about the reach of "applies retroactively" in the second
> paragraph.
>
> Suppose the following scenario:
>
> A file is opened and receives stateid OS1.
> A layout is gotten using OS1 and stateid LS1 is returned.
> logr_return_on_close set to FALSE.
> OS1 is closed and there is no open state for the file in question.
> The file is opened again and receives stateid OS2.
> A layout is gotten using OS2 and stateid LS2 is returned.
> logr_return_on_close set to TRUE.
> OS2 is closed and there is (again) no open state for the file in question.
>
> Is it the intention that LS1 is now invalid and MUST NOT be used?  That
> seems to be the case from my reading of the proposed text but I'd like to
> check.

My interpretation of RFC5661 is that return-on-close is applied only
at the layout segment level.
IOW: in the scenario above, LS2 is subject to return-on-close, and so
cannot be used after OS2 is closed, but LS1 is not, and so can be used
until recalled or an explicit LAYOUTRETURN is sent.

The intention of the errata is not to change that interpretation, but
rather to clarify:

a) the timing of when the return-on-close is applied.

The main use case for return-on-close is to allow the server to cut
down the layout metadata operations by limiting the amount of time
that the client is allowed to hold onto the layout.
Since the client is only allowed by RFC5661 to hold one layout at a
time for any given file, if the layout were attached to the specific
open stateid that was used in the first OPEN, this would mean that the
client would have to issue a new set of LAYOUTGET calls on every CLOSE
of that open stateid.

Marc's errata 3226 attempted to clarify this issue by stating that the
client is allowed to hold onto the layout until it no longer holds any
open state.

b) what happens in the presence of a delegation.

Note in particular that RFC 5661 Section 18.43.3. calls out:

"   The loga_stateid field specifies a valid stateid.  If a layout is not
   currently held by the client, the loga_stateid field represents a
   stateid reflecting the correspondingly valid open, byte-range lock,
   or delegation stateid."

meaning that we could in theory get a return-on-close for a byte range
lock stateid or a delegation stateid. RFC5661 leaves it undefined what
happens in that case.

Marc's errata, although it could be said to cover the byte range lock
stateid case, did not consider what happens in the case of
delegations. Hence the need for errata number 3901, which replaced his
initial proposal with a superset that includes delegations.

Cheers
  Trond

>
> On Wed, Dec 16, 2015 at 5:12 PM, RFC Errata System
> <[email-masked]> wrote:
>>
>> The following errata report has been rejected for RFC5661,
>> "Network File System (NFS) Version 4 Minor Version 1 Protocol".
>>
>> --------------------------------------
>> You may review the report below and at:
>> http://www.rfc-editor.org/errata_search.php?rfc=5661&eid=3901
>>
>> --------------------------------------
>> Status: Rejected
>> Type: Technical
>>
>> Reported by: Trond Myklebust <[email-masked]>
>> Date Reported: 2014-02-25
>> Rejected by: Martin Stiemerling (IESG)
>>
>> Section: 18.43.3
>>
>> Original Text
>> -------------
>> The logr_return_on_close result field is a directive to return the
>> layout before closing the file.  When the metadata server sets this
>> return value to TRUE, it MUST be prepared to recall the layout in the
>> case in which the client fails to return the layout before close.
>> For the metadata server that knows a layout must be returned before a
>> close of the file, this return value can be used to communicate the
>> desired behavior to the client and thus remove one extra step from
>> the client's and metadata server's interaction.
>>
>> Corrected Text
>> --------------
>> The logr_return_on_close result field is a directive to return
>> or forget the layout when the client returns all open/delegation
>> state for that file.
>>
>> Once a LAYOUTGET operation returns with logr_return_on_close
>> set to TRUE for a given file, then all subsequent LAYOUTGET
>> requests by that client for the same file and layout type, MUST
>> reply with logr_return_on_close set to TRUE until the client returns
>> all its open state for that file using CLOSE and DELEGRETURN.
>> Note that return_on_close also applies retroactively to all layout
>> segments retrieved by the client for that file and layout type.
>>
>> After the client has closed all open stateids and returned the
>> delegation stateids for a file for which logr_return_on_close
>> was set to TRUE, the server MUST  invalidate all layout segments
>> that were  issued to the client for that file. The client MUST NOT
>> attempt to use that layout or the layout stateid.
>>
>> If the server needs to revoke all open stateids and delegation
>> stateids owned by the client for a file for which logr_return_on_close
>> was set to TRUE, then it MUST also revoke all layout segments of
>> type loga_layout_type that were issued for that file to that client,
>> and take action to fence the access to the DSes.
>>
>> Notes
>> -----
>> This is intended as a replacement for the errata with id 3226, which is
>> incomplete in that it does not discuss how return-on-close is supposed to
>> work with delegations or with layout revoking.
>>  --VERIFIER NOTES--
>>  Please get an agreement in the WG and submit an up to date errata, as
>> this text part seems to be under discussion.
>>
>> --------------------------------------
>> RFC5661 (draft-ietf-nfsv4-minorversion1-29)
>> --------------------------------------
>> Title               : Network File System (NFS) Version 4 Minor Version 1
>> Protocol
>> Publication Date    : January 2010
>> Author(s)           : S. Shepler, Ed., M. Eisler, Ed., D. Noveck, Ed.
>> Category            : PROPOSED STANDARD
>> Source              : Network File System Version 4
>> Area                : Transport
>> Stream              : IETF
>> Verifying Party     : IESG
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/nfsv4
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: David Noveck <[email-masked]>
Date: Thu, 28 Jan 2016 14:46:53 -0500

Here's my version of the corrected text:

The logr_return_on_close result field acts as a directive to the client.  A
client receiving this flag is to return or forget the layout once it returns
all open/delegation state for that file.

Once a LAYOUTGET operation returns with logr_return_on_close set to TRUE
for a given file, then, for all subsequent LAYOUTGET requests by that
client for the same file and layout type, the server MUST reply with
logr_return_on_close set to TRUE until the client returns all its open
state for that file using CLOSE and DELEGRETURN.

It should be noted that the return_on_close directive applies to layout
segments on which it was not set.  In the event that one request returns
with this bit not set and a later one returns with it set, the latter has
priority, as long as there has been no occasion between the receipt of the
earlier layout segment and the newer one in which the client held no open
or delegation state for the file in question.

After the client has closed all open stateids and returned the delegation
stateids for a file for which logr_return_on_close was set to TRUE, the
server MUST  invalidate all layout segments to which the directive applies that
were  issued to the client for that file. These include layout segments
which were received with the directive set and  those to which it applies
as specified in the paragraph above. The client MUST NOT attempt to use
that layout or the layout stateid.

If the server needs to revoke all open stateids and delegation stateids
owned by the client for a file for which logr_return_on_close was set to
TRUE, then it MUST also revoke all layout segments of type loga_layout_type
that were issued for that file to that client, and take action to fence the
access to the DSes.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 28 Jan 2016 15:00:00 -0500

> or delegation state for the file in question.
This last paragraph differs from the previous version of the errata. In
that version, the return-on-close bit was said to apply only to the layout
segments for which it was explicitly set. Is there any reason for the
change? It would require changes to the Linux client.

> that were issued for that file to that client, and take action to fence the
The rest looks OK to me. Thanks!