Subject: [nfsv4] LAYOUTCOMMIT byte range

From: Benny Halevy <[email-masked]>
Date: Wed, 30 Mar 2011 18:57:31 +0200

Following up on discussion over email and today in the IETF meeting
I'd like to make the following proposal.

Background:
RFC5661 defines a byte range for the LAYOUTCOMMIT operation represented
by <loca_offset, loca_length> and then it has restrictions for using it
regarding its overlap with a layout with LAYOUTIOMODE4_RW.

When discussing the implementation of LAYOUTCOMMIT it was suggested to
limit the operation's byte range to particular layout segment
potentially resulting in sending multiple LAYOUTCOMMIT operations,
one for each segment.

Further discussion raised even more issues with the blocks layout type
which contains an extents list in the layoutupdate4 contents that
may be fragmented if the client did not write to all blocks in the
layout range and it was not clear from RFC5661 and RFC5663 if the
client is allowed to send a LAYOUTCOMMIT with a fragmented extents list
or whether it should send a LAYOUTCOMMIT for each contiguous byte range.

More Facts:
1. The loca_stateid argument to LAYOUTCOMMIT should be sufficient for
server to determine an upper bound on the client's layout state
at the time the operation was sent, regardless of the LAYOUTCOMMITted
range. In particular the server knows which RW layout segments the
client has.

2. The server uses the LAYOUTCOMMIT byte range in a generic way
if loca_reclaim is true as follows (see 12.7.4 for more info):
      If the metadata server's
      consistency checks on loca_layoutupdate succeed, then the metadata
      server MUST commit the data (as described by the loca_offset,
      loca_length, and loca_layoutupdate fields of the arguments) that
      was written to the storage device.

3. The file and object layouts ignore the LAYOUTCOMMIT byte range.

4. The blocks layout has all the byte range metadata it needs as part
of the extents list passed down in loca_layoutupdate.

Therefore, I propose a simplification for the protocol that essentially
deprecates loca_offset and loca_length when loca_reclaim is false
and keeps their use if loca_reclaim is true.

For the blocks layout I propose to allow to pass a fragmented extents
list in LAYOUTCOMMIT and only restrict it so the committed extents must
be completely contained in valid RW layout the client holds.

loca_stateid MUST be verified by the server to ensure LAYOUTCOMMIT is
processed before a conflicting LAYOUTRETURN, otherwise return
NFS4ERR_{BAD,OLD}_STATEID as needed. Note that with the blocks layout
type LAYOUTRETURN may deallocate provisionally allocated space that
was not explicitly LAYOUTCOMMITted (a.k.a holes) and the client may
lose data if it did not successfully LAYOUTCOMMITted its changes
before LAYOUTRETURN.

Coming to think about it, I'm not sure that the spec actually defines
the desired order of events, so I think we need to add text instructing
the client to first make sure data written to data servers is synced to
stable storage before sending a LAYOUTCOMMIT.  Yet, since LAYOUTCOMMIT
is not serialized with I/O, this constraint makes sense only for data
at loca_last_write_offset, and for the blocks layout type, the data
in blu_commit_list (taking into assumption possible block protocols with
write behind properties).

Benny

---

From: Suchit Kaura <[email-masked]>
Date: Wed, 30 Mar 2011 12:19:55 -0700

Hi,

> 1. The loca_stateid argument to LAYOUTCOMMIT should be sufficient for server to determine an upper bound on the client's > layout state at the time the operation was sent, regardless of the LAYOUTCOMMITted range. In particular the server knows > which RW layout segments the client has.
I believe the layout range requested can be the complete file so the upper bound is not deterministic.

> 3. The file and object layouts ignore the LAYOUTCOMMIT byte range. 
This is not true for all file layout based servers. The file layout based servers using asymmetric model can use this information as an indication of what was the last client offset that was written. I believe we agreed on this as part of the proposal from Mike after the discussion in interim IETF meeting, the errata for the same was captured by Ricardo and reviewed. 

Regards,
Suchit

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Benny Halevy
Sent: Wednesday, March 30, 2011 9:58 AM
To: nfsv4 list; David Black
Subject: [nfsv4] LAYOUTCOMMIT byte range

Following up on discussion over email and today in the IETF meeting I'd like to make the following proposal.

Background:
RFC5661 defines a byte range for the LAYOUTCOMMIT operation represented by <loca_offset, loca_length> and then it has restrictions for using it regarding its overlap with a layout with LAYOUTIOMODE4_RW.

When discussing the implementation of LAYOUTCOMMIT it was suggested to limit the operation's byte range to particular layout segment potentially resulting in sending multiple LAYOUTCOMMIT operations, one for each segment.

Further discussion raised even more issues with the blocks layout type which contains an extents list in the layoutupdate4 contents that may be fragmented if the client did not write to all blocks in the layout range and it was not clear from RFC5661 and RFC5663 if the client is allowed to send a LAYOUTCOMMIT with a fragmented extents list or whether it should send a LAYOUTCOMMIT for each contiguous byte range.

More Facts:
1. The loca_stateid argument to LAYOUTCOMMIT should be sufficient for server to determine an upper bound on the client's layout state at the time the operation was sent, regardless of the LAYOUTCOMMITted range. In particular the server knows which RW layout segments the client has.

2. The server uses the LAYOUTCOMMIT byte range in a generic way if loca_reclaim is true as follows (see 12.7.4 for more info):
      If the metadata server's

4. The blocks layout has all the byte range metadata it needs as part of the extents list passed down in loca_layoutupdate.

Therefore, I propose a simplification for the protocol that essentially deprecates loca_offset and loca_length when loca_reclaim is false and keeps their use if loca_reclaim is true.

For the blocks layout I propose to allow to pass a fragmented extents list in LAYOUTCOMMIT and only restrict it so the committed extents must be completely contained in valid RW layout the client holds.

loca_stateid MUST be verified by the server to ensure LAYOUTCOMMIT is processed before a conflicting LAYOUTRETURN, otherwise return NFS4ERR_{BAD,OLD}_STATEID as needed. Note that with the blocks layout type LAYOUTRETURN may deallocate provisionally allocated space that was not explicitly LAYOUTCOMMITted (a.k.a holes) and the client may lose data if it did not successfully LAYOUTCOMMITted its changes before LAYOUTRETURN.

Coming to think about it, I'm not sure that the spec actually defines the desired order of events, so I think we need to add text instructing the client to first make sure data written to data servers is synced to stable storage before sending a LAYOUTCOMMIT.  Yet, since LAYOUTCOMMIT is not serialized with I/O, this constraint makes sense only for data at loca_last_write_offset, and for the blocks layout type, the data in blu_commit_list (taking into assumption possible block protocols with write behind properties).

Benny
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Fred Isaman <[email-masked]>
Date: Wed, 30 Mar 2011 15:33:43 -0400

> This is not true for all file layout based servers. The file layout based servers using asymmetric model can use this information as an indication of what was the last client offset that was writte…

Is this mentioned in the errata at all?  I do not see it.  I remember
some discussion of using COMMIT (not LAYOUTCOMMIT) ranges to determine
if the LAYOUTCOMMIT is needed, but that was tabled once the current
errata was proposed.  Also note that Benny is talking about
deprecating loca_offset and loca_length, the thing you seem to care
about is loca_last_write_offset, which is not affected.

Fred

---

From: Suchit Kaura <[email-masked]>
Date: Thu, 31 Mar 2011 06:00:26 -0700

Hi Fred,

Thanks for your response. 
> I remember some discussion of using COMMIT (not LAYOUTCOMMIT) ranges to determine if the LAYOUTCOMMIT is needed, 
Just to be sure http://www.rfc-editor.org/errata_search.php?rfc=5661&eid=2751 is not being considered anymore

>  but that was tabled once the current errata was proposed.
The current errata is Benny's email or is it documented somewhere else?

> Also note that Benny is talking about deprecating loca_offset and loca_length, the thing you seem to care about is loca_last_write_offset, which is not affected
Sounds okay but would like to understand the details prior to commenting on this part.

Questions:
Will the client send LAYOUTCOMMIT to servers supporting file layout in the current errata? 

Also from rfc5661:
   The loca_last_write_offset field specifies the offset of the last
   byte written by the client previous to the LAYOUTCOMMIT.  Note that
   this value is never equal to the file's size (at most it is one byte
   less than the file's size) and MUST be less than or equal to
   NFS4_MAXFILEOFF.  Also, loca_last_write_offset MUST overlap the range
   described by loca_offset and loca_length.  The metadata server may
   use this information to determine whether the file's size needs to be
   updated.  If the metadata server updates the file's size as the
   result of the LAYOUTCOMMIT operation, it must return the new size
   (locr_newsize.ns_size) as part of the results.

I think if we deprecate loca_offset and loca_length the checks for, loca_last_write_offset MUST overlap the range described by loca_offset and loca_length need to be removed. There is more text in rfc 5661 pertaining to mentioned fields up for deprecation.

Regards,
Suchit

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Fred Isaman
Sent: Wednesday, March 30, 2011 12:34 PM
To: Suchit Kaura
Cc: Benny Halevy; nfsv4 list; David Black
Subject: Re: [nfsv4] LAYOUTCOMMIT byte range

> This is not true for all file layout based servers. The file layout based servers using asymmetric model can use this information as an indication of what was the last client offset that was writte…

Is this mentioned in the errata at all?  I do not see it.  I remember some discussion of using COMMIT (not LAYOUTCOMMIT) ranges to determine if the LAYOUTCOMMIT is needed, but that was tabled once the current errata was proposed.  Also note that Benny is talking about deprecating loca_offset and loca_length, the thing you seem to care about is loca_last_write_offset, which is not affected.

Fred

---

From: Fred Isaman <[email-masked]>
Date: Thu, 31 Mar 2011 09:29:23 -0400

> The current errata is Benny's email or is it documented somewhere else?

The current errata I was referring to, the implementation of which is
now in the upstream kernel, is the the one you reference above
(eid=2751).

> Will the client send LAYOUTCOMMIT to servers supporting file layout in the current errata?

Yes, if commit_to_mds is not set and a write returns something less
than FILE_SYNC.

Fred

---

From: <[email-masked]>
Date: Thu, 31 Mar 2011 10:23:11 -0400

> Just to be sure http://www.rfc-editor.org/errata_search.php?rfc=5661&eid=2751 is not being

That is incorrect, the above (2751) errata is still being considered - it affects only the pNFS file layout.

This email thread about byte range concerns a different problem that has arisen for the pNFS block layout.  To oversimplify, the problem arises from the byte ranges to be committed being described in two places for the block layout, leading Benny to ask: do we need the layout-independent description for other layout types? The resolution of this problem may lead to another errata (which cannot be written until we agree on how to solve the problem).

Thanks,
--David (who talked to Benny about this in Prague).

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Suchit Kaura
> Sent: Thursday, March 31, 2011 9:00 AM
> To: Fred Isaman
> Cc: Benny Halevy; nfsv4 list; Black, David; [email-masked]
> > Also note that Benny is talking about deprecating loca_offset and loca_length, the thing you seem
> to care about is loca_last_write_offset, which is not affected
> I think if we deprecate loca_offset and loca_length the checks for, loca_last_write_offset MUST
> overlap the range described by loca_offset and loca_length need to be removed. There is more text in
> rfc 5661 pertaining to mentioned fields up for deprecation.
> 
> Regards,
> Suchit
> 
> > This is not true for all file layout based servers. The file layout based servers using asymmetric
> model can use this information as an indication of what was the last client offset that was written.
> I believe we agreed on this as part of the proposal from Mike after the discussion in interim IETF
> meeting, the errata for the same was captured by Ricardo and reviewed.
> 
> 
> Is this mentioned in the errata at all?  I do not see it.  I remember some discussion of using
> COMMIT (not LAYOUTCOMMIT) ranges to determine if the LAYOUTCOMMIT is needed, but that was tabled
> once the current errata was proposed.  Also note that Benny is talking about deprecating loca_offset
> and loca_length, the thing you seem to care about is loca_last_write_offset, which is not affected.
> 
> Fred