Subject: Re: [nfsv4] LAYOUTCOMMIT (or absence of thereof) semantics

From: Spencer <[email-masked]>
Date: Fri, 5 Mar 2010 19:46:41 -0600

Benny,

I haven't seen any followup and am curious if we need to
target this for rfc5661 errata?

Spencer

On Feb 24, 2010, at 6:32 PM, Benny Halevy wrote:

> Mike,
> 
> The excerpts from rfc5661 below show that the server has the authority
> to update the file size using loca_last_write_offset as provided by
> the client on LAYOUTCOMMIT or ignore it altogether and use an internal
> mechanism such as polling its data servers (having a cluster-coherent
> file system would be another example)
> 
> ========================================================================
> 12.5.4.2. LAYOUTCOMMIT and size
> 
> 
>   The size of a file may be updated when the LAYOUTCOMMIT operation is
>   used by the client.  One of the fields in the argument to
>   LAYOUTCOMMIT is loca_last_write_offset; this field indicates the
>   highest byte offset written but not yet committed with the
>   LAYOUTCOMMIT operation.  The data type of loca_last_write_offset is
>   newoffset4 and is switched on a boolean value, no_newoffset, that
>   indicates if a previous write occurred or not.  If no_newoffset is
>   FALSE, an offset is not given.  If the client has a layout with
>   LAYOUTIOMODE4_RW iomode on the file, with a byte-range (denoted by
>   the values of lo_offset and lo_length) that overlaps
>   loca_last_write_offset, then the client MAY set no_newoffset to TRUE
>   and provide an offset that will update the file size.  Keep in mind
>   that offset is not the same as length, though they are related.  For
>   example, a loca_last_write_offset value of zero means that one byte
>   was written at offset zero, and so the length of the file is at least
>   one byte.
> 
>   The metadata server may do one of the following:
> 
>   1.  Update the file's size using the last write offset provided by
>       the client as either the true file size or as a hint of the file
>       size.  If the metadata server has a method available, any new
>       value for file size should be sanity-checked.  For example, the
>       file must not be truncated if the client presents a last write
>       offset less than the file's current size.
> 
>   2.  Ignore the client-provided last write offset; the metadata server
>       must have sufficient knowledge from other sources to determine
>       the file's size.  For example, the metadata server queries the
>       storage devices with the control protocol.
> 
> 18.42. Operation 49: LAYOUTCOMMIT - Commit Writes Made Using a Layout
> 
>   The loca_last_write_offset field specifies the offset of the last
>   byte written by the client previous to the LAYOUTCOMMIT.  Note that
>   this value is never equal to the file's size (at most it is one byte
>   less than the file's size) and MUST be less than or equal to
>   NFS4_MAXFILEOFF.  Also, loca_last_write_offset MUST overlap the range
>   described by loca_offset and loca_length.  The metadata server may
>   use this information to determine whether the file's size needs to be
>   updated.
> 
> ========================================================================
> 
> In case the client restart case we say:
>   In this
>   scenario, the data written by the client but not covered by a
>   successful LAYOUTCOMMIT is in an undefined state; it may have been
>   written or it may now be lost.  This is acceptable behavior and it is
>   the client's responsibility to use LAYOUTCOMMIT to achieve the
>   desired level of stability.
> 
> That said, the spec does not seem to define the server's behavior
> for the case the client returns the layout before sending LAYOUTCOMMIT,
> either via an explicit LAYOUTRETURN or by returning NFS4ERR_NOMATCHING_LAYOUT
> to CB_LAYOUTRECALL, and the case where the client's layout was revoked before
> sending LAYOUTCOMMIT.
> 
> In either case, I think it's fair to presume that the state of any data
> written to the DS, whether it was COMMITted there or not is undefined,
> similar to the client restart case, and the spec. should encourage
> that server SHOULD use an alternative mechanism, similar to the ones
> described in 12.5.4.2. item 2. to recover any data committed to the DSs,
> in particular data written and committed to stable storage on the DS.
> 
> Benny
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mike Eisler" <[email-masked]>
Date: Sat, 6 Mar 2010 10:17:50 -0800

We need errata. My Action Item.

The context is that I discussed this at Cthon. I.e. the fact that
RFC5661 has the absurd implication that to append to a file, one must
send a LAYOUTCOMMIT after each WRITE to a data server. IOW,
there's no way to use log files over pNFS if one accepts this absurd
implication.

I will file an errata that says that if the MDS revokes a layout, it
MUST (if not done so already) recover the highest file offset and mtime
from each of the data servers. In NFSv4.1. If we need a quick I-D ->
RFC that makes this clarification, that's fine too.

I sensed strong consensus among implementers at cthon. Benny said that it
was certainly the intent of the original pNFS community to do so.

>> file system would be another example)

Agreed but authority does not imply duty when a layout is revoked.

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: <[email-masked]>
Date: Mon, 8 Mar 2010 02:21:17 -0500

Mike,

We need to figure out what to do for the block layout, where the data
server doesn't know the max offset (i.e., that MUST can't apply to the
block layout).

My initial suggestion is that the client will do a LAYOUTCOMMIT as part
of processing the server's revocation of the layout, and that
LAYOUTCOMMIT may be the only way to get the new max offset.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of Mike Eisler
> Sent: Saturday, March 06, 2010 1:18 PM
> To: [email-masked]
> Subject: Re: [nfsv4] LAYOUTCOMMIT (or absence of thereof) semantics
> 
> >> to CB_LAYOUTRECALL, and the case where the client's layout was
revoked
> >> before

---

From: "Muntz, Daniel" <[email-masked]>
Date: Mon, 8 Mar 2010 09:36:02 -0800

Considering the different requirements files, objects, and blocks place
on layouts (and layoutcommit) perhaps this could be refactored into
layout-type-specific wording, with updates to the files layout section
of the 4.1 spec and references to the blocks and objects specs.

  -Dan

> From: [email-masked] [mailto:[email-masked]] 
> Sent: Sunday, March 07, 2010 11:21 PM
> To: [email-masked]; [email-masked]
> > From: [email-masked] 
> [mailto:[email-masked]] On Behalf
> Of Mike Eisler
> > I sensed strong consensus among implementers at cthon. 
> Benny said that
> it
> > >>   The size of a file may be updated when the 
> LAYOUTCOMMIT operation
> is
> > >>   LAYOUTCOMMIT operation.  The data type of 
> loca_last_write_offset
> is
> > >> In either case, I think it's fair to presume that the 
> state of any
> data
> > >> described in 12.5.4.2. item 2. to recover any data 
> committed to the
> DSs,

---

From: sfaibish <[email-masked]>
Date: Mon, 08 Mar 2010 14:01:51 -0500

Dan,

I guess this is the right approach. The block needs the layoutcommit
sent to the server as the DS are just LUNs and do not play in the
protocol. We can make this specific to block and leave it optional
for the other layout types. So, the block client will always send
commit to get filesize updated. We can leave this for the RFC5663
to reinforce.

/Sorin

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Network Storage Group

         EMC²
where information lives

Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Benny Halevy <[email-masked]>
Date: Mon, 08 Mar 2010 23:13:39 +0200

> sent to the server as the DS are just LUNs and do not play in the

LAYOUTCOMMIT is always sent to the MDS.
The DSs may be sent SYNCHRONIZE CACHE commands for committing
data to stable storage, and this equivalent to sending NFS COMMITs
to the DS in the files layout or OSD FLUSH commands for the objects layout.

> commit to get filesize updated. We can leave this for the RFC5663

This is true for the objects layout as well.

Benny