Subject: RE: [nfsv4] Proposal for adding reclaim bit to pNFS COMMITLAYOUT

From: "Halevy, Benny" <[email-masked]>
Date: Thu, 26 Jan 2006 15:42:07 -0500

This looks good.

In addition, I think we should now add NFS4ERR_GRACE as a possible error
return code to all other operations that are not allowed during the grace
period (e.g. LAYOTUGET or LAYOUTRETURN).

Benny

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]]On Behalf Of
> [email-masked]
> Sent: Thursday, January 26, 2006 8:59 PM
> To: [email-masked]
> Subject: [nfsv4] Proposal for adding reclaim bit to pNFS COMMITLAYOUT
> 
> 
> Hi all---
> As we discussed on the latest conference call, I'd like to
> propose adding a reclaim bit to the pNFS COMMITLAYOUT operation.
> I've drafted the required changes to the pnfs4.1 doc and
> included them below.  Take a look and please send comments/
> corrections.
> 
> -Steve
> 
> 
> 
> ----------------------------------------------------------------
> 
> We require changes in section 7.7.3 (Meta-data server recovery)
> and section 14.15 (description of LAYOUTCOMMIT operation).
> I have included new version of both of these sections below.
> New lines are marked with "#>".
> 
> 
> 
> 7.7.3  Metadata Server Recovery
> 
>    The server recovery case is slightly more complex.  In general, the
>    recovery process again follows the standard NFSv4 recovery 
> model: the
>    client will discover that the metadata server has rebooted when it
>    receives an unexpected STALE_STATEID or STALE_CLIENTID 
> reply from the
>    server; it will then proceed to try to reclaim its previous
>    delegations during the server's recovery grace period.  However,
>    layouts are not reclaimable in the same sense as data delegations;
>    there is no reclaim bit, thus no guarantee of continuity 
> between the
>    previous and new layout.  This is not necessarily required since a
>    layout is not required to perform I/O; I/O can always be performed
>    through the metadata server.
> 
>    If the client has dirty data that it needs to write out, or an
>    outstanding LAYOUTCOMMIT, the client should try to obtain a new
>    layout segment covering the byte range covered by the 
> previous layout
>    segment.  However, the client might not not get the same layout
>    segment it had.  The range might be different or it might get the
>    same range but the content of the layout might be different.  For
>    example, if using a block/volume-based layout, the blocks
>    provisionally assigned by the layout might be different, in which
>    case the client will have to write the corresponding 
> blocks again; in
>    the interest of simplicity, the client might decide to always write
>    them again.  Alternatively, the client might be unable to obtain a
>    new layout and thus, must write the data using normal NFSv4 through
> #> the metadata server.  Also note that the meta-data server will not
> #> allow any operations that could potentially change the state of the
> #> file-system during the recovery grace period.  Thus reclaiming of
> #> pNFS layouts and writing of dirty data must wait until the recovery
> #> grace period is over.
> 
> #> It is also possible that the client has written dirty data to a
> #> provisionally allocated region of the layout, but was unable to
> #> commit the layout changes for this data before the server rebooted.
> #> In this case, depending on the data storage protocol, the 
> client may
> #> be unable to reliably re-read the data from the date 
> storage devices
> #> in order to write it again via the meta-data server.  In this case
> #> the client needs to inform the meta-data server that the layout has
> #> changed, before the server has completed its recovery grace period
> #> and starts allowing updates to the file-system.  For this purpose,
> #> the LAYOUTCOMMIT operation contains a "reclaim" field.  During the
> #> meta-data server's recovery grace period (and only during the
> #> recovery grace period) the client may send a LAYOUTCOMMIT request
> #> with the "reclaim" field set to "true".  This indicates that the
> #> client is attempting to commit changes to the file layout that
> #> occurred prior to the reboot of the meta-data server.  The "layout
> #> update" field of the request should contain the portion of the
> #> layout that the client held prior to the meta-data server reboot
> #> which covers the outstanding writes.  The meta-data server is free
> #> to apply consistency checks on the layout update provided by the
> #> client, and reject the request if the checks fail.  If the checks
> #> do not fail, then the server will commit the changes to the file
> #> layout contained in the "layoutupdate" field of the LAYOUTCOMMIT
> #> request, ensuring that the clients outstanding writes are not lost.
> 
>    There is an important safety concern associated with layouts that
>    does not come into play in the standard NFSv4 case.  If a standard
>    NFSv4 client makes use of a stale delegation, while reading, the
>    consequence could be to deliver stale data to an application.  If
>    writing, using a stale delegation or a stale state stateid for an
>    open or lock would result in the rejection of the client's 
> write with
>    the appropriate stale stateid error.
> 
>    However, the pNFS layout enables the client to directly access the
>    file system storage---if this access is not properly managed by the
>    NFSv4 server the client can potentially corrupt the file 
> system data
>    or metadata.  Thus, it is vitally important that the 
> client discover
>    that the metadata server has rebooted, and that the client stops
>    using stale layouts before the metadata server gives them away to
>    other clients.  To ensure this, the client must be implemented so
>    that layouts are never used to access the storage after 
> the client's
>    lease timer has expired.  It is crucial that clients have precise
>    knowledge of the lease periods of their layouts.  For specific
>    details on lease renewal and client lease timers, see 
> Section 7.7.1.
> 
>    The prohibition on using stale layouts applies to all 
> layout related
>    accesses, especially the flushing of dirty data to the storage
>    devices.  If the client's lease timer expires because the client
>    could not contact the server for any reason, the client MUST
>    immediately stop using the layout until the server can be contacted
>    and the layout can be officially recovered or reclaimed.  However,
>    this is only part of the solution.  It is also necessary 
> to deal with
>    the consequences of I/Os already in flight.
> 
>    The issue of the effects of I/Os started before lease 
> expiration and
>    possibly continuing through lease expiration is the 
> responsibility of
>    the data storage protocol and as such is layout type 
> specific.  There
>    are two approaches the data storage protocol can take.  
> The protocol
>    may adopt a global solution which prevents all I/Os from being
>    executed after the lease expiration and thus is safe 
> against a client
>    who issues I/Os after lease expiration.  This is the preferred
>    solution and the solution used by NFSv4 file based layouts (see
>    Section 9.6); as well, the object storage device protocol allows
>    storage to fence clients after lease expiration.  
> Alternatively, the
>    storage protocol may rely on proper client operation and only deal
>    with the effects of lingering I/Os.  These solutions may impact the
>    client layout-driver, the metadata server layout-driver, and the
>    control protocol.
> 
> 
> 
> 
> 
> 14.15  LAYOUTCOMMIT - Commit writes made using a layout
> 
>    SYNOPSIS
> 
> #>   (cfh), clientid, reclaim, offset, length, last_write_offset,
>      time_modify, time_access, layoutupdate -> newsize
> 
> 
>    ARGUMENT
> 
>      union newtime4 switch (bool timechanged) {
>              case TRUE:
>                      nfstime4           time;
>              case FALSE:
>                      void;
>      };
> 
>      union newsize4 switch (bool sizechanged) {
>              case TRUE:
>                      length4            size;
>              case FALSE:
>                      void;
>      };
> 
>      struct LAYOUTCOMMIT4args {
>              /* CURRENT_FH: file */
>              clientid4               clientid;
> #>           bool                    reclaim;
>              offset4                 offset;
>              length4                 length;
>              length4                 last_write_offset;
>              newtime4                time_modify;
>              newtime4                time_access;
>              pnfs_layoutupdate4      layoutupdate;
>      };
> 
>    RESULT
> 
>      struct LAYOUTCOMMIT4resok {
>             newsize4                 newsize;
>      };
> 
>      union LAYOUTCOMMIT4res switch (nfsstat4 status) {
>              case NFS4_OK:
>                      LAYOUTCOMMIT4resok  resok4;
>              default:
>                      void;
>      };
> 
>    DESCRIPTION
> 
>    Commits changes in the layout segment represented by the current
>    filehandle, clientid, and byte range.  Since layouts are sub-
>    dividable, a smaller portion of a layout, retrieved via LAYOUTGET,
>    may be committed.  The region being committed is specified through
>    the byte range (length and offset).  Note: the "layoutupdate"
>    structure does not include the length and offset, as they 
> are already
>    specified in the arguments.
> 
>    The LAYOUTCOMMIT operation indicates that the client has completed
>    writes using a layout obtained by a previous LAYOUTGET.  The client
>    may have only written a subset of the data range it previously
>    requested.  LAYOUTCOMMIT allows it to commit or discard 
> provisionally
>    allocated space and to update the server with a new end of 
> file.  The
>    layout referenced by LAYOUTCOMMIT is still valid after the 
> operation
>    completes and can be continued to be referenced by the clientid,
>    filehandle, byte range, and layout type.
> 
> #> The "reclaim" field specifies that the client is attempting to
> #> commit changes to a layout after the reboot of the meta-data server
> #> during the meta-data server's recovery grace period.  The client
> #> uses this request when it has un-committed writes to provisionally
> #> allocated regions of a file which were sent to the storage servers
> #> before the reboot of the meta-data server.  In this case the layout
> #> provided by the client must be a subset of a writable layout that
> #> the client held immediately before the reboot of the 
> meta-data server.
> #> The meta-data server is free to accept or reject this request based
> #> on its own internal meta-data consistency checks.  If the meta-data
> #> server finds that the layout provided by the client does not pass
> #> its consistency checks, it should reject the request with 
> the status
> #> NFS4ERR_RECLAIM_BAD.  If the meta-data server returns success, it
> #> does not mean that the client possesses any layout for the file.
> #> It only means that the changes to the file layout specified in the
> #> "layoutupdate" field are committed to the file.  If the meta-data
> #> server receives a LAYOUTCOMMIT request with "reclaim" set to true
> #> when the meta-data server is not in its recovery grace period, it
> #> must reject the request with the status NFS4ERR_NO_GRACE.
> 
>    The "last_write_offset" field specifies the offset of the last byte
>    written by the client previous to the LAYOUTCOMMIT.  Note: 
> this value
>    is never equal to the file's size (at most it is one byte less than
>    the file's size).  The metadata server may use this information to
>    determine whether the file's size needs to be updated.  If the
>    metadata server updates the file's size as the result of the
>    LAYOUTCOMMIT operation, it must return the new size as part of the
>    results.
> 
>    The "time_modify" and "time_access" fields allow the client to
>    suggest times it would like the metadata server to set.  
> The metadata
>    server may use these time values or it may use the time of the
>    LAYOUTCOMMIT operation to set these time values.  If the metadata
>    server uses the client provided times, it should sanity check the
>    values (e.g., to ensure time does not flow backwards).  If 
> the client
>    wants to force the metadata server to set an exact time, the client
>    should use a SETATTR operation in a compound right after
>    LAYOUTCOMMIT.  See Section 7.4 for more details.  If the new client
>    desires the resultant mtime or atime, it should issue a GETATTR
>    following the LAYOUTCOMMIT; e.g., later in the same compound.
> 
>    The "layoutupdate" argument to LAYOUTCOMMIT provides a 
> mechanism for
>    a client to provide layout specific updates to the metadata server.
>    For example, the layout update can describe what regions of the
>    original layout have been used and what regions can be deallocated.
>    There is no NFSv4 file layout specific layoutupdate structure.
> 
>    The layout information is more verbose for block devices than for
>    objects and files because the latter hide the details of block
>    allocation behind their storage protocols.  At the minimum, the
>    client needs to communicate changes to the end of file 
> location back
>    to the server, and, if desired, its view of the file modify and
>    access time.  For block/volume layouts, it needs to 
> specify precisely
>    which blocks have been used.
> 
>    If the layout identified in the arguments does not exist, the error
>    NFS4ERR_BADLAYOUT is returned.  The layout being committed may also
>    be rejected if it does not correspond to an existing layout with an
>    iomode of RW.
> 
>    On success, the current filehandle retains its value.
> 
>    ERRORS
> 
>       NFS4ERR_BADLAYOUT
>       NFS4ERR_BADIOMODE
>       NFS4ERR_FHEXPIRED
>       NFS4ERR_INVAL
>       NFS4ERR_NOFILEHANDLE
>       NFS4ERR_STALE
>       NFS4ERR_STALE_CLIENTID
>       NFS4ERR_UNKNOWN_LAYOUTTYPE
> #>    NFS4ERR_NO_GRACE
> #>    NFS4ERR_RECLAIM_BAD
> 
> 
> 
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: [email-masked]
Date: Thu, 26 Jan 2006 16:20:37 -0500

Good point.  Of course, a "mean" server is free to always return
"NFS4ERR_RECLAIM_BAD" whenever a client tries to send a COMMITLAYOUT
with the reclaim bit set.  But for the client, finding out that the
server doesn't support "reclaim commits" only when the request is
rejected is a little too late!

A "nice" server would somehow advertise to the client that this
feature was unsupported, which would allow the client to take corrective
action:  for instance, enter a degraded performance mode where all
writes were buffered in memory until successfully committed.

Is there a standard way for the meta-data server to advertise its
"capabilities"?  Can we define a global boolean attribute whose value
will reflect whether the server supports "reclaim commits"?
-Steve 

-----Original Message-----
From: Garth Goodson [mailto:[email-masked]] 
Sent: Thursday, January 26, 2006 4:05 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Proposal for adding reclaim bit to pNFS COMMITLAYOUT

The text looks good, but I have clarifying question/comment.  From this 
it isn't clear whether a server can not support reclaimable layouts.  It 
may return a NFS4ERR_RECLAIM_BAD error if "the layout provided by the 
client does not pass its [the metadata server's] consistency checks".

Do we want to allow certain layout types/implementations to have the 
ability to not support reclaimable layouts?  If so, are these error 
return codes good enough or do we need to add something else.  As well, 
we may want to clarify this issue in the text.

-Garth

[email-masked] wrote:
>    recovery process again follows the standard NFSv4 recovery model: the
>    receives an unexpected STALE_STATEID or STALE_CLIENTID reply from the
>    there is no reclaim bit, thus no guarantee of continuity between the
>    layout segment covering the byte range covered by the previous layout
>    case the client will have to write the corresponding blocks again; in
> #> In this case, depending on the data storage protocol, the client may
> #> be unable to reliably re-read the data from the date storage devices
>    open or lock would result in the rejection of the client's write with
>    NFSv4 server the client can potentially corrupt the file system data
>    or metadata.  Thus, it is vitally important that the client discover
>    that layouts are never used to access the storage after the client's
>    details on lease renewal and client lease timers, see Section 7.7.1.
> 
>    The prohibition on using stale layouts applies to all layout related
>    this is only part of the solution.  It is also necessary to deal with
>    The issue of the effects of I/Os started before lease expiration and
>    possibly continuing through lease expiration is the responsibility of
>    the data storage protocol and as such is layout type specific.  There
>    are two approaches the data storage protocol can take.  The protocol
>    executed after the lease expiration and thus is safe against a client
>    storage to fence clients after lease expiration.  Alternatively, the
>    structure does not include the length and offset, as they are already
>    requested.  LAYOUTCOMMIT allows it to commit or discard provisionally
>    allocated space and to update the server with a new end of file.  The
>    layout referenced by LAYOUTCOMMIT is still valid after the operation
> #> the client held immediately before the reboot of the meta-data server.
> #> its consistency checks, it should reject the request with the status
>    written by the client previous to the LAYOUTCOMMIT.  Note: this value
>    suggest times it would like the metadata server to set.  The metadata
>    values (e.g., to ensure time does not flow backwards).  If the client
>    The "layoutupdate" argument to LAYOUTCOMMIT provides a mechanism for
>    client needs to communicate changes to the end of file location back
>    access time.  For block/volume layouts, it needs to specify precisely

---

From: [email-masked]
Date: Thu, 26 Jan 2006 16:25:07 -0500

These "tweaks" sound good to me... thanks!  I'll merge them into
my local copy.  In fact, I guess I should volunteer to gather all
the edits that people have for this issue.  I'll capture changes
for a few days and issue an updated, merged version of the proposal
on Monday...
-Steve  

-----Original Message-----
From: Brent Welch [mailto:[email-masked]] 
Sent: Thursday, January 26, 2006 3:53 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Proposal for adding reclaim bit to pNFS COMMITLAYOUT 

I think the first paragraph could be tweaked slightly to better introduce
the reclaim-on-layoutcommit notion.  See below.

>>>[email-masked] said:
 > 
 > Hi all---

However, layouts have a slightly different mechanism for reclaim.
The problem is that a client might not get the same layout it
had previously. The range might be different or it might get the
same range but the content of the layout might be different.  For
example, if using a block/volume-based layout, the blocks
provisionally assigned by the layout might be different, in which
case the client will have to write the corresponding blocks again.

Instead of reclaiming a layout with LAYOUTGET, a client can attempt
to commit data written before the file server crash by setting a
reclaim bit on the LAYOUTCOMMIT operation.  This should only be
done for data that the client has already written using a layout
obtained before the server restart.  For data still
dirty in the client memory, the client should get a new layout
segment after the server's grace period has elapsed.  Alternatively,
the client can write that data through the metadata server.  Note
that the meta-data server will not
allow any operations that could potentially change the state of the
file-system during the recovery grace period.  Thus getting a new
pNFS layout and writing of dirty data must wait until the recovery
grace period is over.

[The above replaces the following text]
 > #> pNFS layouts and writing of dirty data must wait until the recovery
[down to here]

[And then tweak the first two sentences of this paragraph slightly]

In the case that the client has written dirty data to a
provisionally allocated region of the layout, but was unable to
commit the layout changes for this data before the server rebooted,
the client may be unable to reliably re-read the data from the date storage 
devices
in order to write it again via the meta-data server.  In this case...

 > #> in order to write it again via the meta-data server.  In this case

[pick up here - the rest looks OK]

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: "Halevy, Benny" <[email-masked]>
Date: Fri, 27 Jan 2006 17:58:21 -0500

> Sent: Saturday, January 28, 2006 12:31 AM
> To: [email-masked]; [email-masked]; [email-masked]
> Cc: [email-masked]
> bfields at fieldses.org wrote:
> > On Fri, Jan 27, 2006 at 12:42:13PM -0500, fridella_stephen 
> at emc.com
> wrote:
> > > One other point---isn't this already a requirement of 
> NFSv4? During
> > > the grace period, aren't writes disallowed due to 
> potential conflicts
> > > with locks that have yet to be reclaimed?
> >
> > I think that should only be the case with special-stateid-writes or
> > writes to files that may have mandatory locking in force.
> >
> > If a write is performed with a "real" stateid (derived from an open)
> > then we know it can't conflict with a still-to-be-reclaimed open
> > (otherwise the open couldn't legitimately have been 
> reclaimed), so the
> > only possible conflict is with a mandatory lock.
> > --b. 
> 
> 
> 
> In light of these comments I went back to the NFSv4 RFC and
> carefully re-read the recovery section (hmmm... maybe I should
> have done this before my original post, huh?  ;) )  The RFC takes
> the attitude:  "If the server can guarantee that a read or write
> does not conflict with any potential un-reclaimed lock, it can
> honor the request during the grace period."
> 
> So if we take the same approach here, then depending on the storage
> protocol used, the meta-data server can allow standard NFSv4
> writes to proceed during the grace period---as long as it can
> guarantee that the write will not conflict/interfere with some
> un-reclaimed provisional allocation for another file's layout.
> 
> As a practical matter, block/volume servers which don't support
> persistent provisional allocation would probably just reject
> write requests during the grace period for any file that may
> have un-reclaimed layouts.  File and Object servers may be able
> to provide the required guarantee (since allocation generally
> happens at the storage server) and start accepting write requests
> earlier.

This limitation applies also for objects striped with redundancy
(e.g. RAID-5) when the server does not keep persistent state about
all outstanding write layouts. The server MUST NOT commit data to 
such objects if there are valid outstanding write layouts conflicting
with the affected parity stripes.  If the server can recall or revoke
these layouts it MUST do that before committing any data to the storage
servers.

> 
> 
> So I propose the following text:
> 
> change:
> 
> "Note that the
> meta-data server will not allow any operations that could 
> potentially change
> the state of the file-system during the recovery grace 
> period. Thus getting
> a new pNFS layout and writing of dirty data must wait until 
> the recovery
> grace period is over."
> 
> 
> to:
> 
> "Note that during the recovery grace period, the meta-data 
> server cannot
> allow operations that could potentially conflict with uncommitted
> layout changes that a client may yet attempt to reclaim.  
> Some meta-data
> servers (e.g. those that persistently save the provisional allocation
> state for each file, or those where the allocation for each file is
> handled by the data storage device) may be able to guarantee for a
> particular WRITE or GETLAYOUT request, that there is no way that the
> request will interfere with some client's attempt to reclaim 
> uncommitted
> layout changes.  These servers are free to service such 
> requests during
> the recovery grace period.  If the meta-data server cannot make such
> a guarantee for a request, it must reject the request with the status
> NFS4ERR_GRACE."
> 
> 
> Make sense to everybody?
> -Steve
>