Subject: Re: [nfsv4] Proposal for adding reclaim bit to pNFS COMMITLAYOUT

From: [email-masked]
Date: Mon, 13 Feb 2006 16:31:00 -0500

Hi All---
Here is the final proposed text for adding a reclaim flag to
the pNFS LAYOUTCOMMIT operation.  Final edits included text
to make supporting LAYOUTCOMMIT reclaim operations mandatory
and stamping out the remaining uses of "COMMITLAYOUT".

Spencer---do you need this in any particular format or is the
text below ok?

-Steve

------------------------------------------------------

We require changes in section 7.7.3 (Meta-data server recovery)
and section 14.15 (description of LAYOUTCOMMIT operation).
I have included new version of both of these sections below.
New lines are marked with "#>".

7.7.3  Metadata Server Recovery

   The server recovery case is slightly more complex.  In general, the
   recovery process again follows the standard NFSv4 recovery model: the
   client will discover that the metadata server has rebooted when it
   receives an unexpected STALE_STATEID or STALE_CLIENTID reply from the
   server; it will then proceed to try to reclaim its previous
   delegations during the server's recovery grace period.
#> However, layouts have a slightly different mechanism for reclaim.
#> The problem is that a client might not get the same layout it
#> had previously. The range might be different or it might get the
#> same range but the content of the layout might be different.  For
#> example, if using a block/volume-based layout, the blocks
#> provisionally assigned by the layout might be different, in which
#> case the client will have to write the corresponding blocks again.

#> Instead of reclaiming a layout with LAYOUTGET, a client can attempt
#> to commit data written before the file server crash by setting a
#> reclaim bit on the LAYOUTCOMMIT operation.  This should only be
#> done for data that the client has already written using a layout
#> obtained before the server restart.  For data still
#> dirty in the client memory, the client should get a new layout
#> segment after the server's grace period has elapsed.  Alternatively,
#> the client can write that data through the metadata server using
#> the standard NFSv4 WRITE. 

#> In the case that the client has written dirty data to a provisionally
#> allocated region of the layout, but was unable to commit the layout
#> changes for this data before the server rebooted, the client may be
#> unable to reliably re-read the data from the date storage devices in
#> order to write it again via the meta-data server.  In this case
#> the client needs to inform the meta-data server that the layout has
#> changed, before the server has completed its recovery grace period
#> and starts allowing updates to the file-system.  For this purpose,
#> the LAYOUTCOMMIT operation contains a "reclaim" field.  During the
#> meta-data server's recovery grace period (and only during the
#> recovery grace period) the client may send a LAYOUTCOMMIT request
#> with the "reclaim" field set to "true".  This indicates that the
#> client is attempting to commit changes to the file layout that
#> occurred prior to the reboot of the meta-data server.  The "layout
#> update" field of the request must contain the portion of the
#> layout that the client held prior to the meta-data server reboot
#> which covers the outstanding writes.  The meta-data server is free
#> to apply consistency checks on the layout update provided by the
#> client, and reject the request if the checks fail.  If the checks
#> do not fail, then the server MUST commit the changes to the file
#> layout contained in the "layoutupdate" field of the LAYOUTCOMMIT
#> request, ensuring that the clients outstanding writes are not lost.

#> During the recovery grace period the meta-data server should apply
#> the standard approach to handling WRITE and GETLAYOUT
#> requests.  That is, if the server can reliably determine that
#> servicing such a request will not conflict with an impending
#> LAYOUTCOMMIT reclaim request, it may choose to service the request.
#> If the server is unable to offer this guarantee, it MUST reject
#> the request with status NFS4ERR_GRACE.   

#> For a meta-data server to provide simple, valid handling during
#> the grace period with respect to pNFS layouts, the easiest method
#> is to simply reject all non-reclaim pNFS requests and WRITE
#> operations by returning the NFS4ERR_GRACE error.  However,
#> depending on the storage protocol and server implementation, the
#> server may be able to determine that a particular request is safe.
#> For example, a server may save provisional allocation mappings for
#> each file to stable storage, and use this information during the
#> recovery grace period to determine that a WRITE request is safe.
#> Under such circumstances, the WRITE request MAY be serviced.
#> To re-iterate, for a server to allow non-reclaim pNFS requests
#> and WRITE operations to be serviced during the recovery grace
#> period, it MUST determine that the request will not conflict
#> with any subsequent LAYOUTCOMMIT with reclaim request.
 
   There is an important safety concern associated with layouts that
   does not come into play in the standard NFSv4 case.  If a standard
   NFSv4 client makes use of a stale delegation, while reading, the
   consequence could be to deliver stale data to an application.  If
   writing, using a stale delegation or a stale state stateid for an
   open or lock would result in the rejection of the client's write with
   the appropriate stale stateid error.

   However, the pNFS layout enables the client to directly access the
   file system storage---if this access is not properly managed by the
   NFSv4 server the client can potentially corrupt the file system data
   or metadata.  Thus, it is vitally important that the client discover
   that the metadata server has rebooted, and that the client stops
   using stale layouts before the metadata server gives them away to
   other clients.  To ensure this, the client must be implemented so
   that layouts are never used to access the storage after the client's
   lease timer has expired.  It is crucial that clients have precise
   knowledge of the lease periods of their layouts.  For specific
   details on lease renewal and client lease timers, see Section 7.7.1.

   The prohibition on using stale layouts applies to all layout related
   accesses, especially the flushing of dirty data to the storage
   devices.  If the client's lease timer expires because the client
   could not contact the server for any reason, the client MUST
   immediately stop using the layout until the server can be contacted
   and the layout can be officially recovered or reclaimed.  However,
   this is only part of the solution.  It is also necessary to deal with
   the consequences of I/Os already in flight.

   The issue of the effects of I/Os started before lease expiration and
   possibly continuing through lease expiration is the responsibility of
   the data storage protocol and as such is layout type specific.  There
   are two approaches the data storage protocol can take.  The protocol
   may adopt a global solution which prevents all I/Os from being
   executed after the lease expiration and thus is safe against a client
   who issues I/Os after lease expiration.  This is the preferred
   solution and the solution used by NFSv4 file based layouts (see
   Section 9.6); as well, the object storage device protocol allows
   storage to fence clients after lease expiration.  Alternatively, the
   storage protocol may rely on proper client operation and only deal
   with the effects of lingering I/Os.  These solutions may impact the
   client layout-driver, the metadata server layout-driver, and the
   control protocol.

14.15  LAYOUTCOMMIT - Commit writes made using a layout

   SYNOPSIS

#>   (cfh), clientid, reclaim, offset, length, last_write_offset,
     time_modify, time_access, layoutupdate -> newsize

   ARGUMENT

     union newtime4 switch (bool timechanged) {
             case TRUE:
                     nfstime4           time;
             case FALSE:
                     void;
     };

     union newsize4 switch (bool sizechanged) {
             case TRUE:
                     length4            size;
             case FALSE:
                     void;
     };

     struct LAYOUTCOMMIT4args {
             /* CURRENT_FH: file */
             clientid4               clientid;
#>           bool                    reclaim;
             offset4                 offset;
             length4                 length;
             length4                 last_write_offset;
             newtime4                time_modify;
             newtime4                time_access;
             pnfs_layoutupdate4      layoutupdate;
     };

   RESULT

     struct LAYOUTCOMMIT4resok {
            newsize4                 newsize;
     };

     union LAYOUTCOMMIT4res switch (nfsstat4 status) {
             case NFS4_OK:
                     LAYOUTCOMMIT4resok  resok4;
             default:
                     void;
     };

   DESCRIPTION

   Commits changes in the layout segment represented by the current
   filehandle, clientid, and byte range.  Since layouts are sub-
   dividable, a smaller portion of a layout, retrieved via LAYOUTGET,
   may be committed.  The region being committed is specified through
   the byte range (length and offset).  Note: the "layoutupdate"
   structure does not include the length and offset, as they are already
   specified in the arguments.

   The LAYOUTCOMMIT operation indicates that the client has completed
   writes using a layout obtained by a previous LAYOUTGET.  The client
   may have only written a subset of the data range it previously
   requested.  LAYOUTCOMMIT allows it to commit or discard provisionally
   allocated space and to update the server with a new end of file.  The
   layout referenced by LAYOUTCOMMIT is still valid after the operation
   completes and can be continued to be referenced by the clientid,
   filehandle, byte range, and layout type.

#> The "reclaim" field specifies that the client is attempting to
#> commit changes to a layout after the reboot of the meta-data server
#> during the meta-data server's recovery grace period.  The client
#> uses this request when it has un-committed writes to provisionally
#> allocated regions of a file which were sent to the storage servers
#> before the reboot of the meta-data server.  In this case the layout
#> provided by the client must be a subset of a writable layout that
#> the client held immediately before the reboot of the meta-data server.
#> The meta-data server is free to accept or reject this request based
#> on its own internal meta-data consistency checks.  If the meta-data
#> server finds that the layout provided by the client does not pass
#> its consistency checks, it should reject the request with the status
#> NFS4ERR_RECLAIM_BAD.  If the meta-data server returns success, it
#> does not mean that the client possesses any layout for the file.
#> It only means that the changes to the file layout specified in the
#> "layoutupdate" field are committed to the file.  If the meta-data
#> server receives a LAYOUTCOMMIT request with "reclaim" set to true
#> when the meta-data server is not in its recovery grace period, it
#> MUST reject the request with the status NFS4ERR_NO_GRACE.

   The "last_write_offset" field specifies the offset of the last byte
   written by the client previous to the LAYOUTCOMMIT.  Note: this value
   is never equal to the file's size (at most it is one byte less than
   the file's size).  The metadata server may use this information to
   determine whether the file's size needs to be updated.  If the
   metadata server updates the file's size as the result of the
   LAYOUTCOMMIT operation, it must return the new size as part of the
   results.

   The "time_modify" and "time_access" fields allow the client to
   suggest times it would like the metadata server to set.  The metadata
   server may use these time values or it may use the time of the
   LAYOUTCOMMIT operation to set these time values.  If the metadata
   server uses the client provided times, it should sanity check the
   values (e.g., to ensure time does not flow backwards).  If the client
   wants to force the metadata server to set an exact time, the client
   should use a SETATTR operation in a compound right after
   LAYOUTCOMMIT.  See Section 7.4 for more details.  If the new client
   desires the resultant mtime or atime, it should issue a GETATTR
   following the LAYOUTCOMMIT; e.g., later in the same compound.

   The "layoutupdate" argument to LAYOUTCOMMIT provides a mechanism for
   a client to provide layout specific updates to the metadata server.
   For example, the layout update can describe what regions of the
   original layout have been used and what regions can be deallocated.
   There is no NFSv4 file layout specific layoutupdate structure.

   The layout information is more verbose for block devices than for
   objects and files because the latter hide the details of block
   allocation behind their storage protocols.  At the minimum, the
   client needs to communicate changes to the end of file location back
   to the server, and, if desired, its view of the file modify and
   access time.  For block/volume layouts, it needs to specify precisely
   which blocks have been used.

   If the layout identified in the arguments does not exist, the error
   NFS4ERR_BADLAYOUT is returned.  The layout being committed may also
   be rejected if it does not correspond to an existing layout with an
   iomode of RW.

   On success, the current filehandle retains its value.

   ERRORS

      NFS4ERR_BADLAYOUT
      NFS4ERR_BADIOMODE
      NFS4ERR_FHEXPIRED
      NFS4ERR_INVAL
      NFS4ERR_NOFILEHANDLE
      NFS4ERR_STALE
      NFS4ERR_STALE_CLIENTID
      NFS4ERR_UNKNOWN_LAYOUTTYPE
#>    NFS4ERR_NO_GRACE
#>    NFS4ERR_RECLAIM_BAD

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Mon, 13 Feb 2006 14:10:07 -0800

Hi Steve,

A couple of nitpicks and some text suggestions:

1) can we use metadata instead of meta-data (just to be consistent with 
rest of pNFS spec)

2) NFS4ERRs are in alphabetic order

3) can we change the position of the reclaim argument to either before 
clientid or after length (<clientid, offset, length> identify the layout).

4) change "storage server(s)" to "storage device(s)", again just to be 
consistent (These may change during a final pass through the doc, but a 
single search/replace will be easier).

In 7.7.3

- Sentence "The problem is that a client might not get the same layout 
it had previously."  needs a little more explanation.  Maybe add 
something like ", if a client uses LAYOUTGET to re-get a layout it 
previously held."  It seems like the real problem is that the client has 
uncommited metadata which describes newly written data.

- 3rd modified paragraph typo: "the date <-- storage devices"

- 4th modified paragraph: GETLAYOUT should be LAYOUTGET

In op. description:

- Change: "The client uses this request when..." to something like "The 
client uses a LAYOUTCOMMIT with the reclaim argument set to true when..."

- I think "provided by the client must be a subset" should have a 
capital must (MUST).

- We should probably explicitly say that a LAYOUTCOMMIT with reclaim 
does not give the client a layout; a LAYOUTGET must be done to reget the 
layout.

-Garth