Subject: Re: [nfsv4] Proposal for adding reclaim bit to pNFS COMMITLAYOUT

From: Benny Halevy <[email-masked]>
Date: Wed, 22 Feb 2006 16:26:29 +0200

I've added some text about usage of the reclaim field in LAYOUTCOMMIT
for RAID over objects and added a reclaim field to LAYOUTRETURN.
My proposed changes are marked below with "##> "

Benny

-------- Original Message --------
Subject: 	Re: [nfsv4] Proposal for adding reclaim bit to pNFS COMMITLAYOUT
Date: 	Wed, 15 Feb 2006 17:41:56 -0500
From: 	[email-masked]
To: 	[email-masked]

Hi All---
Here is the latest version of the proposal text.  I have
updated it to reflect Garth's comments.

Benny---
Please use this text as the basis for your text about
adding a reclaim field for LAYOUTRETURN.  Thanks!

-Steve
  
------------------------------------------------------

9.3  LAYOUTRETURN - Release Layout Information

   SYNOPSIS

##>  (cfh), clientid, offset, length, reclaim, iomode, layout_type -> -

   ARGUMENT

     struct LAYOUTRETURN4args {
             /* CURRENT_FH: file */
             clientid4               clientid;
             offset4                 offset;
             length4                 length;
##>          bool                    reclaim;
             pnfs_layoutiomode4      iomode;
             pnfs_layouttype4        layout_type;
     };

   RESULT

     struct LAYOUTRETURN4res {
             nfsstat4        status;
     };

   DESCRIPTION

   Returns the layout segment represented by the current filehandle,
   clientid, byte range, iomode, and layout type.  After this call, the
   client MUST NOT use the layout and the associated storage protocol to
   access the file data.  The layout being returned may be a subdivision
   of a layout previously fetched through LAYOUTGET.  As well, it may be
   a subset or superset of a layout specified by CB_LAYOUTRECALL.
   However, if it is a subset, the recall is not complete until the full
   byte range has been returned.  It is also permissible, and no error
   should result, for a client to return a byte range covering a layout
   it does not hold.  If the length is all 1s, the layout covers the
   range from offset to EOF.  An iomode of ANY specifies that all
   layouts that match the other arguments to LAYOUTRETURN (i.e.,
   clientid, byte range, and type) are being returned.

##> The "reclaim" field set to "true" in a LAYOUTRETURN request specifies
##> that the client is attempting to return a layout that was acquired
##> before the reboot of the metadata server during the metadata server's
##> grace period.   Returning layouts that were acquired during the metadata
##> server's grace period MUST set the "reclaim" field to "false".  (See
##> section 14.15 for more details)

   Layouts may be returned when recalled or voluntarily (i.e., before
   the server has recalled them).  In either case the client must
   properly propagate state changed under the context of the layout to
   storage or to the server before returning the layout.

   If a client fails to return a layout in a timely manner, then the
   metadata server should use its control protocol with the storage
   devices to fence the client from accessing the data referenced by the
   layout.  See Section 3.5 for more details.

   If the layout identified in the arguments does not exist, the error
   NFS4ERR_BADLAYOUT is returned.  If a layout exists, but the iomode
   does not match, NFS4ERR_BADIOMODE is returned.

##> It the LAYOUTRETURN request sets the "reclaim" field to "true" after
##> the metadata server's grace period, NFS4ERR_NO_GRACE is returned.

   On success, the current filehandle retains its value.

   [OPEN ISSUE: Should LAYOUTRETURN be modified to handle FSID
   callbacks?]

   ERRORS

      NFS4ERR_BADLAYOUT
      NFS4ERR_BADIOMODE
      NFS4ERR_FHEXPIRED
      NFS4ERR_INVAL
      NFS4ERR_NOFILEHANDLE
##>   NFS4ERR_NO_GRACE
      NFS4ERR_STALE
      NFS4ERR_STALE_CLIENTID
      NFS4ERR_UNKNOWN_LAYOUTTYPE

We require changes in section 7.7.3 (Metadata server recovery)
and section 14.15 (description of LAYOUTCOMMIT operation).
I have included new version of both of these sections below.
New lines are marked with "#>".

7.7.3  Metadata Server Recovery

   The server recovery case is slightly more complex.  In general, the
   recovery process again follows the standard NFSv4 recovery model: the
   client will discover that the metadata server has rebooted when it
   receives an unexpected STALE_STATEID or STALE_CLIENTID reply from the
   server; it will then proceed to try to reclaim its previous
   delegations during the server's recovery grace period.
#> However, layouts have a slightly different mechanism for reclaim.
#> The problem is that a client which uses LAYOUTGET to reclaim a layout
#> might not get the same layout it had previously. The range might be
#> different or it might get the same range but the content of the layout
#> might be different.  For example, if using a block/volume-based layout,
#> the blocks provisionally assigned by the layout might be different,
#> in which case the client will have to write the corresponding blocks
#> again.

#> Instead of reclaiming a layout with LAYOUTGET, a client can attempt
#> to commit data written before the file server crash by setting a
#> reclaim bit on the LAYOUTCOMMIT operation.  This should only be
#> done for data that the client has already written using a layout
#> obtained before the server restart.  For data still
#> dirty in the client memory, the client should get a new layout
#> segment after the server's grace period has elapsed.  Alternatively,
#> the client can write that data through the metadata server using
#> the standard NFSv4 WRITE. 

#> In the case that the client has written dirty data to a provisionally
#> allocated region of the layout, but was unable to commit the layout
#> changes for this data before the server rebooted, the client may be
#> unable to reliably re-read the data from the data storage devices in
#> order to write it again via the metadata server.  In this case
#> the client needs to inform the metadata server that the layout has
#> changed, before the server has completed its recovery grace period
#> and starts allowing updates to the file-system.  For this purpose,
#> the LAYOUTCOMMIT operation contains a "reclaim" field.  During the
#> metadata server's recovery grace period (and only during the
#> recovery grace period) the client may send a LAYOUTCOMMIT request
#> with the "reclaim" field set to "true".  This indicates that the
#> client is attempting to commit changes to the file layout that
#> occurred prior to the reboot of the metadata server.  The "layout
#> update" field of the request must contain the portion of the
#> layout that the client held prior to the metadata server reboot
#> which covers the outstanding writes.  The metadata server is free
#> to apply consistency checks on the layout update provided by the
#> client, and reject the request if the checks fail.  If the checks
#> do not fail, then the server MUST commit the changes to the file
#> layout contained in the "layoutupdate" field of the LAYOUTCOMMIT
#> request, ensuring that the clients outstanding writes are not lost.

#> During the recovery grace period the metadata server should apply
#> the standard approach to handling WRITE and LAYOUTGET
#> requests.  That is, if the server can reliably determine that
#> servicing such a request will not conflict with an impending
#> LAYOUTCOMMIT reclaim request, it may choose to service the request.
#> If the server is unable to offer this guarantee, it MUST reject
#> the request with status NFS4ERR_GRACE.   

#> For a metadata server to provide simple, valid handling during
#> the grace period with respect to pNFS layouts, the easiest method
#> is to simply reject all non-reclaim pNFS requests and WRITE
#> operations by returning the NFS4ERR_GRACE error.  However,
#> depending on the storage protocol and server implementation, the
#> server may be able to determine that a particular request is safe.
#> For example, a server may save provisional allocation mappings for
#> each file to stable storage, and use this information during the
#> recovery grace period to determine that a WRITE request is safe.
#> Under such circumstances, the WRITE request MAY be serviced.
#> To re-iterate, for a server to allow non-reclaim pNFS requests
#> and WRITE operations to be serviced during the recovery grace
#> period, it MUST determine that the request will not conflict
#> with any subsequent LAYOUTCOMMIT with reclaim request.
 
   There is an important safety concern associated with layouts that
   does not come into play in the standard NFSv4 case.  If a standard
   NFSv4 client makes use of a stale delegation, while reading, the
   consequence could be to deliver stale data to an application.  If
   writing, using a stale delegation or a stale state stateid for an
   open or lock would result in the rejection of the client's write with
   the appropriate stale stateid error.

   However, the pNFS layout enables the client to directly access the
   file system storage---if this access is not properly managed by the
   NFSv4 server the client can potentially corrupt the file system data
   or metadata.  Thus, it is vitally important that the client discover
   that the metadata server has rebooted, and that the client stops
   using stale layouts before the metadata server gives them away to
   other clients.  To ensure this, the client must be implemented so
   that layouts are never used to access the storage after the client's
   lease timer has expired.  It is crucial that clients have precise
   knowledge of the lease periods of their layouts.  For specific
   details on lease renewal and client lease timers, see Section 7.7.1.

   The prohibition on using stale layouts applies to all layout related
   accesses, especially the flushing of dirty data to the storage
   devices.  If the client's lease timer expires because the client
   could not contact the server for any reason, the client MUST
   immediately stop using the layout until the server can be contacted
   and the layout can be officially recovered or reclaimed.  However,
   this is only part of the solution.  It is also necessary to deal with
   the consequences of I/Os already in flight.

   The issue of the effects of I/Os started before lease expiration and
   possibly continuing through lease expiration is the responsibility of
   the data storage protocol and as such is layout type specific.  There
   are two approaches the data storage protocol can take.  The protocol
   may adopt a global solution which prevents all I/Os from being
   executed after the lease expiration and thus is safe against a client
   who issues I/Os after lease expiration.  This is the preferred
   solution and the solution used by NFSv4 file based layouts (see
   Section 9.6); as well, the object storage device protocol allows
   storage to fence clients after lease expiration.  Alternatively, the
   storage protocol may rely on proper client operation and only deal
   with the effects of lingering I/Os.  These solutions may impact the
   client layout-driver, the metadata server layout-driver, and the
   control protocol.

14.15  LAYOUTCOMMIT - Commit writes made using a layout

   SYNOPSIS

#>   (cfh), clientid, offset, length, reclaim, last_write_offset,
     time_modify, time_access, layoutupdate -> newsize

   ARGUMENT

     union newtime4 switch (bool timechanged) {
             case TRUE:
                     nfstime4           time;
             case FALSE:
                     void;
     };

     union newsize4 switch (bool sizechanged) {
             case TRUE:
                     length4            size;
             case FALSE:
                     void;
     };

     struct LAYOUTCOMMIT4args {
             /* CURRENT_FH: file */
             clientid4               clientid;
             offset4                 offset;
             length4                 length;
#>           bool                    reclaim;
             length4                 last_write_offset;
             newtime4                time_modify;
             newtime4                time_access;
             pnfs_layoutupdate4      layoutupdate;
     };

   RESULT

     struct LAYOUTCOMMIT4resok {
            newsize4                 newsize;
     };

     union LAYOUTCOMMIT4res switch (nfsstat4 status) {
             case NFS4_OK:
                     LAYOUTCOMMIT4resok  resok4;
             default:
                     void;
     };

   DESCRIPTION

   Commits changes in the layout segment represented by the current
   filehandle, clientid, and byte range.  Since layouts are sub-
   dividable, a smaller portion of a layout, retrieved via LAYOUTGET,
   may be committed.  The region being committed is specified through
   the byte range (length and offset).  Note: the "layoutupdate"
   structure does not include the length and offset, as they are already
   specified in the arguments.

   The LAYOUTCOMMIT operation indicates that the client has completed
   writes using a layout obtained by a previous LAYOUTGET.  The client
   may have only written a subset of the data range it previously
   requested.  LAYOUTCOMMIT allows it to commit or discard provisionally
   allocated space and to update the server with a new end of file.  The
   layout referenced by LAYOUTCOMMIT is still valid after the operation
   completes and can be continued to be referenced by the clientid,
   filehandle, byte range, and layout type.

#> The "reclaim" field set to "true" in a LAYOUTCOMMIT request specifies
#> that the client is attempting to commit changes to a layout after
#> the reboot of the metadata server during the metadata server's
#> recovery grace period.  This type of request may be necessary
#> when the client has uncommitted writes to provisionally allocated
#> regions of a file which were sent to the storage devices
#> before the reboot of the metadata server.  In this case the layout
#> provided by the client MUST be a subset of a writable layout that
#> the client held immediately before the reboot of the metadata server.
#> The metadata server is free to accept or reject this request based
#> on its own internal metadata consistency checks.  If the metadata
#> server finds that the layout provided by the client does not pass
#> its consistency checks, it MUST reject the request with the status
#> NFS4ERR_RECLAIM_BAD.  The successful completion of the LAYOUTCOMMIT
#> request with "reclaim" set to true does NOT provide the client with
#> a layout for the file.  It simply commits the changes to the file layout
#> specified in the "layoutupdate" field.  To obtain a layout for the
#> file the client must issue a LAYOUTGET request to the server after
#> the server's grace period has expired.  If the metadata
#> server receives a LAYOUTCOMMIT request with "reclaim" set to true
#> when the metadata server is not in its recovery grace period, it
#> MUST reject the request with the status NFS4ERR_NO_GRACE.

##> Setting the "reclaim" field to "true" is required if and only if
##> the committed layout was acquired before the metadata server reboot.
##> Committing layouts that were acquired during the metadata server's
##> grace period MUST set the "reclaim" field to "false".

   The "last_write_offset" field specifies the offset of the last byte
   written by the client previous to the LAYOUTCOMMIT.  Note: this value
   is never equal to the file's size (at most it is one byte less than
   the file's size).  The metadata server may use this information to
   determine whether the file's size needs to be updated.  If the
   metadata server updates the file's size as the result of the
   LAYOUTCOMMIT operation, it must return the new size as part of the
   results.

   The "time_modify" and "time_access" fields allow the client to
   suggest times it would like the metadata server to set.  The metadata
   server may use these time values or it may use the time of the
   LAYOUTCOMMIT operation to set these time values.  If the metadata
   server uses the client provided times, it should sanity check the
   values (e.g., to ensure time does not flow backwards).  If the client
   wants to force the metadata server to set an exact time, the client
   should use a SETATTR operation in a compound right after
   LAYOUTCOMMIT.  See Section 7.4 for more details.  If the new client
   desires the resultant mtime or atime, it should issue a GETATTR
   following the LAYOUTCOMMIT; e.g., later in the same compound.

   The "layoutupdate" argument to LAYOUTCOMMIT provides a mechanism for
   a client to provide layout specific updates to the metadata server.
   For example, the layout update can describe what regions of the
   original layout have been used and what regions can be deallocated.
   There is no NFSv4 file layout specific layoutupdate structure.

   The layout information is more verbose for block devices than for
   objects and files because the latter hide the details of block
   allocation behind their storage protocols.  At the minimum, the
   client needs to communicate changes to the end of file location back
   to the server, and, if desired, its view of the file modify and
   access time.  For block/volume layouts, it needs to specify precisely
   which blocks have been used.

   If the layout identified in the arguments does not exist, the error
   NFS4ERR_BADLAYOUT is returned.  The layout being committed may also
   be rejected if it does not correspond to an existing layout with an
   iomode of RW.

##> It the LAYOUTCOMMIT request sets the "reclaim" field to "true" after
##> the metadata server's grace period, NFS4ERR_NO_GRACE is returned.

   On success, the current filehandle retains its value.

##> IMPLEMENTATION
##> 
##> Optionally, the client can also use LAYOUTCOMMIT with the "reclaim"
##> field set to "true" to convey hints to modified file attributes or
##> to report I/O errors in the object-based pNFS layout type as normally
##> done during normal operation. Doing so may help the metadata server to
##? recover the file's attributes and parity more efficiently after reboot.
##> For example, when using RAID over multiple objects, the metadata server
##> must recover the file's attributes and verify the file's parity
##> consistency if it does not have a positive indication from all clients
##> that could have modified the file that they have successfully completed
##> all their writes.  Sending a LAYOUTCOMMIT (if required) and then following
##> with LAYOUTRETURN can provide such an indication and allow for graceful
##> and efficient recovery.

   ERRORS

      NFS4ERR_BADLAYOUT
      NFS4ERR_BADIOMODE
      NFS4ERR_FHEXPIRED
      NFS4ERR_INVAL
      NFS4ERR_NOFILEHANDLE
#>    NFS4ERR_NO_GRACE
#>    NFS4ERR_RECLAIM_BAD
      NFS4ERR_STALE
      NFS4ERR_STALE_CLIENTID
      NFS4ERR_UNKNOWN_LAYOUTTYPE

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Wed, 22 Feb 2006 09:13:41 -0800

The only objection I have is with the specific mention of the RAID over 
objects within the Implementation section of LAYOUTCOMMIT.  Since we 
never mention RAID over objects in the pNFS doc.  I believe that it is 
more appropriate to place this text in the object specific layout doc. 
If you can replace the text with something more general to the pNFS 
protocol (rather than specific to a layout type and scheme) that would 
also be ok with me.

-Garth

Benny Halevy wrote:
> ##> ##> Optionally, the client can also use LAYOUTCOMMIT with the "reclaim"

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Feb 2006 14:48:06 +0200

Take two...
See new text below.

Thanks,

Benny

Garth Goodson wrote:

> The only objection I have is with the specific mention of the RAID 
> over objects within the Implementation section of LAYOUTCOMMIT.  Since 
> we never mention RAID over objects in the pNFS doc.  I believe that it 
> is more appropriate to place this text in the object specific layout 
> doc. If you can replace the text with something more general to the 
> pNFS protocol (rather than specific to a layout type and scheme) that 
> would also be ok with me.
>
> -Garth
>
> Benny Halevy wrote:
>
>> #> the request with status NFS4ERR_GRACE.  #> For a metadata server 
>> to provide simple, valid handling during
##> Optionally, the client can also use LAYOUTCOMMIT with the "reclaim"
##> field set to "true" to convey hints to modified file attributes or
##> to report layout-type specific information such as I/O errors for 
object-based
##> storage layouts, as normally done during normal operation. Doing so 
may help
##> the metadata server to recover files more efficiently after reboot.
##> For example, some file system implementations may require expansive 
recovery
##> of filesystem objects if the metadata server does not get  a 
positive indication from
##> all clients holding a write layout that they have successfully 
completed
##> all their writes.  Sending a LAYOUTCOMMIT (if required) and then 
following
##> with LAYOUTRETURN can provide such an indication and allow for graceful
##> and efficient recovery.