Subject: [nfsv4] Proposal for adding reclaim bit to pNFS COMMITLAYOUT

From: [email-masked]
Date: Thu, 26 Jan 2006 13:59:13 -0500

Hi all---
As we discussed on the latest conference call, I'd like to
propose adding a reclaim bit to the pNFS COMMITLAYOUT operation.
I've drafted the required changes to the pnfs4.1 doc and
included them below.  Take a look and please send comments/
corrections.

-Steve

----------------------------------------------------------------

We require changes in section 7.7.3 (Meta-data server recovery)
and section 14.15 (description of LAYOUTCOMMIT operation).
I have included new version of both of these sections below.
New lines are marked with "#>".

7.7.3  Metadata Server Recovery

   The server recovery case is slightly more complex.  In general, the
   recovery process again follows the standard NFSv4 recovery model: the
   client will discover that the metadata server has rebooted when it
   receives an unexpected STALE_STATEID or STALE_CLIENTID reply from the
   server; it will then proceed to try to reclaim its previous
   delegations during the server's recovery grace period.  However,
   layouts are not reclaimable in the same sense as data delegations;
   there is no reclaim bit, thus no guarantee of continuity between the
   previous and new layout.  This is not necessarily required since a
   layout is not required to perform I/O; I/O can always be performed
   through the metadata server.

   If the client has dirty data that it needs to write out, or an
   outstanding LAYOUTCOMMIT, the client should try to obtain a new
   layout segment covering the byte range covered by the previous layout
   segment.  However, the client might not not get the same layout
   segment it had.  The range might be different or it might get the
   same range but the content of the layout might be different.  For
   example, if using a block/volume-based layout, the blocks
   provisionally assigned by the layout might be different, in which
   case the client will have to write the corresponding blocks again; in
   the interest of simplicity, the client might decide to always write
   them again.  Alternatively, the client might be unable to obtain a
   new layout and thus, must write the data using normal NFSv4 through
#> the metadata server.  Also note that the meta-data server will not
#> allow any operations that could potentially change the state of the
#> file-system during the recovery grace period.  Thus reclaiming of
#> pNFS layouts and writing of dirty data must wait until the recovery
#> grace period is over.

#> It is also possible that the client has written dirty data to a
#> provisionally allocated region of the layout, but was unable to
#> commit the layout changes for this data before the server rebooted.
#> In this case, depending on the data storage protocol, the client may
#> be unable to reliably re-read the data from the date storage devices
#> in order to write it again via the meta-data server.  In this case
#> the client needs to inform the meta-data server that the layout has
#> changed, before the server has completed its recovery grace period
#> and starts allowing updates to the file-system.  For this purpose,
#> the LAYOUTCOMMIT operation contains a "reclaim" field.  During the
#> meta-data server's recovery grace period (and only during the
#> recovery grace period) the client may send a LAYOUTCOMMIT request
#> with the "reclaim" field set to "true".  This indicates that the
#> client is attempting to commit changes to the file layout that
#> occurred prior to the reboot of the meta-data server.  The "layout
#> update" field of the request should contain the portion of the
#> layout that the client held prior to the meta-data server reboot
#> which covers the outstanding writes.  The meta-data server is free
#> to apply consistency checks on the layout update provided by the
#> client, and reject the request if the checks fail.  If the checks
#> do not fail, then the server will commit the changes to the file
#> layout contained in the "layoutupdate" field of the LAYOUTCOMMIT
#> request, ensuring that the clients outstanding writes are not lost.

   There is an important safety concern associated with layouts that
   does not come into play in the standard NFSv4 case.  If a standard
   NFSv4 client makes use of a stale delegation, while reading, the
   consequence could be to deliver stale data to an application.  If
   writing, using a stale delegation or a stale state stateid for an
   open or lock would result in the rejection of the client's write with
   the appropriate stale stateid error.

   However, the pNFS layout enables the client to directly access the
   file system storage---if this access is not properly managed by the
   NFSv4 server the client can potentially corrupt the file system data
   or metadata.  Thus, it is vitally important that the client discover
   that the metadata server has rebooted, and that the client stops
   using stale layouts before the metadata server gives them away to
   other clients.  To ensure this, the client must be implemented so
   that layouts are never used to access the storage after the client's
   lease timer has expired.  It is crucial that clients have precise
   knowledge of the lease periods of their layouts.  For specific
   details on lease renewal and client lease timers, see Section 7.7.1.

   The prohibition on using stale layouts applies to all layout related
   accesses, especially the flushing of dirty data to the storage
   devices.  If the client's lease timer expires because the client
   could not contact the server for any reason, the client MUST
   immediately stop using the layout until the server can be contacted
   and the layout can be officially recovered or reclaimed.  However,
   this is only part of the solution.  It is also necessary to deal with
   the consequences of I/Os already in flight.

   The issue of the effects of I/Os started before lease expiration and
   possibly continuing through lease expiration is the responsibility of
   the data storage protocol and as such is layout type specific.  There
   are two approaches the data storage protocol can take.  The protocol
   may adopt a global solution which prevents all I/Os from being
   executed after the lease expiration and thus is safe against a client
   who issues I/Os after lease expiration.  This is the preferred
   solution and the solution used by NFSv4 file based layouts (see
   Section 9.6); as well, the object storage device protocol allows
   storage to fence clients after lease expiration.  Alternatively, the
   storage protocol may rely on proper client operation and only deal
   with the effects of lingering I/Os.  These solutions may impact the
   client layout-driver, the metadata server layout-driver, and the
   control protocol.

14.15  LAYOUTCOMMIT - Commit writes made using a layout

   SYNOPSIS

#>   (cfh), clientid, reclaim, offset, length, last_write_offset,
     time_modify, time_access, layoutupdate -> newsize

   ARGUMENT

     union newtime4 switch (bool timechanged) {
             case TRUE:
                     nfstime4           time;
             case FALSE:
                     void;
     };

     union newsize4 switch (bool sizechanged) {
             case TRUE:
                     length4            size;
             case FALSE:
                     void;
     };

     struct LAYOUTCOMMIT4args {
             /* CURRENT_FH: file */
             clientid4               clientid;
#>           bool                    reclaim;
             offset4                 offset;
             length4                 length;
             length4                 last_write_offset;
             newtime4                time_modify;
             newtime4                time_access;
             pnfs_layoutupdate4      layoutupdate;
     };

   RESULT

     struct LAYOUTCOMMIT4resok {
            newsize4                 newsize;
     };

     union LAYOUTCOMMIT4res switch (nfsstat4 status) {
             case NFS4_OK:
                     LAYOUTCOMMIT4resok  resok4;
             default:
                     void;
     };

   DESCRIPTION

   Commits changes in the layout segment represented by the current
   filehandle, clientid, and byte range.  Since layouts are sub-
   dividable, a smaller portion of a layout, retrieved via LAYOUTGET,
   may be committed.  The region being committed is specified through
   the byte range (length and offset).  Note: the "layoutupdate"
   structure does not include the length and offset, as they are already
   specified in the arguments.

   The LAYOUTCOMMIT operation indicates that the client has completed
   writes using a layout obtained by a previous LAYOUTGET.  The client
   may have only written a subset of the data range it previously
   requested.  LAYOUTCOMMIT allows it to commit or discard provisionally
   allocated space and to update the server with a new end of file.  The
   layout referenced by LAYOUTCOMMIT is still valid after the operation
   completes and can be continued to be referenced by the clientid,
   filehandle, byte range, and layout type.

#> The "reclaim" field specifies that the client is attempting to
#> commit changes to a layout after the reboot of the meta-data server
#> during the meta-data server's recovery grace period.  The client
#> uses this request when it has un-committed writes to provisionally
#> allocated regions of a file which were sent to the storage servers
#> before the reboot of the meta-data server.  In this case the layout
#> provided by the client must be a subset of a writable layout that
#> the client held immediately before the reboot of the meta-data server.
#> The meta-data server is free to accept or reject this request based
#> on its own internal meta-data consistency checks.  If the meta-data
#> server finds that the layout provided by the client does not pass
#> its consistency checks, it should reject the request with the status
#> NFS4ERR_RECLAIM_BAD.  If the meta-data server returns success, it
#> does not mean that the client possesses any layout for the file.
#> It only means that the changes to the file layout specified in the
#> "layoutupdate" field are committed to the file.  If the meta-data
#> server receives a LAYOUTCOMMIT request with "reclaim" set to true
#> when the meta-data server is not in its recovery grace period, it
#> must reject the request with the status NFS4ERR_NO_GRACE.

   The "last_write_offset" field specifies the offset of the last byte
   written by the client previous to the LAYOUTCOMMIT.  Note: this value
   is never equal to the file's size (at most it is one byte less than
   the file's size).  The metadata server may use this information to
   determine whether the file's size needs to be updated.  If the
   metadata server updates the file's size as the result of the
   LAYOUTCOMMIT operation, it must return the new size as part of the
   results.

   The "time_modify" and "time_access" fields allow the client to
   suggest times it would like the metadata server to set.  The metadata
   server may use these time values or it may use the time of the
   LAYOUTCOMMIT operation to set these time values.  If the metadata
   server uses the client provided times, it should sanity check the
   values (e.g., to ensure time does not flow backwards).  If the client
   wants to force the metadata server to set an exact time, the client
   should use a SETATTR operation in a compound right after
   LAYOUTCOMMIT.  See Section 7.4 for more details.  If the new client
   desires the resultant mtime or atime, it should issue a GETATTR
   following the LAYOUTCOMMIT; e.g., later in the same compound.

   The "layoutupdate" argument to LAYOUTCOMMIT provides a mechanism for
   a client to provide layout specific updates to the metadata server.
   For example, the layout update can describe what regions of the
   original layout have been used and what regions can be deallocated.
   There is no NFSv4 file layout specific layoutupdate structure.

   The layout information is more verbose for block devices than for
   objects and files because the latter hide the details of block
   allocation behind their storage protocols.  At the minimum, the
   client needs to communicate changes to the end of file location back
   to the server, and, if desired, its view of the file modify and
   access time.  For block/volume layouts, it needs to specify precisely
   which blocks have been used.

   If the layout identified in the arguments does not exist, the error
   NFS4ERR_BADLAYOUT is returned.  The layout being committed may also
   be rejected if it does not correspond to an existing layout with an
   iomode of RW.

   On success, the current filehandle retains its value.

   ERRORS

      NFS4ERR_BADLAYOUT
      NFS4ERR_BADIOMODE
      NFS4ERR_FHEXPIRED
      NFS4ERR_INVAL
      NFS4ERR_NOFILEHANDLE
      NFS4ERR_STALE
      NFS4ERR_STALE_CLIENTID
      NFS4ERR_UNKNOWN_LAYOUTTYPE
#>    NFS4ERR_NO_GRACE
#>    NFS4ERR_RECLAIM_BAD

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Brent Welch <[email-masked]>
Date: Thu, 26 Jan 2006 12:52:31 -0800

I think the first paragraph could be tweaked slightly to better introduce
the reclaim-on-layoutcommit notion.  See below.

>>>[email-masked] said:
 > 
 > Hi all---

However, layouts have a slightly different mechanism for reclaim.
The problem is that a client might not get the same layout it
had previously. The range might be different or it might get the
same range but the content of the layout might be different.  For
example, if using a block/volume-based layout, the blocks
provisionally assigned by the layout might be different, in which
case the client will have to write the corresponding blocks again.

Instead of reclaiming a layout with LAYOUTGET, a client can attempt
to commit data written before the file server crash by setting a
reclaim bit on the LAYOUTCOMMIT operation.  This should only be
done for data that the client has already written using a layout
obtained before the server restart.  For data still
dirty in the client memory, the client should get a new layout
segment after the server's grace period has elapsed.  Alternatively,
the client can write that data through the metadata server.  Note
that the meta-data server will not
allow any operations that could potentially change the state of the
file-system during the recovery grace period.  Thus getting a new
pNFS layout and writing of dirty data must wait until the recovery
grace period is over.

[The above replaces the following text]
 > #> pNFS layouts and writing of dirty data must wait until the recovery
[down to here]

[And then tweak the first two sentences of this paragraph slightly]

In the case that the client has written dirty data to a
provisionally allocated region of the layout, but was unable to
commit the layout changes for this data before the server rebooted,
the client may be unable to reliably re-read the data from the date storage 
devices
in order to write it again via the meta-data server.  In this case...

 > #> in order to write it again via the meta-data server.  In this case

[pick up here - the rest looks OK]

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Garth Goodson <[email-masked]>
Date: Thu, 26 Jan 2006 13:05:23 -0800

The text looks good, but I have clarifying question/comment.  From this 
it isn't clear whether a server can not support reclaimable layouts.  It 
may return a NFS4ERR_RECLAIM_BAD error if "the layout provided by the 
client does not pass its [the metadata server's] consistency checks".

Do we want to allow certain layout types/implementations to have the 
ability to not support reclaimable layouts?  If so, are these error 
return codes good enough or do we need to add something else.  As well, 
we may want to clarify this issue in the text.

-Garth

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Fri, 27 Jan 2006 09:06:44 -0500

[email-masked] said:
> Instead of reclaiming a layout with LAYOUTGET, a client can attempt to commit
> data written before the file server crash by setting a reclaim bit on the
> LAYOUTCOMMIT operation.  This should only be done for data that the client
> has already written using a layout obtained before the server restart.  For
> data still dirty in the client memory, the client should get a new layout
> segment after the server's grace period has elapsed.  Alternatively, the
> client can write that data through the metadata server. 
> Note that the
> meta-data server will not allow any operations that could potentially change
> the state of the file-system during the recovery grace period. Thus getting
> a new pNFS layout and writing of dirty data must wait until the recovery
> grace period is over.  

the last sentence is unclear - writing of dirty data does not have to wait 
until the recovery grace period is over, as suggested, the client can recover 
the OPEN(write) and write the dirty data through the meta-data server a la 
v4.0.

so change 

"Thus getting a new pNFS layout and writing of dirty data must wait until the 
recovery grace period is over. "

to  

"Thus getting a new pNFS layout and writing of dirty data via the data storage 
protocol must wait until the recovery grace period is over."

-->Andy

---

From: Benny Halevy <[email-masked]>
Date: Fri, 27 Jan 2006 17:22:29 +0200

I agree that writing through the MDS may work in most cases, just note 
that for some servers (possibly depending also on the layout type or map 
type) writing to the storage servers while clients might have valid 
write layouts might cause data corruption.

My understanding of the spec is that outstanding layouts are implicitly 
invalidated when the server reboots since clients cannot reclaim or 
renew their layouts during the grace period, but the server must make 
sure that any outstanding write layout is either recalled, revoked, or 
expired. If the server does not keep persistent state about which 
clients had write layouts on which file before the reboot it might have 
to wait until the grace period ends before COMMITting any writes.

Even if the server COMMITs writes to local stable storage (e.g. local 
disk and/or nvram) during the grace period and sync them to the storage 
servers later to avoid data corruption due to unsynchronized concurrent 
writing, pnfs readers may read directly from storage and will not see 
these writes, breaking distributed data coherency.

Benny