Subject: [nfsv4] Issue 58: pnfs: layout return flag for fsid returns

From: Benny Halevy <[email-masked]>
Date: Wed, 02 Aug 2006 16:18:09 +0300

Here's the updated proposal for the new flags for layout
recall and return based on my previous proposal and our
discussion in Montreal.

Please provide any feedback soon as I'd like to declare consensus
by 8/9.

Benny

16.5.1.  Basic Operation

    Since a layout protects a client's access to a file via a direct
    client-storage-device path, a layout need only be recalled when it is
    semantically unable to serve this function.  Typically, this occurs
    when the layout no longer encapsulates the true location of the file
    over the byte range it represents.  Any operation or action (e.g.,
    server driven restriping or load balancing) that changes the layout
    will result in a recall of the layout.  A layout is recalled by the
    CB_LAYOUTRECALL callback operation (see Section 24.3).  This callback
-   can either recall a layout segment identified by a byte range, or all
-   the layouts associated with a file system (FSID).  However, there is
-   no single operation to return all layouts associated with an FSID;
-   multiple layout segments may be returned in a single compound
+   can either recall a layout segment identified by a byte range, all
+   the layouts associated with a file system (FSID), or all layouts.
+   Multiple layout segments may be returned in a single compound
    operation.  Section 16.5.3 discusses sequencing issues surrounding
    the getting, returning, and recalling of layouts.

 16.5.2.  Recall Callback Robustness

    For simplicity, the discussion thus far has assumed that pNFS client
    state for a file exactly matches the pNFS server state for that file
    and client regarding layout ranges and permissions.  This assumption
    leads to the implicit assumption that any callback results in a
@@ -58,168 +57,232 @@
    for a client to return ranges it does not hold.  A pNFS client must
    always return layout segments that comprise the full range specified
    by the recall.  Note, the full recalled layout range need not be
    returned as part of a single operation, but may be returned in
    segments.  This allows the client to stage the flushing of dirty
    data, layout commits, and returns.  Also, it indicates to the
    metadata server that the client is making progress.

    In order to ensure client/server convergence on the layout state, the
    final LAYOUTRETURN operation in a sequence of returns for a
-   particular recall, SHOULD specify the entire range being recalled,
+   particular recall, MUST specify the entire range being recalled,
+   echoing the recalled layout type, iomode, recall/return type (FILE, 
FSID,
+   or ALL), and byte range;
    even if layout segments pertaining to partial ranges were previously
    returned.  In addition, if the client holds no layout segment that
    overlaps the range being recalled, the client should return the
    NFS4ERR_NOMATCHING_LAYOUT error code.  This allows the server to
    update its view of the client's layout state.

 22.48.  Operation 50: LAYOUTRETURN - Release Layout Information

 22.48.1.  SYNOPSIS

-   (cfh), clientid, offset, length, reclaim, iomode, layout_type -> -
+   (cfh), clientid, layout_type, iomode, layoutreturn, reclaim -> -

 22.48.2.  ARGUMENT

+
+   enum layoutreturn_type4 {
+       RETURN_FILE = 1,
+       RETURN_FSID = 2,
+       RETURN_ALL  = 3
+   };
+
+
+   struct layoutreturn_file4 {
+       offset4         offset;
+       length4         length;
+   };
+
+
+   union layoutreturn4 switch(layoutreturn_type4 returntype) {
+       case RETURN_FILE:
+           layoutreturn_file4 layout;
+       default:
+           void;
+   };
+
+
    struct LAYOUTRETURN4args {
        /* CURRENT_FH: file */
        clientid4               clientid;
-       offset4                 offset;
-       length4                 length;
        bool                    reclaim;
-       pnfs_layoutiomode4      iomode;
        pnfs_layouttype4        layout_type;
+       pnfs_layoutiomode4      iomode;
+       layoutreturn4           layoutreturn;
    };

 22.48.3.  RESULT

    struct LAYOUTRETURN4res {
        nfsstat4        status;
    };

 22.48.4.  DESCRIPTION

-   Returns the layout segment represented by the current filehandle,
-   clientid, byte range, iomode, and layout type.  After this call, the
-   client MUST NOT use the layout and the associated storage protocol to
+   Returns one or more layout segments represented by the clientid,
+   layout_type, and iomode.  When layoutreturn is RETURN_FILE the
+   returned layout segment is identified by the current filehandle,
+   offset, and length.  When layoutreturn is RETURN_FSID the current
+   filehandle is used to identify the file system and all layouts
+   matching the clientid, layout_type, and iomode are returned.
+   When layoutreturn is RETURN_ALL all layouts matching the clientid,
+   layout_type, and iomode are returned and the current filehandle is
+   not used.
+   After this call, the client MUST NOT use the returned layout(s) and
+   the associated storage protocol to
    access the file data.  The layout being returned may be a subdivision
    of a layout previously fetched through LAYOUTGET.  As well, it may be
    a subset or superset of a layout specified by CB_LAYOUTRECALL.
    However, if it is a subset, the recall is not complete until the full
-   byte range has been returned.  It is also permissible, and no error
+   recalled scope (FILE byte range, FSID, or ALL) has been returned.
+   It is also permissible, and no error
    should result, for a client to return a byte range covering a layout
    it does not hold.  If the length is all 1s, the layout covers the
    range from offset to EOF.  An iomode of ANY specifies that all
    layouts that match the other arguments to LAYOUTRETURN (i.e.,
-   clientid, byte range, and type) are being returned.
+   clientid, type, filehandle and byte range, fsid, or all) are being 
returned.
+
+   When returntype is set to RETURN_FSID or RETURN_ALL the client also
+   invalidates all the devices in the affected filesystem(s).  Any device
+   identifier returned by a subsequent LAYOUTGET in the affected 
filesystem(s)
+   will have to be resolved using either GETDEVICEINFO or GETDEVICELIST.

    The "reclaim" field set to "true" in a LAYOUTRETURN request specifies
    that the client is attempting to return a layout that was acquired
    before the reboot of the metadata server during the metadata server's
    grace period.  Returning layouts that were acquired during the
    metadata server's grace period MUST set the "reclaim" field to
-   "false".  See LAYOUTCOMMIT (Section 22.46) for more details.
+   "false".  The "reclaim" field MUST be set to false also when 
layoutreturn
+   is for RETURN_FSID or RETURN_ALL. See LAYOUTCOMMIT (Section 22.46) 
for more details.

    Layouts may be returned when recalled or voluntarily (i.e., before
    the server has recalled them).  In either case the client must
    properly propagate state changed under the context of the layout to
    storage or to the server before returning the layout.

    If a client fails to return a layout in a timely manner, then the
    metadata server should use its control protocol with the storage
    devices to fence the client from accessing the data referenced by the
    layout.  See Section 16.5 for more details.

    If the layout identified in the arguments does not exist, the error
    NFS4ERR_BADLAYOUT is returned.  If a layout exists, but the iomode
    does not match, NFS4ERR_BADIOMODE is returned.

    If the LAYOUTRETURN request sets the "reclaim" field to "true" after
    the metadata server's grace period, NFS4ERR_NO_GRACE is returned.

+   If the LAYOUTRETURN request sets the "reclaim" field to "true" and
+   returntype is set to RETURN_FSID or RETURN_ALL, NFS4ERR_INVAL is 
returned.
+
    On success, the current filehandle retains its value.

 22.48.5.  IMPLEMENTATION

+   Returning all layouts in a file system using RETURN_FSID is typically
+   done in response to a CB_LAYOUTRECALL for that file system as the final
+   return operation. Similarly, RETURN_ALL is used in response to a recall
+   callback for all layouts.
+   It is possible that the client already returned some outstanding
+   layouts via individual LAYOUTRETURN calls and the call for RETURN_FSID
+   or RETURN_ALL marks the end of the layoutreturn sequence.
+   See "Recall Callback Robustness" (Section 16.5.2) for more details.
+

 24.3.  Operation 5: CB_LAYOUTRECALL

 24.3.1.  SYNOPSIS

    layout_type, iomode, layoutchanged, layoutrecall -> -

 24.3.2.  ARGUMENT

    enum layoutrecall_type4 {
        RECALL_FILE = 1,
-       RECALL_FSID = 2
+       RECALL_FSID = 2,
+       RECALL_ALL  = 3
    };

    struct layoutrecall_file4 {
        nfs_fh4         fh;
        offset4         offset;
        length4         length;
    };

    union layoutrecall4 switch(layoutrecall_type4 recalltype) {
        case RECALL_FILE:
            layoutrecall_file4 layout;
        case RECALL_FSID:
            fsid4              fsid;
+       case RECALL_ALL:
+           void;
    };

-   struct CB_LAYOUTRECALLargs {
+   struct CB_LAYOUTRECALL4args {
        pnfs_layouttype4        layout_type;
        pnfs_layoutiomode4      iomode;
        bool                    layoutchanged;
        layoutrecall4           layoutrecall;
    };

 24.3.3.  RESULT

-   struct CB_LAYOUTRECALLres {
+   struct CB_LAYOUTRECALL4res {
        nfsstat4        status;
    };

 24.3.4.  DESCRIPTION

    The CB_LAYOUTRECALL operation is used to begin the process of
-   recalling a layout, a portion thereof, or all layouts pertaining to a
-   particular file system (FSID).  If RECALL_FILE is specified, the
+   recalling a layout, a portion thereof, all layouts pertaining to a
+   particular file system (FSID), or layouts in all filesystems (ALL).
+   If RECALL_FILE is specified, the
    offset and length fields specify the portion of the layout to be
-   returned.  The iomode specifies the set of layouts to be returned.
+   returned.
+   If a length of all 1s is specified then
+   the layout corresponding to the byte range from "offset" to the end-
+   of-file MUST be returned.
+   The iomode specifies the set of layouts to be returned.
    An iomode of ANY specifies that all matching layouts, regardless of
    iomode, must be returned; otherwise, only layouts that exactly match
    the iomode must be returned.

    If the "layoutchanged" field is TRUE, then the client SHOULD not
    flush its dirty data to the devices specified by the layout being
    recalled.  Instead, it is preferable for the client to flush the
    dirty data through the metadata server.  Alternatively, the client
    may attempt to obtain a new layout.  Note: in order to obtain a new
    layout the client must first return the old layout.  Since obtaining
    a new layout is not guaranteed to succeed, the client must be ready
    to flush its dirty data through the metadata server.

-   If RECALL_FSID is specified, the fsid specifies the file system for
-   which any outstanding layouts must be returned.  Layouts are returned
-   through the LAYOUTRETURN operation.
+   Layouts are returned through the LAYOUTRETURN operation.

    If the client does not hold any layout segment either matching or
    overlapping with the requested layout, it returns
-   NFS4ERR_NOMATCHING_LAYOUT.  If a length of all 1s is specified then
-   the layout corresponding to the byte range from "offset" to the end-
-   of-file MUST be returned.
+   NFS4ERR_NOMATCHING_LAYOUT.
+
+   If RECALL_FSID is specified, the fsid specifies the file system for
+   which any outstanding layouts MUST be returned.
+   If RECALL_ALL is specified, all outstanding layouts MUST be returned.
+   In addition, RECALL_FSID and RECALL_ALL specify that all the devices
+   in the affected filesystem(s) are also recalled. The respective
+   LAYOUTRETURN with either RETURN_FSID or RETURN_ALL acknowledges to
+   the server that the client invalidated the said devices.
+   Devices are invalidated also when no layouts are found for RECALL_FSID
+   or RECALL_ALL and NFS4ERR_NOMATCHING_LAYOUT is returned.

 24.3.5.  IMPLEMENTATION

    The client should reply to the callback immediately.  Replying does
    not complete the recall except when an error is returned.  The recall
    is not complete until the layout(s) are returned using a
    LAYOUTRETURN.

    The client should complete any in-flight I/O operations using the
    recalled layout(s) before returning it/them via LAYOUTRETURN.  If the
@@ -232,13 +295,13 @@
    after first returning the recalled layout, using the new layout to
    flush the dirty data.  Regardless of whether the client is holding a
    layout, it may always write data through the metadata server.

    If dirty data is flushed while the layout is held, the client must
    still issue LAYOUTCOMMIT operations at the appropriate time,
    especially before issuing the LAYOUTRETURN.  If a large amount of
    dirty data is outstanding, the client may issue LAYOUTRETURNs for
    portions of the layout being recalled; this allows the server to
    monitor the client's progress and adherence to the callback.
-   However, the last LAYOUTRETURN in a sequence of returns, SHOULD
+   However, the last LAYOUTRETURN in a sequence of returns, MUST
    specify the full range being recalled (see Section 16.5.2 for
    details).

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Wed, 02 Aug 2006 10:56:04 -0700

Can you add a sentence to 16.5.1 about the client flushing its device 
cache on FSID or ALL recall?

-Garth

---

From: Benny Halevy <[email-masked]>
Date: Wed, 02 Aug 2006 21:04:49 +0300

> cache on FSID or ALL recall?
Sure.  I used "invalidate" rather than "flush" in other places since 
typically
flushing involves writing dirty cache contents onto primary storage and
I don't think anything can go dirty in this cache...

Benny
>> +   echoing the recalled layout type, iomode, recall/return type 
>> (FILE, FSID,
>> +   clientid, type, filehandle and byte range, fsid, or all) are 
>> being returned.
>> +
>> +   is for RETURN_FSID or RETURN_ALL. See LAYOUTCOMMIT (Section 
>> 22.46) for more details.
>>

---

From: Benny Halevy <[email-masked]>
Date: Tue, 08 Aug 2006 12:45:46 +0300

Attached is the edited proposal incorporating your feedback.
Hoping to declare wg consensus in today's conference call.

Benny

>>>    leads to the implicit assumption that any callback results in a

--- layout_return_fsid-04.orig	2006-08-02 16:22:45.000000000 +0300
+++ layout_return_fsid-04.txt	2006-08-08 12:36:27.000000000 +0300
@@ -1,31 +1,32 @@
 16.5.  Recalling a Layout
 
 16.5.1.  Basic Operation
 
+   the layouts associated with a filesystem (FSID), or all layouts.
+   Recalling all layouts or all the layouts associated with a filesystem
+   also invalidates the client's device cache for the affected filesystems.

    For simplicity, the discussion thus far has assumed that pNFS client
    state for a file exactly matches the pNFS server state for that file
    and client regarding layout ranges and permissions.  This assumption
    leads to the implicit assumption that any callback results in a
    LAYOUTRETURN or set of LAYOUTRETURNs that exactly match the range in
    the callback, since both client and server agree about the state
    being maintained.  However, it can be useful if this assumption does
    not always hold.  For example:
 
@@ -53,192 +54,256 @@
       to issue the LAYOUTRETURN that the server is expecting in response
       to a recall.
 
    Thus, in light of the above, it is useful for a server to be able to
    issue callbacks for layout ranges it has not granted to a client, and
    for a client to return ranges it does not hold.  A pNFS client must
    always return layout segments that comprise the full range specified
    by the recall.  Note, the full recalled layout range need not be
    returned as part of a single operation, but may be returned in
    segments.  This allows the client to stage the flushing of dirty
    data, layout commits, and returns.  Also, it indicates to the
    metadata server that the client is making progress.
 
+   echoing the recalled layout type, iomode, recall/return type (FILE, FSID,

 
 22.48.1.  SYNOPSIS

 22.48.2.  ARGUMENT
 
+

 22.48.3.  RESULT

 22.48.4.  DESCRIPTION
 
+   clientid, type, filehandle and byte range, fsid, or all) are being returned.
+   identifier returned by a subsequent LAYOUTGET in the affected filesystem(s)
 
    The "reclaim" field set to "true" in a LAYOUTRETURN request specifies
    that the client is attempting to return a layout that was acquired
    before the reboot of the metadata server during the metadata server's
    grace period.  Returning layouts that were acquired during the
    metadata server's grace period MUST set the "reclaim" field to
-   "false".  See LAYOUTCOMMIT (Section 22.46) for more details.
+   "false".  The "reclaim" field MUST be set to false also when layoutreturn
+   is for RETURN_FSID or RETURN_ALL. See LAYOUTCOMMIT (Section 22.46) for more details.

 

+   If the LAYOUTRETURN request sets the "reclaim" field to "true" and
+   returntype is set to RETURN_FSID or RETURN_ALL, NFS4ERR_INVAL is returned.
+
    On success, the current filehandle retains its value.
 
 22.48.5.  IMPLEMENTATION

 
 24.3.1.  SYNOPSIS

 24.3.2.  ARGUMENT

 

 24.3.3.  RESULT

 24.3.4.  DESCRIPTION

 

 24.3.5.  IMPLEMENTATION

    The client should complete any in-flight I/O operations using the
    recalled layout(s) before returning it/them via LAYOUTRETURN.  If the
    client has buffered dirty data there are a number of options for
    flushing that data.  If "layoutchanged" is false, the client may
    choose to write dirty data directly to storage before calling
    LAYOUTRETURN.  However, if "layoutchanged" is true, the client may
    either choose to write it later using normal NFSv4 WRITE operations
    to the metadata server or it may attempt to obtain a new layout,
    after first returning the recalled layout, using the new layout to
    flush the dirty data.  Regardless of whether the client is holding a
    layout, it may always write data through the metadata server.
 
+   However, the last LAYOUTRETURN in a sequence of returns MUST

---

From: Todd Pisek <[email-masked]>
Date: Tue, 08 Aug 2006 09:26:27 -0500

Benny Halevy wrote On 08/08/06 04:45 AM,:
:

>+   layout_type, and iomode are returned and the current filehandle is
I understand the need to match iomode and clientid, but why layout_type?

---

From: Todd Pisek <[email-masked]>
Date: Tue, 08 Aug 2006 09:46:59 -0500

In the case of multiple layout types in the same filesystem, server, etc.,
a server wanting to recall all layouts, regardless of layout type, must
issue a layout recall for each outstanding type. Correct?

Note: I'm not saying this is good or bad. Just looking for clarity on my
behalf.

--- Todd

Benny Halevy wrote On 08/08/06 09:34 AM,:

> It is possible (though unlikely I think) that the client will hold 
> layouts of different layout_types for the same file
> and from the same server.
>
> Benny
>
> Todd Pisek wrote:
>

---

From: Benny Halevy <[email-masked]>
Date: Tue, 08 Aug 2006 17:50:44 +0300

> issue a layout recall for each outstanding type. Correct?
Yes, that's correct.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 08 Aug 2006 17:34:22 +0300

Benny