Subject: [nfsv4] Issue 103: keeping the client and server layout state in sync

From: Benny Halevy <[email-masked]>
Date: Wed, 02 Aug 2006 15:24:10 +0300

I opened this issue and assigned to myself to track the changes needed 
to make sure the client's layout state never diverges from the 
server's.  Originally, I thought of including these changes as part of 
issue 58, but the changes are just too broad to be part of that, hence a 
new issue.  Proposed text coming soon (before 8/8/06 :).

Benny

--

Currently the nfsv4.1 draft assumes that the client and server can go 
out of sync with respect to the state of the layouts a particular client 
holds and that there are mechanisms that allow that state to converge 
back.  There is concesus that the layout state must never diverge 
(although a "wildcard" recall and return of FSID or ALL can be used to 
reset the state in case the state has diverged for an unexpected reason, 
e.g. a bug)

http://www1.ietf.org/mail-archive/web/nfsv4/current/msg03329.html 
summarizes the current discussion regarding the proposed mechanism for 
keeping the client and server layout state in sync.

I will send a separate message with the proposed changes to the latest 
v1 draft.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Tue, 08 Aug 2006 14:24:31 +0300

Here's the proposed change (on top of my changes in issue 58):

 16.5.2.  Recall Callback Robustness

-   For simplicity, the discussion thus far has assumed that pNFS client
+   It is assumed that pNFS client
    state for a file exactly matches the pNFS server state for that file
    and client regarding layout ranges and permissions.  This assumption
    leads to the implicit assumption that any callback results in a
    LAYOUTRETURN or set of LAYOUTRETURNs that exactly match the range in
    the callback, since both client and server agree about the state
    being maintained.  However, it can be useful if this assumption does
    not always hold.  For example:

    o  It may be useful for clients to be able to discard layout
       information without calling LAYOUTRETURN.  If conflicts that
       require callbacks are very rare, and a server can use a multi-file
       callback to recover per-client resources (e.g., via a FSID recall,
       or a multi-file recall within a single compound), the result may
       be significantly less client-server pNFS traffic.

    o  It may be similarly useful for servers to enhance information
       about what layout ranges are held by a client beyond what a client
       actually holds.  In the extreme, a server could manage conflicts
       on a per-file basis, only issuing whole-file callbacks even though
       clients may request and be granted sub-file ranges.

-   o  As well, the synchronized state assumption is not robust to minor
-      errors.  A more robust design would allow for divergence between
-      client and server and the ability to recover.  It is vital that a
+   o  It is vital that a
       client not assign itself layout permissions beyond what the server
       has granted and that the server not forget layout permissions that
       have been granted in order to avoid errors.  On the other hand, if
       a server believes that a client holds a layout segment that the
       client does not know about, it's useful for the client to be able
       to issue the LAYOUTRETURN that the server is expecting in response
       to a recall.

    Thus, in light of the above, it is useful for a server to be able to
    issue callbacks for layout ranges it has not granted to a client, and
    for a client to return ranges it does not hold.  A pNFS client must
    always return layout segments that comprise the full range specified
    by the recall.  Note, the full recalled layout range need not be
    returned as part of a single operation, but may be returned in
    segments.  This allows the client to stage the flushing of dirty
    data, layout commits, and returns.  Also, it indicates to the
    metadata server that the client is making progress.

    In order to ensure client/server convergence on the layout state, the
    final LAYOUTRETURN operation in a sequence of returns for a
    particular recall, MUST specify the entire range being recalled,
    echoing the recalled layout type, iomode, recall/return type (FILE, 
FSID,
    or ALL), and byte range;
    even if layout segments pertaining to partial ranges were previously
    returned.  In addition, if the client holds no layout segment that
    overlaps the range being recalled, the client should return the
    NFS4ERR_NOMATCHING_LAYOUT error code.  This allows the server to
    update its view of the client's layout state.

-16.5.3.  Recall/Return Sequencing
+16.6.  Serialization of Layout Operations

    As with other stateful operations, pNFS requires the correct
    sequencing of layout operations. pNFS relies on the sessions feature
    of NFSv4.1 to provide the correct sequencing between regular
    operations and callbacks.  It is the server's responsibility to avoid
    inconsistencies regarding the layouts it hands out and the client's
-   responsibility to properly serialize its layout requests.
+   responsibility to properly serialize its layout requests and
+   layout returns.
+
+16.6.1.  Get/Return Serialization
+
+   The protocol allows the client to send concurrent LAYOUTGET and
+   LAYOUTRETURN operations to the server.  However, the protocol does
+   not provide any means for the server to process the requests
+   in the same order in which they were created, nor it provides a way
+   for the client to determine the order in which parallel outstanding
+   operations were processed by the server. Thus, when a layout segment
+   retrieved by an outstanding LAYOUTGET operation intersects with
+   a layout segment returned by an outstanding LAYOUTRETURN the order
+   in which the two conflicting operations are processed determines
+   the final state of the overlapping segment.  To disambiguate between
+   the two cases the client MUST serialize LAYOUTGET operations and 
voluntary
+   LAYOUTRETURN operations for the same file.
+
+   It is permissible for the client to send in parallel multiple LAYOUTGET
+   operations for the same file or multiple LAYOUTRETURN operations for the
+   same file; but never a mix of both.  It is also permissible for the 
client
+   to combine LAYOUTRETURN and LAYOUTGET operations for the same file 
in the
+   same compound RPC as the server must process these in order.
+
+16.6.2.  Recall/Return Sequencing

    One critical issue with operation sequencing concerns callbacks.  The
    protocol must defend against races between the reply to a LAYOUTGET
    operation and a subsequent CB_LAYOUTRECALL.  A client MUST NOT
    process a CB_LAYOUTRECALL that identifies an outstanding LAYOUTGET
    operation to which the client has not yet received a reply.
    Conflicting LAYOUTGET operations are identified in the CB_SEQUENCE
    preceding the CB_LAYOUTRECALL.

    The callback races section (Section 11.10.3) describes the sessions
    mechanism for allowing the client to detect such situations in order
    to not process such a CB_LAYOUTRECALL.  The server MUST reference all
    conflicting LAYOUTGET operations in the CB_SEQUENCE that precedes the
    CB_LAYOUTRECALL.  A zero length array of referenced operations is
    used by the server to tell the client that the server does not know
    of any LAYOUTGET operations that conflict with the recall.

-16.5.3.1.  Client Side Considerations
+16.6.2.1.  Client Side Considerations

    Consider a pNFS client that has issued a LAYOUTGET and then receives
    an overlapping recall callback for the same file.  There are two
    possibilities, which in the absence of a session, the client cannot
    distinguish when the callback arrives:

    1.  The server processed the LAYOUTGET before issuing the recall, so
        the LAYOUTGET response is in flight, and must be waited for
        because it may be carrying layout info that will need to be
        returned to deal with the recall callback.

    2.  The server issued the callback before receiving the LAYOUTGET.
        The server will not respond to the LAYOUTGET until the recall
        callback is processed.

    This can cause deadlock, as the client must wait for the LAYOUTGET
    response before processing the recall in the first case, but that
    response will not arrive until after the recall is processed in the
    second case.  In the presence of a session, the server will provide
    the client with the { slotid , sequenceid } of any earlier LAYOUTGET
    operations which remain unconfirmed at the server by the session slot
    usage rules.  This allows the client to disambiguate between the two
    cases, in case 1, the server will provide the operation reference(s),
    whereas in case 2 it will not (because there are no dependent client
    operations).  Therefore, the action at the client will only require
-   waiting in the case that the client has not yet seen the sever's
+   waiting in the case that the client has not yet seen the server's
    earlier responses to the LAYOUTGET operation(s).

    The following requirements apply to avoid this deadlock: by adhering
    to the following requirements:

    o  A LAYOUTGET MUST be rejected with an error (i.e.,
       NFS4ERR_RECALLCONFLICT) if there's an overlapping outstanding
-      recall callback to the same client
+      recall callback to the same client.

    o  When processing a recall, the client MUST wait for a response to
       all conflicting outstanding LAYOUTGETs that are referenced in the
       CB_SEQUENCE for the recall before performing any RETURN that could
       be affected by any such response.

    o  The client SHOULD wait for responses to all operations required to
       complete a recall before sending any LAYOUTGETs that would
       conflict with the recall because the server is likely to return
       errors for them.

    o  Before sending a new LAYOUTGET for a range covered by a layout
       recall, the client SHOULD wait for responses to any outstanding
       LAYOUTGET that overlaps any portion of the new LAYOUTGET's range .
       This is because it is possible (although unlikely) that the prior
       operation may have arrived at the server after the recall
       completed and hence will succeed.

+   o  The recall process can be considered as done by the client
+      when the final LAYOUTRETURN operation for the recalled range
+      is replied.
+
-16.5.3.2.  Server Side Considerations
+16.6.2.2.  Server Side Considerations

    Consider a related situation from the pNFS server's point of view.
    The server has issued a recall callback and receives an overlapping
    LAYOUTGET for the same file before the LAYOUTRETURN(s) that respond
    to the recall callback.  Again, there are two cases:

    1.  The client issued the LAYOUTGET before processing the recall
        callback.

    2.  The client issued the LAYOUTGET after processing the recall
        callback, but it arrived before the LAYOUTRETURN that completed
        that processing.

    The simplest approach is to always reject the overlapping LAYOUTGET;
-   te pNFS server MUST do this.  The client has two ways to avoid this
+   the pNFS server MUST do this.  The client has two ways to avoid this
    result - it can issue the LAYOUTGET as a subsequent element of a
    COMPOUND containing the LAYOUTRETURN that completes the recall
    callback, or it can wait for the response to that LAYOUTRETURN.

    There is little a session can do to disambiguate between these two
    cases, because both operations are independent of one another.  They
    are simply asynchronous events which crossed.  The situation can even
    occur if the session is configured to use a single connection for
    both operations and callbacks.

-   This leads to a more general problem; in the absence of a callback if
-   a client issues concurrent overlapping LAYOUTGET and LAYOUTRETURN
-   operations, it is possible for the server to process them in either
-   order.  Again, a client must take the appropriate precautions in
-   serializing its actions.

Benny Halevy wrote:
> issue 58, but the changes are just too broad to be part of that, hence 
> a new issue.  Proposed text coming soon (before 8/8/06 :).
>
> Benny
>
> -- 
>
> out of sync with respect to the state of the layouts a particular 
> client holds and that there are mechanisms that allow that state to 
> converge back.  There is concesus that the layout state must never 
> diverge (although a "wildcard" recall and return of FSID or ALL can be 
> used to reset the state in case the state has diverged for an 
> unexpected reason, e.g. a bug)
>