Subject: [nfsv4] pnfs: keeping the client and server layouts in sync

From: Benny Halevy <[email-masked]>
Date: Thu, 22 Jun 2006 19:54:52 +0300

While working on Issue 58 I became quite concerned about keeping the client and
server layout state in sync with the current layout get/recall/return protocol
when there are concurrent conflicting get and return or recall calls in flight
even if we use sessions to order the calls.  My concern arose from the fuzziness
of LAYOUTRETURN that allows "a subset or superset of a layout specified by
CB_LAYOUTRECALL" and "It is also permissible, and no error should result, for a
client to return a byte range covering a layout it does not hold".

When a layoutget is in flight the state of holding a layout segment transfers
from the server to the client and until the client receives the response for the
layoutget it doesn't really know if it already holds the layout or not.
Moreover, since the server can return a layout segment outside the segment the
client asked for, even if the outstanding requested range and the
about-to-be-sent layoutreturn do not intersect, there might be a potential
conflict on the "extended" range that the server is about to give to the client.
  For example:
1. client holds a layout for offset 0, length 3
2. client returns offset 0, length 2
3. client sends a layoutget for offset 0, length 1
4. client sends a layoutreturn for offset 1, length 2
5. server replies (3) for offset 0, length 2

note the conflict on the byte at offset 1.  At stages 3 and 4 the client does
not know it is going to be the proud owner of the layout segment at byte 1 since
it didn't get (5) in time. When (4) is received the server will think that the
client holds a layout for <offset=0, length=1> and the client will think, after
receiving (5) that it holds a layout for <offset=1, length=2>. Oops.

This becomes more obvious if the client does a return for RETURN_FSID (as
proposed in Issue 58) in which conflicts are more abundant.  Unfortunately, not
allowing RETURN_FSID helps only to hide problems by allowing the client to
remain out of sync with the server without that being noticed.  Also the text
for "Recalling a layout" (Section 14.5.1) is challenging for a CB_LAYOUTRECALL
for RECALL_FSID

    In order to ensure client/server convergence on the layout state, the final
    LAYOUTRETURN operation in a sequence of returns for a particular recall,
    SHOULD specify the entire range being recalled, even if layout segments
    pertaining to partial ranges were previously returned.

This adds to LAYOUTRETURN semantics (Section 22.48)

    However, if it is a subset, the recall is not complete until the full
    byte range has been returned.

My interpretation is that there's a hidden assumption that there's a recall
meta-operation that starts when a CB_LAYOUTRECALL is issued and is complete when
all layout segments covered by the layout recall scope are returned.  Since we
don't mandate an explicit return operation that matches the respective recall
operation there's a potential for the client and server to go or remain out of
sync here.

I think that to allow the server to clean up the client state in a sensible way
we need to require such a final response and add some other restrictions on the
client as described below so that the state of the client's layout is safely
synchronized between the client and the server when the client receives the
reply for the final return it sent in response to a CB_LAYOUTRECALL.

1) The server sends CB_LAYOUTRECALL for the layouts it wants back
    (RECALL_FILE, RECALL_FSID, or RECALL_ALL).

    1.1) This triggers a recall meta-operation at the client in which it will
         return all layouts matching the CB_LAYOUTRECALL parameters.

    1.2) The meta-operation completes when either CB_LAYOUTRECALL is replied with
         an error, or when the final LAYOUTRETURN reply is received by the
         client.

    1.2) If at any point during the meta-operation the server restarts, then
         regular server recovery applies and the client implicitly loses all its
         layouts.

2) While the recall meta-operation is in progress the client MUST NOT issue
    any possibly conflicting LAYOUTGET operation.

3) The client MUST wait for any outstanding LAYOUTGET that possibly conflicts
    with the recalled layout to complete before returning any layout.

    3.1) Wait time can be bound to the server lease time as the layouts will be
         revoked by the server if CB_LAYOUTRECALL is not replied by then.

4) Possible conflicts are determined by matching clientid, layout_type, iomode,
    and depending on the recall scope (RECALL_{FILE,FSID,ALL}) outstanding
    LAYOUTGETs for the respective {FILE,FSID,ALL} are possibly conflicting.
    Note that since the server can return a layout segment which is outside
    the requested byte range in LAYOUTGET, it may conflict with the
    CB_LAYOUTRECALL even if the recalled byte range does not overlap with the
    byte range requested in LAYOUTGET, hence the conflict is assumed on any
    LAYOUTGET to the same file, with a conflicting layout_type, iomode and
    clientid.

5) At this point, if the client finds no matching layout it returns a
    NFS4ERR_NOMATCHING_LAYOUT error to CB_LAYOUTRECALL.

6) Otherwise, the client starts returning the matching layouts.
    6.1) It can do this in a single or multiple LAYOUTRETURN operations as long
         as the final LAYOUTRETURN call echoes exactly the same parameters of the
         respective CB_LAYOUTRECALL.

    6.1) To make sure that the final LAYOUTRETURN is received last, the client
         SHOULD wait for any outstanding LAYOUTRETURNs to be replied (if it sent
         any) before sending the final one.  With sessions, if all respective
         LAYOUTRETURN operations are serialized on the same slot, the final
         LAYOUTRETURN will naturally happen last.

7) The server should rely only on the LAYOUTRETURNs it receives from the client
    so it does not have to maintain any state that is associated with a
    CB_LAYOUTRECALL it sent to a client.
    7.1) If the server wants to sync with the client layout state it can send a
         "wildcard" CB_LAYOUTRECALL (whole file, fsid, or all layouts)
         and wait until the final LAYOUTRETURN is received from the client. At
         this point the server and client states (for the given clientid,
         layout_type, iomode, and scope) can be safely reset.

I believe that doing the above will save us from relying on sessions to resolve 
recall/get races.

Benny

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Thu, 22 Jun 2006 20:28:41 +0300

> receiving (5) that it holds a layout for <offset=1, length=2>. Oops.

that should be <offset=0, length=2>, sorry.

>         revoked by the server if CB_LAYOUTRECALL is not replied by then.

One point that I had in an earlier version of this proposal, that I realize is 
still needed is:

      3.2) To prevent a deadlock at this state, after sending a CB_LAYOUTRECALL
           the server should reject any conflicting LAYOUTGET from that client
           with NFS4ERR_LAYOUTRECALL_IN_PROGRESS.  This way if the server
           replied to a LAYOUTGET before sending CB_LAYOUTRECALL the client
           should typically get the reply before or shortly after receiving the
           CB_LAYOUTRECALL.  If the server received the LAYOUTGET after sending
           the CB_LAYOUTRECALL then the client will typically receive the error
           shortly after receiving the CB_LAYOUTRECALL.
> I believe that doing the above will save us from relying on sessions to 
> resolve recall/get races.
> 
> Benny
> 
>