Subject: Re: [nfsv4] Session-based fix for callback races

From: "Talpey, Thomas" <[email-masked]>
Date: Fri, 03 Mar 2006 10:39:27 -0500

Here's the text as applied to the pNFS section, my additions are
bracketed in >> / << pairs. I have also included my earlier text
additions which I have tweaked slightly.

The only other text to add is a referring sentence in the delegations
text to refer to the new section 3.10.3 (or whatever # it gets).
Because this text isn't in the minorversion doc yet, I'll leave it to
Spencer to insert it. But it should say "An important issue regarding
races between delegation granting and delegation recall operations
is discussed in 3.10.3".

I'll be on a plane later today but will check in over the weekend for
any comments or needed changes to this text.

Tom.

-----

7.5.3  Recall/Return Sequencing

   As with other stateful operations, pNFS requires the correct
   sequencing of layout operations.  This proposal assumes that sessions
   will precede or accompany pNFS into NFSv4.x and thus, pNFS will
   require the use of sessions.  If the sessions proposal does not
   precede pNFS, then this proposal needs to be modified to provide for
   the correct sequencing of pNFS layout operations.  Also, this
   specification is reliant on the sessions protocol to provide the
   correct sequencing between regular operations and callbacks.  It is
   the server's responsibility to avoid inconsistencies regarding the
   layouts it hands out and the client's responsibility to properly
   serialize its layout requests.

   One critical issue with operation sequencing concerns callbacks.  The
   protocol must defend against races between the reply to a LAYOUTGET
   operation and a subsequent CB_LAYOUTRECALL.  It MUST NOT be possible
   for a client to process the CB_LAYOUTRECALL for a layout that it has
   not received in a reply message to a LAYOUTGET.

   The "N.M.n Callback races" section describes the sessions mechanism
   for allowing the client to detect such situations in order to not
   process such a CB_LAYOUTRECALL. The LAYOUTGET operation is in this
   case the dependent operation which the server should reference in
   any layout recall, if it remains active in the server's slot table.
<<

7.5.3.1  Client Side Considerations

   Consider a pNFS client that has issued a LAYOUTGET and then receives
   an overlapping recall callback for the same file.  There are two
   possibilities, which
   in the absence of a session,
<<
   the client cannot distinguish when the callback
   arrives:

   1.  The server processed the LAYOUTGET before issuing the recall, so
       the LAYOUTGET response is in flight, and must be waited for
       because it may be carrying layout info that will need to be
       returned to deal with the recall callback.

   2.  The server issued the callback before receiving the LAYOUTGET.
       The server will not respond to the LAYOUTGET until the recall
       callback is processed.

   This can cause deadlock, as the client must wait for the LAYOUTGET
   response before processing the recall in the first case, but that
   response will not arrive until after the recall is processed in the
   second case.  
   In the presence of a session, the server will provide the client
   with the { slotid , sequenceid } of any earlier LAYOUTGET which
   remains unconfirmed at the server by the session slot usage rules.
   This allows the client to disambiguate between the two cases, in
   case 1, the server will provide the reference, whereas in case 2
   it will not (because there is no dependent client operation).
   Therefore, the action at the client will only require waiting
   in the case that the client has not yet seen the sever's earlier
   reply to the LAYOUTGET.

   Without the session, 
<<
   This deadlock can be avoided by adhering to the
   following requirements:

   o  A LAYOUTGET MUST be rejected with an error (i.e.,
      NFS4ERR_RECALLCONFLICT) if there's an overlapping outstanding
      recall callback to the same client

   o  When processing a recall, the client MUST wait for a response to
      all conflicting outstanding LAYOUTGETs before performing any
      RETURN that could be affected by any such response.

   o  The client SHOULD wait for responses to all operations required to
      complete a recall before sending any LAYOUTGETs that would
      conflict with the recall because the server is likely to return
      errors for them.

   Now the client can wait for the LAYOUTGET response, as it will be
   received in both cases.

7.5.3.2  Server Side Considerations

   Consider a related situation from the pNFS server's point of view.
   The server has issued a recall callback and receives an overlapping
   LAYOUTGET for the same file before the LAYOUTRETURN(s) that respond
   to the recall callback.  Again, there are two cases:

   1.  The client issued the LAYOUTGET before processing the recall
       callback.

   2.  The client issued the LAYOUTGET after processing the recall
       callback, but it arrived before the LAYOUTRETURN that completed
       that processing.

   The simplest approach is to always reject the overlapping LAYOUTGET.
   The client has two ways to avoid this result - it can issue the
   LAYOUTGET as a subsequent element of a COMPOUND containing the
   LAYOUTRETURN that completes the recall callback, or it can wait for
   the response to that LAYOUTRETURN.
   There is little a session can do to disambiguate between these two
   cases, because both operations are independent of one another. They
   are simply asynchronous events which crossed. The situation can even
   occur if the session is configured to use a single connection for
   both operations and callbacks.
<<

   This leads to a more general problem; in the absence of a callback if
   a client issues concurrent overlapping LAYOUTGET and LAYOUTRETURN
   operations, it is possible for the server to process them in either
   order.  Again, a client must take the appropriate precautions in
   serializing its actions.

   [ASIDE: HighRoad forbids a client from doing this, as the per-file
   layout stateid will cause one of the two operations to be rejected
   with a stale layout stateid.  This approach is simpler and produces
   better results by comparison to allowing concurrent operations, at
   least for this sort of conflict case, because server execution of
   operations in an order not anticipated by the client may produce
   results that are not useful to the client (e.g., if a LAYOUTRETURN is
   followed by a concurrent overlapping LAYOUTGET, but executed in the
   other order, the client will not retain layout extents for the
   overlapping range).]

>>> Here's the related text I proposed adding previously, I've made
>>> some minor tuneups.

3.10.2 Slot Identifiers and Server Duplicate Request Cache  . . .

<please see existing section>

3.10.3 Resolving server callback races with sessions [new]

It is possible for server callbacks to arrive at the client before the
reply from related forward channel operations. For example, a client may
have been granted a delegation to a file it has opened, but the reply
to the OPEN (informing the client of the granting of the delegation) may
be delayed in the network. If a conflicting operation arrives at the
server, it will recall the delegation using the callback channel, which
may be on a different TCP connection, perhaps even a different network.
If the callback request arrives before the related reply, the client may
reply to the server with an error.

The presence of a session between client and server can be used to
alleviate this issue. When a session is in place, each client request
is uniquely identified by its { slotid, sequenceid } pair. By the rules
under which slot entries (duplicate request cache entries) are retired,
the server has knowledge whether the client has "seen" each of the server's
replies. The server can therefore provide sufficient information to the
client to allow it to disambiguate between an erroneous or conflicting
callback and a race condition.

To implement this, the CB_SEQUENCE operation which begins each server
callback may optionally carry a related { slotid, sequenceid } identifier.
If the client finds this identifier to be currently outstanding (the
server's reply has not been seen by the client), it can determine that
the callback has raced the reply, and act accordingly.

The client must not simply wait forever for the expected server reply
to arrive any of the session's operations channels, because it is possible
that they will be delayed indefinitely. However, it should endeavor to wait
for a period of time, and if the time expires it can provide a more
meaningful error such as NFS4ERR_DELAY.

[We need to consider the clients' options here, and describe them...
 NFS4ERR_DELAY has been discussed as a legal reply to CB_RECALL? ]

There are other scenarios under which callbacks may race replies, among
them pnfs layout recalls, described in 7.5.3, [fill in the blanks w/others,
etc...]

Therefore, for each client operation which might result in some sort of
server callback, the server should "remember" the { slotid, sequenceid }
pair of the client request until the slotid retirement rules allow the 

server to determine that the client has, in fact, seen the server's
reply. During this time, any recalls of the associated object should
carry these identifiers, for the benefit of the client. After this time,
it is not necessary for the server to provide this information in
related callbacks, since it is certain that a race condition can no
longer occur.

-----

14.10.  Callback operation: CB_SEQUENCE - Supply callback channel
sequencing and control

     SYNOPSIS

          control -> control

     ARGUMENT

          typedef uint32_t sequenceid4;
          typedef uint32_t slotid4;

          struct CB_SEQUENCE4args {
               sessionid4     sessionid;
               sequenceid4    sequenceid;
               slotid4        slotid;
               slotid4        maxslot;
>>             sequenceid4    referring_sequenceid;
>>             slotid4        referring_slotid;
          };

     RESULT

          struct CB_SEQUENCE4resok {
               sessionid4     sessionid;
               sequenceid4    sequenceid;
               slotid4        slotid;
               slotid4        maxslot;
               slotid4        target_maxslot;
          };

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4