Subject: [nfsv4] Comments on text for issues 103 and 108

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 22 Aug 2006 17:07:06 -0400

Nothing major on these.
 
Issue 103:
 
16.5.2 para 1: "It is assumed" --> "It has been assumed thus far"
 
16.5.2 para 3: "enhance information about what ranges are held by a
client beyond what a client actually holds"  --> "maintain information
about what ranges are held by a client on a coarse-grained basis,
leading to the server's layout ranges being beyond those actually held
by the client."
 
16.5.2 para 4: Add "In order to avoid errors, " to the front of the
first sentence and delete that phrase from the end fo the first
sentence.
 
16.5.2 para 6: The sentence about NFS4ERR_NOMATCHING_LAYOUT seems to be
at variance with what is said in para 4, which seems to suggest that the
client should instead issue a LAYOUTRETURN in this situation.  Perhaps
change "to issue the LAYOUTRETURN that the server is expecting in
response to a recall" to "to cleanly indicate completion of the
requested recall either by issuing a LAYOUITRETURN for the entire
requested range to by returning an NFS4ERR_NOMATCHING_LAYOUT to the
layout recall callback."
 
16.6.1 para 2: "compound RPC" --> "COMPOUND request".
 
16.6.1 para 2:  Add at the end: "If a client does issue such requests,
it must not have more than one outstanding for the same file at the same
time and must not have other LAYOUTGET or LAYOUTREURN oeprations
outstanding at the same time for that same file."
 
16.6.2.1 para 1: "which in the absence of a session, the client cannot
distinguish when the callback arrives:" --> "which the client would be
unable to distinguish without additional information provided by the
sessions implementation".
 
16.6.2.1 para 4: "can cause" --> "could cause".
 
16.6.2.1 para 4: "In the prsence of a session" --> "Via the a
CB_SEQUENCE operation"
 
16.6.2.1 para 4: "will provide" --> "provides"
 
16.6.2.1 last para: "is replied" --> "is issued"
 
16.6.2.2 para 4: "result - it"  --> 'result.  It"
 
16.6.2.2 para 5: "a session" ---> "session sequence logic"
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Wed, 23 Aug 2006 19:14:02 +0300

Thanks Dave!

Please find in the attached the corrected text (as diff's from the xml 
files)

Benny

Noveck, Dave wrote:
> first sentence and delete that phrase from the end fo the first sentence.
>  
> 16.5.2 para 6: The sentence about NFS4ERR_NOMATCHING_LAYOUT seems to 
> be at variance with what is said in para 4, which seems to suggest 
> that the client should instead issue a LAYOUTRETURN in this 
> situation.  Perhaps change "to issue the LAYOUTRETURN that the server 
> is expecting in response to a recall" to "to cleanly indicate 
> completion of the requested recall either by issuing a LAYOUITRETURN 
> for the entire requested range to by returning an 
> NFS4ERR_NOMATCHING_LAYOUT to the layout recall callback."
>  
> it must not have more than one outstanding for the same file at the 
> same time and must not have other LAYOUTGET or LAYOUTREURN oeprations 
> ------------------------------------------------------------------------
>

--- nfsv41-05_middle_pnfs.orig.xml	2006-08-23 17:24:47.000000000 +0300
+++ nfsv41-05_middle_pnfs.xml	2006-08-23 19:00:10.000000000 +0300
@@ -743,7 +743,7 @@
 
 <section title="Recall Callback Robustness" anchor="recall_robustness">
 <t>
-  For simplicity, the discussion thus far has assumed that pNFS client
+  It has been assumed thus far that pNFS client
   state for a file exactly matches the pNFS server state for that file
   and client regarding layout ranges and permissions.  This assumption
   leads to the implicit assumption that any callback results in a
@@ -763,23 +763,24 @@
   significantly less client-server pNFS traffic.
 </t>
 <t>
-  It may be similarly useful for servers to enhance information about
-  what layout ranges are held by a client beyond what a client
-  actually holds.  In the extreme, a server could manage conflicts on
+  It may be similarly useful for servers to maintain information about
+  what ranges are held by a client on a coarse-grained basis, leading
+  to the server's layout ranges being beyond those actually held by
+  the client.
+  In the extreme, a server could manage conflicts on
   a per-file basis, only issuing whole-file callbacks even though
   clients may request and be granted sub-file ranges.
 </t>
 <t>
-  As well, the synchronized state assumption is not robust to minor
-  errors.  A more robust design would allow for divergence between
-  client and server and the ability to recover.  It is vital that a
-  client not assign itself layout permissions beyond what the server
-  has granted and that the server not forget layout permissions that
-  have been granted in order to avoid errors.  On the other hand, if a
+  In order to avoid errors, it is vital that a client not assign
+  itself layout permissions beyond what the server has granted and
+  that the server not forget layout permissions that have been granted.
+  On the other hand, if a
   server believes that a client holds a layout segment that the client
-  does not know about, it's useful for the client to be able to issue
-  the LAYOUTRETURN that the server is expecting in response to a
-  recall.
+  does not know about, it's useful for the client to cleanly indicate
+  completion of the requested recall either by issuing a LAYOUTRETURN
+  for the entire requested range or by returning an
+  NFS4ERR_NOMATCHING_LAYOUT error to the layout recall callback.
 </t>
 </list>
 </t>
@@ -826,16 +827,47 @@
 </t>
 </section>
 
-
-<section title="Recall/Return Sequencing" anchor="pnfs_operation_sequencing">
+<section title="Serialization of Layout Operations" anchor="pnfs_operation_sequencing">
 <t>
   As with other stateful operations, pNFS requires the correct
   sequencing of layout operations.  pNFS relies on the sessions
   feature of NFSv4.1 to provide the correct sequencing between regular
   operations and callbacks.  It is the server's responsibility to
   avoid inconsistencies regarding the layouts it hands out and the
-  client's responsibility to properly serialize its layout requests.
+  client's responsibility to properly serialize its layout requests
+  and layout returns.
+</t>
+
+<section title="Get/Return Serialization">
+<t>
+  The protocol allows the client to send concurrent LAYOUTGET and
+  LAYOUTRETURN operations to the server. However, the protocol does
+  not provide any means for the server to process the requests
+  in the same order in which they were created, nor it provides a way
+  for the client to determine the order in which parallel outstanding
+  operations were processed by the server. Thus, when a layout segment
+  retrieved by an outstanding LAYOUTGET operation intersects with
+  a layout segment returned by an outstanding LAYOUTRETURN the order
+  in which the two conflicting operations are processed determines
+  the final state of the overlapping segment. To disambiguate between
+  the two cases the client MUST serialize LAYOUTGET operations and
+  voluntary LAYOUTRETURN operations for the same file.
+</t>
+<t>
+  It is permissible for the client to send in parallel multiple
+  LAYOUTGET operations for the same file or multiple LAYOUTRETURN
+  operations for the same file; but never a mix of both. It is also
+  permissible for the client to combine LAYOUTRETURN and LAYOUTGET
+  operations for the same file in the same COMPOUND request as the
+  server must process these in order.
+  If a client does issue such requests, it must not have more than one
+  outstanding for the same file at the same time and must not have
+  other LAYOUTGET or LAYOUTRETURN operations outstanding at the same
+  time for that same file.
 </t>
+</section>
+
+<section title="Recall/Return Sequencing">
 <t>
   One critical issue with operation sequencing concerns callbacks.
   The protocol must defend against races between the reply to a
@@ -859,8 +891,9 @@
 <t>
   Consider a pNFS client that has issued a LAYOUTGET and then receives
   an overlapping recall callback for the same file.  There are two
-  possibilities, which in the absence of a session, the client cannot
-  distinguish when the callback arrives:
+  possibilities, which the client would be unable to distinguish
+  without additional information provided by the sessions
+  implementation.
 <list style="numbers">
 <t>
   The server processed the LAYOUTGET before issuing the recall, so the
@@ -874,10 +907,10 @@
   is processed.
 </t>
 </list>
-  This can cause deadlock, as the client must wait for the LAYOUTGET
+  This could cause deadlock, as the client must wait for the LAYOUTGET
   response before processing the recall in the first case, but that
   response will not arrive until after the recall is processed in the
-  second case.  In the presence of a session, the server will provide
+  second case.  Via the CB_SEQUENCE operation, the server provides
   the client with the { slotid , sequenceid } of any earlier LAYOUTGET
   operations which remain unconfirmed at the server by the session
   slot usage rules.  This allows the client to disambiguate between
@@ -885,7 +918,7 @@
   reference(s), whereas in case 2 it will not (because there are no
   dependent client operations).  Therefore, the action at the client
   will only require waiting in the case that the client has not yet
-  seen the sever's earlier responses to the LAYOUTGET operation(s).
+  seen the server's earlier responses to the LAYOUTGET operation(s).
 
   <vspace blankLines='1' /> 
 
@@ -896,7 +929,7 @@
   <t>
     A LAYOUTGET MUST be rejected with an error (i.e.,
     NFS4ERR_RECALLCONFLICT) if there's an overlapping outstanding recall
-    callback to the same client
+    callback to the same client.
   </t>
   <t>
     When processing a recall, the client MUST wait for a response to
@@ -918,6 +951,10 @@
     operation may have arrived at the server after the recall
     completed and hence will succeed.
   </t>
+    The recall process can be considered as done by the client when
+    the final LAYOUTRETURN operation for the recalled range is issued.
+  <t>
+  </t>
   </list>
 </t>
 </section>
@@ -942,25 +979,19 @@
 </t>
 <t>
   The simplest approach is to always reject the overlapping LAYOUTGET;
-  te pNFS server MUST do this.  The client has two ways to avoid this
-  result - it can issue the LAYOUTGET as a subsequent element of a
+  the pNFS server MUST do this.  The client has two ways to avoid this
+  result.  It can issue the LAYOUTGET as a subsequent element of a
   COMPOUND containing the LAYOUTRETURN that completes the recall
   callback, or it can wait for the response to that LAYOUTRETURN.
 </t>
 <t>
-   There is little a session can do to disambiguate between these two
+   There is little session sequence logic can do to disambiguate between these two
    cases, because both operations are independent of one another. They
    are simply asynchronous events which crossed. The situation can even
    occur if the session is configured to use a single connection for
    both operations and callbacks.
 </t>
-<t>
-  This leads to a more general problem; in the absence of a callback
-  if a client issues concurrent overlapping LAYOUTGET and LAYOUTRETURN
-  operations, it is possible for the server to process them in either
-  order.  Again, a client must take the appropriate precautions in
-  serializing its actions.
-</t>
+</section>
 </section>
 </section>
 </section>
--- nfsv41-05_middle_op_layoutreturn.orig.xml	2006-08-23 19:06:33.000000000 +0300
+++ nfsv41-05_middle_op_layoutreturn.xml	2006-08-23 19:08:49.000000000 +0300
@@ -128,6 +128,16 @@
   </section>
   <section toc="exclude" anchor="OP_LAYOUTRETURN_IMPLEMENTATION" title="IMPLEMENTATION">
   <t>
+    The final LAYOUTRETURN operation in response to a CB_LAYOUTRECALL
+    callback MUST be serialized with any outstanding, intersecting
+    layoutreturn operations.  Note that it is possible that while a
+    client is returning the layout for some recalled range the server
+    may recall a superset of that range (e.g. RECALL_ALL); the final
+    return operation for the latter must block until the former layout
+    recall is done - when its corresponding final return operation is
+    replied.
+  </t>
+  <t>
     Returning all layouts in a file system using LAYOUTRETURN_FSID is
     typically done in response to a CB_LAYOUTRECALL for that file
     system as the final return operation. Similarly, LAYOUTRETURN_ALL
@@ -136,15 +146,7 @@
     via individual LAYOUTRETURN calls and the call for
     LAYOUTRETURN_FSID or LAYOUTRETURN_ALL marks the end of the
     layoutreturn sequence.  See <xref target="recall_robustness" />
-    for more details.  Returning all layouts in a file system using
-    LAYOUTRETURN_FSID is typically done in response to a
-    CB_LAYOUTRECALL for that file system as the final return
-    operation. Similarly, LAYOUTRETURN_ALL is used in response to a
-    recall callback for all layouts.  It is possible that the client
-    already returned some outstanding layouts via individual
-    LAYOUTRETURN calls and the call for LAYOUTRETURN_FSID or
-    LAYOUTRETURN_ALL marks the end of the layoutreturn sequence.  See
-    <xref target="recall_robustness" /> for more details.
+    for more details.
   </t>
   </section>
 </section>
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4