Subject: [nfsv4] Draft 21: parallel LAYOUTGET operations and the layout stateid

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Fri, 29 Feb 2008 16:38:48 -0500

I brought this up with Spencer at the Linux Storage and File System
Workshop. There is no description of how parallel LAYOUTGET operations are
processed past the parallel use of a non-layout stateid (when no layout
stateid has been established).

12.5.3.  Layout Stateid

   As with all other stateids, the stateid consists of a "seqid" and
   "other" field.  Once a layout stateid is changed, the "other" field
   will stay constant unless the stateid is revoked, or the client
   returns all layouts on the file and the server disposes of the
   stateid.  The "seqid" field is initially set to one, and is never
   zero on any NFSv4.1 operation that uses layout stateids, whether it
   is fore channel or backchannel operation.  After the layout stateid
   is established, the "seqid" is incremented by the server in each
   subsequent LAYOUTGET and LAYOUTRETURN response, and in each
   CB_LAYOUTRECALL request.  When the client fully processes the
   response to a LAYOUTGET or LAYOUTRETURN, or fully processes the
   arguments of a CB_LAYOUTRECALL, it MUST use the seqid of the stateid
   of the reply from LAYOUTGET and LAYOUTRETURN, or the seqid of the
   stateid in the arguments of CB_LAYOUTRECALL, on subsequent calls to
   LAYOUTGET or LAYOUTRETURN.

The last sentence above allows for the client layout stateid to decrease -
if parallel LAYOUTGET operations are sent, the order of the returns could be
such that a lower stateid.seqid arrives from the server after a higher
stateid.seqid. Not good.

                                                              The client and
server use the "seqid" of
   the layout stateid for the following purposes:

   o  Permit the client to send parallel LAYOUTGET operations on the
      same file.  As with parallel opens (see Section 9.10) the use of
      the sequence ID allows a client to avoid serializing LAYOUTGET
       ^^^^^^^^
the zero sequence ID

      operations.  If LAYOUTGETs were serialized, especially non-
      overlapping LAYOUTGETs, then non-overlapping I/Os to storage
      devices would in turn be effectively serialized with each other.
      In the event parallel LAYOUTGET operations are sent with a non-
      layout stateid (because the client does not yet have a layout
      stateid), the successful responses MUST have the same "other"
      field in the LAYOUTSTATEID, and each response with a unique
      "seqid", where the lowest "seqid" is one, and the highest "seqid"
      is equal to the count of parallel LAYOUTGET operations invoked on
      the non-layout stateid.

What about after the layout stateid is established? Can the client still
send LAYOUTGET operations in parallel? If so, what are the rules - does the
client keep higher stateid.seqids and ignore lower stateid.seqids? The
server will receive multiple LAYOUTGET operations with the same layout
stateid.seqid. Is this permitted?

-->Andy
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 29 Feb 2008 17:04:11 -0500

I don't understand this.  There are all these references to parallel
opens as a model but parallel opens handle this by using zero seqid as a
wildcard but the layout stuff explicitly rules out zero being sent as a
wildcard seqid.  It isn't clear to me why this is ruled out however.  

	-----Original Message-----
	From: William A. (Andy) Adamson [mailto:[email-masked]] 
	Sent: Friday, February 29, 2008 4:39 PM
	To: NFSv4
	Subject: [nfsv4] Draft 21: parallel LAYOUTGET operations and the
layoutstateid

	I brought this up with Spencer at the Linux Storage and File
System Workshop. There is no description of how parallel LAYOUTGET
operations are processed past the parallel use of a non-layout stateid
(when no layout stateid has been established).
	
	12.5.3.  Layout Stateid
	
	   As with all other stateids, the stateid consists of a "seqid"
and
	   "other" field.  Once a layout stateid is changed, the "other"
field
	   will stay constant unless the stateid is revoked, or the
client
	   returns all layouts on the file and the server disposes of
the
	   stateid.  The "seqid" field is initially set to one, and is
never
	   zero on any NFSv4.1 operation that uses layout stateids,
whether it
	   is fore channel or backchannel operation.  After the layout
stateid
	   is established, the "seqid" is incremented by the server in
each
	   subsequent LAYOUTGET and LAYOUTRETURN response, and in each
	   CB_LAYOUTRECALL request.  When the client fully processes the
	   response to a LAYOUTGET or LAYOUTRETURN, or fully processes
the
	   arguments of a CB_LAYOUTRECALL, it MUST use the seqid of the
stateid
	   of the reply from LAYOUTGET and LAYOUTRETURN, or the seqid of
the
	   stateid in the arguments of CB_LAYOUTRECALL, on subsequent
calls to
	   LAYOUTGET or LAYOUTRETURN.  
	
	The last sentence above allows for the client layout stateid to
decrease - if parallel LAYOUTGET operations are sent, the order of the
returns could be such that a lower stateid.seqid arrives from the server
after a higher stateid.seqid. Not good.

The client and server use the "seqid" of
	   the layout stateid for the following purposes:
	
	   o  Permit the client to send parallel LAYOUTGET operations on
the
	      same file.  As with parallel opens (see Section 9.10) the
use of
	      the sequence ID allows a client to avoid serializing
LAYOUTGET
	       ^^^^^^^^
	the zero sequence ID
	
	      operations.  If LAYOUTGETs were serialized, especially
non-
	      overlapping LAYOUTGETs, then non-overlapping I/Os to
storage
	      devices would in turn be effectively serialized with each
other.
	      In the event parallel LAYOUTGET operations are sent with a
non-
	      layout stateid (because the client does not yet have a
layout
	      stateid), the successful responses MUST have the same
"other"
	      field in the LAYOUTSTATEID, and each response with a
unique
	      "seqid", where the lowest "seqid" is one, and the highest
"seqid"
	      is equal to the count of parallel LAYOUTGET operations
invoked on
	      the non-layout stateid.
	
	What about after the layout stateid is established? Can the
client still send LAYOUTGET operations in parallel? If so, what are the
rules - does the client keep higher stateid.seqids and ignore lower
stateid.seqids? The server will receive multiple LAYOUTGET operations
with the same layout stateid.seqid. Is this permitted?

	-->Andy

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 29 Feb 2008 18:54:24 -0600

On Feb 29, 2008, at 4:04 PM, Noveck, Dave wrote:

> I don't understand this.  There are all these references to  
> parallel opens as a model but parallel opens handle this by using  
> zero seqid as a wildcard but the layout stuff explicitly rules out  
> zero being sent as a wildcard seqid.  It isn't clear to me why this  
> is ruled out however.

I worked through the scenarios that make the wording in
this section plausible (even the non-zero requirement) but
have not had time to get it written down.  Will do so as
soon as I get the time.

Spencer

> Workshop. There is no description of how parallel LAYOUTGET  
> operations are processed past the parallel use of a non-layout  
> stateid (when no layout stateid has been established).
>
> 12.5.3.  Layout Stateid
>
> decrease - if parallel LAYOUTGET operations are sent, the order of  
> the returns could be such that a lower stateid.seqid arrives from  
> the server after a higher stateid.seqid. Not good.
>
>
>                                                               The  
> What about after the layout stateid is established? Can the client  
> still send LAYOUTGET operations in parallel? If so, what are the  
> stateid.seqids? The server will receive multiple LAYOUTGET  
> operations with the same layout stateid.seqid. Is this permitted?
>
>
> -->Andy

---

From: Mike Eisler <[email-masked]>
Date: Fri, 29 Feb 2008 18:26:31 -0800 (PST)

> > zero seqid as a wildcard but the layout stuff explicitly rules out

???

OPEN does not take a stateid as an argument. It takes a seqid as an
argument, but the NFSv4.1 server MUST ignore it.

Here is the text parallel opens:

  Unlike the case of NFSv4.0, in which OPEN operations for the same
open-owner are inherently serialized because of the owner-based seqid,
multiple OPENs for the same open-owner may be done in parallel. When
clients do this, they may encounter situations in which, because of the
existence of hard links, two OPEN operations may turn out to open the
same file, with a later OPEN performed being an upgrade of the first,
with this fact only visible to the client once the operations complete.

   In this situation, clients may determine the order in which the
OPENs were performed by examining the stateids returned by the OPENs.
Stateids that share a common value of the "other" field can be
recognized as having opened the same file, with the order of the
operations determinable from the order of the "seqid" fields, mod any
possible wraparound of the 32-bit field.

   When the possibility exists that the client will send multiple OPENs
for the same open-owner in parallel, it may be the case that an open
upgrade may happen without the client knowing beforehand that this
could happen. Because of this possibility, CLOSEs and OPEN_DOWNGRADEs,
should generally be sent with a non-zero seqid in the stateid, to avoid
the possibility that the status change associated with an open upgrade
is not inadvertently lost. 

> > zero being sent as a wildcard seqid.  It isn't clear to me why this

Wildcard seqids are not part of the plan for detecting the issue
discussed in section 9.10. Non-zero seqids are there to detect order.
That's what seqids do for layouts too. What is there now in draft-21 is
for dealing with parallel LAYOUTGETs sent after the first 
LAYOUTGET. 

> have not had time to get it written down.  Will do so as

Thanks. Do find the time soon, like before March 7.

> > stateid (when no layout stateid has been established).

I never considered that this was a requirement to support, since
unlike parallel OPENs, LAYOUTGET only takes a filehandle, and so
a client ought to be able to serialize the first LAYOUTGET.
We are getting in true crunch time on the specification ...
so pray for a miracle from St. Spencer.

> > the server after a higher stateid.seqid. Not good.

It isn't obvious to me why this is bad. However, assuming
we allowed parallel LAYOUTGETs before a layout stateid was established,
what would happen is that one LAYOUTGET would have a seqid of 2, and
the
other would be 1. This would unambiguously tell the client what
happenned. So I think it is good.

> >       the sequence ID allows a client to avoid serializing

Again I am confused. No my intent was not to write "zero sequence ID".
(however, the use of the term "sequence ID" is very bad form here, 
because it could be confused with a "sequence ID" on a slot. I
fixed that just now).

> > still send LAYOUTGET operations in parallel? If so, what are the 

Yes, that's the whole point of layout stateid.
 
> > stateid.seqids? The server will receive multiple LAYOUTGET  

``The client MUST always use a "seqid" that was returned by the server
in a reply to a LAYOUTGET or LAYOUTRETURN operation, or sent by the
server in a CB_LAYOUTRECALL operation. It MUST only use such a seqid
after processing the LAYOUTGET and LAYOUTRETURN results, or the
CB_LAYOUTRECALL request. Simply seeing the result or the
CB_LAYOUTRECALL request is not sufficient cause to use the seqid. ''

> > operations with the same layout stateid.seqid. Is this permitted?

Not just permitted, but REQUIRED in some cases.

BTW, the thread "layout races" that we had in December of 2007
went over all this (except for parallel LAYOUTGETs when there is
no layout stateid). I see that only three people participated:

    * Benny Halevy 
    * Mike Eisler 
    * [email-masked]