Subject: Re: [nfsv4] layout stateid handling/processing updates

From: Spencer Shepler <[email-masked]>
Date: Wed, 02 Apr 2008 09:48:40 -0500

On Mar 31, 2008, at 7:24 PM, Mike Eisler wrote:
>>
>> On Mar 28, 2008, at 9:43 AM, Noveck, Dave wrote:
>>>> I also did not specify a valid range for
>>>> BAD_STATEID usage because of the various caveats that we have
>>>> identified.
>
>>> BAD_STATEID would result.
>>>
>>> Clearly any fixed number like 50 is not going to work and if I
>>> suggested
>>> adding a new attribute to specify that value at this point, I'd  
>>> worry
>>> about Mike Eisler being rather upset at me.  So this is where the
>
> Indeed Dave should worry if he goes that route. :-)
>
>>> number
>>> of slots on all sessions for the client comes in.  It is not an
>>> automatically enforced limit, since, as Jason has pointed out,  
>>> you can
>>> issue multiple layou operations in a single COMPOUND.  But it would
>>> be a
>>> value that would naturally scale with the amount of parallelism that
>>> people wanted and so if you used this value (or twice this value or
>>> half
>>> of this value) and made it obligatory on the client to stay  
>>> within it,
>>> then we could establish a valid range that the server could enforce,
>>> while still allowing as much LAYOUTGET parallelism as any reasonable
>>> client could want.
>
> Let's pursue this notion a bit more though. As Jason points out,
> technically it is the summation of i=1..N of  ca_maxrequests(i) *
> (ca_maxoperations(i) - 1) where N is the number of forechannels
> (the - 1 is because SEQUENCE uses a operation in a COMPOUND).
> That however might be too large a number. Or it might not. It  
> depends on
> the server (keeping in mind that this is a global limit per client
> ID, not per client ID/fh pair). So the range of parallelism is
> thus between summation of i=1..N ca_maxoperations(i) - 1,
> and summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)
>
> It is fair game for the client to attempt to play in this range, with
> the understanding that the server MUST support summation of i=1..N
> ca_maxoperations(i) - 1, and MAY support up to summation of i=1..N
> (ca_maxoperations(i) - 1) * ca_requests(i).
>
> Of course,
>  ca_maxrequestsize;
>  ca_maxresponsesize;
>  ca_maxresponsesize_cached;
>
> might limit the MAY further, because layout operations are not
> exactly compact.
>
> But regardless server can limit the range by simply
> returning NFS4ERR_DELAY on a layout operation that
> returns a layout stateid.
>
>
>>
>> The client knows what to do in that it is required to use the
>> highest seqid from the set of fully processed responses to
>> LAYOUTGET/LAYOUTRETURN and the call of CB_LAYOUTRECALL.
>>
>> As you say, we want to allow for the maximum amount of parallelism.
>> As I said, BAD_STATEID will be returned when the the seqid == 0
>> and when seqid is "in the future".  The problem with the "in the
>> future" is with wraparound, as we have noted.  I would suggest
>> that the server use something very large, like 10 million, as the
>> limit for the number of seqid's "in the past" and everything else
>> is "in the future".  Of course, I don't want to put this in the spec.
>
> I am having trouble imagining servers that support 10 million
> outstanding, parallel layout operations. But I think we are
> talking about four separate concepts:
>
> 1. the minimum number of parallel layout operations a client
>    MUST be allowed
>
> 2. the maximum number a parallel layout operations a server MAY
>    allow
>
> 3. what values of seqid get _BAD_STATEID returned?
>
> 4. what values of seqid get _OLD_STATEID returned?
>
>
> I described 1 and 2.

Note that parallel in the above definition is the current over-the-wire
parallelism.  In the case of LAYOUT processing, the specification speaks
of fully processed responses and requests.  Allowing for flexibility
in the client implementation, the client may have reused a slot
for the next request without fully processing the response to
LAYOUTGET/LAYOUTRETURN.  Technically, the range of parallelism is
expanded beyond that of the calculations based on the session's
parameters.

> I think 3 and 4 determined by the number of seqids in the
> past. This number, call it, PAST_SEQ, is determined by the maximum  
> value of
> summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)
> that has ever existed on the client ID. This allows for additional
> fore channels to be added to the client ID and then deleted without
> introducing problems, as well as the highest slot number on a fore  
> channel
> to be increased and reduced on a server.
>
> Of course, a server implementer might decide: "to heck with that,
> I'm just going to set PAST_SEQ to 10 million." Fine, as long as it  
> is never
> less than maximum value of
>   summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i)
>
> I willing to believe that the "to heck with that" server dude
> can make that guarantee by doing stuff like limiting the maximum  
> number
> of fore channels to 100, and the maximum number of slots per
> fore channel to 100. 10K is much less than 1M, and 10K slots is
> probably enough for most MDS clients. Though I think THWT ought to
> be more elegant with his #define for PAST_SEQ.
>
>> The major point is that the server should be be very broad in
>> its interpretation of what is in the past to allow for the
>> parallelism.
>
> But the point that trumps it is that one server's definition of
> broad will inevitably be at odds with that of one client's. Since
> we don't want to codify 1M in the spec, and instead
> can deterministically define PAST_SEQ, and without .x changes,
> we should do it.

Even with my statement above, I agree that defining PAST_SEQ
is useful.  That value can then be used in describing the
processing of layout stateid.seqid values and defining
the client's recovery from receipt of OLD_STATEID (client
has exceeded degree of allowable parallelism).

Spencer

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 2 Apr 2008 17:01:48 -0700 (PDT)

>> returns a layout stateid.

[...]
> expanded beyond that of the calculations based on the session's

OK, so the range of parallelism is not
between summation of i=1..N ca_maxoperations(i) - 1,
and summation of i=1..N (ca_maxoperations(i) - 1) * ca_requests(i). Instead
it is between

MIN( (summation of i=1..N ca_maxoperations(i) - 1), 2 ^ 32 - 1 )

   and

2 ^ 32 - 1

E.g., let's say client issues a LAYOUTGET after an OPEN, and
gets a layout stateid with seqid set to 1. The client then
issues 2^32 parallel LAYOUTGETs, each with the layout stateid with seqid
set to 1.

The server has to return a unique seqid for all the LAYOUTGETs.
After sending a reply with with the seqid set to 2^32 - 1, the next
reply has to have seqid set to 1. The server cannot do that because
the client has not fully processed the reply that has seqid set to 2,
so seqid = 1 is not available. So
the server has to return NFS4ERR_DELAY. The client has just learned the
server's current value of PAST_SEQ.

So nothing fundamental is altered in my proposal.

> has exceeded degree of allowable parallelism).

OK. New text coming soon?

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 04 Apr 2008 14:13:53 -0500

Updated section based on recent discussion:

-----

>
> 12.5.5.2.1.4.  Wraparound and validation of sequence id
>
>    The rules for layout stateid processing differ from other  
> stateids in
>    the protocol because the "seqid" value can not be zero and the
>    stateid's "seqid" value changes in a CB_LAYOUTRECALL operation.   
> The
>    non-zero requirement combined with the inherent parallelism of  
> layout
>    operations means that a set of LAYOUTGET and LAYOUTRETURN  
> operations
>    may contain the same value for "seqid".  The server uses a slightly
>    modified version of the modulo arithmetic as described in
>    Section 2.10.5.1 when incrementing the layout stateid's  
> "seqid".  The
>    modification to that modulo arithmetic description is to not use
>    zero.  The modulo arithmetic is also used for the comparisons of
>    "seqid" values in the processing of CB_LAYOUTRECALL events as
>    described above in Section 12.5.5.2.1.3.
>
>    Just as the server validates the "seqid" in the event of
>    CB_LAYOUTRECALL usage, as described in Section 12.5.5.2.1.3, the
>    server also validates the "seqid" value to ensure that it is within
>    an appropriate range.  This range represents the degree of
>    parallelism the server supports for layout stateids.  If the client
>    is sending multiple layout operations to the server in parallel, by
>    definition, the "seqid" value in the supplied stateid will not  
> be the
>    current "seqid" as held by the server.  The range of parallelism
>    spans from the highest or current "seqid" to a "seqid" value in the
>    past.  To assist in the discussion, the server's current "seqid"
>    value for a layout stateid is defined as: SERVER_CURRENT_SEQID.   
> The
>    lowest "seqid" value that is acceptable to the server is  
> represented
>    by PAST_SEQID.  And the value for the range of valid "seqid"s or
>    range of parallelism is VALID_SEQID_RANGE.  Therefore, the  
> following
>    holds: VALID_SEQID_RANGE = SERVER_CURRENT_SEQID - PAST_SEQID.   
> In the
>    following, all arithmetic is the modulo arithmetic as described
>    above.
>
>    The server MUST support a minimum VALID_SEQID_RANGE.  The  
> minimum is
>    defined as: VALID_SEQID_RANGE = summation of 1..N of
>    (ca_maxoperations(i) - 1) where N is the number of session fore
>    channels and ca_maxoperations is the value returned from
>    CREATE_SESSION.  The reason for minus 1 is to allow for the  
> required
>    SEQUENCE operation.  The server MAY support a VALID_SEQID_RANGE  
> value
>    larger than the minimum.  The maximum VALID_SEQID_RANGE is (2 ^  
> 32 -
>    2) (accounts for 0 not being a valid "seqid" value).
>
>    If the server finds the "seqid" is zero, the NFS4ERR_BAD_STATEID
>    error is returned to the client.  The server further validates the
>    "seqid" to ensure it is within the range of parallelism,
>    VALID_SEQID_RANGE.  If the "seqid" value is outside of that range,
>    the error NFS4ERR_OLD_STATEID is returned to the client.  Upon
>    receipt of NFS4ERR_OLD_STATEID, the client updates the stateid  
> in the
>    layout request based on processing of other layout requests and re-
>    sends the operation to the server.
>

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 08 Apr 2008 09:27:30 -0500

Assuming no discussion means rough consensus for these changes, they
will be part of the next I-D.

Spencer

On Apr 4, 2008, at 2:13 PM, Spencer Shepler wrote:
>>    stateid's "seqid" value changes in a CB_LAYOUTRECALL  
>> operation.  The
>>    value for a layout stateid is defined as:  
>> SERVER_CURRENT_SEQID.  The

---

From: Benny Halevy <[email-masked]>
Date: Tue, 08 Apr 2008 17:56:52 +0300

To make sure, we intend to implement this feature in linux,
server and client for the May Connectathon.

Who else is planning that?

Benny