Subject: [nfsv4] Review of draft-ietf-nfsv4-rfc5667bis-09

From: David Noveck <[email-masked]>
Date: Tue, 18 Apr 2017 15:21:48 -0400

*Overall Evaluation*

Major improvement over RFC5667.  Almost ready to ship.  No technical issues.

A lot of my comments are basically editorial and are offered on a
take-it-or-lease-it basis.

I think some clarification in Section 5.4.1 is needed although not
necessarily in the ways suggested below,

*Comments by Section*

*5.4.1.  Multiple DDP-eligible Data Items*

Given that READ_PLUS no longer has any DDP-eligible data items, the
situation described in the fifth bullet can no longer arise.  I
suggest deleting the bullet.

The penultimate paragraph can be read as applying to some situations
in which it shouldn't and where the extra chunks would very naturally
ignored.  For example, if you had on write chunk together with a READ
operation which failed, the server would have more chunks (i.e. one)
than the number it is prepared to process (i.e. zero).  Suggest, as a
possible replacement:

Normally, when an NFS version 4 client sends an RPC Call with a Write
list that contains multiple chunks. each such, when matched with a
DDP-eligible data item in the response, directs the placement of the
data item as specified by [I.D.-nfsv4-rfc5666bis].  When there are
DDP-eligible data items matched to write chunks that an NFS version 4
server is not prepared to process, the server MUST reject the RPC  by
responding with an RDMA_ERROR message with the rdma_err value set to
ERR_CHUNK.

With regard to the last paragraph, I am curious that this paragraph, unlike
the previous one, allows GARBGEARGS. Is this so because that would be
allowed if the chunks in question had offsets other than those that
correspond to DDP-eligible data items? If so, please consider the following
possible replacement.

Normally, when an NFS version 4 client sends an RPC Call with a Read
list that contains multiple chunks, each such, when properly matched
with a DDP-eliigible data item in the request, directs the fetching of
the the data item as specified by [I.D.-nfsv4-rfc5666bis]. When there
are DDP-eligible data items matched to read chunks that an NFS version
4 server is not prepared to process, the server MUST reject the RPC
by responding with an RDMA_ERROR message with the rdma_err value set
to ERR_CHUNK.

*5.6.  Session-Related Considerations*

In the third sentence of the second paragraph, suggest replacing "no
different" by "not different".

In the last sentence of the last  paragraph, suggest replacing "is
not" by "were not"

---

From: karen deitke <[email-masked]>
Date: Tue, 18 Apr 2017 16:32:09 -0600

Hi Chuck, its unclear what you mean by "is prepared to process" in the text below.
Other than that, looks good.

Karen

  5.4.1
    If an NFS version 4 client sends an RPC Call with a Write list that
    contains more chunks than an NFS version 4 server is prepared to
    process, the server MUST reject the RPC by responding with an
    RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.

    If an NFS version 4 client sends an RPC Call with a Read list that
    contains more chunks than an NFS version 4 server is prepared to
    process, the server MUST reject the RPC by responding with an
    RDMA_MSG message containing an RPC Reply with an accept status of
    GARBAGE_ARGS, or with an RDMA_ERROR message with the rdma_err value
    set to ERR_CHUNK.

On 4/18/2017 1:21 PM, David Noveck wrote:
> *
> *
> Giventhat READ_PLUS no longer has any DDP-eligible data items, the 
> situation described in the fifth bullet can no longer arise. I suggest 
> deleting the bullet.
>     Normally, when an NFS version 4 client sends an RPC Call with a
>     Write list that contains multiple chunks. each such, when matched
>     with a DDP-eligible data item in the response, directs the
>     placement of the data item as specified by
>     [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items
>     matched to write chunks that an NFS version 4 server is not
>     prepared to process, the server MUST reject the RPC by responding
>     with an RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>
> With regard to the last paragraph, I am curious that this paragraph, 
> unlike the previous one, allows GARBGEARGS. Is this so because that 
> would be allowed if the chunks in question had offsets other than 
> those that correspond to DDP-eligible data items? If so, please 
> consider the following possible replacement.
>
>     Normally, when an NFS version 4 client sends an RPC Call with a
>     Read list that contains multiple chunks, each such, when properly
>     matched with a DDP-eliigible data item in the request, directs the
>     fetching of the the data item as specified by
>     [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items
>     matched to read chunks that an NFS version 4 server is not
>     prepared to process, the server MUST reject the RPC by responding
>     with an RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>
> In the last sentence of the last paragraph, suggest replacing "is not" 
> by "were not"
>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Tom Talpey <[email-masked]>
Date: Tue, 18 Apr 2017 23:08:52 -0400

I noticed the same thing, and I'll add that the MUST reject condition
is very confusing because it allows an "or". In my opinion a MUST is
always a single requirement, never ambiguous.

Is there some reason you want to allow such a choice? I think you'll
find that, worded properly, it becomes actually much less implementable
and interoperable than you may think.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Wed, 19 Apr 2017 11:14:21 -0400

Hi Tom-

> On Apr 18, 2017, at 11:08 PM, Tom Talpey <[email-masked]> wrote:
> 

I agree this kind of thing is tricky. I wrote it as "the server MUST
reject the RPC". That's the single requirement. The choice is how the
rejection is conveyed to the client.

> and interoperable than you may think.

The Solaris server can return an RPC-level error in cases like this.
I think there are similar choices allowed in rfc5666bis. Let's say
that in a perfect world, I would go with only ERR_CHUNK, but I'm
documenting existing implementation behavior here.

I'm not sure it matters to the client: both errors are permanent and
the RPC is terminated on the client.

I'm open to alternatives.

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Wed, 19 Apr 2017 11:18:04 -0400

Hi Karen-

> On Apr 18, 2017, at 6:32 PM, karen deitke <[email-masked]> wrote:
> 

Fair enough, that's an awkward turn of phrase, and I had some
difficulty coming up with the right language.

Suppose this limit is based on what's in the COMPOUND? Like the
server supports 6 Read chunks for WRITE operations, but only 1
Read chunk for SYMLINK operations?

Can you give some alternatives? That would help me understand
what other readers find ambiguous.

>> Overall Evaluation
>> A lot of my comments are basically editorial and are offered on a take-it-or-lease-it basis.
>> 
>> I think some clarification in Section 5.4.1 is needed although not necessarily in the ways suggested below, 
>> 
>> Comments by Section
>> 5.4.1.  Multiple DDP-eligible Data Items
>> Given that READ_PLUS no longer has any DDP-eligible data items, the situation described in the fifth bullet can no longer arise.  I suggest deleting the bullet.
>> The penultimate paragraph can be read as applying to some situations in which it shouldn't and where the extra chunks would very naturally ignored.  For example, if you had on write chunk together with a READ operation which failed, the server would have more chunks (i.e. one) than the number it is prepared to process (i.e. zero).  Suggest, as a possible replacement:
>> Normally, when an NFS version 4 client sends an RPC Call with a Write list that contains multiple chunks. each such, when matched with a DDP-eligible data item in the response, directs the placement of the data item as specified by [I.D.-nfsv4-rfc5666bis].  When there are DDP-eligible data items matched to write chunks that an NFS version 4 server is not prepared to process, the server MUST reject the RPC  by responding with an RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>> With regard to the last paragraph, I am curious that this paragraph, unlike the previous one, allows GARBGEARGS.  Is this so because that would be allowed if the chunks in question had offsets other than those that correspond to DDP-eligible data items?  If so, please consider the following possible replacement.
>> Normally, when an NFS version 4 client sends an RPC Call with a Read list that contains multiple chunks, each such, when properly matched with a DDP-eliigible data item in the request, directs the fetching of the the data item as specified by [I.D.-nfsv4-rfc5666bis]. When there are DDP-eligible data items matched to read chunks that an NFS version 4 server is not prepared to process, the server MUST reject the RPC  by responding with an RDMA_ERROR message with the rdma_err value set to ERR_CHUNK.
>> 5.6.  Session-Related Considerations
>> In the third sentence of the second paragraph, suggest replacing "no different" by "not different".
>> In the last sentence of the last  paragraph, suggest replacing "is not" by "were not"
>> 
>> 
>> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 19 Apr 2017 11:24:49 -0400

My interpretation was that, since chunks direct the placement of
DDP-eligible data items, the error was to be returned in cases in which the
server was unable to obtain or return the data item, in the fashion
specified by rfc5666bis.

On Wed, Apr 19, 2017 at 11:18 AM, Chuck Lever <[email-masked]>
wrote:

> >> The penultimate paragraph can be read as applying to some situations in
> which it shouldn't and where the extra chunks would very naturally
> operation which failed, the server would have more chunks (i.e. one) than
> the number it is prepared to process (i.e. zero).  Suggest, as a possible
> replacement:
> DDP-eligible data item in the response, directs the placement of the data
> item as specified by [I.D.-nfsv4-rfc5666bis].  When there are DDP-eligible
> data items matched to write chunks that an NFS version 4 server is not
> prepared to process, the server MUST reject the RPC  by responding with an
> unlike the previous one, allows GARBGEARGS.  Is this so because that would
> be allowed if the chunks in question had offsets other than those that
> correspond to DDP-eligible data items?  If so, please consider the
> following possible replacement.
> list that contains multiple chunks, each such, when properly matched with a
> DDP-eliigible data item in the request, directs the fetching of the the
> DDP-eligible data items matched to read chunks that an NFS version 4 server
> is not prepared to process, the server MUST reject the RPC  by responding

---

From: Tom Talpey <[email-masked]>
Date: Thu, 20 Apr 2017 17:28:26 -0400

> rejection is conveyed to the client.

The statement "MUST reject" is not testable. So, while it may be
understood what is intended, there is nothing implementable in the
MUST. The "or" is a similar situation, it prescribes a choice, which
does not define a protocol.

> The Solaris server can return an RPC-level error in cases like this.

Well, this is happening because the Solaris server is (probably) just
handing the chunk list up to the RPC layer, and it's the RPC (XDR)
processing that detects any problem.

On the other hand, an implementation could do the opposite, it could
process the chunks at the lower layer, before ever invoking RPC
processing. This would naturally lead to a non-RPC error.

The challenge in defining the protocol is to hide these possibilities.

> I'm open to alternatives.

The icky way to do this is to split into two weak requirements.

    If an NFS version 4 client sends an RPC Call with a Read list that
    contains more chunks than an NFS version 4 server is prepared to
    process, the server SHOULD reject the request by responding with an
    RDMA_ERROR message with the rdma_err value set to ERR_CHUNK. The
    server MAY reject the RPC with an RDMA_MSG message containing an RPC
    Reply with an accept status of GARBAGE_ARGS.

This at least makes it clear which response is "preferred". And one
would hope a future draft would decide.

I have a second question though. How does the client determine what is
the actual error? As in, how many chunks were allowed? Does the upper
layer have to recover, and if so what are the implications?

Yes, I know 5667 did not explore this very well. Mea culpa.

Tom.

---

From: David Noveck <[email-masked]>
Date: Thu, 20 Apr 2017 19:21:43 -0400

> The "or" is a similar situation, it prescribes a choice, which

Fair enough, but the point that needs to be made is that, with
regard to Version One, Chuck and  the working group is not
free to define a protocol.  As a result we have the kind of
ugliness you object to, but it is inherent in the choice to try to
revive Version One as-is.

>   Reply with an accept status of GARBAGE_ARGS.

I think I know what you intend here and I've seen stuff like this in RFCs
but I don't
wthink e can do this because this is not in line with the definitions of
"SHOULD"
and "MAY" that appear in RFC2119.

"SHOULD" means that you are supposed to do something but can avoid it if
you have a good reason and are aware of the consequences of not doing it.
In this case the "good" reason is that someone coded the implementation
to do something else, which is not all that good a reason.  The
consequences of
returning the GARBAGEARGS are exactly zero, since the client has to be
prepared
for either it or ERR_CHUNK.

"MAY" means the implementation can choose to do the action or not, which is
line
with the reality here but essentially contradicts the SHOULD.

> This at least makes it clear which response is "preferred".

But it is isn't really the job of the RFC2119 terms to say which is
"preferred" or
"'preferred'".  These terms are supposed to describe interoperability and
the
interoperability situation is that the server MUST return ERR_CHUNK or
 GARBAGEARGS and the client needs to be prepared for either.  That is the
unpleasant reality.  If you want to indicate a preference, you can say
something
like:

   - Returning ERR_CHUNK is preferrable.
   - Returinng ERR_CHUNK is more in line with the appropriate protocol
   layering since this issue relates to a limitation of the transport
   implementation.
   - Use of GARBAGEARGS is an unfortunate artifact of inappropriately
   layered implementations and is only allowed for reasons of compatibility
   with existing implementations.  It is desirable to avoid it.

> And one would hope a future draft would decide.

Not sure what draft you are thinking of.  I don't see us doing an
rfc5667bisbis (rfc5667tris).

By the time we did that, the implementations with these restrictions will
probably be gone.

> the actual error? As in, how many chunks were allowed?

This is not fixable in Version One.  It would be in Version Two, but by then
the need will probably be gone.

> Does the upper

I think something could be put in to indicate that clients should break up
COMPOUNDS
so the only have a single chunk each.

> Yes, I know 5667 did not explore this very well.

It didn't explore it at all.  And 5666's error reporting facilities were
extremely limited.

> Mea culpa.

I don'tt think you have anything to apologize for.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 21 Apr 2017 10:43:50 -0400

I agree that SHOULD/MAY makes things cloudier, and does not
seem to align with well-defined RFC2119 usage.

Another way we've dealt with similar disagreements between
specification and implementation is to decide that one of
the implementations is incorrect.

Can we agree that:

- GARBAGE_ARGS is a bit of a layering violation, though it's
understandable why it might be returned

- RPC clients are already prepared for GARBAGE_ARGS

- In RPC-over-RDMA Version One, we are not trying to recover
(in the sense of resending a simpler COMPOUND) but are rather
trying to ensure the offending RPC is properly terminated on
the client, and does not further block other RPCs or deadlock
the transport

Thus I claim it is harmless if a server returns GARBAGE_ARGS
instead of ERR_CHUNK.

As a result, I can change the Read list text in S5.4.1 to be
the same as the Write list text, removing the mention of
GARBAGE_ARGS.

Would that sit comfortably with everyone?

> On Apr 20, 2017, at 7:21 PM, David Noveck <[email-masked]> wrote:
> 
> I think I know what you intend here and I've seen stuff like this in RFCs but I don't
> wthink e can do this because this is not in line with the definitions of "SHOULD" 
> to do something else, which is not all that good a reason.  The consequences of 
> returning the GARBAGEARGS are exactly zero, since the client has to be prepared
> "MAY" means the implementation can choose to do the action or not, which is line
> But it is isn't really the job of the RFC2119 terms to say which is "preferred" or
> "'preferred'".  These terms are supposed to describe interoperability and the 
> unpleasant reality.  If you want to indicate a preference, you can say something
> like:
> 	• Returning ERR_CHUNK is preferrable.
> 	• Returinng ERR_CHUNK is more in line with the appropriate protocol layering since this issue relates to a limitation of the transport implementation.
> 	• Use of GARBAGEARGS is an unfortunate artifact of inappropriately layered implementations and is only allowed for reasons of compatibility with existing implementations.  It is desirable to avoid it.
> Not sure what draft you are thinking of.  I don't see us doing an rfc5667bisbis (rfc5667tris).
> 
> By the time we did that, the implementations with these restrictions will probably be gone.
> 
> I think something could be put in to indicate that clients should break up COMPOUNDS
> It didn't explore it at all.  And 5666's error reporting facilities were extremely limited.
> 
> > Mea culpa.
> 

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Mon, 24 Apr 2017 07:56:38 -0400

> - RPC clients are already prepared for GARBAGE_ARGS

Are you certain of this? And out of curiosity, what is returned
to the consumer for GARBAGE_ARGS versus ERR_CHUNK?

> Thus I claim it is harmless if a server returns GARBAGE_ARGS

"Harmless" is a bit relative. The operation fails, through no fault
of the consumer. And, frankly, in a very mysterious way.

Again, I think there is more to say here. It's a limitation of the
protocol whose implications should be made clear (contraining the
complexity of COMPOUNDs, limiting scatter/gather lengths, etc).

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 24 Apr 2017 10:30:24 -0400

> On Apr 24, 2017, at 7:56 AM, Tom Talpey <[email-masked]> wrote:
> 
> Are you certain of this?

GARBAGE_ARGS has been part of the RPC protocol for decades.
The two Unix-flavored clients that have NFS/RDMA support can
both handle this error.

> And out of curiosity, what is returned

RFC 5531:
> GARBAGE_ARGS  = 4, /* procedure can’t decode params   */

GARBAGE_ARGS is an RPC-level error. The reply is "accepted"
with accept_stat GARBAGE_ARGS. An XID is available in the
header.

rfc5666bis:
> If the rdma_vers field contains a recognized value, but an
> XDR parsing error occurs, the responder MUST reply with an
> RDMA_ERROR procedure and set the rdma_err value to ERR_CHUNK.

ERR_CHUNK is a transport level error. An XID is available
in the header.

The difference is that the RPC layer v. the transport layer
are reporting they don't understand the contents of the
message (Call). There is nothing more in either type of
message.

> of the consumer. And, frankly, in a very mysterious way.

We have no richer way of communicating failure in RPC-over-RDMA
Version One. We are not looking for recovery here, so I don't
believe any more information would be useful. If the server
wishes, it can log the failure with a message explaining what
went wrong.

> complexity of COMPOUNDs, limiting scatter/gather lengths, etc).

I'd welcome any suggested text.

Honestly, I'm not sure what can be said. Neither NFSv4.0 nor
RPC-over-RDMA have a sophisticated mechanism to communicate this
kind of limitation. The best an NFSv4 server can do is return
NFS4ERR_RESOURCE, which also carries little extra information
about what a client should do to recover.

So are you comfortable with eliminating GARBAGE_ARGS if we can
come up with more detail about the impact of not knowing how
complex a COMPOUND can be?

--
Chuck Lever

---

From: karen deitke <[email-masked]>
Date: Mon, 24 Apr 2017 08:36:42 -0600

> to the consumer for GARBAGE_ARGS versus ERR_CHUNK?
Consumer will see EIO for both.
>>>     • Returinng ERR_CHUNK is more in line with the appropriate 
>>> protocol layering since this issue relates to a limitation of the 
>>> transport implementation.
>>>     • Use of GARBAGEARGS is an unfortunate artifact of 
>>> inappropriately layered implementations and is only allowed for 
>>> reasons of compatibility with existing implementations. It is 
>>> desirable to avoid it.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 24 Apr 2017 10:46:04 -0400

> On Apr 24, 2017, at 10:36 AM, karen deitke <[email-masked]> wrote:
> 
> 
> 

On Solaris.

On Linux, EIO for ERR_CHUNK, and EACCES for GARBAGE_ARGS
(historical behavior).

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Wed, 26 Apr 2017 09:18:23 -0700

> On Apr 24, 2017, at 7:30 AM, Chuck Lever <[email-masked]> wrote:
> 
>> 

I've confirmed that the only other known NFS/RDMA client
(Oracle dNFS) properly recognizes GARBAGE_ARGS.

> complex a COMPOUND can be?

I've come up with some possible replacement text for
the final two paragraphs of S5.4.1 in an attempt to
address comments from Tom, David, and Karen. The
normative requirements have been removed, and a (brief)
discussion of the consequences of not handling complex
COMPOUNDs was introduced.

5.4.2.  Complexity Considerations

   As mentioned above, an NFS version 4 COMPOUND procedure can contain
   more than one operation that carries a DDP-eligible data item.  The
   RPC-over-RDMA Version One protocol does not place any limit on the
   number of chunks that may appear in the Read or Write lists.
   Therefore an NFS version 4 client MAY construct an RPC-over-RDMA
   Version One message containing more than one Read chunk or Write
   chunk.

   However, implementations have practical limits on the number of
   chunks or segments they are prepared to process in one of these
   lists.  There are several ways an NFS Version 4 server might indicate
   that an RPC Call message constructed by a client is valid but cannot
   be processed because of implementation limitations:

   o  If the problem is detected in the upper layer (i.e., by the NFS
      version 4 implementation), the server returns an NFS status of
      NFS4ERR_RESOURCE.

   o  If the problem is detected during XDR decoding of the request
      (e.g., during re-assembly of the Call message by the RPC layer),
      the server returns an RPC accept_stat of GARBAGE_ARGS.

   o  If the problem is detected at the transport layer (i.e., during
      transport header processing), the server returns an RDMA_ERROR
      message with the err_code field set to ERR_CHUNK.

   Such implementation limits can constrain the complexity of NFS
   version 4 COMPOUNDs, limit the number of elements in scatter-gather
   operations, or prevent the use of Kerberos integrity or privacy
   services.

Comments, opinions on this approach?

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 26 Apr 2017 13:05:52 -0400

Works for me.

On Wed, Apr 26, 2017 at 12:18 PM, Chuck Lever <[email-masked]>
wrote:

> protocol layering since this issue relates to a limitation of the transport
> implementation.
> >>>>    • Use of GARBAGEARGS is an unfortunate artifact of inappropriately

---

From: karen deitke <[email-masked]>
Date: Wed, 26 Apr 2017 13:37:28 -0600

How does this prevent kerberos integrity or privacy?  Couldnt that still 
be done with simply just 1 offset 0 read chunk?

Karen

---

From: Chuck Lever <[email-masked]>
Date: Wed, 26 Apr 2017 13:34:56 -0700

Hi Karen-

The language also now mentions a limit on the maximum number
of chunk segments the server is prepared to process. Long
messages use chunks that can be broken into multiple
segments, and thus could encounter a maximum segment count
limit.

> On Apr 26, 2017, at 12:37 PM, karen deitke <[email-masked]> wrote:
> 
> How does this prevent kerberos integrity or privacy?  Couldnt that still be done with simply just 1 offset 0 read chunk?
> 
> Karen
> 
> 

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Thu, 27 Apr 2017 10:20:49 -0400

>snip:

I like the approach, and the lead-in language looks good. The text
quoted above is just a little bit dark, especially that bit about
preventing krb5i/krb5p. I'd suggest a more active statement to replace
the above, including the more prescriptive SHOULD rather than "can".
How about:

  "Client implementations SHOULD be prepared to provide mechanisms for
   reporting the above errors, and optionally provide configuration to
   limit the complexity of NFS version 4 COMPOUNDs, limit the number
   of elements in scatter-gather operations, and to avoid other possible
   sources of RPC-over-RDMA chunk overruns at the peer.

   These become especially important when Kerberos integrity or privacy
   is in place for the RPC connection. These facilities add payload to
   the RPC headers, potentially increasing the complexity of the chunk
   manipulation, independent of the upper layer NFS operation. The
   implementation SHOULD consider such RPC payload requirements in
   addition to the NFS considerations."

Feel free to wordsmith further.

Tom.

---

From: karen deitke <[email-masked]>
Date: Thu, 27 Apr 2017 08:38:28 -0600

>   addition to the NFS considerations."
  This wording about preventing the use of krb5 is confusing to me. Are 
you saying that clients whose krb5 implementation causes "extraneous" 
number of chunks in an rdma header may not be supported by a server and 
thus prevent krb usage?

Also:

  "However, implementations have practical limits on the number of
   chunks_or_  _segments_  they are prepared to process in one of these
   lists."

Agreed, but what is the practical limit?  There are cases where a solaris client can send up to 16 segments in one write chunk.  WOuld that be considered under the "practical limit"?

Karen

>>>>>> not"

---

From: Tom Talpey <[email-masked]>
Date: Thu, 27 Apr 2017 11:13:05 -0400

> number of chunks in an rdma header may not be supported by a server and

As the protocol stands today, yes, there is no requirement that the
server MUST be prepared to handle any certain number of chunks. And,
since krb can increase the size of RPC headers, potentially pushing
them beyond the maximum message size, that can potentially add new
payload to the chunk list.

The implication is that NFS COMPOUNDs may need to shrink, if this is
the case. However, the "limit" is server implementation dependent,
and the chunks are built at the client. So my suggestion, short of
writing a new protocol, is to inform the client implementer of this
possibility.

> Agreed, but what is the practical limit?  There are cases where a
> solaris client can send up to 16 segments in one write chunk.  WOuld
> that be considered under the "practical limit"?

I'm afraid there is not enough experience to define such a limit. This
document could attempt to define one, but that may be problematic. The
stated goal of this -bis update is to clarify, not to make new
requirements.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 27 Apr 2017 08:35:44 -0700

> On Apr 27, 2017, at 7:38 AM, karen deitke <[email-masked]> wrote:
> 
> Agreed, but what is the practical limit?

There has to be a limit based on rsize/wsize constraints, page
size, client memory registration mode, the maximum size of
transport headers, and so on.

"Implementations have practical limits" means implementers
get to choose to support only a certain number of segments or
chunks to keep their implementations as simple as possible. The
choice of the limit is based on the features they want their
implementation to support. The protocol specification doesn't
define a limit.

It's OK that the protocol doesn't define a limit. The problem
is there's no way for a client and server to discover the
other's actual limits. Since we're documenting the protocol
here, not defining it, we're stuck with that for the moment.

We could do something clever like add these limits to the
RDMA CM private data.

> There are cases where a solaris client can send up to 16 segments in one write chunk.  WOuld that be considered under the "practical limit"?

No, that would _be_ the practical limit for your client.

For example, the Linux server has a limit of about 260 segments,
because it supports 1MB NFS READs and WRITEs, and 256 pages (plus
a few to accommodate non-aligned payloads) is needed with 4KB
pages.

The Linux server has a limit of 1 normal Read chunk or 1
Position-zero Read chunk. That's totally arbitrary, and was an
implementation choice based on what the Linux client of 2008
might send. Clearly, the Linux server will need to be improved
to support more general clients like Solaris.

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Thu, 27 Apr 2017 08:44:34 -0700

> On Apr 27, 2017, at 7:20 AM, Tom Talpey <[email-masked]> wrote:
> 
> >snip:

Sure, I can work this in.

When you say "Client implementations SHOULD ... [report] the above
errors" you are talking about reporting them to administrators
and/or RPC consumers? I don't think we can use SHOULD in that case.
This feels like implementation advice, not protocol.

Would "recommend" be enough for this section?

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Thu, 27 Apr 2017 11:46:58 -0400

> We could do something clever like add these limits to the

Please, no. Placing a message in that data is in itself a protocol,
and the size and support for that data is implementation defined,
in the RDMA layer.

If such a parameter exchange is required (and remember it needs to
go both ways), just define it as a full-blown negotiation, at the
RPC-RDMA, or better, the NFS layer. There are already some RDMA
values exchanged in the NFSv4.1 Session establishment, which could
be extended.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 27 Apr 2017 08:53:29 -0700

> On Apr 27, 2017, at 8:46 AM, Tom Talpey <[email-masked]> wrote:
> 

So you would object strenuously to

https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpcrdma-cm-pvt-msg/

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Thu, 27 Apr 2017 11:54:33 -0400

> and/or RPC consumers? I don't think we can use SHOULD in that case.

I am agnostic about who to inform. The important thing is that some
visibility of the error be surfaced. I absolutely don't think an
arbitrary GARBAGE_ARGS returned to an application that may simply
choke on it, qualifies.

> This feels like implementation advice, not protocol.

Correct. But since the protocol creates the problem, the protocol
definition needs to say something about dealing with it. So I believe
SHOULD is best.

> Would "recommend" be enough for this section?

The RFC2119 term RECOMMENDED is basically a synonym for SHOULD. It's
perfectly permissible.

Tom.

---

From: karen deitke <[email-masked]>
Date: Thu, 27 Apr 2017 10:01:51 -0600

> writing a new protocol, is to inform the client implementer of this
Or the client can simply send the entire packet in 1 offset 0 read chuck 
as the solaris client does.  I'm guessing that the linux client can not 
do this based on where the packets are created, but just pointing out 
there are other ways to accomplish this.

Karen
>>>>>>>> Major improvement over RFC5667.  Almost ready to ship. No 
>>>>>>>> technical
>>>>>>>> issues.
>>>>>>>>

---

From: David Noveck <[email-masked]>
Date: Thu, 27 Apr 2017 12:09:17 -0400

> Would "recommend" be enough for this section?

I agree that what we want here is implementation advice but I would prefer
something stronger than "recommend".  Note that I don't consider the
normative "SHOULD" strong enough, particularly in the context "SHOULD
be prepared to provide".

How about:

Given that there are existing servers with quite restrictive limits, there
will

be situations in which such limits will be exceeded, making it necessary

for clients to provide reporting mechanisms that will bring these
situations
to the attention of those in a position to prevent further occurrences.
One
important way of allowing effective adaptation to such client limitations
would be to provide configuration options to limit the complexity of NFS
version 4 COMPOUNDs,  limit the number of elements in scatter-gather
operations, and to avoid other  possible sources of RPC-over-RDMA chunk
overruns at the peer.

---

From: Tom Talpey <[email-masked]>
Date: Thu, 27 Apr 2017 12:09:38 -0400

> https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpcrdma-cm-pvt-msg/

Experimentally, I'm fine with it. As a Standard, I'm not as thrilled,
but you are correct I have not previously commented on your draft.
I'll try to find time to do so.

Tom.

---

From: Tom Talpey <[email-masked]>
Date: Thu, 27 Apr 2017 12:14:54 -0400

> there are other ways to accomplish this.

Indeed it can, but the penalty is high since it requires registration
of the message, and five trips on the wire even before transferring
any bulk data. That penalty will be incurred for every request, plus,
servers may choose to not implement the exchange.

It's worth mentioning. But I feel it is far short of a solution.

Tom.

---

From: David Noveck <[email-masked]>
Date: Thu, 27 Apr 2017 12:17:32 -0400

> The RFC2119 term RECOMMENDED is basically a synonym for SHOULD. It's

Not if it's implementation advice.

On Thu, Apr 27, 2017 at 11:54 AM, Tom Talpey <[email-masked]> wrote:

>>>>>>>> inappropriately layered implementations and is only allowed for reasons of
>>>>>>>> compatibility with existing implementations.  It is desirable to avoid it.
>>>>>>>>

---

From: Chuck Lever <[email-masked]>
Date: Thu, 27 Apr 2017 09:27:34 -0700

> On Apr 27, 2017, at 8:54 AM, Tom Talpey <[email-masked]> wrote:
> 

Right, no client surfaces a protocol element like GARBAGE_ARGS
as it is. The failure of the RPC on the server is reported to
consumers in a way that conforms with the API they use to drive
the requests. So an application running on a POSIX client might
use read(2) or write(2) and get back EIO in this case, because
POSIX allows only a narrow range of error status codes for these
APIs.

That translation is a normal part of RPC client implementations,
so I didn't think I needed to state that explicitly, and thus
was confused about why "Client implementations" was mentioned
above.

> definition needs to say something about dealing with it.

Philosophical agreement with that.

> So I believe SHOULD is best.

> The RFC2119 term RECOMMENDED is basically a synonym for SHOULD. It's

To be clear, I was suggesting lower-case "recommended". IMO
RFC 2119 terminology isn't appropriate for describing
internal APIs or mechanisms that do not appear on the wire.

The typical approach has been to say "The RPC client then
terminates the RPC with an appropriate error status." Which
crisply yet generically describes desired behavior while
avoiding the use of MUST or SHOULD.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 27 Apr 2017 13:24:56 -0400

> definition needs to say something about dealing with it.

The protocol did not create this problem.  Up until about a month ago,
the protocol allowed multiple chunks and we believed that the
implementations did as well.

Then we found out that a lot implementations have these restrcctions and we
are
trying to deal with that situation.  The protocol has stayed the same but
we now
know that some implementations have these restrictions.

Even though the protocol did not create this situation, this document is
the only
opportunity we have to tell clients how to deal with these restrictions.

---

From: David Noveck <[email-masked]>
Date: Fri, 28 Apr 2017 11:44:18 -0400

> We could do something clever like add these limits to the

I don't think this is all that "clever".  Calling it "clever' may strengthen
the perception that is a hack, which I don't think it is.  On the plus
side,
it is a reasonable extension of a mechanism that already exists
(i.e. there is running code) and could be reasonably extended.

> Please, no.

If this extension were to be made, it would be done after the existing
I-D becomes a working group draft.   I think your general
objections to this way of passing information can be made as that
issue is considered.

> Placing a message in that data is in itself a protocol,

Agreed

> and the size and support for that data is implementation defined,

That's an issue already deal with in the existing I-D.  If
the implementation
does not support this or the size limits are too restrictive, a deafault is
used.  In the event of an extension, the same would apply.  There would
have to be defaults to deal with the case in which extended message is too
long
to be sent.

> If such a parameter exchange is required

The question is not whether it is "required" or "REQUIRED" :-).  The real
question is whether anyone wants to bother to author the needed document.
In other words, I'm looking for volunteers.

> (and remember it needs to go both ways)

I don't see that.  To me, this something that the responder need to
inform the requester of, so that the requester can do the appropriate
thing to avoid getting a mysterious error.

I'm assuming that there is no real need for this on the callback path,
although
there might eventually be.

> just define it as a full-blown negotiation,

I'm not sure there is a place for an actual negotiation, even though many
things
where there is no actual negotiation are described that way.  To me, the
appropriate  approach is for the responder to make its final/only offer on a
take-it-or-leave-it-basis.  that's as close to "negotiation" as it gets.

> at the RPC-RDMA,

This could be done using the framework for transport properties defined in
draft-cel-nfsv4-rpcrdma-version-two-03.

> or better, the NFS layer.

I think this the best choice because the extension mechanism is furthest
along;

   - draft-cel-nfsv4-rpcrdma-version-two is an I-D and moving it to be a WG
   document has even been discussed yet
   - draft-cel-nfsv4-rpcrdma-cm-pvt-msg is an I-D and while there has been
   a discussion of making it a WG document, it still might controversial.
   - draft-ietf-nfsv4-versioning-09 has been through WGLC and is 14 days
   into AD Evaluation.  The fastest way of making this extension available is
   by defining an NFSv4.2 extension referencing the versioining draft
   normatively extension, this doesn't help v4.0 clients.  Sigh!

> There are already some RDMA

I don't think you can extend this without creating CREATE_SESSION2,
CREATE_SESSION_PLUSPLUS, or, my favorite CREATE_SESSION_2017.

Again, we need a volunteer.  If you think we need this, the way to make it
happen
is to submit and I-D and try to make it happen.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 28 Apr 2017 10:32:22 -0700

> On Apr 27, 2017, at 10:24 AM, David Noveck <[email-masked]> wrote:
> 
> The protocol did not create this problem.

I respectfully disagree.

The transport protocol design allows arbitrarily complex Read
and Write lists. This is a common practice in protocol design
to permit wide latitude for innovation by implementers.

NFSv4 also allows arbitrarily complex COMPOUNDs for similar
reasons, but a) clients do not today make use of this, and b)
newer minor versions recognize that implementation limits have
to be communicated.

The protocol problem we are trying to address is that there
is no mechanism (either via a specified limit or via a run-time
negotiation) that allows implementations to choose limits that
are less than infinity while allowing acceptable interoperability.

> Up until about a month ago,

It's true that the transport protocol hasn't changed, but
I think up until a month ago, we simply didn't realize that
permissive chunk list complexity limits was an interopera-
bility issue.

The Linux implementations take a lot of short cuts because
they are really advanced prototypes, not fully mature
implementations of the transport. One of the short cuts
has been that they implement just the minimum number of
chunk combinations needed for most NFS operations. They do
not implement all possible chunk combinations, nor do they
support arbitrarily long chunk lists, because they never
had to.

I'm aware that other implementers have taken a philoso-
phically similar approach to shorten the time it takes to
get a working client and server, and I'm sure that will be
the case for implementations in the future. As protocol
designers I don't think we should ignore this kind of
expediency.

In practice, for now chunk list complexity limits really
aren't a problem at all, because of a) above, except for the
desire to have servers accept a Read list that contains a
Position Zero Read chunk and a normal Read chunk at once.

So this is a situation that is hazardous, but might not be
encountered in practice for years. I don't want to spill a
lot more electrons or brain cells on fixing something that
has minimal consequences for the set of implementations and
ULPs we have today.

> Then we found out that a lot implementations have these restrcctions and we are
> trying to deal with that situation.  The protocol has stayed the same but we now
> Even though the protocol did not create this situation, this document is the only

Or we could give implementers a base set of chunk list
capabilities that must be observed. For all versions of NFS on
RPC-over-RDMA Version One, make the limit one normal Read chunk
plus one PZRC (both with multiple Read segments), and one Write
chunk (with multiple segments). Replace discussion of handling
NFSv4 COMPOUNDs with more than one DDP-eligible element with
a few rules that determine which single operation in a multiple
READ or WRITE COMPOUND gets to use DDP.

Real support for multiple chunks with NFSv4 COMPOUNDs will have
to wait until another version of RPC-over-RDMA is available.

Not sure what to do about segment count limits.

> On Thu, Apr 27, 2017 at 12:27 PM, Chuck Lever <[email-masked]> wrote:
> 

--
Chuck Lever