Subject: [nfsv4] [Technical Errata Reported] RFC8166 (6528)

From: RFC Errata System <[email-masked]>
Date: Mon, 12 Apr 2021 09:40:24 -0700 (PDT)

The following errata report has been submitted for RFC8166,
"Remote Direct Memory Access Transport for Remote Procedure Call Version 1".

--------------------------------------
You may review the report below and at:
https://www.rfc-editor.org/errata/eid6528

--------------------------------------
Type: Technical
Reported by: David Noveck <[email-masked]>

Section: 4.2.4

Original Text
-------------
A Requester MUST NOT send an RPC-over-RDMA header with the RDMA_ERROR
procedure.  A Responder MUST silently discard RDMA_ERROR procedures

Corrected Text
--------------
An RDMA_ERROR header cannot be sent without an assurance that, the receiver 
has posted a receive operation which its sending will satisfy.  In most cases, 
this means that a Requester (i.e. one sending RPC Calls) MUST NOT send an 
RPC-over-RDMA header with the RDMA_ERROR procedure.  Similarly, a Responder 
(i.e. one sending RPC Replies) MUST silently discard RDMA_ERROR procedures.

However, in the case of providing an RDMA_ERROR headers containing an error 
code of ERR_VERS, such a schema is not realizable, since there is no way for
a receiver who does not support a particular version, to determine whether 
an RPC Call or Reply is being sent, leaving the receiver uncertain as to whether 
it is being Addressed as a Requester or a Responder, leaving it unable to
participate in version negotiation.  In the case of version errors, the
implementation is to rely on the assumption that  forward direction requests 
are being done and reserve direction requests only done once the version is
properly negotiated.   As a result, such messages MUST NOT be sent by the 
client and MUST be silently discarded by servers.

Notes
-----
Even if one feels that this is not an appropriate correction, the existing text must be fixed somehow.   In assuming that the terms Requester and Responder can be used this way in this context is likely to result in some implementers concluding that version errors can never be sent while other might be unabble to coonclude that given the effort expended in the spec to make such errors be interpretable by anf rpc-over-rdma version.

Instructions:
-------------
This erratum is currently posted as "Reported". If necessary, please
use "Reply All" to discuss whether it should be verified or
rejected. When a decision is reached, the verifying party  
can log in to change the status and edit the report, if necessary. 

--------------------------------------
RFC8166 (draft-ietf-nfsv4-rfc5666bis-11)
--------------------------------------
Title               : Remote Direct Memory Access Transport for Remote Procedure Call Version 1
Publication Date    : June 2017
Author(s)           : C. Lever, Ed., W. Simpson, T. Talpey
Category            : PROPOSED STANDARD
Source              : Network File System Version 4
Area                : Transport
Stream              : IETF
Verifying Party     : IESG

---

From: David Noveck <[email-masked]>
Date: Tue, 20 Apr 2021 06:22:36 -0400

I believe this should be verified, but of course, having submitted it, I
would think that.  We need to hear from others who have looked at
rpc-over-rdma versioning issues and particularly from the authors of
rfc8166.

On Mon, Apr 12, 2021, 12:40 PM RFC Errata System <[email-masked]>
wrote:

> Even if one feels that this is not an appropriate correction, the existing
> text must be fixed somehow.   In assuming that the terms Requester and
> Responder can be used this way in this context is likely to result in some
> implementers concluding that version errors can never be sent while other
> might be unabble to coonclude that given the effort expended in the spec to
> make such errors be interpretable by anf rpc-over-rdma version.
>
> Instructions:
> -------------

---

From: Chuck Lever III <[email-masked]>
Date: Tue, 20 Apr 2021 15:17:54 +0000

Sorry for the delay, I wasn't sure where to begin further conversation
on this errata.

> On Apr 12, 2021, at 12:40 PM, RFC Errata System <[email-masked]> wrote:
> 

Stated as a co-author of this document, I agree that this text is
problematic for version negotiation, which uses the RDMA_ERROR
procedure/header type.

One vote for "verified".

> Corrected Text

I don't recall that the issue that prompted the addition of the
original text was the requirement for a posted Receive. That
might be slightly revisionist.

Though it is true sending an RDMA_ERROR when there is no Receive
waiting will likely result in loss of connection, I think the
typical scenario is that RDMA_ERROR is in response to a previous
message; thus missing a Receive would be an implementation bug.

"In most cases," is not appropriate with MUST/MUST NOT. We could
stipulate that the exact cases where these requirements hold is
when sending ERR_CHUNK. These requirements do not hold for ERR_VERS.

> participate in version negotiation.

The particular issue is that either:

- the Sender of the message with the unsupported version might be
using a procedure of the new protocol version that is not supported
in the protocol version the Receiver supports, or

- the Sender of the message with the unsupported version might be
using a procedure that does not bear an RPC message or any other
indication of message direction.

Thus the Receiver would not be able to determine the direction of
the incoming message.

I'm not convinced that RFC 8166 Section 4.2 is an appropriate place
to explain this reasoning. (Stating it here in the errata is OK).
The above could be dropped in the Corrected text.

> In the case of version errors, the
> properly negotiated.

Shorter:

"For the purpose of version negotiation, the receiver is to assume
that reverse-direction operation is started only once the connection's
transport protocol version has been fixed using forward direction
operations."

One problem with this is that reverse-direction operation is defined
in a separate RFC (8167).

> As a result, such messages MUST NOT be sent by the 

I'd like to confirm that what the reporter intended in the second
sentence above was "client" and "servers" and not "Requester" and
"Responders".

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Tue, 20 Apr 2021 11:36:46 -0400

> message; thus missing a Receive would be an implementation bug.

Why is this particular RDMA_ERROR so important? It seems to me like
a ton of normative text for a version mismatch, which basically
should never happen. If anything, changing "silently discard" to
"close the connection" would be just as appripriat.

Can someone motivate the scenario for me?

> when sending ERR_CHUNK. These requirements do not hold for ERR_VERS.

I'll go further - "in most cases" isn't appropriate for a spec.

Tom.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever III <[email-masked]>
Date: Tue, 20 Apr 2021 17:13:18 +0000

> On Apr 20, 2021, at 11:36 AM, Tom Talpey <[email-masked]> wrote:
> 
> should never happen.

It will happen now that we have rpcrdma-version-two.

> If anything, changing "silently discard" to
> "close the connection" would be just as appripriate.

I prefer to avoid "close the connection" as a first resort since
it can impact other RPC transactions that are ongoing. That might
not be a concern for ERR_VERS, but it could very well be a
problem for ERR_CHUNK.

> Can someone motivate the scenario for me?

The problem is that RFC 8166 defines only forward-direction
operation, where the Requester/Responder roles are clear. With
the introduction of reverse-direction operation and control
plane operations, it becomes impossible to implement the
requirements stipulated in S4.2.4 because there are now scenarios
where the Receiver has no way to determine whether it is a
Requester or a Responder.

I think Dave is concerned about how these requirements impact the
operation of versions subsequent to v1, or during version negotiation
between a v1-only implementation and a polyglot.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 21 Apr 2021 07:47:14 -0400

> a ton of normative text for a version mismatch, which basically

If we need to upgrade to v2, then we will have version mismatches.  The
spec as a whole devotes a lot of attention to version negotiation,
Including requirements that certain structures and values be maintained as
they are, for all r-o-r versions.

> "close the connection" would be just as appripriat.

No.  That would be a substantive change which we cannot make via an errata.
With the submitted errata we clarify the existing statement, and can move
forward with such a basic change as you are suggesting.

> Can someone motivate the scenario for me?

We have a number of weaknesses in v1, and we adopted your suggestions
(thanks) to correct them in v2.   To effect that transition effectively we
need a workable version negotiation scheme.

> >> Even if one feels that this is not an appropriate correction, the
> existing text must be fixed somehow.   In assuming that the terms Requester
> and Responder can be used this way in this context is likely to result in
> some implementers concluding that version errors can never be sent while
> other might be unabble to coonclude that given the effort expended in the
> spec to make such errors be interpretable by anf rpc-over-rdma version.
> >>
> >> Instructions:
> >> -------------

---

From: Tom Talpey <[email-masked]>
Date: Wed, 21 Apr 2021 09:20:10 -0400

> Requester or a Responder.

Oh. So, the idea is that the ULP reverse channel might attempt to
establish an RPCRDMA version which might be different from the forward
channel? How is that useful, or even plausible? I think the answer will
guide the proposed fix.

I don't think, in any case, that it's at all useful to allow the
version to change after both sides have exchanged messages with
matching versions. If that happens, closing the connection is the
only valid action.

Tom.

---

From: Chuck Lever III <[email-masked]>
Date: Wed, 21 Apr 2021 13:25:05 +0000

> On Apr 21, 2021, at 9:20 AM, Tom Talpey <[email-masked]> wrote:
> 
> channel?

Or it could be due to a bug. The point is this text in RFC 8166
can't work.

> How is that useful, or even plausible? I think the answer will
> guide the proposed fix.
> 

This errata deals strictly with RFC 8166. For an RPC/RDMA v1 client,
why wouldn't sending an ERR_VERS work for v1?

For later versions, we'll need a new error code to indicate that
the Receiver detected a supported but unexpected version.

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Wed, 21 Apr 2021 09:51:41 -0400

> the Receiver detected a supported but unexpected version.

I'll find time to take a closer look. I fully agree that an erratum
needs to be tightly scoped. Longer term, I'm skeptical of the value
of extending error codes for this. A v1 RDMA_ERROR reply seems like
the most universal approach, on the face of it. But heroically
attempting to recover from this is, to me, a complete non-goal.

Tom.

---

From: Chuck Lever III <[email-masked]>
Date: Wed, 21 Apr 2021 14:10:36 +0000

> On Apr 21, 2021, at 9:51 AM, Tom Talpey <[email-masked]> wrote:
> 
> needs to be tightly scoped.

Thanks for having a look!

> Longer term, I'm skeptical of the value

I'm not necessarily interested in recovery. There are two basic
principals that guide behavior regarding protocol errors:

- Try hard not to break the connection because that can perturb
other ongoing RPC transactions.

- Try hard not to leave pending RPC transactions hanging on the
Requester. In general, dropping a connection does nothing to
help the Requester retire the erroneous transaction.

The purpose of an explicit error response here would be to enable
the Requester to terminate the transaction instead of leaving it
hanging. RDMA2_ERR_SYSTEM would be adequate for that, but that
would imply that the Responder is at fault.

More extensive recovery is a quality-of-implementation issue, and
I tend to think heroic recovery in this case is probably not of
high value.

--
Chuck Lever