Subject: [nfsv4] Issue in RFC8166

From: David Noveck <[email-masked]>
Date: Wed, 7 Apr 2021 12:36:29 -0400

The document contains the following text:

   A Requester MUST NOT send an RPC-over-RDMA header with the RDMA_ERROR
   procedure.  A Responder MUST silently discard RDMA_ERROR procedures.

One problem with stating things this way is that that is not clear to
the receiver whether he is

a requester or responder or neither until he actually looks at the
first word of the RPC message.

This is a relatively minor issue until we consider the possibility of
version mismatches.   If I

receive an RPC-over-RDMA header for a version I don't support, there
is no way I can find the

first word of the RPC message to tell me whether I am a
requester/responder in this case.  It

seems that I wouldn't be allowed to send a version error since the
implementation cannot

be sure it is the responder.

I think we need to make an exception for version errors at least with
regard to the first message

sent.   I hope we can do this via an errata report rather than
requiring a new RFC.   I will file the

report soon if that is OK with you.

---

From: Chuck Lever III <[email-masked]>
Date: Wed, 7 Apr 2021 16:42:46 +0000

> On Apr 7, 2021, at 12:36 PM, David Noveck <[email-masked]> wrote:
> 
> One problem with stating things this way is that that is not clear to the receiver whether he is
> a requester or responder or neither until he actually looks at the first word of the RPC message.
> 
> This is a relatively minor issue until we consider the possibility of version mismatches.   If I
> receive an RPC-over-RDMA header for a version I don't support, there is no way I can find the 
> first word of the RPC message to tell me whether I am a requester/responder in this case.  It 
> seems that I wouldn't be allowed to send a version error since the implementation cannot 

Well, another way of looking at this is that if you receive an RDMA_ERROR
then you MUST be a Requester, and therefore there must be an XID waiting
for a response. If there isn't, then the ERROR can be discarded.

> I think we need to make an exception for version errors at least with regard to the first message 
> sent.   I hope we can do this via an errata report rather than requiring a new RFC.   I will file the

Before discussing the procedural issues, I need to have a better understanding
of the problem that is being solved by weakening the compliance requirements.
Can you post a ladder diagram that shows an interaction that violates the
requirements?

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 7 Apr 2021 13:37:12 -0400

On Wed, Apr 7, 2021 at 12:42 PM Chuck Lever III <[email-masked]>
wrote:

> then you MUST be a Requester,

  That deals with the case of receiving one, but the big issue here (see
below) concerns  the fact that  sending one seems to be forbidden in a lot
of cases:

> and therefore there must be an XID waiting
> for a response.

> If there isn't, then the ERROR can be discarded.
>

If there must be there is.   That's the way "must" (as opposed to "MUST")
is defined.  in any case, the big issue concerns sending the ERROR.

> Can you post a ladder diagram that shows an interaction that violates the

I don't see how a trivial ladder diagram will help things but here goes.
 If this is a ladder, it leads downward, and that first step is a doozy.

-----> node A sends  a message with version != 1 to node B.

Node B receives that message and would like do so but RFC8166 tells him he
can only do so if he is the responder.   Since he doesn't understand the
message, he has no way of determining that he is the responder since the
format of the header is not known to him.   The spec defines that the error
fields stay the same for all versions but the requester/responder
distinction is not the same for all transport versions.

---

From: Chuck Lever III <[email-masked]>
Date: Wed, 7 Apr 2021 20:06:03 +0000

> On Apr 7, 2021, at 1:37 PM, David Noveck <[email-masked]> wrote:
> 
> 
> 
> On Wed, Apr 7, 2021 at 12:42 PM Chuck Lever III <[email-masked]> wrote:
> 
> 
>   That deals with the case of receiving one, but the big issue here (see below) concerns  the fact that  sending one seems to be forbidden in a lot of cases:

Right, so which cases? Can you provide an example or two?

> If there must be there is.   That's the way "must" (as opposed to "MUST") is defined.  in any case, the big issue concerns sending the ERROR. 
> 
> 
> 
> I don't see how a trivial ladder diagram will help things but here goes.   If this is a ladder, it leads downward, and that first step is a doozy.
> 
> Node B receives that message and would like do so but RFC8166 tells him he can only do so if he is the responder.   Since he doesn't understand the message, he has no way of determining that he is the responder since the format of the header is not known to him.

This leaves out some context. For example:

When would a sender send a Version 2 message to a receiver it
does not know can support it? My impression was that would be
only the first exchange right after a connection is established.

Moreover that initial exchange is going to be a forward-direction
exchange. Reverse-direction operation has to use the same version
as forward direction, according to Section 5.1 of RFC8167.

Why would a server send a Version 2 reply to a Version 1 Call?
Generally servers copy the version number from the Call to the
Reply header. I thought there was some language that required
that, but I can't find it now. Maybe that's all that is missing?

> The spec defines that the error fields stay the same for all versions but the requester/responder distinction is not the same for all transport versions.

Understood, but I'm just not seeing in what scenarios this
specification gap might cause a practical issue. I think we need
to understand this better to figure out what to file the errata
against. Is it 8166, 8167, or an issue against rpcrdma-version-two ?

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 7 Apr 2021 19:20:17 -0400

> Right, so which cases? Can you provide an example or two?

you receive a version two message when you don't support version two.

> > If there must be there is.   That's the way "must" (as opposed to
> "MUST") is defined.  in any case, the big issue concerns sending the ERROR.
> >
> >
> >
> > I don't see how a trivial ladder diagram will help things but here
> goes.   If this is a ladder, it leads downward, and that first step is a
> doozy.
> >
> > Node B receives that message and would like do so but RFC8166 tells him
> he can only do so if he is the responder.   Since he doesn't understand the
> format of the header is not known to him.
>
> does not know can support it?

My impression was that would be

That's my impression as well.

> exchange.

It's most likely to  be a message sent to report transport parameters.  I
don't consider that as either a forward or a reverse direction exchange.
YMMV.

Reverse-directionn operation has to use the same version
> as forward direction, according to Section 5.1 of RFC8167.

Good point.

> Why would a server send a Version 2 reply to a Version 1 Call?

The only case I can think of is if the recveiver does not support version
one.

> that, but I can't find it now. Maybe that's all that is missing?

That might be missing but you still have an issue for the first message you
receive.

>
>
> > The spec defines that the error fields stay the same for all versions
> but the requester/responder distinction is not the same for all transport
> versions.
>
> to understand this better to figure out what  file the errata
> against. Is it 8166,

Yes.  That has the troublesome text I quoted.

8167,

i never looked at 8167 and don't see why it would be an issue once 8166 is
xealt with.

orrr an issue against rpcrdma-version-two ?

I think we need to look at the upgrade path (to v3, for example) but there
is time for that. Nevertheless, for the v1 receiver we need to fix 8166.

---

From: "Marc Eshel" <[email-masked]>
Date: Wed, 7 Apr 2021 17:45:45 -0700

Hi All,

What is the statues of draft-hellwig-nfsv4-rdma-layout-00.txt ? 
Did it expire ?

Thanks, Marc.

---

From: David Noveck <[email-masked]>
Date: Wed, 7 Apr 2021 20:55:13 -0400

On Wed, Apr 7, 2021, 8:45 PM Marc Eshel <[email-masked]> wrote:

> Hi All,
>
> What is the statues of draft-hellwig-nfsv4-rdma-layout-00.txt?
>

as far as I know, ot is not being worked on.

Did it expire?

it did over two years ago.

Thanks, Marc.

---

From: Chuck Lever III <[email-masked]>
Date: Thu, 8 Apr 2021 16:20:36 +0000

> On Apr 7, 2021, at 7:20 PM, David Noveck <[email-masked]> wrote:
> 
> 
> 
> It's most likely to  be a message sent to report transport parameters.  I don't consider that as either a forward or a reverse direction exchange.  YMMV.
> 
> The only case I can think of is if the recveiver does not support version one.
> 
> That might be missing but you still have an issue for the first message you receive.
> 
> 
> i never looked at 8167 and don't see why it would be an issue once 8166 is xealt with.
> 
> I think we need to look at the upgrade path (to v3, for example) but there is time for that. Nevertheless, for the v1 receiver we need to fix 8166.

The problem statement is still nebulous and needs to be clarified
in order to file an actionable errata, IMO.

It's not clear to me yet that the only path forward is to change
RFC 8166. That kind of change might be difficult given that there
are implementations of v1 in the field, and some of them might be
immutable.

It's possible that v2 clients can be made to discover the server's
range of transport protocol versions in a way that aligns with
the current RFC 8166 restriction.

For example, the first operation on the wire could be an RPC NULL
message that is a a v1-style RDMA_MSG with an impossible value
in the rdma_vers field. A v1-only server would be able to sniff
the direction as it expects and respond with an ERR_VERS. The
client can then proceed using the highest transport version it
supports.

rpcrdma-version-two can be made to stipulate that a v2 server
MUST NOT send any message to the client peer until the transport
protocol version is sorted.

--
Chuck Lever

---

From: Christoph Hellwig <[email-masked]>
Date: Sat, 10 Apr 2021 08:19:01 +0200

> as far as I know, ot is not being worked on.

It did expire, but there is some very slow work going on to resurrect
it between me, David Black and Sorin Faibish.

---

From: David Noveck <[email-masked]>
Date: Mon, 12 Apr 2021 06:14:35 -0400

On Thu, Apr 8, 2021, 12:20 PM Chuck Lever III <[email-masked]>
wrote:

> > >> >    procedure.  A Responder MUST silently discard RDMA_ERROR
> procedures
> > >> > .
> > >> >
> > >   That deals with the case of receiving one, but the big issue here
> (see below) concerns  the fact that  sending one seems to be forbidden in a
> lot of cases:
> >
> > > Node B receives that message and would like do so but RFC8166 tells
> him he can only do so if he is the responder.   Since he doesn't understand
> the message, he has no way of determining that he is the responder since
> the format of the header is not known to him.
> >
> > It's most likely to  be a message sent to report transport parameters.
> I don't consider that as either a forward or a reverse direction exchange.
> YMMV.
> >
> > I think we need to look at the upgrade path (to v3, for example) but
> there is time for that. Nevertheless, for the v1 receiver we need to fix
> 8166.
>

Let me try to be as clear as I can. I hope this helps:

  - the penultimate paragraph of section 4.2.4 contains requirements on
ERROR messages that, while unfortunate, have been accepted because such
errors are rare and there is a possibility of fixing this issue in v2.

 - the existing text is not clear how these restrictions apply to version
errors since the existing text, relying on requester/responder roles, does
not provide any clear way to decide whether a message of a version other
than v1 is a request or a response.

  - the inability to make such a determination could make it impossible to
send a Version error in any circumstances, which would hobble rpc-over-rdma
vesion negotiation despite the fact that RFC8166 has devoted considerable
effort to make sure that version errors can be sent and interpreted by all
rpc-over-rdma implementations, irrespective of versions.

  - Clear restrictions on sending version errors need to be provided to
enable version negotiation. This will be important to enable prototyping of
v2.

> in order to file an actionable errata, IMO.

I guess I'll file an errata report and discuss how it might be acted upon.
That will give us something concrete to discuss.

> RFC 8166.

It seems to me that that is the only reasonable path.  The erratta report
will explain that path and we can discuss other proposals

Thatt kind of change might be difficult given that there
> are implementations of v1 in the field, and some of them might be

I find it hard to believe that there are rpc-over- rdma servers whose
implementers read the sentence in question as forbidding sending the client
a version error.  i think that if people had read it that way, we would
have had the discussion we are now having when those implementations were
written.

If such implementations exist, it would be a problem but there is no
indication that they do.  Once we start testing v2 clients, we will face
the issue but if there is a server that doesn't send a version error and is
immutable, then it will not support the version negotiation as described in
RFC 8166.  That would be unfortunate, but is not, in my opinion, a basis
for making this unfortunate behavior mandatory for all servers.

> the current RFC 8166 restriction.

Such hacks are possible but we should stay away from them even apart from
consideration of their metastatic potential.

We have some workable alternatives, but they all involve recognizing that
the current text of RFC8166 is wrong. This should have beem caught in
document review (mea culpa) but it wasn't.

> client can then proceed using the highest transport version it

For v2 it makes most sense to start with transport parameters including
authorization and it has been designed on that basis.  Going down this
hacky path would set v2 back in a major way.

> protocol version is sorted.

I have no objections to this sort of restriction and it appears unavoidable
:-(

However, I don't see doing this in a v2-specific way, since:

  - it would apply to all upgrades to versions beyond v1.

  - it needs to be relied on in explaining upgrade from v1 or as part of
the genaral description of version upgrade. (The latter would require a
bis, which I would like to avoid) for now

---

From: "Black, David" <[email-masked]>
Date: Mon, 12 Apr 2021 14:30:54 +0000

Actually, the layout draft that we're currently working on is the NVMe-oF layout (draft-hellwig-nfsv4-scsi-layout-nvme), not the RDMA layout draft.

Thanks, --David

-----Original Message-----
From: Christoph Hellwig <[email-masked]> 
Sent: Saturday, April 10, 2021 2:19 AM
To: David Noveck
Cc: Marc Eshel; NFSv4; [email-masked]; Black, David
Subject: Re: [nfsv4] draft-hellwig-nfsv4-rdma-layout-00.txt

[EXTERNAL EMAIL]

---

From: Christoph Hellwig <[email-masked]>
Date: Mon, 12 Apr 2021 16:46:16 +0200

> Actually, the layout draft that we're currently working on is the NVMe-oF layout (draft-hellwig-nfsv4-scsi-layout-nvme), not the RDMA layout draft.

Oh.  Indeed.  The RDMA layout has expired and I currently have no
short or mid-term plans to resurrect it.  It still is a good idea, but
it will need a lot of work.

---

From: Chuck Lever III <[email-masked]>
Date: Mon, 12 Apr 2021 15:07:50 +0000

On Apr 12, 2021, at 6:14 AM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:

On Thu, Apr 8, 2021, 12:20 PM Chuck Lever III <[email-masked]<mailto:[email-masked]>> wrote:

> On Apr 7, 2021, at 7:20 PM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
>
>
>
> On Wed, Apr 7, 2021, 4:06 PM Chuck Lever III <[email-masked]<mailto:[email-masked]>> wrote:
>
>
> > On Apr 7, 2021, at 1:37 PM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
> >
> >
> >
> > On Wed, Apr 7, 2021 at 12:42 PM Chuck Lever III <[email-masked]<mailto:[email-masked]>> wrote:
> >
> >
> >> > On Apr 7, 2021, at 12:36 PM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
> >> >

It does.

  - the penultimate paragraph of section 4.2.4 contains requirements on ERROR messages that, while unfortunate, have been accepted because such errors are rare and there is a possibility of fixing this issue in v2.

 - the existing text is not clear how these restrictions apply to version errors since the existing text, relying on requester/responder roles, does not provide any clear way to decide whether a message of a version other than v1 is a request or a response.

The context is that RFC 8166 defines a transport where the Requester and
Responder roles are unambiguous. The mistake is that we added reverse
direction support for v1, which introduced Requester/Responder ambiguity,
in a separate document.

I recall that the restriction at issue was an attempt to address that ambiguity
for RPC/RDMA procedures that do not carry an RPC message, like
RDMA_ERROR. It was added late in the process.

There's also the issue that Section 4.5 is written so that a version error is
supposed to be returned without looking at other fields in the header. It's
the first sanity check a receiver does before continuing to process the
rest of the header.

Would it be sufficient simply to drop the final sentence of the first
paragraph of Section 4.5?

  - the inability to make such a determination could make it impossible to send a Version error in any circumstances,

which would hobble rpc-over-rdma vesion negotiation despite the fact that RFC8166 has devoted considerable effort to make sure that version errors can be sent and interpreted by all rpc-over-rdma implementations, irrespective of versions.

  - Clear restrictions on sending version errors need to be provided to enable version negotiation. This will be important to enable prototyping of v2.

I guess I'll file an errata report and discuss how it might be acted upon. That will give us something concrete to discuss.

It seems to me that that is the only reasonable path.  The erratta report will explain that path and we can discuss other proposals

I find it hard to believe that there are rpc-over- rdma servers whose implementers read the sentence in question as forbidding sending the client a version error.  i think that if people had read it that way, we would have had the discussion we are now having when those implementations were written.

I'm not following you here. Servers would send a version error and
clients would take action by no longer using that version. I thought
the issue was that a server might send a reverse-direction Call to
a client using a version the client does not support. It is then the
client's problem to determine whether to send an RDMA_ERROR
back to the server.

And now we have the situation where a server can send a non-
RPC-bearing message to the client. The client has to be allowed
to send a rejection - "don't understand that version".

If such implementations exist, it would be a problem but there is no indication that they do.  Once we start testing v2 clients, we will face the issue but if there is a server that doesn't send a version error and is immutable, then it will not support the version negotiation as described in RFC 8166.  That would be unfortunate, but is not, in my opinion, a basis for making this unfortunate behavior mandatory for all servers.

Such hacks are possible but we should stay away from them even apart from consideration of their metastatic potential.

We have some workable alternatives, but they all involve recognizing that the current text of RFC8166 is wrong. This should have beem caught in document review (mea culpa) but it wasn't.

For v2 it makes most sense to start with transport parameters including authorization and it has been designed on that basis.  Going down this hacky path would set v2 back in a major way.

What makes it hacky?

I have no objections to this sort of restriction and it appears unavoidable :-(

  - it needs to be relied on in explaining upgrade from v1 or as part of the genaral description of version upgrade. (The latter would require a bis, which I would like to avoid) for now

Sorry, I didn't intend it to be a v2-only thing. I still lean towards the approach
that rpcrdma-version-two should update RFC 8166 in this area rather than
piling on more places that implementers have to look (ie, separate documents
and errata) to get their implementation working.

--
Chuck Lever

---

From: Chuck Lever III <[email-masked]>
Date: Mon, 12 Apr 2021 15:15:32 +0000

On Apr 12, 2021, at 11:07 AM, Chuck Lever III <[email-masked]<mailto:[email-masked]>> wrote:

It does.

No that's silly.

I meant dropping the entirety of the penultimate paragraph of Section 4.2.4.

What makes it hacky?

--
Chuck Lever

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

--
Chuck Lever