Subject: [nfsv4] Expert review of NFSv4 RDMA private data draft (draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05)

From: "Black, David" <[email-masked]>
Date: Thu, 12 Dec 2019 22:06:20 +0000

With credit to Magnus (AD responsible for nfsv4 WG) for pinging me behind the scenes, I was able to obtain an expert review (see below) for the RDMA private data draft.

Please keep Dave Minturn  and Sean Hefty on cc: and take it easy on email volume, as they're doing the WG a favor.

Thanks, --David

From: Hefty, Sean <[email-masked]<mailto:[email-masked]>>
Sent: Monday, December 9, 2019 6:08 PM
To: Minturn, Dave B <[email-masked]<mailto:[email-masked]>>
Subject: RE: Question regarding RDMA CM Private Data

Just reading your comments, CM private data is the correct name.  The amount of data present is defined by the spec, with a portion of the data consumed by the RDMA CM protocol itself.  I *think* there's something like 48 bytes to play with on connection requests.

I agree that the format specifier is useless.  There are many existing users which could be setting the private data to anything.  You really need to rely on the port numbers being correct, with checks on the other fields detecting mismatched protocols.  The private data is and has been exposed directly to user space applications for years, so anything there is fair play.

In order to standardize the private data format, you would need to change the version of the RDMA CM header carried in the private data of the underlying transport.

The field sizes for the send/recv sizes look too small to me, unless those fields are scaled somehow.

From: Minturn, Dave B <[email-masked]<mailto:[email-masked]>>
Sent: Monday, December 9, 2019 5:03 PM
To: Hefty, Sean <[email-masked]<mailto:[email-masked]>>
Subject: Question regarding RDMA CM Private Data

Hi Sean,

David Black contacted me and asked if I could review an IETF RFC for NFS/RDMA's use of RDMA CM's private data.  The RFC reference is:  https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.pdf
It would be great if you could take a look.  I called out some items I saw but you are the expert in this area.
..Dave

Here are some items that I had questions/concerns with: (Section 4)

When an RPC-over-RDMA version 1 transport connection is
established, the client (which actively establishes connections) and
the server (which passively accepts connections) populate the CM
Private Data field exchanged as part of CM connection establishment.

>>  Is "CM Private Data" the proper name?

For RPC-over-RDMA version 1, the CM Private Data field is formatted
as described in the following subsection. RPC clients and servers
use the same format. If the capacity of the Private Data field is
too small to contain this message format, the underlying RDMA
transport is not managed by a Connection Manager, or the underlying
RDMA transport uses Private Data for its own purposes, the CM Private
Data field cannot be used on behalf of RPC-over-RDMA version 1.

>> This statement seemed unnecessary because it's transparent to the RFC ULP if the RDMA transport
>> is using Private Data.  Seems more appropriate to say something about the <capacity> of the Private Data
>> field length being RDMA transport dependent.
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Format Identifier                                                                                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version           | Flags         | Send Size               | Receive Size      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Format Identifier: This field contains a fixed 32-bit value that
identifies the content of the Private Data field as an RPC-over-
RDMA version 1 CM Private Data message. In RPC-over-RDMA version
1 Private Data, the value of this field is always 0xf6ab0e18, in
network byte order. The use of this field is further expanded
upon in Section 4.1.
4.1.2. Amongst Implementations of Other Upper-Layer Protocols
The Format Identifier field in the message format defined in this
document is provided to enable implementations to distinguish RPC-
over-RDMA version 1 Private Data from application-specific private
data inserted by applications other than RPC-over RDMA version 1.
Examples of other applications that make use of CM Private Data
include iWARP, via the MPA enhancement described in [RFC6581], and
iSCSI extensions for RDMA (iSER), as defined in [RFC7145].
During connection establishment, an implementation of the extension
described in this document checks the Format Identifier field before
decoding subsequent fields. If the RPC-over-RDMA version 1 CM
Private Data Format Identifier is not present as the first 4 octets,
an RPC-over-RDMA version 1 receiver MUST ignore the CM Private Data,
behaving as if no RPC-over-RDMA version 1 Private Data has been
provided (see above).
>>  It seems wrong to assume that no other RDMA ULP will put that "magic #" in.   Maybe
>> if combined with a well know port, but even then it's for sure.  Now that I said that,
>> I found the following text proposing using IANA to standardize the RDMA_CM private data
>>
. IANA Considerations
In accordance with [RFC8126], the author requests that IANA create a
new registry in the "Remote Direct Data Placement" Protocol Category
Group. The new registry is to be called the "RDMA-CM Private Data
Lever Expires May 11, 2020 [Page 8]
Internet-Draft RPC-Over-RDMA CM Private Data November 2019
Identifier Registry". This is a registry of 32-bit numbers that
identify the upper-layer protocol associated with data that appears
in the application-specific RDMA-CM Private Data area. The fields in
this registry include: Format Identifier, Description, and Reference.
The initial contents of this registry are a single entry:
+------------------+------------------------------------+-----------+
| Format | Format Description | Reference |
| Identifier | | |
+------------------+------------------------------------+-----------+
| 0xf6ab0e18 | RPC-over-RDMA version 1 CM Private | [RFC-TBD] |
| | Data | |
+------------------+------------------------------------+-----------+
Table 1: RDMA-CM Private Data Identifier Registry
IANA is to assign subsequent new entries in this registry using the
Expert Review policy as defined in Section 4.5 of [RFC8126].

---

From: Chuck Lever <[email-masked]>
Date: Thu, 12 Dec 2019 17:30:46 -0500

> On Dec 12, 2019, at 5:06 PM, Black, David <[email-masked]> wrote:
> 
> Please keep Dave Minturn  and Sean Hefty on cc: and take it easy on email volume, as they’re doing the WG a favor.

My thanks to the reviewers. A few comments inline.

> Thanks, --David
>  
>  <>From: Hefty, Sean <[email-masked] <mailto:[email-masked]>> 
> To: Minturn, Dave B <[email-masked] <mailto:[email-masked]>>
> Just reading your comments, CM private data is the correct name.  The amount of data present is defined by the spec, with a portion of the data consumed by the RDMA CM protocol itself.  I *think* there’s something like 48 bytes to play with on connection requests.

If that 48-byte limit is documented somewhere, this draft can easily reference it. I was never able to find such a document.

> I agree that the format specifier is useless.  There are many existing users which could be setting the private data to anything.  You really need to rely on the port numbers being correct, with checks on the other fields detecting mismatched protocols.

Other reviewers have been ambivalent about the format identifier, but we don't have a better solution at the moment. The problem is the port numbers are all arbitrary. AFAIK they are not used in a way that identify the connection as RPC.

The 2049 destination port number means "NFS" but that's a layer above RPC/RDMA.

RPC/RDMA also does not define a ToS. That would be a sure way to distinguish the traffic the connection is to carry.

> The private data is and has been exposed directly to user space applications for years, so anything there is fair play.
>  

Which is outside the purview of the IETF.

> The field sizes for the send/recv sizes look too small to me, unless those fields are scaled somehow.

The values in the inline threshold fields are scaled; see Section 5.2. The actual size range of these buffers is 1KB to 256KB.

> From: Minturn, Dave B <[email-masked] <mailto:[email-masked]>> 
> To: Hefty, Sean <[email-masked] <mailto:[email-masked]>>
> David Black contacted me and asked if I could review an IETF RFC for NFS/RDMA’s use of RDMA CM’s private data.  The RFC reference is:  https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.pdf <https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.pdf>
> >>  Is “CM Private Data” the proper name? 
>  
> >> This statement seemed unnecessary because it’s transparent to the RFC ULP if the RDMA transport
> >>  It seems wrong to assume that no other RDMA ULP will put that “magic #” in.   Maybe
> >> if combined with a well know port, but even then it’s for sure.  Now that I said that,
> Identifier Registry".. This is a registry of 32-bit numbers that
> _______________________________________________
> nfsv4 mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>
--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Thu, 12 Dec 2019 20:04:43 -0500

On 12/12/2019 5:30 PM, Chuck Lever wrote:
> 
>> On Dec 12, 2019, at 5:06 PM, Black, David <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>> With credit to Magnus (AD responsible for nfsv4 WG) for pinging me 
>> behind the scenes, I was able to obtain an expert review (see below) 
>> for the RDMA private data draft.
>> Please keep Dave Minturn  and Sean Hefty on cc: and take it easy on 
>> email volume, as they’re doing the WG a favor.
> 
>> *From:*Hefty, Sean <[email-masked] <mailto:[email-masked]>>
>> *Sent:*Monday, December 9, 2019 6:08 PM
>> *To:*Minturn, Dave B <[email-masked] 
>> <mailto:[email-masked]>>
>> *Subject:*RE: Question regarding RDMA CM Private Data
>> Just reading your comments, CM private data is the correct name.  The 
>> amount of data present is defined by the spec, with a portion of the 
>> data consumed by the RDMA CM protocol itself.  I **think** there’s 
>> something like 48 bytes to play with on connection requests.
> 
> If that 48-byte limit is documented somewhere, this draft can easily 
> reference it. I was never able to find such a document.

It's not necessary, in my opinion. The limit is different for each
transport protocol, and all that matters for this RFC is that there
is "enough" to carry the payload it defines. Any such reference would
be purely informative, IOW.

>> I agree that the format specifier is useless. There are many existing 
>> users which could be setting the private data to anything.  You really 
>> need to rely on the port numbers being correct, with checks on the 
>> other fields detecting mismatched protocols.
> 
> Other reviewers have been ambivalent about the format identifier, but we 
> don't have a better solution at the moment. The problem is the port 
> numbers are all arbitrary. AFAIK they are not used in a way that 
> identify the connection as RPC.
> 
> The 2049 destination port number means "NFS" but that's a layer /above/ 
> RPC/RDMA.

This is not the purpose of the identifier. It is there to signal to the
receiver that the payload is the one defined in the document. If the
identifier is not present, the receiver will ignore the private data.
This means the identifier must be defined, and present. It's not meant
to be universal. Yes, there's a risk that another layer injects its own
private data, with a payload somehow containing the identifier. That
might warrant a sentence or two.

> RPC/RDMA also does not define a ToS. That would be a sure way to 
> distinguish the traffic the connection is to carry.

Nor do all the RDMA transports. Out of scope.

Tom.

>> The private data is and has been exposed directly to user space 
>> applications for years, so anything there is fair play.
>> In order to standardize the private data format, you would need to 
>> change the version of the RDMA CM header carried in the private data 
>> of the underlying transport.
> 
> The values in the inline threshold fields are scaled; see Section 5.2. 
> The actual size range of these buffers is 1KB to 256KB.
> 
> 
>> *From:*Minturn, Dave B <[email-masked] 
>> <mailto:[email-masked]>>
>> *Sent:*Monday, December 9, 2019 5:03 PM
>> *To:*Hefty, Sean <[email-masked] <mailto:[email-masked]>>
>> *Subject:*Question regarding RDMA CM Private Data
>> Hi Sean,
>> David Black contacted me and asked if I could review an IETF RFC for 
>> NFS/RDMA’s use of RDMA CM’s private data.  The RFC reference is: 
>> https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.pdf 
>> <https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05..pdf>
>> /When an RPC-over-RDMA version 1 transport connection is/
>> /established, the client (which actively establishes connections) and/
>> /the server (which passively accepts connections) populatethe CM/
>> /Private Data//field exchanged as part of CM connection establishment./
>> //
>> /For RPC-over-RDMA version 1, the CM Private Data field is formatted/
>> /as described in the following subsection. RPC clients and servers/
>> /use the same format. If the capacity of the Private Data field is/
>> /too small to contain this message format, the underlying RDMA/
>> /transport is not managed by a Connection Manager,or the underlying/
>> /RDMA transport uses Private Data for its own purposes,//the CM Private/
>> /Data field cannot be used on behalf of RPC-over-RDMA version 1./
>> //
>> /0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1/
>> /+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
>> /| Format Identifier 
>>                                                                                |/
>> /+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
>> /| Version           | Flags         | Send Size               | 
>> Receive Size      |/
>> /+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+/
>> /Format Identifier: This field contains a fixed 32-bit value that/
>> /identifies the content of the Private Data field as an RPC-over-/
>> /RDMA version 1 CM Private Data message. In RPC-over-RDMA version/
>> /1 Private Data, the value of this field is always 0xf6ab0e18, in/
>> /network byte order. The use of this field is further expanded/
>> /upon in Section 4.1./
>> /4.1.2. Amongst Implementations of Other Upper-Layer Protocols/
>> /The Format Identifier field in the message format defined in this/
>> /document is provided to enable implementations to distinguish RPC-/
>> /over-RDMA version 1 Private Data from application-specific private/
>> /data inserted by applications other than RPC-over RDMA version 1.///
>> /Examples of other applications that make use of CM Private Data/
>> /include iWARP, via the MPA enhancement described in [RFC6581], and/
>> /iSCSI extensions for RDMA (iSER), as defined in [RFC7145]./
>> /During connection establishment, an implementation of the extension/
>> /described in this document checks the Format Identifier field before/
>> /decoding subsequent fields. If the RPC-over-RDMA version 1 CM/
>> /Private Data Format Identifier is not present as the first 4 octets,/
>> /an RPC-over-RDMA version 1 receiver MUST ignore the CM Private Data,/
>> /behaving as if no RPC-over-RDMA version 1 Private Data has been/
>> /provided (see above)./
>> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> --
> Chuck Lever
> 
> 
> 
> 
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Tom Talpey <[email-masked]>
Date: Thu, 12 Dec 2019 20:07:26 -0500

[forgot to cc the reviewers]

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 13 Dec 2019 13:29:54 -0500

> On Dec 12, 2019, at 8:07 PM, Tom Talpey <[email-masked]> wrote:
> 
>>> On Dec 12, 2019, at 5:06 PM, Black, David <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> *To:*Minturn, Dave B <[email-masked] <mailto:[email-masked]>>
>>> Just reading your comments, CM private data is the correct name.  The amount of data present is defined by the spec, with a portion of the data consumed by the RDMA CM protocol itself.  I **think** there’s something like 48 bytes to play with on connection requests.
>> The 2049 destination port number means "NFS" but that's a layer /above/ RPC/RDMA.
> 

Then I think the only modification needed for this document is in
Section 4.1.2. That is fairly young language anyway. We can take
that editorial work back to only [email-masked], and incorporate
the change into the document as the IESG review proceeds.

Everyone OK with that?

>>> *From:*Minturn, Dave B <[email-masked] <mailto:[email-masked]>>
>>> David Black contacted me and asked if I could review an IETF RFC for NFS/RDMA’s use of RDMA CM’s private data.  The RFC reference is: https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05.pdf <https://tools.ietf.org/pdf/draft-ietf-nfsv4-rpcrdma-cm-pvt-data-05..pdf>
>>> /| Format Identifier                                                                                |/
>>> /| Version           | Flags         | Send Size               | Receive Size      |/

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 13 Dec 2019 14:46:45 -0500

On Fri, Dec 13, 2019, 1:30 PM Chuck Lever <[email-masked]> wrote:

> Section 4.1.2.

I think so.

That is fairly young language anyway. We can take
> the change into the document as the IESG review proceeds.

I'm OK with it.

--