Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5666bis-03.txt

From: [email-masked]
Date: Mon, 25 Jan 2016 12:18:24 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Network File System Version 4 Working Group of the IETF.

        Title           : Remote Direct Memory Access Transport for Remote Procedure Call
        Authors         : Charles Lever
                          William Allen Simpson
                          Tom Talpey
	Filename        : draft-ietf-nfsv4-rfc5666bis-03.txt
	Pages           : 51
	Date            : 2016-01-25

Abstract:
   This document specifies a protocol for conveying Remote Procedure
   Call (RPC) messages on physical transports capable of Remote Direct
   Memory Access (RDMA).  It requires no revision to application RPC
   protocols or the RPC protocol itself.  This document obsoletes RFC
   5666.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5666bis/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-nfsv4-rfc5666bis-03

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-rfc5666bis-03

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Chuck Lever <[email-masked]>
Date: Mon, 25 Jan 2016 15:24:38 -0500

Begin forwarded message:

> From: [email-masked]
> Date: January 25, 2016 at 3:18:24 PM EST
> To: <[email-masked]>
> Cc: [email-masked]
> Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5666bis-03.txt
> 
> 

This will be the final revision for a while. Section 9 has been restructured
and rewritten based on recent comments from Dave Noveck. It probably
needs further fleshing out.

The rest of the document may need some word-smithing but I consider
it organizationally complete.

As always, comments are encouraged.

Meanwhile, I think it is time for one or more conference calls to discuss
this document and other RDMA-related work. Spencer, can you host a
call sometime in the next few weeks (ideally before Connectathon)?

---

From: William Allen Simpson <[email-masked]>
Date: Sun, 31 Jan 2016 17:22:12 -0500

> needs further fleshing out.
Or not.  In fact, I'd prefer it to be made smaller.  I didn't
see it before posting, and apparently a fair amount of
discussion was private.

> it organizationally complete.
Folks, please read this draft! Chuck has done an outstanding job.

I've found a couple of "the use of the" (should just be "use of the"),
but it holds together really well now.

> call sometime in the next few weeks (ideally before Connectathon)?
My hope is we can all make sure our code matches this draft, or
document where it doesn't match, and test at Connectathon, and
then have one or at most two drafts before publication.
3

---

From: William Allen Simpson <[email-masked]>
Date: Sun, 31 Jan 2016 17:56:29 -0500

This was extensively re-written, and I didn't see it until now.

9.1.  Extending RPC-over-RDMA Header XDR

    The first four fields in the RPC-over-RDMA header must remain aligned
    at the same fixed offsets for all versions of the RPC-over-RDMA
    protocol.  The version number must be in a fixed place in order for
    version mismatches to be detected.  For version mismatches to be
    reported in a fashion that all future version implementations can
    reliably decode, the rdma_proc field must be in a fixed place, the
    value of RDMA_ERR_VERS must always remain the same, and the field
    placement of the RDMA_ERR_VERS arm of the rpcrdma1_error union must
    always remain the same.

So far, so good!

    Given these constraints, one way to extend RPC-over-RDMA is to add
    new values to the rdma_proc enumerated type and new components (arms)
    to the rpcrdma1_body union.  New argument and result types may be
    introduced for each new procedure defined this way.  These extensions
    would be specified by new Internet Drafts with appropriate Working
    Group and IESG review to ensure continued interoperation with
    existing implementations.

This is a bit of a problem.  The recent change introduced:

       union rpcrdma1_body switch (rpcrdma1_proc rdma_proc) {
               case RDMA_MSG:
               case RDMA_NOMSG:
                 rpcrdma1_chunks rdma_chunks;
               case RDMA_MSGP:
                 uint32          rdma_align;
                 uint32          rdma_thresh;
                 rpcrdma1_chunks rdma_achunks;
               case RDMA_DONE:
                 void;
               case RDMA_ERROR:
                 rpcrdma1_error rdma_error;
       };

First of all, RDMA_MSGP and RDMA_DONE are now reserved, and
their old definitions should be removed.

Secondly, having the union body may be syntactically correct,
but extensions really need the rpcrdma1_proc field to stand
alone.  The above means the body has to be duplicated in
each extension document.  As soon as we have two extensions,
they will conflict and won't compile.

    XDR extensions may introduce only optional features to an existing
    RPC-over-RDMA protocol version.  To detect when an optional rdma_proc
    value is supported by a receiver, it is desirable to have a specific
    value of the rdma_err field, say, RDMA_ERR_PROC, that indicates when
    the receiver does not recognize an rdma_proc value.

It should be added in this document.  Obviously missing protocol
errors should be fixed.

Older implementations won't recognize the error, but (to quote
section 5.5.  Error Handling,

    To avoid an infinite loop, a receiver should drop an RDMA_ERROR
    procedure that is malformed.

We continue:

    In RPC-over-RDMA Version One, a receiver can indicate that it does
    not recognize an rdma_proc enum value only by returning an RDMA_ERROR
    procedure with the rdma_err field set to RDMA_ERR_CHUNK (see
    Section 5.5.2).  This is indistinguishable from a situation where the
    receiver does indeed support the procedure, but the XDR is malformed.

Correct.  Also, RDMA_ERR_CHUNK causes the connection to terminate.

    To resolve this problem, an RPC-over-RDMA Version One sender uses the
    following convention.  If the first time the sender uses an optional
    rdma_proc value the receiver returns an RDMA_ERROR procedure with
    RDMA_ERR_CHUNK in the rdma_err field, the sender simply marks that
    feature as unsupported and does not send it again on the current
    connection instance.  Subsequent to an initial successful result,
    receiving RDMA_ERR_CHUNK retains its more relaxed meaning of "generic
    XDR parsing error."

The threads could very well return out of order.  I've got no
indicators about the order the requests originally arrived.
Overloading this doesn't seem like a good idea.

    To ensure backwards compatibility when such an extension mechanism is
    in place, the value of RDMA_ERR_CHUNK must remain the same for all
    versions of the RPC-over-RDMA protocol.

Agreed, but not for the reasons stated.