Subject: [nfsv4] Simpson RPC over RDMA introduction

From: William Allen Simpson <[email-masked]>
Date: Wed, 10 Aug 2016 06:52:59 -0400

I've been largely unable to participate due to health issues.  But
I've been asked to post my paper notes for discussion.  Someday,
I'll incorporate these into a complete replacement draft for V1,
which I've termed RFC-5666ter.

It does not seem to be useful to merely issue slight enhancements
for v2.  If we're going to do the work, we should make all the
known changes that are required to make RPC over RMDA more robust.

* Better error handling and recovery is more important than any
other feature.  This was the major downfall of V1.  Worse,
superficially compliant implementations didn't actually properly
implement the error handling of V1, necessitating V2.

* Divide the procedure from the header fields, in a backward
compatible manner.  This allows more than 1 procedure per send.

* Support larger sends.  There is already strong agreement in the WG.
This is much the same as Chuck's proposal, but not the same syntax.

* Support call and reply chaining.

* Always fill the send buffers for transmission efficiency, before
chaining.

* Consistent syntax for reply, write, and read buffer lists that
better support chaining.  These require new procedures and syntax.

These changes are extensive enough that Chuck has suggested they
should be version 3.  They cannot be version 3 and be backward
compatible with version 1, because the version 1 version error code
requires a range.  Because it cannot skip v2, this version either
MUST NOT support v1 or v2, or it MUST *be* v2.

I'd prefer to support v1 in the interim, as long as that interim
is relatively short.

Therefore, this is a proposed replacement for Chuck's personal
draft v2.

---

From: William Allen Simpson <[email-masked]>
Date: Wed, 10 Aug 2016 07:03:35 -0400

Fixed Header Fields

    The RPC-over-RDMA header begins with three fixed 32-bit fields that
    control the RDMA interaction, immediately followed by the first
    Procedure Number.

    These four fields must remain in the same positions and with the
    same semantics in all versions of the RPC-over-RDMA protocol.

    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                             XID                               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                        Version Number                         |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                         Credit Value                          |
    +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
    |                       Procedure Number                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

---

From: William Allen Simpson <[email-masked]>
Date: Wed, 10 Aug 2016 08:04:23 -0400

Procedure Numbers required

The following Procedure Numbers MUST be supported:

    o  0 = RDMA_HALT indicates that the list of Procedures is complete.

       Where the final procedure exactly fills the Inline Threshold,
       this MUST be omitted.

       This is backward compatible with Version One, as the procedure
       value has no Length field, and ensures that erroneous Version
       One Procedure 0 will terminate the request or response.

    o  1 is reserved.  All Version Two implementations fill the
       Inline Threshold as completely as possible.

    o  2 = RDMA_PAD allows alignment of following procedures on
       arbitrary 64-bit or 128-bit boundaries.

       Where the final procedure exactly fills the Inline Threshold,
       this MUST be omitted.

       This is backward compatible with Version One, as the procedure
       value has no Length field.  However, it MUST be followed by a
       later RDMA_HALT, or the Inline Threshold.

    o  3 is reserved.

    o  4 = RDMA_ERROR is used to signal an encoding error in the RPC-
       over-RDMA header.

    /// rpcrdma2_error {
    ///     rpcrdma_htype rdma_htype; /* 4 */
    ///     union rpcrdma2_errors switch (rpcrdma2_errcode rdma_err) {
    ///         case RDMA_ERR_VERS:
    ///             rpcrdma_err_vers rdma_vrange;  /* Immutable */
    ///         case RDMA_ERR_BADHEADER:
    ///             uint32           rdma_position;
    ///     };
    /// };

       Version Two RDMA_ERR_VERS is identical to Version One. This
       allows fall back to Version One.

       Version Two RDMA_ERR_BADHEADER has an additional rdma_position
       that indicate where in the stream an error occurs.  Where the
       Procedure Number is not recognized or supported, the position
       will indicate the rpcrdma_htype.  Where the value of a field in
       the procedure is not supported, the position will indicate the
       first byte of the value field.

       For transaction efficiency, multiple back-to-back rpcrdma2_error
       reports MUST be supported.  Version One RDMA_ERR_BADHEADER has
       no rdma_position field, but will ignore the trailing field, as
       Version One will not process multiple error procedures.

       This is backward compatible with Version One, as the procedure
       value has no Length field.

---

From: David Noveck <[email-masked]>
Date: Wed, 10 Aug 2016 08:09:32 -0400

> should be version 3.

I haven't seen them but I have no reason to doubt Chuck's judgment in this
regard.

> They cannot be version 3 and be backward

I don't understand your logic here.  The existing fixed fields allow you to
determine
if  your peer is using a version you don't support.  With regard to the
range returned
on version mismatch in the case, I don't see how it justifies your
conclusion.

All that is required is that if you, as an implementation that attempts to
use V1,
accesses an implementation that does not support V1, that implementation
must
give you back a range of versions it does support.  this is compatible with
implentations
that support v2 only, v3 only, or v2 and v3.  this independent of the
compatibility of
the protocols, once a common version has been found.

If you do want to have implementations that support V1 and V3, then a small
changes would
be needed to V2.  I think that could be done as as extension within v2,
when and if the
working group decides to go forward with a plan for version two.

> Therefore, this is a proposed replacement for Chuck's personal

We can discuss this in detail when there is a draft to discuss.  However, I
don't believe that the
working group should be constrained to pick one or the other.  I think
there will be people
who will be happy with the incremental improvements presented in the v2
framework and
others whose instincts are more revolution-based.  If there is sufficient
interest, I could
see both going forward.

On Wed, Aug 10, 2016 at 6:52 AM, William Allen Simpson <
[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: David Noveck <[email-masked]>
Date: Wed, 10 Aug 2016 08:22:09 -0400

I'd like to see an I-D when you are ready.  It should have  an introduction
explaining your goals in making this major restructuring.

---

From: William Allen Simpson <[email-masked]>
Date: Wed, 10 Aug 2016 08:38:48 -0400

[Resending with a little more detail for clarity.]

    /// rpcrdma2_halt {
    ///     rpcrdma_htype rdma_htype; /* 0 */
    /// };

    /// rpcrdma2_pad {
    ///     rpcrdma_htype rdma_htype; /* 2 */
    /// };

    o  3 is reserved.

    /// rpcrdma2_error {

---

From: William Allen Simpson <[email-masked]>
Date: Wed, 10 Aug 2016 09:38:30 -0400

Optional Chunk List Procedure Numbers

The following Procedure Numbers SHOULD be supported.  However, there
is no requirement that direct data placement be supported in the
presence of a sufficiently large Inline Threshold.

All procedure numbers greater than 4 have a rdma_hlength, so that
unrecognized procedures can be processed without knowing their format.

All RDMA chunk lists use a consistent format, so that such entries
can be chained and properly indicated in the reply:

    /// struct rpcrdma2_segment {
    ///         uint32 rdma_handle;
    ///         uint32 rdma_length;
    ///         uint64 rdma_offset;
    /// };
    ///
    /// struct rpcrdma2_chunks {
    ///         uint32                  rdma_position;
    ///         struct rpcrdma2_segment rdma_target<>;
    /// };
    ///
    /// struct rpcrdma2_chunk_list {
    ///         struct rpcrdma2_chunks     rdma_entry;
    ///         struct rpcrdma2_chunk_list *rdma_next;
    /// };
    ///

    o  5 = RDMA_RESULT indicates that an extended result is supported.

    /// rpcrdma2_result {
    ///     rpcrdma_htype rdma_htype; /* 5 */
    ///     uint32 rdma_hlength;
    ///     struct rpcrdma2_chunks result_chunks;
    /// };

       To avoid a dining philosopher problem, the optional RDMA_RESULT
       MUST be specified before other chunk lists.  This ensures that
       buffers for the potentially long result are able to be
       allocated before processing the message.

    o  6 = RDMA_WRITE indicates that RDMA Write is supported.

    /// rpcrdma2_write {
    ///     rpcrdma_htype rdma_htype; /* 6 */
    ///     uint32 rdma_hlength;
    ///     struct rpcrdma2_chunk_list write_list;
    /// };

       The optional RDMA_WRITE immediately follows any RDMA_RESULT,
       or is the first chunk list Procedure specified.

    o  7 = RDMA_READ indicates that RDMA Read is supported.

    /// rpcrdma2_read {
    ///     rpcrdma_htype rdma_htype; /* 7 */
    ///     uint32 rdma_hlength;
    ///     struct rpcrdma2_chunk_list read_list;
    /// };

       The optional RDMA_READ immediately follows any RDMA_RESULT and
       RDMA_WRITE or is the first chunk list Procedure specified.

       Because the RDMA_READ is not specified in the reply Send,
       the same buffer can be easily copied or reused for the
       preceding RDMA_RESULT and/or RDMA_WRITE without wasting a
       scatter/gather operation.

---

From: William Allen Simpson <[email-masked]>
Date: Wed, 10 Aug 2016 10:46:26 -0400

Optional Inline Threshold procedure

The following Procedure Number MAY be supported.

    o  8 = RDMA_THRESHOLD indicates that an extended RDMA Inline
       Threshold is supported.

    /// rpcrdma2_threshold{
    ///     rpcrdma_htype rdma_htype; /* 8 */
    ///     uint32 rdma_hlength;      /* 1 word */
    ///     uint32 rdma_threshold;    /* 1,000 .. 64,000 */
    /// };

       When present, the optional RDMA_THRESHOLD is always the first
       Procedure specified.  The RDMA_THRESHOLD can be changed at any
       time to reflect current conditions, but SHOULD NOT be sent
       during periods of unchanged conditions.

       Note that the RDMA_THRESHOLD in a request takes effect for the
       following result, while the RDMA_THRESHOLD in a result takes
       effect on subsequent requests.

       Due to the importance of acknowledging a significant change in
       setup, upon processing RDMA_THRESHOLD in a request, the result
       will also send RDMA_THRESHOLD (even with a default value).

---

From: Chuck Lever <[email-masked]>
Date: Wed, 10 Aug 2016 11:28:46 -0400

> On Aug 10, 2016, at 6:52 AM, William Allen Simpson <[email-masked]> wrote:
> 

RFC 5666bis is a documentation and archive project of a protocol
that is basically now read-only. I don't see how it's helpful
to create new protocol and document it in a new version of 5666.

> known changes that are required to make RPC over RMDA more robust.

I disagree with that statement as a whole and in its particulars.

I get the sense that you haven't read either the V2 draft or
Dave's rtrext draft at all. Much of this has already been proposed
using the protocol extensibility feature that was added to V2.

We've given you a cat-free sandbox to play in, but you are
ignoring it.

> implement the error handling of V1, necessitating V2.

I don't think it's fair to ding implementations for being
prototypes. Development was largely stopped after 2010,
and has only recently been picked up again. The protocol
issues are being addressed in new releases of the
implementations.

The problem here is that there are still old releases being
used that have bugs. That is a problem no matter what protocol
you care to discuss.

That said, there are some rare cases where the protocol
allows deadlocks in abnormal situations that might be
avoided with more error reporting precision.

More error fields are probably not necessary to fix that,
but a few new error codes and better spec language would
help.

> compatible manner.  This allows more than 1 procedure per send.

Please make a technical case for the need to send multiple
messages per send.

For example: Are we approaching some dire technological limit?
How is this helpful in a world that is full of storage that
is accessible only by explicit RDMA Read and Write operations?

A similar proposal was introduced in NFSv4, called COMPOUND.
So far, it is being used largely to send single operations
that are decorated in slightly different ways. Clients are
sending largely what they always have, COMPOUNDs are costly
to construct and parse, and server complexity is high.

Do you have a reason to believe that such an approach would
be used any differently in RPC-over-RDMA, and that it would
guarantee better transport efficiency?

> This is much the same as Chuck's proposal, but not the same syntax.

Already in V2.

To make those sends efficient, structured Receive buffers
would also be needed. That is proposed in rtrext.

> * Support call and reply chaining.

If you mean message continuation, it is already proposed
in rtrext.

However, some of us don't feel that a full implementation
is necessary. Most performance gain will arise with larger
inline thresholds and/or Remote Invalidation (ie the "simple
enhancements" being added to base V2).

Reply chunks, for example, are nearly as efficient as inline
sends when used with Remote Invalidation.

Also continuation adds complexity to RPC-over-RDMA credit
accounting that is difficult to get right.

A very clear data-based justification is needed here as well.

> * Always fill the send buffers for transmission efficiency, before

Again, this adds a lot of complexity to credit management;
it can be proposed as an extension to V2; and I believe
Dave already suggested this at one point.

And note that the price of combining messages into fewer
buffers is often additional latency and latency jitter.

If you have some prototypes or experiments or any other data
that demonstrates the need for this additional complexity,
please publish it or cite it.

> better support chaining.  These require new procedures and syntax.

Already proposed in rtrext.

> I'd prefer to support v1 in the interim, as long as that interim

Proposing a "yuge new protocol that will be great, believe me"
nearly guarantees that the interim will be long.

> Therefore, this is a proposed replacement for Chuck's personal

If there's something that needs to be in V2, make a case for
it or propose an extension.

Otherwise, it's senseless to talk about V3 right now,
especially because V2 extensibility enables everything you
have proposed here.

No-one I know is interested in a complete replacement of V1.
Replacing V1 throws away years of implementation experience
and a lot of working code. Good filesystem engineers are very
cautious and prefer incremental change over scorched earth.

Ganesha is in a position where there is no working code yet.
A flashy new protocol would not encounter the inertia of a
code or customer base who have data they would like to
retrieve reliably. It's easy to propose big changes from that
position. It's much more difficult to sell such changes to
enterprise customers who abhor change and cannot tolerate the
pain of immature implementations.

Also, I would prefer to read the proposals in I-Ds, not in
e-mails. Thanks.

--
Chuck Lever