Subject: [nfsv4] roadmap for completion of rfc5667bis

From: Chuck Lever <[email-masked]>
Date: Mon, 27 Mar 2017 10:49:49 -0400

Hi-

Recently there were some open questions about the completeness of rfc5667bis
(now titled "Network File System (NFS) Upper Layer Binding To RPC-Over-RDMA
Version One"). Here are the issues I feel that remain to be addressed before
WG LC.

1. There is a binding in Section 4.1 provided for MOUNT, NLM, and NSM. This
was added because the bindings are straightforward.

However no RPC-over-RDMA implementations of these protocols exist. All
existing NFS/RDMA implementations use TCP for these protocols. This is
because these protocols depend on rpcbind, and no binding for rpcbind
exists.

One or more of the following things could happen:

- The bindings for MOUNT, NLM, and NSM could be removed from rfc5667bis.
This reflects current practice.

- A remark could be added that is similar to the remark in Section 8.2.2.1
of rfc5666bis that requires NFS/RDMA implementations to support these
protocols on UDP and TCP. This documents current practice.

- The RPC binding of these protocols must be fixed and known in advance
when using a RPC-over-RDMA transport. This document could request that
IANA assign fixed ports for MOUNT, NLM, and NSM on RPC-over-RDMA.

2. The original RFC 5667 permits implementations to use multiple Read and
Write chunks.

Because it hasn't had NFS/RDMA with Kerberos support until recently, one
implementation currently does not support certain configurations of chunks:
- Write + Reply
- PZRC + normal Read
- multiple normal Read

However, one implementation does support all of these. Therefore, IMO that
resolves the issue of whether rfc5667bis documents current implementation
practice by allowing multiple chunks per RPC: it does.

The question remains how implementations might discover what is supported
by their peers (and perhaps, what is the peer's limit on number of chunks
per RPC).

Server implementations should report ERR_CHUNK (or perhaps GARBAGE_ARGS)
whenever a client has sent more chunks than is supported, or a configuration
of chunks that is unsupported. Implementation guidance should be added to
rfc5667bis to discuss this scenario.

3. rfc5667bis introduces a DDP-eligibility statement about the NFSv4.2
READ_PLUS operation. However, after further thought, I have concluded that
perhaps READ_PLUS CONTENT cannot reasonably be a candidate for Direct Data
Placement.

A READ_PLUS reply requires more marshaling and unmarshaling activity on
both peers than a READ reply. Section 6.1 of rfc5666bis says a data item
is a candidate for DDP eligibility when:

   o  Transport-level processing of the XDR data item is not needed.
      For example, the data item is an opaque byte array, which requires
      no XDR encoding and decoding of its content.

CONTENT itself is an opaque array, but it is contained in a larger data
structure which does need to be packed and unpacked. Further:

   o  The content of the XDR data item is sensitive to address
      alignment.  For example, pullup would be required on the receiver
      before the content of the item can be used.

When returning a Write chunk, the server is required to start writing at
offset zero in that chunk. This is not problematic when READ_PLUS returns
a single CONTENT arm.

If the returned data starts with a hole, however, the client would be
responsible for moving subsequent CONTENT arms and then inserting that
hole in the receive buffer, thus negating any possible benefits of direct
data placement.

Thus I propose that READ_PLUS should not have any DDP-eligible data items.
Section 5.1.1 should be removed, and mention of READ_PLUS should be
stricken from Section 5.4.1.

Next Steps:

It is possible these issues can be resolved here on the mailing list. If
they are not, a design conference call can be scheduled in the next few
weeks.

Subsequent to the resolution of these issues and submission of a fresh
document revision, I feel this document will be ready for WG LC. I'd
like to see WG LC completed in mid-May.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 27 Mar 2017 14:46:36 -0400

> stricken from Section 5.4.1.

Agree.  You also might delete the fifth bullet from section 5.4.1 since
READ_PLUS
was the only possible case this could have been referring to.

On Mon, Mar 27, 2017 at 10:49 AM, Chuck Lever <[email-masked]>
wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Mike Kupfer <[email-masked]>
Date: Tue, 28 Mar 2017 11:17:21 -0700

> because these protocols depend on rpcbind, and no binding for rpcbind

I don't see much point in using RDMA for these protocols (though I can't
claim a deep understanding of the issues).  So I'm inclined to remove
them from 5667bis.

> protocols on UDP and TCP. This documents current practice.

That sounds okay to me.

mike

---

From: Chuck Lever <[email-masked]>
Date: Tue, 28 Mar 2017 13:37:21 -0500

> On Mar 28, 2017, at 1:17 PM, Mike Kupfer <[email-masked]> wrote:
> 
> Chuck Lever wrote:
> 
> claim a deep understanding of the issues).

There is no opportunity for direct data placement,
but using RDMA Send for basic RPC operations is
significantly faster than sending the same operations
on a TCP socket. For NLM, at least, there would be
a performance advantage.

> So I'm inclined to remove them from 5667bis.

That is probably safe; if the WG ever wants to make
this work, a document describing the operation of
rpcbind and the other protocols on RPC-over-RDMA
could be created. It could even update 5667bis,
but that doesn't seem necessary.

--
Chuck Lever

---

From: Mike Kupfer <[email-masked]>
Date: Tue, 28 Mar 2017 13:27:11 -0700

> on a TCP socket. For NLM, at least, there would be

Thanks; I didn't know that about RDMA Send.  Does anyone have numbers
that break out the performance of sample NLM calls into over-the-wire
time and service procedure time?

> but that doesn't seem necessary.

That sounds fine to me.

mike