Subject: [nfsv4] Requirements for bi-directional RPC/RDMA

From: Chuck Lever <[email-masked]>
Date: Thu, 23 Oct 2014 12:29:29 -0400

An active discussion about how to implement NFSv4.1 backchannel
for RDMA-capable NFS clients is going on over on
[email-masked]. See:

  http://marc.info/?l=linux-nfs&m=141407115607977&w=2

Some questions for this group are:

Does RFC 5661, 5666, or 5667 contain a requirement or
recommendation for clients or servers to support bi-directional
RPC/RDMA ? Is there a RECOMMENDED approach, or is it entirely
at an implementer’s discretion?

If there is no requirement for bi-directional RPC/RDMA, how is
interoperation to be guaranteed between an implementation that
supports bi-directional RPC/RDMA and one that does not? In
particular, are RDMA-capable NFSv4.1 servers required to support
bi-directional RPC/RDMA?

If servers are not required to support bi-directional RPC/RDMA,
one recovery method is to establish a TCP connection to the
server and use that for backchannel traffic. Would require a
minimalist form of session trunking.

If the client does not support bi-directional RPC/RDMA and
chooses not to use the above approach, an alternative is to
squelch client recovery when SEQ4_STATUS_CB_PATH_DOWN is
asserted in a SEQUENCE reply on the RDMA forechannel. Would
disable features that require a callback channel, such as
delegation; and some servers might waste effort attempting
to check the client’s callback service on every forechannel
operation.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Chuck Lever <[email-masked]>
Date: Fri, 24 Oct 2014 14:21:34 -0400

On Oct 23, 2014, at 12:29 PM, Chuck Lever <[email-masked]> wrote:

> RPC/RDMA ?

I’ve tried to answer the first question myself. I’d
like a sanity check on this abstract analysis.

RFC 5666 seems to contain nothing relevant.

RFC 5667 section 5.1 discusses NFSv4 callbacks.

The first paragraph discusses how to encode NFSv4 CB
RPCs, regardless of whether they use legacy RPC or
bi-directional RPC.

The remaining paragraphs discuss various implementation
pitfalls and requirements regarding how the direction
of a message is determined in the bi-directional case.

I don’t see a requirement or recommendation for a
particular mechanism for supporting the NFSv4.1
backchannel when the forechannel is RDMA.

RFC 5661 section 2.9 discusses transport requirements,
and 2.10 discusses requirements around NFSv4.1 sessions.

I see two possible relevant requirements:

A legal NFSv4.1 transport is required to support reliable
and in-order transmission of data. I found no stated
requirement that a legal NFSv4.1 transport must support
bi-directional RPC.

Section 2.10 requires that all NFSv4.1+ servers support
session backchannel, and they must support session
trunking when the server IP address is the same on all
connections associated with a session. (This requires
NFSv4.1 servers to support a side car backchannel).

There is also a statement that it is permissible for the
session backchannel and forechannel to use not only
different connections (trunking) but different transports
(eg, RDMA and TCP).

I do not see an explicit requirement that any particular
transport must support bi-directional RPC. Is there any
_implied_ requirement that an NFSv4.1 server
implementation that supports RPC/RDMA must support
bi-directional RPC/RDMA?

Even the TCP transport is not required to support
bi-directional RPC. There is an indirect recommendation
that bi-directional RPC should be supported on at least
one valid transport supported by a server; the text
strongly recommends implementation of a backchannel,
which works only with bi-directional RPC on NFSv4.1.

> Is there a RECOMMENDED approach, or is it entirely

Because CREATE_SESSION enables a client to detect
whether the server supports bi-directional RPC, and
because there appear to be no other restrictions in
the specifications, IMO NFSv4.1 client and server
implementers are free to choose whether or not to
implement bi-directional RPC/RDMA.

Today, the Linux client will fail to mount any server
that does not have bi-directional RPC/TCP. But it
seems to me that is a perfectly legal configuration
(though never used in practice).