Subject: [nfsv4] outline for NFS/RDMA specification updates

From: Chuck Lever <[email-masked]>
Date: Mon, 20 Apr 2015 12:52:32 -0400

During the nfsv4 WG meeting at IETF 92, an update of the NFS/RDMA
standards was discussed. The chair (or someone? don’t remember)
requested an outline of the work. This e-mail contains my impression
of the room consensus and a proposed strategy for accomplishing the
work.

Deliverables:

1. A replacement for RFC 5666; RFC 5666 would become obsolete

   - Remove ambiguities that might allow spec-compliant
     implementations that do not interoperate
   - Call this effort “RFC 5666bis”

2. A replacement for RFC 5667; RFC 5667 would become obsolete

   - Replace section 5, which specifies an Upper Layer
     Binding for NFSv4.0
   - Clarify discussion of NFSv4 COMPOUND, callback, and
     RDMA-eligible operations
   - Fill in Upper Layer Binding for NFSv4.1, especially pNFS
     operations and NFSv4.1 backchannel
   - Possibly add an Upper Layer Binding for NFSv4.2
   - Call this effort “RFC 5667bis”

3. An experimental draft describing bi-directional RPC/RDMA

   - Enable prototype implementations for proof of concept
   - Identify capabilities that can be implemented in
     RPC/RDMA version 1, separate from capabilities that must
     be handled by a protocol version bump
   - Call this effort “RPC/RDMA bi-direction”

I’d like to propose a framework for getting all this done.

===

First, start work on deliverable 3.

RFC 5667bis will depend on 3., if we want to include a comprehensive
NFSv4.1 update in 5667bis. I’m also aware of other NFSv4.1
implementations (besides Linux and Solaris) that are showing some
interest in NFSv4.1 on RDMA now.

I believe a size-limited backwards direction capability is possible
without changing the RPC/RDMA version 1 XDR. A more generic capability
would likely require XDR changes and thus a protocol version bump.

This effort would determine if version 1 bi-direction is something
that can be standardized, and would describe the conventions well
enough to allow the construction of independent prototypes.

Dai Ngo and I have prototyped bi-direction with a Solaris server and
a Linux client, which pass basic interoperability tests. I have also
constructed an RPC/RDMA bi-direction Linux server prototype which is
undergoing initial testing now.

The week of April 27, I will deliver a first revision of a draft
document that describes what has been prototyped.

===

Once bi-direction in RPC/RDMA version one is underway, we can start
on RFC 5666bis (deliverable 1). RFC 5667bis will have normative
references to RFC 5666bis, so it needs to be completed concurrently
with or before RFC 5667bis.

There are two main clarifications:

  - RDMA_NOMSG and position zero chunks

  - padding and other details around inline content following a chunk

I’d like to consider other changes, like removing discussion of the
deprecated READ/READ mechanism, addition of an extensive glossary,
and possibly a description of a limited RPC bi-direction capability
(included from the above experimental draft if that is appropriate).

This work can be started any time, but completion of 5666bis is
strongly dependent on deliverable 3. With the exception of the
bi-direction piece, this should be almost entirely document
clean-up work.

===

After the disposition of the RPC components is well understood, RFC
5667bis (deliverable 2.) can be undertaken. Main purpose of 5667bis
is to replace RFC 5667 Section 5 in its entirety:

  - Add discussion of upper layer binding for NFSv4 COMPOUND

  - Clarify discussion of NFSv4 READ and WRITE COMPOUNDs

  - Replace NFSv4 SYMLINK with NFSv4 CREATE(symlink)

  - Update discussion of NFSv4.1 backchannel on RDMA transport

  - Provide an upper layer binding for NFSv4 callback operations

  - Provide an upper layer binding for pNFS-specific operations

Also I’d like to consider making additional NFSv4 operations eligible
for RDMA (namely {SET,GET}ACL, but others might be interesting too).

This work can be started any time, but completion of 5667bis is
strongly dependent on deliverables 1 and 3. Note also there may
be some prototyping effort for operations that are newly made
RDMA-eligible.

===

After the smoke has cleared, we should consider two additional
questions:

1. An NFSv4.2 Upper Layer Binding might be taken up as part of
RFC5667bis, but at this time NFSv4.2 is still seeing some changes.
Moreover, the additional prototyping might delay RFC 5667bis
unacceptably. An alternate plan would include only a provisional
NFSv4.2 Upper Layer Binding in RFC 5667bis.

2. There are several advanced RPC/RDMA features that might be
considered after the existing documents are updated. These would
likely require a protocol version bump, thus would be inappropriate
for RFC 5666bis.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 20 Apr 2015 18:14:23 +0000

Appreciate the level of detail, Chuck.  Everything that you mention looks appropriate and appropriately sequenced.

Any other comments from the WG?

Spencer

-----Original Message-----
From: nfsv4 [mailto:[email-masked]] On Behalf Of Chuck Lever
Sent: Monday, April 20, 2015 9:53 AM
To: NFSv4
Subject: [nfsv4] outline for NFS/RDMA specification updates

During the nfsv4 WG meeting at IETF 92, an update of the NFS/RDMA standards was discussed. The chair (or someone? don't remember) requested an outline of the work. This e-mail contains my impression of the room consensus and a proposed strategy for accomplishing the work.

Deliverables:

   - Remove ambiguities that might allow spec-compliant
     implementations that do not interoperate
   - Call this effort "RFC 5666bis"

   - Call this effort "RFC 5667bis"

   - Call this effort "RPC/RDMA bi-direction"

I'd like to propose a framework for getting all this done.

===

RFC 5667bis will depend on 3., if we want to include a comprehensive
NFSv4.1 update in 5667bis. I'm also aware of other NFSv4.1 implementations (besides Linux and Solaris) that are showing some interest in NFSv4.1 on RDMA now.

I believe a size-limited backwards direction capability is possible without changing the RPC/RDMA version 1 XDR. A more generic capability would likely require XDR changes and thus a protocol version bump.

This effort would determine if version 1 bi-direction is something that can be standardized, and would describe the conventions well enough to allow the construction of independent prototypes.

Dai Ngo and I have prototyped bi-direction with a Solaris server and a Linux client, which pass basic interoperability tests. I have also constructed an RPC/RDMA bi-direction Linux server prototype which is undergoing initial testing now.

The week of April 27, I will deliver a first revision of a draft document that describes what has been prototyped.

===

Once bi-direction in RPC/RDMA version one is underway, we can start on RFC 5666bis (deliverable 1). RFC 5667bis will have normative references to RFC 5666bis, so it needs to be completed concurrently with or before RFC 5667bis.

I'd like to consider other changes, like removing discussion of the deprecated READ/READ mechanism, addition of an extensive glossary, and possibly a description of a limited RPC bi-direction capability (included from the above experimental draft if that is appropriate).

This work can be started any time, but completion of 5666bis is strongly dependent on deliverable 3. With the exception of the bi-direction piece, this should be almost entirely document clean-up work.

===

After the disposition of the RPC components is well understood, RFC 5667bis (deliverable 2.) can be undertaken. Main purpose of 5667bis is to replace RFC 5667 Section 5 in its entirety:

Also I'd like to consider making additional NFSv4 operations eligible for RDMA (namely {SET,GET}ACL, but others might be interesting too).

This work can be started any time, but completion of 5667bis is strongly dependent on deliverables 1 and 3. Note also there may be some prototyping effort for operations that are newly made RDMA-eligible.

===

1. An NFSv4.2 Upper Layer Binding might be taken up as part of RFC5667bis, but at this time NFSv4.2 is still seeing some changes.
Moreover, the additional prototyping might delay RFC 5667bis unacceptably. An alternate plan would include only a provisional
NFSv4.2 Upper Layer Binding in RFC 5667bis.

2. There are several advanced RPC/RDMA features that might be considered after the existing documents are updated. These would likely require a protocol version bump, thus would be inappropriate for RFC 5666bis.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: David Noveck <[email-masked]>
Date: Mon, 20 Apr 2015 15:57:45 -0400

I think there was a short mention of RDMA as it applied to protocol
extension.  I think the conclusion was that draft-ietf-nfsv4-versioning
would have to state that it is responsibility of the definition document
for the new feature to specify where an added operation has chunkable
arguments/results.  I'm assuming that RFC5667bis would be written to be
compatible with that approach.

On Mon, Apr 20, 2015 at 2:14 PM, Spencer Shepler <[email-masked]>
wrote:

> standards was discussed. The chair (or someone? don't remember) requested
> an outline of the work. This e-mail contains my impression of the room
> consensus and a proposed strategy for accomplishing the work.
>
>
> Deliverables:
>
> NFSv4.1 update in 5667bis. I'm also aware of other NFSv4.1 implementations
> (besides Linux and Solaris) that are showing some interest in NFSv4.1 on
> RDMA now.
>
> This effort would determine if version 1 bi-direction is something that
> can be standardized, and would describe the conventions well enough to
> allow the construction of independent prototypes.
>
> Dai Ngo and I have prototyped bi-direction with a Solaris server and a
> Linux client, which pass basic interoperability tests. I have also
> Once bi-direction in RPC/RDMA version one is underway, we can start on RFC
> 5666bis (deliverable 1). RFC 5667bis will have normative references to RFC
> 5666bis, so it needs to be completed concurrently with or before RFC
> 5667bis.
>
> I'd like to consider other changes, like removing discussion of the
> deprecated READ/READ mechanism, addition of an extensive glossary, and
> possibly a description of a limited RPC bi-direction capability (included
> from the above experimental draft if that is appropriate).
>
> This work can be started any time, but completion of 5666bis is strongly
> dependent on deliverable 3. With the exception of the bi-direction piece,
> this should be almost entirely document clean-up work.
>
>
> ===
>
> 5667bis (deliverable 2.) can be undertaken. Main purpose of 5667bis is to
> replace RFC 5667 Section 5 in its entirety:
>
> Also I'd like to consider making additional NFSv4 operations eligible for
> RDMA (namely {SET,GET}ACL, but others might be interesting too).
>
> This work can be started any time, but completion of 5667bis is strongly
> dependent on deliverables 1 and 3. Note also there may be some prototyping
> effort for operations that are newly made RDMA-eligible.
>
>
> ===
>
> 1. An NFSv4.2 Upper Layer Binding might be taken up as part of RFC5667bis,
> but at this time NFSv4.2 is still seeing some changes.
> Moreover, the additional prototyping might delay RFC 5667bis unacceptably.
> An alternate plan would include only a provisional
> 2. There are several advanced RPC/RDMA features that might be considered
> after the existing documents are updated. These would likely require a
> protocol version bump, thus would be inappropriate for RFC 5666bis.
>
> --
> Chuck Lever