Subject: [nfsv4] Fwd: New Version Notification for draft-cel-nfsv4-rpcrdma-bidirection-01.txt

From: Chuck Lever <[email-masked]>
Date: Wed, 20 May 2015 15:19:17 -0400

Hi-

This draft has been revised. The revisions include:

 - A discussion on avoiding XID collision in the forward
   and backward direction has been added

 - The “Payload Size” subsection has been rewritten for clarity

 - Review comments since -00 have been addressed (thanks!)

Begin forwarded message:

> From: [email-masked]
> Subject: New Version Notification for draft-cel-nfsv4-rpcrdma-bidirection-01.txt
> Date: May 20, 2015 at 3:09:57 PM EDT
> To: "Charles Lever" <[email-masked]>, "Chuck Lever" <[email-masked]>
> 
> 
> A new version of I-D, draft-cel-nfsv4-rpcrdma-bidirection-01.txt
> has been successfully submitted by Charles Lever and posted to the
> IETF repository.
> 
> Name:		draft-cel-nfsv4-rpcrdma-bidirection
> Revision:	01
> Title:		Size-Limited Bi-directional Remote Procedure Call On Remote Direct Memory Access Transports
> Document date:	2015-05-20
> Group:		Individual Submission
> Pages:		14
> URL:            https://www.ietf.org/internet-drafts/draft-cel-nfsv4-rpcrdma-bidirection-01.txt
> Status:         https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpcrdma-bidirection/
> Htmlized:       https://tools.ietf.org/html/draft-cel-nfsv4-rpcrdma-bidirection-01
> Diff:           https://www.ietf.org/rfcdiff?url2=draft-cel-nfsv4-rpcrdma-bidirection-01
> 
> Abstract:
>   Recent minor versions of NFSv4 work best when ONC RPC transports can
>   send ONC RPC transactions in both directions.  This document
>   describes conventions that enable RPC-over-RDMA version 1 transport
>   endpoints to interoperate when operation in both directions is
>   necessary.
> 
> 
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> The IETF Secretariat
> 

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: David Noveck <[email-masked]>
Date: Thu, 21 May 2015 14:46:56 -0400

I think this document is in good shape and now is an appropriate time to
convert it to a working-group document.

I gave the document a detailed review and have come up with the editorial
issues/suggestions below.

*Section 1.2:*

In the last sentence of the first paragraph, "from and" needs to be
replaced by "and from".

Suggest rewriting the first sentence of the second paragraph as follows:

These conventions can be observed when using the existing the
RPC-over-RDMA version 1 protocol definition.

In the third paragraph, suggest inserting "use of" before "NFSv4.1".

In the fourth paragraph, suggest replacing "not in" by "outside".

*Section 1.3.2:*

I have a number of suggestions regarding the third paragraph:

   - Given how "traditional" is used in section 1.3.1, suggest rewriting
   the first sentence as "ONC RPC clients and services are generally
not prepared
   for backward operation".
   - In the second sentence, suggest deleting "heavily".
   - In the third sentence, suggest replacing "any" by "a".

*Section 1.3.4.1 <http://1.3.4.1>:*

In the last sentence, suggest replacing "entirely unique" by "distinct",
"separate", or "entirely separate".

*Section 1.4.1:*

Suggest rewriting the last paragraph as follows:

NFSv4.0  functions correctly without a backchannel in place.  In this
case, the server does not grant file delegations.  This might result
in a negative performance effect, but functional correctness is
unaffected.

*Section 1.4.2:*

If the first sentence of the sixth paragraph, suggest replacing "prefer to
use" by "find it more convenient to use".

In the last sentence of the last paragraph, suggest replacing "impossible"
by "not possible".

*Section 1.5.1:*

In the first sentence of the second paragraph, suggest replacing "We wish
to avoid" by "The approach taken here avoids".

*Section 1.5.3: *
Suggest rewriting the last paragraph as follows:

Because RDMA transfers are not used for backward direction operations,
there is no need for servers to expose their memory to clients.
Further, this avoids any need to introduce the client complexity
required to drive RDMA transfers.

*Section 2.2:*

I don't think "MUST" is correct in the second sentence since the sender
cannot be depending (for interoperability) on conditions internal to the
receiver.

I would replace the last clause of this sentence by:

the receiver will generally have to drop the transport connection.

*Section 2.2.3:*

In the last sentence of the third paragraph, suggest replacing "results" by
"would result".

In the last sentence of the last paragraph, suggest replacing "not in" by
"outside".

*Section 3.3:*

Suggest the following replacement:

To address these limitations in the long run, a new version of the
RPC-over-RDMA protocol would be required.  Given this context, use of
the conventions described here to enable backward direction operation
is a transitional approach which is appropriate only while
RPC-over-RDMA version 1 is the predominantly deployed version of the
RPC-over-RDMA protocol.

On Wed, May 20, 2015 at 3:19 PM, Chuck Lever <[email-masked]> wrote:

> > Subject: New Version Notification for
> draft-cel-nfsv4-rpcrdma-bidirection-01.txt
> > Title:                Size-Limited Bi-directional Remote Procedure Call
> On Remote Direct Memory Access Transports
> > URL:
> https://www.ietf.org/internet-drafts/draft-cel-nfsv4-rpcrdma-bidirection-01.txt
> > Status:
> https://datatracker.ietf.org/doc/draft-cel-nfsv4-rpcrdma-bidirection/
> > Htmlized:
> https://tools.ietf.org/html/draft-cel-nfsv4-rpcrdma-bidirection-01
> > Diff:
> https://www.ietf.org/rfcdiff?url2=draft-cel-nfsv4-rpcrdma-bidirection-01
> >
> > Abstract:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever <[email-masked]>
Date: Fri, 22 May 2015 11:34:07 -0400

Hi Dave-

Thanks for your time! I’ve incorporated your comments into the document
with one exception, which I’d like to discuss further below.

On May 21, 2015, at 2:46 PM, David Noveck <[email-masked]> wrote:

> I think this document is in good shape and now is an appropriate time to convert it to a working-group document.
> 
> I gave the document a detailed review and have come up with the editorial issues/suggestions below.
> 
> Section 1.2:
> 
> In the last sentence of the first paragraph, "from and" needs to be replaced by "and from".
> 
> These conventions can be observed when using the existing the RPC-over-RDMA version 1 protocol definition.
> Section 1.3.2:
> 	• Given how "traditional" is used in section 1.3.1, suggest rewriting the first sentence as "ONC RPC clients and services are generally not prepared for backward operation".
> 	• In the second sentence, suggest deleting "heavily".
> 	• In the third sentence, suggest replacing "any" by "a".
> Section 1.3.4.1:
> 
> In the last sentence, suggest replacing "entirely unique" by "distinct", "separate", or "entirely separate".
> 
> Section 1.4.1:
> 
> NFSv4.0  functions correctly without a backchannel in place.  In this case, the server does not grant file delegations.  This might result in a negative performance effect, but functional correctness is unaffected.
> Section 1.4.2:
> 
> If the first sentence of the sixth paragraph, suggest replacing "prefer to use" by "find it more convenient to use".
> 
> In the last sentence of the last paragraph, suggest replacing "impossible" by "not possible".
> 
> Section 1.5.1:
> 
> In the first sentence of the second paragraph, suggest replacing "We wish to avoid" by "The approach taken here avoids".
> 
> Section 1.5.3:
> 
> 
> Because RDMA transfers are not used for backward direction operations, there is no need for servers to expose their memory to clients.  Further, this avoids any need to introduce the client complexity required to drive RDMA transfers.
> Section 2.2:
> 
> I don't think "MUST" is correct in the second sentence since the sender cannot be depending (for interoperability) on conditions internal to the receiver.
> 

Receive buffer overrun is an area I’m having difficulty with.

I used MUST because if the receiver does not have resources available,
either the sender has sent more messages than there are advertised credits,
or the receiver has not set up resources that match his advertised credit
value. Thus the protocol is violated.

The RPC message is essentially lost when this happens. Quick recovery is
desirable, thus the connection is forcibly dropped to give the sender an
opportunity to retransmit immediately. That could be just a best
implementation practice rather than a part of the protocol?

An NFSv4 ULP would likely drop the connection anyway. Indeed, at the
transport level, an RNR NAK that occurs when the RNR retry count is zero
will result in a connection loss. (That’s reflected in your suggested
replacement text).

The question of interoperability arises because it’s not clear how the
receiver should indicate to the sender that an RPC message was lost.
Perhaps this is entirely a concern for the ULP.

Maybe the text could say that the RPC message is lost in this case, but
nothing else is done at the RPC/RDMA level in response? The resulting
behavior is that message flow stops, at least until the RPC is
retransmitted, and its not clear to observers what happened.

I’m interested in hearing other opinions.

> Section 2.2.3:
> 
> In the last sentence of the third paragraph, suggest replacing "results" by "would result".
> 
> In the last sentence of the last paragraph, suggest replacing "not in" by "outside".
> 
> Section 3.3:
> 
> To address these limitations in the long run, a new version of the RPC-over-RDMA protocol would be required.  Given this context, use of the conventions described here to enable backward direction operation is a transitional approach which is appropriate only while RPC-over-RDMA version 1 is the predominantly deployed version of the RPC-over-RDMA protocol.
>

---

From: David Noveck <[email-masked]>
Date: Fri, 22 May 2015 12:38:43 -0400

> Perhaps this is entirely a concern for the ULP.

It doesn't seem worth it to try to address this at the RPC-RDMA level,
especially
since this situation will only arise when the sender is broken.

> retransmitted, and its not clear to observers what happened.

Makes sense to me.

On Fri, May 22, 2015 at 11:34 AM, Chuck Lever <[email-masked]>
wrote:

> >       • Given how "traditional" is used in section 1.3.1, suggest
> rewriting the first sentence as "ONC RPC clients and services are generally
> not prepared for backward operation".
> case, the server does not grant file delegations.  This might result in a
> negative performance effect, but functional correctness is unaffected.
> > Section 1.4.2:
> >
> > If the first sentence of the sixth paragraph, suggest replacing "prefer
> to use" by "find it more convenient to use".
> >
> > In the first sentence of the second paragraph, suggest replacing "We
> wish to avoid" by "The approach taken here avoids".
> >
> > Section 1.5.3:
> >
> >
> there is no need for servers to expose their memory to clients.  Further,
> this avoids any need to introduce the client complexity required to drive
> RDMA transfers.
> > Section 2.2:
> >
> RPC-over-RDMA protocol would be required.  Given this context, use of the
> conventions described here to enable backward direction operation is a
> transitional approach which is appropriate only while RPC-over-RDMA version
> 1 is the predominantly deployed version of the RPC-over-RDMA protocol.
> >
> > > Title:                Size-Limited Bi-directional Remote Procedure
> Call On Remote Direct Memory Access Transports

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 22 May 2015 19:49:32 +0000

If there are no objections to Dave’s idea, I will ask Chuck to submit the next version of the I-D as a WG item.

Spencer

From: nfsv4 [mailto:[email-masked]] On Behalf Of David Noveck
Sent: Friday, May 22, 2015 9:39 AM
To: Chuck Lever
Cc: NFSv4
Subject: Re: [nfsv4] New Version Notification for draft-cel-nfsv4-rpcrdma-bidirection-01.txt

> Perhaps this is entirely a concern for the ULP.

It doesn't seem worth it to try to address this at the RPC-RDMA level, especially
since this situation will only arise when the sender is broken.

Makes sense to me.

On Fri, May 22, 2015 at 11:34 AM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:
Hi Dave-

On May 21, 2015, at 2:46 PM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:

> Section 1.3.4.1<http://1.3.4.1>:
>

> On Wed, May 20, 2015 at 3:19 PM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:
> Hi-
>
> > From: [email-masked]<mailto:[email-masked]>
> > To: "Charles Lever" <[email-masked]<mailto:[email-masked]>>, "Chuck Lever" <[email-masked]<mailto:[email-masked]>>
> >
> >
> > until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.
> >
> > The IETF Secretariat
> >
>
> --
> Chuck Lever
> [email-masked]<mailto:[email-masked]>

---

From: Chuck Lever <[email-masked]>
Date: Tue, 26 May 2015 10:18:45 -0400

Hi Dave-

I will rework the language in this document related to receive
buffer overflow to make connection loss a recommended implementation
practice rather than an interoperability requirement.

On May 22, 2015, at 12:38 PM, David Noveck <[email-masked]> wrote:

> On Fri, May 22, 2015 at 11:34 AM, Chuck Lever <[email-masked]> wrote:
> Hi Dave-
>