Subject: [nfsv4] AD Evaluation for draft-ietf-nfsv4-rpcrdma-bidirection-05

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 10 Jan 2017 15:32:45 -0600

Hi, Spencer (S),

I think this catches me up on nfsv4 AD Evaluations ...

Again, this draft was pretty clear to me, but I did have some questions
about wording, that I'd like people to look at, before requesting IETF Last
Call.

Spencer (S), could you let me know when this draft is good to go?

Thanks!

Spencer (D)

--- AD Evaluation

Could you spell out "Parallel NFS" on first use of pNFS? Or whatever pNFS
stands for :-)

Probably just for my benefit, but since you mention NAT routers in this
text,

   To facilitate operation through NAT routers, all NFSv4.1 transport
   connections are initiated by NFSv4.1 clients.  Therefore NFSv4.1
   servers send callbacks to clients in the backward direction on
   connections established by NFSv4.1 clients.

is it obvious which end has responsibility for NAT binding keep-alives? I'm
guessing that's a client responsibility, but that's just a guess.

In Sections 4.3.1 and 4.3.2, I THINK you're saying that clients and servers
both need to pre-post more receive buffers to accommodate bidirectional
operation, but that's more clear in 4.3.1,

   To receive incoming backward direction Calls, an RPC-over-RDMA client
   endpoint must pre-post enough additional receive buffers to match its
   advertised backward direction credit value.  Each outstanding forward
   direction RPC requires an additional receive buffer above this
   minimum.

than the corresponding text in 4.3.2,

   To receive incoming backward direction replies, an RPC-over-RDMA
   server endpoint must pre-post a receive buffer for each backward
   direction Call it sends.

because "additional" appears twice in the text on 4.3.1, but doesn't appear
at all in 4.3.2. If this is as symmetrical as I'm thinking it is, perhaps
it's clearer to use similar wording in both places.

In this text,

   When message direction is not fully determined by context (e.g.,
   suggested by the definition of the RPC-over-RDMA version that is in
   use) or by an accompanying RPC message payload with a call direction
   field, it is not possible for the receiver to tell with certainty
   whether the header credit value is a request or grant.  In such
   cases, the receiver MUST NOT use the header's credit value.

does RDMA work at all, if the credit value can't be used?

---

From: Chuck Lever <[email-masked]>
Date: Tue, 10 Jan 2017 16:47:29 -0500

Again, thanks for your review! Responses below.

> On Jan 10, 2017, at 4:32 PM, Spencer Dawkins at IETF <[email-masked]> wrote:
> 
> Hi, Spencer (S), 
> 
> Again, this draft was pretty clear to me, but I did have some questions about wording, that I'd like people to look at, before requesting IETF Last Call. 
> 
> Could you spell out "Parallel NFS" on first use of pNFS? Or whatever pNFS stands for :-)

Will do.

> Probably just for my benefit, but since you mention NAT routers in this text,
> 
> is it obvious which end has responsibility for NAT binding keep-alives? I'm guessing that's a client responsibility, but that's just a guess.

I don't understand what NAT binding keep-alive means, exactly, but
based on a naive assumption, I think the client is responsible here.
If anyone knows of a cite-able summary of the NFSv4.1 backchannel
redesign goals, I can reference it here.

> In Sections 4.3.1 and 4.3.2, I THINK you're saying that clients and servers both need to pre-post more receive buffers to accommodate bidirectional operation, but that's more clear in 4.3.1,
> 
> because "additional" appears twice in the text on 4.3.1, but doesn't appear at all in 4.3.2. If this is as symmetrical as I'm thinking it is, perhaps it's clearer to use similar wording in both places.

I will use similar wording in both places.

> does RDMA work at all, if the credit value can't be used?

What this means is the receiver MUST NOT update its credit accounting
based on the information in this header. These foggy situations should
be exceptionally rare.

"MUST ignore" might be more appropriate.

--
Chuck Lever

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 10 Jan 2017 16:13:20 -0600

Hi, Chuck,

> Again, thanks for your review! Responses below.

Oh, thank YOU. You responded while I still have this draft in my solid
state memory :-) ...

It looks like we're good except for the last point.

> > is it obvious which end has responsibility for NAT binding keep-alives?
> I'm guessing that's a client responsibility, but that's just a guess.
>
> > In Sections 4.3.1 and 4.3.2, I THINK you're saying that clients and
> servers both need to pre-post more receive buffers to accommodate
> bidirectional operation, but that's more clear in 4.3.1,
> >
> > because "additional" appears twice in the text on 4.3.1, but doesn't
> appear at all in 4.3.2. If this is as symmetrical as I'm thinking it is,
> perhaps it's clearer to use similar wording in both places.
>

I think what I was thinking about, is whether this situation can lead to
deadlock.

I guess I should back up and ask a more basic question, which is whether
you'd be able to recognize that this situation applies from looking at the
definition of the RPC-over-RDMA version, so you could just say "I'm not
going to do bidirectional" when a transport connection is established,
rather than trying to figure out that there's a problem during request
processing.

Thanks,

Spencer

---

From: Tom Talpey <[email-masked]>
Date: Tue, 10 Jan 2017 17:13:19 -0500

> redesign goals, I can reference it here.

I think the reference to NAT is a red herring and should be deleted,
or at the very least demoted to an example. The goal of the client-
initiated backchannel was much more general; it included traversing
firewalls but it was not an explicit goal of the 4.1 backchannel to
interoperate through NAT (although it certainly does).

There was some language in the original RFC5661 sections 1.4, 2.10.1,
2.10.3.1 and 2.10.8.1 on this, but RFC5661 specifically never mentioned
NAT.

Tom.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 10 Jan 2017 17:40:31 -0500

> On Jan 10, 2017, at 5:13 PM, Tom Talpey <[email-masked]> wrote:
> 

Fair enough, the mention of NAT will be removed.

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Tue, 10 Jan 2017 18:02:00 -0500

> On Jan 10, 2017, at 5:13 PM, Spencer Dawkins at IETF <[email-masked]> wrote:
> 
> Hi, Chuck,
> 
> Oh, thank YOU. You responded while I still have this draft in my solid state memory :-) ...
> 

[ snipped ]

> I think what I was thinking about, is whether this situation can lead to deadlock.

The credit grant value has to be ignored whenever the forward and backward
grants are not the same value (which is typical in current implementations).

With the current set of protocols, the only case like this is RDMA_ERROR,
which is almost never used. Ignoring the credit value for those messages
doesn't seem problematic.

If the backward credit grant (which is likely to be smaller) is suddenly
used in the forward direction, for that one message, the forward requester
would wait for any outstanding replies before sending more requests. The
next message from the responder would restore the forward credit grant to
its correct value.

I don't think a deadlock could occur unless the grant value went to zero,
and that is already forbidden by rfc5666bis.

Another way to address this I suppose would be to ensure the grant values
in both directions are always the same.

> I guess I should back up and ask a more basic question, which is whether you'd be able to recognize that this situation applies from looking at the definition of the RPC-over-RDMA version, so you could just say "I'm not going to do bidirectional" when a transport connection is established, rather than trying to figure out that there's a problem during request processing.

An implementer would be able to tell where these corner cases are, since
her implementation has to ignore the credit value in those cases. But
maybe I'm missing something.

--
Chuck Lever

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Tue, 10 Jan 2017 21:26:25 -0600

Hi, Chuck,

> and that is already forbidden by rfc5666bis.

I had forgotten that detail. Thanks for the wake-up call!

> maybe I'm missing something.

I'm thinking the guy who's missing something is me ;-)

I think we're good to go on this document.

Spencer (S), would you agree?

Spencer (D)

---

From: spencer shepler <[email-masked]>
Date: Thu, 12 Jan 2017 10:32:20 -0800

Yes, Spencer D.  I believe we are ready with the updates that Chuck has
in-hand.  Thanks for the review.

Spencer S

On Tue, Jan 10, 2017 at 7:26 PM, Spencer Dawkins at IETF <
[email-masked]> wrote:

>> > I guess I should back up and ask a more basic question, which is
>> whether you'd be able to recognize that this situation applies from looking
>> at the definition of the RPC-over-RDMA version, so you could just say "I'm
>> not going to do bidirectional" when a transport connection is established,
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 12 Jan 2017 12:35:26 -0600

Hi, Spencer (S),

On Jan 12, 2017 12:32, "spencer shepler" <[email-masked]> wrote:

Spencer S

Excellent. I'll request Last Call when Chuck submits his update.

Spencer