Subject: [nfsv4] questions about flow control

From: Chuck Lever III <[email-masked]>
Date: Thu, 29 Apr 2021 18:16:48 +0000

Howdy-

We're working on prototyping RPC/RDMA version two. As many of
you know, RPC/RDMA uses credit-based flow control.

I've presented to the WG before on the kinds of improvements
to credit accounting we need to make over version one of
RPC/RDMA in order to support control plane operations and
message continuation -- cases where we no longer have
perfectly symmetrical Call/Reply pairing.

I'm looking at Section 4.2.1.1 of draft-ietf-nfsv4-rpcrdma-version-two
as it is currently constructed and I'm finding it ...
underwhelming.

I'm thinking of replacing it with something more akin to the
original forms of credit-based flow control, as described in
Chapter 4 of:

https://dl.acm.org/doi/pdf/10.1145/190314.190324

and implemented in the form of Chapter 5 of that paper. The
rdma_credits field would be filled in with the sender's Vr,
in both directions, and N2 + N3 would be the credit limit. We
would need to add some kind of "reset credit accounting"
message as well.

I'm not feeling confident about this choice, however. Does
anyone know a person or people who could answer some questions
about flow control design? Or is there a good reference I could
read to help me understand fundamentals and common pitfalls?

Many thanks in advance!

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Fri, 30 Apr 2021 14:47:57 -0400

I'd be happy to discuss it with you, and you and I may have
done so in the past, regarding the approach made by SMB Direct.
This 2012 presentation* discusses some of that design, and
it's perhaps my suggestion to separate the credit protocol
from the credit policy. The protocol is the easy part, honestly.

I'd advise against using the methods in the ATM paper you
cite, at least on their face. These crediting approaches are
largely attempting to meter available bandwidth. They also
are for protocols which are tolerant of loss. This is a very
different goal from the RDMA crediting required to provide a
reliable stream of requests and responses.

Tom.

* 
https://www.snia.org/sites/default/orig/SDC2012/presentations/Revisions/TomTalpeyKramer-High_Performance__File.pdf

On 4/29/2021 2:16 PM, Chuck Lever III wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever III <[email-masked]>
Date: Fri, 30 Apr 2021 21:01:32 +0000

> On Apr 30, 2021, at 2:47 PM, Tom Talpey <[email-masked]> wrote:
> 

A design conference call might be best, once I've had a chance
to digest more literature.

In terms of protocol design I'd like to avoid splitting the
rdma_credits field into two 16-bit fields, as RPC/RDMA v2 does
currently. We did that because of real or perceived restrictions
that RFC 8166 places on the structure of the transport headers
in all RPC/RDMA versions.

The credits in v1 are RPC direction-aware; in v2 they need to
be bound to the peer Receivers, which operate in both directions.

RPC/RDMA v1 implementations ignore the credit request value
completely, so it's not clear it needs to be retained in v2.

> reliable stream of requests and responses.

The ATM paper has a design that veers away from request/grant,
and looks more like a sliding window, but it seems easily
adaptable to a pair of bi-directional streams.

> * https://www.snia.org/sites/default/orig/SDC2012/presentations/Revisions/TomTalpeyKramer-High_Performance__File.pdf

--
Chuck Lever

---

From: Tom Talpey <[email-masked]>
Date: Fri, 30 Apr 2021 20:44:12 -0400

> completely, so it's not clear it needs to be retained in v2.

Well, ok. It's a good approach to avoid unnecessary backwards
compatibility. And the v1 protocol was rather limited, as we've
discovered.

Just as a point of discussion, I'd ask whether you are intending
to design an actual credit algorithm for the protocol, or only
the protocol itself. These are two different things, and in my
view, standardizing an algorithm is something to be avoided. There
are any number of ways to implement a credit scheme, and over-
specifying the details constrains the implementations. This can
quickly lead to v3, v4, ... So, I would suggest a clear goal for
the discussion, if it's to be had.

Tom.