Subject: [nfsv4] Detecting loss of connection with RPC-over-RDMA

From: Chuck Lever <[email-masked]>
Date: Tue, 26 Jan 2016 13:00:22 -0500

Typically on RDMA fabrics, the network layer has very
fast timeouts to detect quickly when a peer HCA is not
responding to RDMA operations.

However, there are some cases where it is difficult to
detect that an RPC-over-RDMA connection has stopped
working. The simplest case is when a client has sent one
or more RPC requests, and then the server crashes after
receiving the last request.

Because the client drives no more RDMA operations, there's
no feedback from the fabric to indicate a loss of function
on the storage target. The client must rely on RPC-level
retransmit timeouts to detect a problem.

Such timeouts are typically a minute or longer. If the
server recovers sooner than that, the client workload is
delayed needlessly until it retransmits. The ideal would
be that the client would use a short timeout in this case
to quickly re-establish the connection.

An NFS client is free to use NFS NULL operations well before
an RPC timeout to assess the health of the connection and
server. An RDMA Send would generate a quick error
completion on the client if the server was unresponsive.

With RPC-over-RDMA, this means the client has to reserve an
RPC-over-RDMA credit in order to guarantee that receive
resources are available to perform the ping. If there are no
RPC-over-RDMA credits available, sending on an otherwise
healthy connection would likely result in connection loss.

One thought was that the client could perform an operation
that does not require an RPC-over-RDMA credit. For example,
it could drive an RDMA Write. But Write does not normally
return an acknowledgement. Would an RDMA Read be a better
choice?

Prototyping with IB cards we have on-hand has demonstrated
Write is effective at detecting when a connection is no
longer usable, at least for simple "server down" scenarios.

With RPC-over-RDMA Version One's transfer model, the
server currently drives all RDMA Read and Write operations.
It would not have registered any memory that the client
can use for this operation. The client would need new,
extra logic to drive the Write.

The client would thus have to depend on using a special
R_key value that the server had not provided in advance
(say, zero) and the Write would have to send no data (be
zero-length). The behavior of a zero-length or zero R_key
Write may not be well-defined for all fabrics and HCAs.

We also wonder if this technique, or something that achieves
the same result, would be appropriate for documentation as
part of the RPC-over-RDMA standards.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 27 Jan 2016 13:03:09 -0500

> If there are no

Sure, but you have to ask how troublesome is that likely to be.
How valuable is an otherwise healthy connection that isn't doing
much?  Essentially you are balancing the time and effort to
re-establish the connection against the delay in ascertaining that
a failed server has in fact failed.  In many cases, it would seem
that trashing the otherwise healthy connection and re-establishing
it would be the better choice, even if the probability of health is
high.  I'm guessing that the health probability is likely to be fairly
low.

> The behavior of a zero-length or zero R_key

The worst case is that it depends on the firmware-level of the hcas of both
client and
server in a not-very-easily-determined way.  If that's the case, it may be
time
to look at something else.

> extra logic to drive the Write.

Even if you added the appropriate logic, with the server registering some
memory for you to read, I'm not sure how helpful the result would be.
It might be that the server cannot respond to requests  (i.e It is
functionally
down) even while the the hardware is functioning so it can respond to
RDMA requests.

In this case, the nice fact that the DMA engine can work autonomously, may
work against you.

On Tue, Jan 26, 2016 at 1:00 PM, Chuck Lever <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever <[email-masked]>
Date: Wed, 27 Jan 2016 16:22:18 -0500

> On Jan 27, 2016, at 1:03 PM, David Noveck <[email-masked]> wrote:
> 
> > If there are no
> much?

Remember that as soon as the credit limit is exhausted, the
client can't send any more requests until the server responds.
So an idle connection does not mean there is no activity.

If there are no credits available, that means there is _a_lot_
of activity going on somewhere. That means there are N pending
RPC requests. Dropping the connection means that the client
must manage resending all of these (pending replies are lost
also) to ensure that they have been properly executed and
retired.

On a client where all the mount points share the same connection
to that server, that means one slow share (or even one slow RPC)
is going to punish all operations against that server.

Consider a scenario where the server is otherwise busy (for
example, it is resilvering a RAID 1 group, or there is a bad
drive that is slowing down I/O). Or, consider a scenario where
the server is about to reply NFS4ERR_DELAY, for instance, if
it is in the middle of migrating one of the shares.

The server is alive and operational, and on RDMA, a ping over
the credit limit would be terminal for that connection.

"Connection Loss Considered Hazardous"

> Essentially you are balancing the time and effort to
> The worst case is that it depends on the firmware-level of the hcas of both client and 
> server in a not-very-easily-determined way.  If that's the case, it may be time
> It might be that the server cannot respond to requests  (i.e It is functionally 

So, I'm guessing the client could detect network partitions
and server reboots fairly reliably.

When a server has crashed, I'm not sure what exactly it
would look like. But eventually the server would restart
and the old connection would be toast fairly quickly.

I agree that either sending an invalid RPC-over-RDMA header
(so the peer responds RDMA_ERROR/ERR_CHUNK) or an NFS NULL
are more preferable options.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 27 Jan 2016 18:28:49 -0500

It sounds to me that the scenario you are describing suggest that credits
are not
being managed appropriately.  Presumably, the server started with a large
number of
credits and somehow the client managed to exhaust them to the point, that he
cannot even do a ping.  Consider the client-side RPC-RDMA implementation
being
asked to do another NFSv4 COMPOUND when it has only one credit outstanding.

Since the client-sde transport presumably has logic to hold the request
pending if
there are zero credits available, it is probably a better use of resources
for him to reserve
the last credit for use by ping-type functionality, i.e.something that will
be responded to
right away if the server is neither dead nor a zombie.

> would be appropriate for documentation as

It doesn't seem like this fits in RFC5666bis if there is no
current implementation.

I think this might make sense as an extension to Version One.  You could
define a new RDMA_PING
header type and describe how the client could deal with servers that either
do or don't support the extension.

---

From: William Allen Simpson <[email-masked]>
Date: Sun, 31 Jan 2016 16:19:15 -0500

On 1/27/16 6:28 PM, David Noveck wrote:
> It sounds to me that the scenario you are describing suggest that credits are not
> being managed appropriately.  Presumably, the server started with a large number of
> cannot even do a ping.  Consider the client-side RPC-RDMA implementation being
A COMPOUND is always handled in one credit.  According to our
current understanding of 5666, credits are per request, because
they are the pre-posted work orders.

Whereas in RDMA-speak, credits are per outstanding segment over
the fabric.  5666 should probably have used a different term, but
we're stuck with it.

That's why there's a big section elucidating the meaning:
"4.3.1.  Credit Limit"

> Since the client-sde transport presumably has logic to hold the request pending if
> there are zero credits available, it is probably a better use of resources for him to reserve
> the last credit for use by ping-type functionality, i.e.something that will be responded to
True.  The client could refrain from using the last credit.

But a better idea might be to measure anticipated round-trip time,
same as we do with TCP.  We have well-known theory and algorithms.

Or we could do both....  Refrain from using the last credit until
the SRTT from the longest outstanding transaction has expired.

> It doesn't seem like this fits in RFC5666bis if there is no current implementation.
>
Agreed.  Let's stick to only what is currently done.

> I think this might make sense as an extension to Version One.  You could define a new RDMA_PING
> header type and describe how the client could deal with servers that either do or don't support the extension.
>
No, a new field value for pinging must be Version Two.  There's no
known response in Version One to a rejected value other than error
and drop connection.

Instead, I'd thought we were going to try NFS NULL operation, which
wouldn't drop the connection.  Why re-invent the wheel?