Subject: [nfsv4] client retry and timeouts

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Fri, 6 Mar 2015 16:33:05 +0100 (CET)

Hi All,

this e-mail is triggered by presentation "Retransmitting on TCP considered harmful"
by Rob Thurlow and Bill Baker at Connectathon-2015.

For people, how wasn't there: the client resend a write request while kernel thread,
which was processing the original request, was blocked by some other activity.

Now, let say we have two hosts the CLIENT and the SERVER. The CLIENT free to
resend any request at any time and the SERVER must process all of them. Yes, yes....
NFSv4.1, EOS and so on... But why SERVER must process both of them in the first place?
The missing part, for me, is that server have no idea that CLIENT is not interested
in the first reply. IOW, if client send a request with a notation 'reply in N sec'
(we actually do this inside the dCache's inter-component message passing layer), then
SERVER is free to drop the request as it knows, that CLIENT is not interested in
it any more.

Well, this sounds like NFSv5 or v4.5, as today there are no way for SERVER to discover
timeout value used by CLIENT (some kind of fort/back channel property). But we can come
up with a 'gentlemen agreement', that CLIENT retry after LEASE_TIME*0.7 
(or so) and SERVER can simply stop processing of an 'expired' requests.

Makes or pure madness?

Tigran.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 6 Mar 2015 10:48:59 -0500

> (or so) and SERVER can simply stop processing of an 'expired' requests.

No.

Cheers
  Trond

---

From: Tom Talpey <[email-masked]>
Date: Fri, 6 Mar 2015 15:50:34 +0000

> -----Original Message-----
> From: nfsv4 [mailto:[email-masked]] On Behalf Of Mkrtchyan, Tigran
> Sent: Friday, March 6, 2015 10:33 AM
> To: NFSv4
> Cc: Bill Baker; Rob Thurlow
> Subject: [nfsv4] client retry and timeouts
> 
> 
> 
> Hi All,
> 

I'd be very interested to see the slides of this presentation. The Connectathon
site doesn't have them (yet?).

This must be an NFSv3 discussion, I assume. For NFSv4 and up, retransmission
on an intact TCP connection, even after a timeout, is disallowed except as stated:

-- 3.1.1.  Client Retransmission Behavior

--   When processing a request received over a reliable transport such as
--   TCP, the NFS version 4 server MUST NOT silently drop the request,
--   except if the transport connection has been broken.  Given such a
--   contract between NFS version 4 clients and servers, clients MUST NOT
--   retry a request unless one or both of the following are true:
--
--   o  The transport connection has been broken
--
--   o  The procedure being retried is the NULL procedure

> Makes [sense] or pure madness?

Pure madness, IMO. The server cannot know the client's state as of the
time it was sent. So it cannot make a decision based on that.

Tom.

---

From: Malahal Naineni <[email-masked]>
Date: Fri, 6 Mar 2015 10:09:56 -0600

> (or so) and SERVER can simply stop processing of an 'expired' requests.

I was there at the conference but didn't get all the details! Someone
said a DRC would catch it, if implemented correctly, but usually things
go into DCR after completing the request. Based on my understanding, the
problem presented there was:

1. Client sent a request.
2. Server received it. It made a call to some user level program prior
   to sending the data to storage.
3. Client sent a new write or retry with different data for the same
   file offset ???
4. Server sent the new write's data to storage
5. Server received response from the user level helper for the original
   request, so it sent OLD data to storage.

At this time, client was expecting new data but has OLD data.

Is that a correct description? If so, you can see where the problem is!
:-)

Regards, Malahal.

---

From: Tom Haynes <[email-masked]>
Date: Fri, 6 Mar 2015 08:41:39 -0800

> On Mar 6, 2015, at 7:50 AM, Tom Talpey <[email-masked]> wrote:
>> 
> 

There were no slides for that talk.

---

From: "Frank Filz" <[email-masked]>
Date: Fri, 6 Mar 2015 09:45:14 -0800

> > this e-mail is triggered by presentation "Retransmitting on TCP
considered
> harmful"
> > For people, how wasn't there: the client resend a write request while
> > kernel thread, which was processing the original request, was blocked by
> some other activity.
> >
> > Now, let say we have two hosts the CLIENT and the SERVER. The CLIENT
> > free to resend any request at any time and the SERVER must process all
of
> them. Yes, yes....
> > The missing part, for me, is that server have no idea that CLIENT is
> > not interested in the first reply. IOW, if client send a request with a
> notation 'reply in N sec'
> > (we actually do this inside the dCache's inter-component message
> > passing layer), then SERVER is free to drop the request as it knows,
> > that CLIENT is not interested in it any more.
> >
> > Well, this sounds like NFSv5 or v4.5, as today there are no way for
> > SERVER to discover timeout value used by CLIENT (some kind of
> > fort/back channel property). But we can come up with a 'gentlemen
> > agreement', that CLIENT retry after LEASE_TIME*0.7 (or so) and SERVER
> can simply stop processing of an 'expired' requests.
> >
> >
> > Makes or pure madness?
> 
> I was there at the conference but didn't get all the details! Someone said
a
> DRC would catch it, if implemented correctly, but usually things go into
DCR
> after completing the request. Based on my understanding, the problem
> presented there was:

I was the one who offered that DRC could catch it (I forget if the issue was
v3 or v4, if it was v4, then it seems like the client was violating spec).
In Ganesha, we create a DRC record as soon as we start processing a request.
This allows us to detect a retried request before the first request has
completed processing. I'm not sure if Ganesha ages these out of the DRC, but
It's reasonable to make such entries immune to aging out (presumably you
have a finite number of threads and therefore a finite number of in progress
requests and therefore a finite number of in progress DRC entries).

> 4. Server sent the new write's data to storage 5. Server received response
> from the user level helper for the original

As I understood it process was:

1. client sent a write request
2. server got lost in user space trying to authenticate request
3. client got impatient and retried the request
4. server processed and responded to retry in a timely fashion
5. client sent another write request modifying the same data range
6. user space finally authenticated the original request and server
completed processing

One potential problem is that even if this was NFS v4 and the client had
dropped the tcp connection, the server would not discover that until sending
the reply, so the server could still process the old write request out of
order unless after coming back from some long delay it checked that the tcp
connection was still alive.

If the tcp connection is dropped and the client retries, is it intended to
use the same XID for the retry? It seems like we still need the XID for the
server to detect a new request is a duplicate of an in progress request so
the server can decide how to handle it.

Frank

---

From: Tom Talpey <[email-masked]>
Date: Fri, 6 Mar 2015 18:24:14 +0000

> From: nfsv4 [mailto:[email-masked]] On Behalf Of Frank Filz
> Sent: Friday, March 6, 2015 12:45 PM
> To: 'Malahal Naineni'; 'Mkrtchyan, Tigran'
> Cc: 'Bill Baker'; 'Rob Thurlow'; 'NFSv4'
> Subject: Re: [nfsv4] client retry and timeouts
> 
>...

Duplicate Request Cache processing has never been documented in any
normative way, certainly not in an IETF draft, so the fact of the matter is
that there are many answers to your question. Many variants of the rules
have been proposed and implemented over the years. I doubt any two
servers are quite alike.

The original DRC was described in Chet Juszczak's 1989 Usenix paper,
"Improving the Performance and Correctness of an NFS Server". Ah,
nostalgia. :-)

Tom.