Subject: [nfsv4] allocate/deallocate of large size on slow I/O subsystem

From: Rick Macklem <[email-masked]>
Date: Wed, 25 Aug 2021 15:38:32 +0000

Hi,

It seems there could be a problem doing large (large aa_length/da_length)
Allocate/Deallocate operations, on certain I/O subsystems, in that the operation
might take a long time.

I have always felt that NFSv4 RPCs should reply within approximately 1sec.

So what should be done if the Allocate/Deallocate is going to take much more
than 1sec.

Possibilities are:
1 - Just allow RPCs that do these operations to take a long time. ie. The client
      will need to handle waiting many seconds for a reply from the server without
      making a new TCP connection and retrying the RPC.
2 - Have the NFSv4.2 client do Allocate/Deallocate operations in small chunks to
     avoid large RPC reply times.
3 - Having the server "guess" how long the operation will take and, if the estimate
     is greater than 1sec, reply an error to indicate that it cannot be completed
     within a reasonable time.
     - Although it is defined specifically for "channel", NFS4ERR_REQ_TOO_BIG is
       listed for both Allocate and Deallocate and seems reasonable.
4 - Define two new operations similar to Allocate/Deallocate, but allowed to
     return partial completion, similar to Copy.
    --> ie. Have ar_offset, ar_length or dr_offset, dr_length in NFS_OK reply,
           which indicates a partial completion if as/dr_length > 0.
5 - ???

What do you think w.r.t this? rick

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 25 Aug 2021 09:40:24 -0700

> On Aug 25, 2021, at 8:38 AM, Rick Macklem <[email-masked]> wrote:
> 
> Hi,
> 

This one, I don’t like artificial limits on the server for replies.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 26 Aug 2021 11:19:02 -0400

>       making a new TCP connection and retrying the RPC.

The client is already required to wait for the reply in v4+. If you
are worried about operations taking too long and I assume the server
can run the task in the background, the server then can return
ERR_DELAY if you are really worried about clients that will
preemptively drop the connection and retry the RPC.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 27 Aug 2021 03:36:20 +0000

Olga Kornievskaia wrote:
>The client is already required to wait for the reply in v4+.
That is true, but the client can retry the request when the
connection the original request is sent on disconnects.
RFC 5661 Sec. 2.10.6.2:
   A requester MUST NOT retry a request, unless the connection it used
   to send the request disconnects.  The requester can then reconnect
   and re-send the request, or it can re-send the request over a
   different connection that is associated with the same session.

If a retry is done when the RPC is in progress, the same section states:
   A retry might be sent while the original request is still in progress
   on the replier.  The replier SHOULD deal with the issue by returning
   NFS4ERR_DELAY as the reply to SEQUENCE or CB_SEQUENCE operation, but
   implementations MAY return NFS4ERR_MISORDERED.
So, if the client retries the RPC while it is in progress on the server, it
might get a NFS4ERR_DELAY reply or it might get NFS4ERR_MISORDERED.

The problem is that the above status reply "completes the RPC on the
session slot". When the RPC is tried later, it will appear to be a "new RPC"
to the session" and that means it will start over again.
--> A retry while an RPC is in progress is the "weak spot" in sessions and
      is not handled well.
(I'm not even going to guess what a client does for NFS4ERR_MISORDERED?)

But, you are correct, in that a timeout/retry should not happen for a
NFSv4.1/4.2 RPC unless the connection disconnects.

I still am not convinced that RPCs that take many seconds to reply are
a good idea, although I agree that, unless the connection disconnects,
they should be "safe to do".

> If you
I don't think this is correct for the initial RPC request.
If a client gets a NFS4ERR_DELAY reply to an initial RPC request (not a retry),
I think it can choose to not try the RPC again.
--> Then the client would not expect the Deallocation to be done at all.

RFC 5661 Sec. 15.1.1.3 defines NFS4ERR_DELAY:
   For any of a number of reasons, the replier could not process this
   operation in what was deemed a reasonable time.  The client should
   wait and then try the request with a new slot and sequence value.

My interpretation of "could not process" is that the operation has not
been done and not "the operation is in progress".

I could argue that there should be a NFS4ERR_INPROGRESS error that
is used as a reply to Sequence when the RPC is in progress, but that does
not exist.

rick

---

From: [email-masked] (J. Bruce Fields)
Date: Fri, 27 Aug 2021 18:14:27 -0400

> What do you think w.r.t this? rick

My knee-jerk reponse is

5) Declare this the I/O subsystem's fault and don't spend a lot of time
thinking about it (in practice that probably means #1?)

--b.

---

From: Rick Macklem <[email-masked]>
Date: Sat, 28 Aug 2021 00:33:37 +0000

>thinking about it (in practice that probably means #1?)
Oh well. I guess all FreeBSD file systems are now faulty;-) None currently
have a "real" deallocator (one for ZFS is in the works, but hasn't been
blessed upstream yet).

I created a 10Gbyte file on a FreeBSD server (slow laptop spinning rust)
and then did this on a Linux 5.12 client (NFSv4.2 mount):
    fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
        65536, 10737418240LL)
- It worked just fine and only took 5min + 18sec to complete.
  (I looked at the packet capture and all that happened during
   the 5min + 18sec between the Deallocate request and reply were
   a few TCP keepalives and some RPCs of only Sequence to maintain
   the lease.)

I'm going to do #2, because the FreeBSD deallocate API allows a reply
with partial completion. I'll use the same tunable size I use for synchronous
Copy operations, which defaults to 100Mbytes/operation.
--> This slow hardware does about 50Mbytes/sec, so the RPC response will
       be about 2 seconds on even this slow server. Doing a separate RPC for
       each 100Mbytes seems an acceptible overhead for me.

I am doing this partly because FreeBSD users don't like processes unresponsive
to <ctrl>C and I strongly discourage "intr" NFSv4.1/4.2 mounts, since they screw
up the session slot, etc.

rick

--b.