Subject: [nfsv4] Question about the server returning ERR_DELAY on SEQUENCE

From: Olga Kornievskaia <[email-masked]>
Date: Mon, 11 Apr 2022 17:47:26 -0400

Hello,

I would like to know if the server is allowed to return ERR_DELAY on a
SEQUENCE to a client request without evaluating anything within the
compound request beyond that it's a legal operation. Then after this
request is retried, is the server allowed to change the error code
based on its evaluation of the insides of the request?

Specifically, here's what's in question:
1. Client sends a request on slot=X seq=Y (doesn't get a reply form
the server).
2. Client sends another request on slot=X seq=Y+1.

According to the spec, when the server is evaluating request #2 it
sees a gap in sequence of more than 2 and MUST reply with
SEQ_MISORDERED.

Now if we combine that the server got a request in step2 and is
allowed to reply to a request with an ERR_DELAY without evaluating it
(as it's internally waiting for something to process it). Then, the
server re-evaluates the request that's been delayed and finds out that
it falls into the group of SEQ_MISORDERED.

IF this is allowed, then it's problematic to the linux client that
assumes that the ERR_DELAY reply means that the server has processed
the slot/seq# number (and deemed them valid. Client updates its
last_acked seq#).

Alternatively one might ask, can the client assume that if it received
ERR_DELAY from the server, then the server already checked the
validity of the SEQUENCE arguments.

Thank you.

---

From: Rick Macklem <[email-masked]>
Date: Tue, 12 Apr 2022 15:14:25 +0000

> based on its evaluation of the insides of the request?
I believe the answer is yes.

I think the only thing you can assume w.r.t.
Sequence returning an error is (from the RFC):
   If SEQUENCE returns an error, then the state of the slot (sequence
   ID, cached reply) MUST NOT change, and the associated lease MUST NOT
   be renewed.

There is this (also from the RFC):
o  If the difference between sa_sequenceid and the server's cached
      sequence ID at the slot ID is two (2) or more, or if sa_sequenceid
      is less than the cached sequence ID (accounting for wraparound of
      the unsigned sequence ID value), then the server MUST return
      NFS4ERR_SEQ_MISORDERED.
But I think this only applies if the server has done the sequence check.
(I do not think the above is guaranteed when NFS4ERR_DELAY is returned.)

> the slot/seq# number (and deemed them valid. Client updates its
Well, issuing multiple concurrent RPCs on the same slot essentially
breaks the "exactly once semantics" for sessions and this client can
expect any of these RPCs to fail (including after a retry due to
NFS4ERR_DELAY) with NFS4ERR_SEQ_MISORDERED.
The client can only determine what the correct next seq# for the
slot is after all of the concurrent RPCs have replied with the reply
for Sequence of NFS_OK or NFS4ERR_MISORDERED. (ie. The seq#
can be advanced by the number of NFS_OK replies for Sequence.)

I think you can see why the FreeBSD client chooses to never send
multiple concurrent RPCs for the same slot#.

> validity of the SEQUENCE arguments.
Since the RFC does not clarify if this is allowed, I think clients must
assume that it can happen.
To be honest, I can't think of why a server would reply NFS4ERR_DELAY
for Sequence after it has done the seq# check. As you noted, it might
return NFS4ERR_DELAY because it cannot access the session info. at this
time (and, therefore, not do the seq# check).
(The FreeBSD server never replies NFS4ERR_DELAY for Sequence, so I
 am not sure when other servers might choose to do so?)

rick

Thank you.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 12 Apr 2022 14:37:59 -0400

>    ID, cached reply) MUST NOT change, and the associated lease MUST NOT

This really confuses me then. If the server is returning an error and,
say, it doesn't know the state of the slot as it never evaluated it,
then how can it keep the state of the slot the same.

> multiple concurrent RPCs for the same slot#.

Linux implementation has no means to wait for a request for which
there is no caller waiting (ie., the systemcall was killed). When that
happens, the slot is left as we say interrupted.

>  am not sure when other servers might choose to do so?)

That's great. But it seems like the spec does allow for this and if
that's the case the linux client isn't handling that case.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 13 Apr 2022 00:41:53 +0000

> then how can it keep the state of the slot the same.
I think you could interpret a NFS4ERR_DELAY reply to Sequence as
meaning something like "not convenient to access/update slot
information at this time", which could imply the server won't change it and
will be able to access/update it a little while".
But, as I noted, the FreeBSD server never replies NFS4ERR_DELAY for
Sequence, so I really do not know why a server would do this?

> happens, the slot is left as we say interrupted.
Good luck. Sounds like a tough one. I can't think of a clever solution...
(Maybe some sort of "housekeeping thread" that processes the
 Sequence reply when nothing else is handling the RPC's reply?)

> that's the case the linux client isn't handling that case.
Yep. NFS4ERR_DELAY is listed as an error reply for Sequence and, if there
are multiple RPCs for the same slot on the fly concurrently, it's not going
to be easy to maintain the seq#, even if you do not care about
"exactly once semantics" for these RPCs.

At least that's how it seems to me at this time.

rick