Subject: [nfsv4] SETATTR/DELEGRETURN deadlock

From: [email-masked] (J. Bruce Fields)
Date: Tue, 13 Apr 2021 14:50:33 -0400

I'm seeing what looks like a deadlock between a 4.1+ client waiting for
a SETATTR reply and a server waiting for a DELEGRETURN.

I seem to remember Trond mentioning it before, as a reason the client
would preemptively return delegations in some cases.

Who's at fault in this case?  Should the server guarantee that it will
never allow a reply to a client to wait on that client returning a
delegation?  (It should be able to in this case--this being 4.1, it
knows which client the SETATTR came from, and shouldn't have to revoke
that client's delegation.)  Or can the client figure out some way to not
make the DELEGRETURN depend on the SETATTR reply?

Cc'ing Trond as I'm pretty sure I remember him discussing exactly this
situation at one point....

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 13 Apr 2021 21:16:13 +0000

> situation at one point....

The Linux NFS client is expected to return the delegation for an NFSv4
server. It will not do so for an NFSv4.1 server, since the SEQUENCE
operation makes it quite trivial for the server to detect that this is
the client that holds the delegation for that file.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email-masked]

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 13 Apr 2021 17:42:48 -0400

> the client that holds the delegation for that file.

OK, and the client's DELEGRETURN does end up waiting for the SETATTR
result in this case?

--b.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 14 Apr 2021 00:16:27 +0000

[email-masked] wrote:
>> > Who's at fault in this case?
Well, here's the para. from pg. 483 of RFC5661:
   If the object whose attributes are being changed has a file
   delegation that is held by a client other than the one doing the
   SETATTR, the delegation(s) must be recalled, and the operation cannot
   proceed to actually change an attribute until each such delegation is
   returned or revoked.  In all cases in which delegations are recalled,
   the server is likely to return one or more NFS4ERR_DELAY errors while
   the delegation(s) remains outstanding, although it might not do that
   if the delegations are returned quickly.

>  Should the server guarantee that it
> > will
> > delegation?
I think that, in general, an NFSv4.1 should always reply to every RPC
in a reasonable amount of time. I've never seen a time limit specified
in the RFCs, but I've always assumed < 1second.

>  (It should be able to in this case--this being 4.1, it
> > that client's delegation.)
I think the server has two choices for the case where the Setattr is from
the same (and only) client as the one that holds the delegation:
- Allow the client to retain the delegation and not expect a Delegreturn.
or
- Start the action of sending a CB_RECALL to the client and then reply
  NFS4ERR_DELAY for the Setattr RPC.
  Then keep replying to the Setattr with NFS4ERR_DELAY until the
  Delegreturn is received and done.

>  Or can the client figure out some way to
> > not
Not sure what you mean here?
If the server did a CB_RECALL, then I think the client is required
to do a Delegreturn, but this may take a while (flushing writes
to the server for a Write delegation, for example), so I do not
think the server can safely wait for the DelegReturn for too long
and may need to reply NFS4ERR_DELAY until it happens.

If the server decides that it is "same client" and chooses to
not CB_RECALL, then it should not expect a DelegReturn.

As Trond notes, there are cases where the client knows that
it needs to return the delegation, but I think it is safer for
the server to be prepared to do a CB_RECALL, if the client does not
precede the RPC that requires a delegation be returned with
a DelegReturn (in a preceding RPC or before the operation that
requires the delegation be returned in the same compound).

I don't recall if the RFCs ever state that the client needs to
return a delegation or if the server needs to recall it, but
I think the server should "play it safe" and do the CB_RECALL
if required.

>OK, and the client's DELEGRETURN does end up waiting for the SETATTR
As above, I think the server should reply NFS4ERR_DELAY while "waiting"
for a delegation to be returned.
--> A really *slow* case is where the CB_RECALL needs to be done to a
      different client and that client's back channel is down, plus it is not
      currently doing normal RPCs to the server.
      --> It eventually does a Sequence by itself, sees the
            SEQ4_STATUS_CB_PATH_DOWN, then does a
            BindConnectionToSession. As a result, it could take up to a
            lease duration to even get the CB_RECALL done. Then the
            client takes its time writing back dirty data before doing the
            DelegReturn.
I don't think a server can wait *forever* for a DelegReturn, but it
has to wait for a lease duration+ before giving up and throwing
the delegation away.

At least that's my thinking on this, rick

--b.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 14 Apr 2021 03:14:45 +0000

On Wed, 2021-04-14 at 00:16 +0000, Rick Macklem wrote:
> > > > I'm seeing what looks like a deadlock between a 4.1+ client
> > > > waiting
> > > > for
> > > > Cc'ing Trond as I'm pretty sure I remember him discussing
> > > > exactly
> > > > this

I agree on all the above points made by Rick.

---

From: David Noveck <[email-masked]>
Date: Wed, 14 Apr 2021 08:12:33 -0400

I agree as well.  However, I'm wondering what  could be done to make this
area clearer in rfc5661bis.  Maybe we need to do more to stress the point
that delegations protect clients from one another so there would be no
recall in the same-client case.