Subject: [nfsv4] "Courtesy locks"

From: <[email-masked]>
Date: Sun, 12 Sep 2010 12:21:43 -0400

The term "courtesy locks" is Trond's.  It refers to the practice of
allowing locks associated with an expired lease to remain around, "as a
courtesy to the client or as an optimization" (see section 8.6.3 or RFC
3530).  The text that is associated with this situation is not as clear
or complete as it should be and a number of RFC3530bis issues are
related to it.

In deciding how to address this, we need to find out what exactly
clients and servers do.  A long while back I promised to send mail and
gather up the requisite information, and now my promise has caught up
with me.  If you can answer the following questions via email that would
be great.  If not, I hope people will at least find out what their
implementations do on these points so we discuss it at the upcoming
bakeathon.

This looks pretty daunting but most people will probably say "Yes" to
(S1) and/or easily deal with (C1), (C2), and (C3).

Make sure you use black pen and completely fill in the circles :-) 

If you have a client and not a server, skip to (C1)

S1) When a lease expires, do you simply release all
    the associated locks?

    If so, go to (C1), or if you don't have a client you 
    are all done.

S2) If you do maintain locks for a longer time, when and if
    there is a conflict before the lease being reactivated,
    what do you do?

    A) Release all the locks associated with that client
    B) Release all the locks held by that client for that 
       same fh.
    C) Something more limited than (B) that includes the lock
       that had the conflict.

    If you answered (A) or (B), skip down to (S7).

S3) If the lock that had a conflict is a byte-range lock,

    S3.1) Is the associated open released as well?
    S3.2) Are all byte-range locks associated with the
          same open released as well?
    S3.3) Are all byte range locks associated with the same
          client/lock-owner released as well?
    S3.4) Are all byte-range locks associated with the 
          same open and owned by the client/lock-owner
          released as well?
    S3.5) If the lock is associated with an open that is
          subordinate to a delegation, is the delegation
          released as well?
    S3.6) Are there any other locks that are released?

S4) If the lock that had a conflict is an open?

    S4.1) Are byte-range locks associated with this open
          released as well?
    S4.2) If the open is associated with a delegation,
          is the delegation released as well.

S5) If the lock that had the conflict a delegation?

    S5.1) Are opens associated with that delegation
          released as well?
    S5.2) Are all opens for that client and for that fh
          released as well?
    S5.3) Are all byte-range locks associated with open
          that are associated with that delegation released
          as well.
    S5.4) Are all byte-range locks for that client and for
          released as well?

S6) Are there any other situations in which release of a 
    lock due to a conflict will cause other locks to be
    released?

S7) After releasing locks due to a conflict, what happen if
    the associated stateid is referenced, assuming that 
    happens relatively quickly?

    S7.1) The client gets NFS4ERR_EXPIRED?
    S7.2) The client gets some other error?  What?
    S7.3) Can it be different for different types of stateids?

S8) Assuming that the release does not cause immediate freeing
    of the stateid (and returning NFS4ERR_BAD_STATEID), what 
    provision is made for eventual deletion of such stateids?

    a) There is an LRU mechanism to eventually delete them.
    b) They are kept around until the client or server 
       reboots.
    c) It depends on the type of lock.

If you have no client, you are done.

C1) When you get NFS4ERR_EXPIRED, what happens?

    a) Assume all locks associated with lease have been released
       that the client needs to create a new clientid?
    b) Assume that it means one or more locks have been lost and
       the client needs to determine what as gone and which are
       still valid.
    c) Something else.  What?

C2) If you didn't answer (a) to (C1), are there any assumptions
    (of the sort mentioned in (S3) to (S6)) whose violation would
    cause problems? 

C3) How do you respond to getting the following errors?  Specifically
    would it be problematic if that error were returned when a
    courtesy lock was released due to a conflict:

    C3.1) NFS4ERR_ADMIN_REVOKED
    C3.2) NFS4ERR_BAD_STATEID

---

From: Rick Macklem <[email-masked]>
Date: Sun, 12 Sep 2010 18:48:39 -0400 (EDT)

> S1) When a lease expires, do you simply release all
No
> there is a conflict before the lease being reactivated,

Expire all open/lock state related to the clientid at that
point. (A) assuming "locks" refers to Open shares as well as
byte range locks.
> S7.1) The client gets NFS4ERR_EXPIRED?
Yes to S7.1
> S7.2) The client gets some other error? What?
No
> S7.3) Can it be different for different types of stateids?
No
> c) It depends on the type of lock.
All stateids related to the clientid are released and all
subsequent uses of them get NFS4ERR_EXPIRED.
> C1) When you get NFS4ERR_EXPIRED, what happens?
A pseudo recovery:
- new SetCLientID, SetClientIDConfirm
- Fresh Opens (Claim_Null) for all Opens that don't have
  byte range locks associated with them (via open_to_lock_owner).
--> If the Open attempt fails or the Open has byte range locks
    associated with it--> Open thrown away and I/O may fail,
    etc.
 
> that the client needs to create a new clientid?
Yes, as above.
> C3.2) NFS4ERR_BAD_STATEID
Both are considered terminal errors and are just passed up to
the syscall. The sysadmin may need to umount/mount to get
around these errors.

These answers apply to the FreeBSD8.1 client/server, rick

---

From: Trond Myklebust <[email-masked]>
Date: Sun, 12 Sep 2010 19:13:43 -0400

>        the client needs to determine what as gone and which are

C) We try to send a RENEW. If that fails, we try to send a SETCLIENTID
that uses the same clientid and verifier as when we originally mounted
the filesystem.
After that, we try to replay all OPEN and LOCK calls.

>     (of the sort mentioned in (S3) to (S6)) whose violation would

In principle not. We are supposed to be able to recover all locks on a
per-open_stateid basis.

IOW: if an open stateid, lock stateid or delegation stateid returns an
error, we should be able to replay just the OPEN+LOCK requests that are
required to allow the application to proceed.

Note that we do not implement a SIGLOST feature in the case where the
application has lost a lock (although we probably should). We never did
it for NFSv3, and since nobody has really requested it for NFSv4 either,
we have deferred implementing it there too.

>     C3.2) NFS4ERR_BAD_STATEID

We replay the OPEN+LOCK requests, and if that fails, we try again using
the zero stateid.

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Sun, 12 Sep 2010 23:22:31 -0400 (EDT)

> After that, we try to replay all OPEN and LOCK calls.

Doesn't expired imply a conflicting lock could have been issued and
then released by now. ie. It doesn't seem safe to re-acquire a byte
range lock. (My client re-acquires Opens because they are all DENY_NONE,
so my client doesn't care about Opens with DENY_xxx, where xxx is not NONE.)

> we have deferred implementing it there too.

So you are saying apps. might believe that they still hold a byte
range lock, even if another client might have acquired/released a
conflicting lock while it was network partitioned?

---

From: Mike Mackovitch <[email-masked]>
Date: Mon, 13 Sep 2010 09:06:43 -0700

On Sun, Sep 12, 2010 at 12:21:43PM -0400, [email-masked] wrote:
> [...]

>        the client needs to determine what as gone and which are

(b)

Upon receipt of NFS4ERR_EXPIRED the client will enter recovery mode
where we will attempt to reclaim all opens, all locks and delegated
state (if we didn't have non-delegated state that already covered it)
and drop/return any lingering delegations we had.

If a file *only* has DENY_NONE opens (no locks), any failure to reclaim
that open will only result in us attempting to open the file again.

>     (of the sort mentioned in (S3) to (S6)) whose violation would

I don't think so.  We try not to assume much.

If any state reclamation for a node fails, we will attempt to return
all state we may have for it (delegation, locks, opens) even though
the server may give us errors for those requests (better to be safe
and try to tell the server we don't think we have that state anymore).

We don't assume that other state is also lost... we just check everything.

For any nodes that fail to properly reclaim state, the node is marked
as "revoked" and attempts to access it will get I/O errors (Darwin
doesn't have SIGLOST).

>     C3.2) NFS4ERR_BAD_STATEID

Both of these are handled just like EXPIRED (above).

HTH
--macko

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 13 Sep 2010 13:02:12 -0400

> so my client doesn't care about Opens with DENY_xxx, where xxx is not NONE.)

We don't ever use deny modes in Linux.

> conflicting lock while it was network partitioned?

Yes. What I said above is that there is no mechanism in Linux (or POSIX
for that matter) for notifying apps that they have lost their locks.
Implementing SIGLOST might be one way to do so, but requires apps to be
rewritten to make use of it.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 13 Sep 2010 13:02:12 -0400

Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 13 Sep 2010 13:16:49 -0400

> S1) When a lease expires, do you simply release all

Yes.

>     If so, go to (C1), or if you don't have a client you 

That was easy!

--b.

---

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 13 Sep 2010 13:27:40 -0400

> -----Original Message-----
> From: Myklebust, Trond
> Sent: Monday, September 13, 2010 1:02 PM
> To: Rick Macklem
> Cc: [email-masked]
> Subject: Re: [nfsv4] "Courtesy locks"
> 
> > > IOW: if an open stateid, lock stateid or delegation stateid
returns
> an
> > > error, we should be able to replay just the OPEN+LOCK requests
that
> > > are
> > > application has lost a lock (although we probably should). We
never
> > > did

An alternative is to indicate the potential loss of lock state by doing
something like failing I/O done with the affected file descriptor.  This
assumes that the application actually intended some semantic content to
be associated with the locks they held.  The file descriptor could start
working again if the lock were released.

Sure, that's something of an application rewrite as well.  But it does
reflect the distributed state to the application.

Now where's my POSIX book...

		Craig

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 13 Sep 2010 13:02:12 -0400

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 13 Sep 2010 14:18:12 -0400

> working again if the lock were released.

That won't necessarily work for applications that want to do I/O on one
file while using a separate 'locking file' on which they take the byte
range locks.

The other point is that POSIX locking isn't really associated to a file
descriptor; it is associated to a file and to a process or set of
threads. A consequence of this is that locks don't follow fork(), even
though open file descriptors do. I'm assuming this is the main reason
why the Solaris folks decided to go with a signalling mechanism
(SIGLOST) back when they were struggling with the same problem on NLM.

> Now where's my POSIX book...

Indeed...

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Mon, 13 Sep 2010 19:26:38 -0400 (EDT)

> We don't ever use deny modes in Linux.

Yep, the FreeBSD8 client doesn't either. Sorry for the confusion.
(ie. It always Opens with DENY_NONE.)

> > > returns an
You could always, as the Governor of California might say:
"terminate them":-).

I haven't solved this for FreeBSD8 either. (It has no SIGLOST.) I
think what happens is:
- any further attempts at lokc ops fail because there isn't any
  open/lock state.
- I/O ops. fall back to using the special stateid of all 0s and
  succeed or fail at the whim of the server.

rick

---

From: Rick Macklem <[email-masked]>
Date: Mon, 13 Sep 2010 19:38:38 -0400 (EDT)

> and drop/return any lingering delegations we had.

The FreeBSD8 server will only allow reclaim Ops during the grace
period right after a reboot. It will return NFS4ERR_NO_GRACE for
all reclaim Ops (Open with Claim_previous or Lock with the reclaim
flag true) received outside of this grace period.

Do other servers allow this? (I thought the above was required
behaviour?)

[good stuff snipped for brevity]

rick

---

From: Mike Mackovitch <[email-masked]>
Date: Mon, 13 Sep 2010 17:00:19 -0700

> Do other servers allow this? (I thought the above was required

I'm not currently working on any NFSv4 server implementation, so I
can't answer your question.  :-)

But if the server is not allowed to process reclaim ops outside the
grace period then there's no hope of nicely handling the case where
the server's grace period passes before the client gets a chance to
reconnect to the server.

In the context of "courtesy locks" I don't think it makes any sense
for a server to keep the state around if it's going to tell the client
that the state expired but not allow the state to be reclaimed.
(Wasn't the point of the "courtesy" to allow them to be reclaimed if
there hadn't been any contention?)

Anyway, I'm simply coding our client to try to allow it to recover as
much state as the server will allow.

--macko

---

From: Rick Macklem <[email-masked]>
Date: Mon, 13 Sep 2010 21:13:30 -0400 (EDT)

> the server's grace period passes before the client gets a chance to

Yep. But grace periods must be greater than a lease duration and I'd
assume clients will be trying hard to re-connect.

> there hadn't been any contention?)

What the FreeBSD server does (and what I assumed was meant by courtesy
lock) is retain the state after the lease has expired (so long as no
conflicting request is received)so that the state is still valid at
the server and NFS4ERR_EXPIRED does not need to be replied to the client.

I'd appreciate hearing, if others have a different understanding of it.

rick

---

From: Mike Mackovitch <[email-masked]>
Date: Tue, 14 Sep 2010 08:57:35 -0700

> assume clients will be trying hard to re-connect.

Yes.  But sometimes NFS client code doesn't have control over all the
network links between it and the server (e.g. a VPN that needs to be
reconnected by the user).

--macko

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 14 Sep 2010 13:07:07 -0400

> I'd appreciate hearing, if others have a different understanding of it.

I think the idea was that you could revoke only those locks that had to
be thrown away due to conflict, return EXPIRED, and then allow reclaims
only of those locks that you hadn't been forced to revoke.

--b.

---

From: <[email-masked]>
Date: Tue, 14 Sep 2010 15:55:30 -0400

> only of those locks that you hadn't been forced to revoke. 

I think the word "reclaim" is being used in a confusing way, or I'm
confused, or both.

While I think you could argue that it is reasonable to use the word
"reclaim" for any situation in which you try to get back a lock that was
lost due to lease expiration or any other cause, I think this adds to
confusion.  I suggest something like "re-establish" and reserve
"reclaim" for the case in which you use a reclaim-type OPEN or LOCK
operation (typically during the grace period).

BTW, the spec does allow you to honor reclaim-type operations after the
grace period if you can be sure that no conflicting lock could have been
granted from the time the server came up until that OPEN.  (Conflicting
locks before the server booted this time are domain of "edge
conditions").  In general, getting that assurance is very difficult, but
in the often common case in which OPENs with anything other than
DENY=NONE are very rare, you can do something very simple.

If the RECLAIM open is with DENY=NONE, and there has never been an open
with anything other than DENY=NONE, then there could have not been any
such lock granted, and you can grant the RECLAIM-type open.  In less
restrictive circumstances things get more difficult but depending on
what information you are willing to save (lists of fh's with various
sorts of locks), you may be able to handle a large proportion of clients
that would otherwise fall victim to expiration of the grace period and a
consequent inability to reclaim their locks. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of J. Bruce Fields
Sent: Tuesday, September 14, 2010 1:07 PM
To: Rick Macklem
Cc: [email-masked]
Subject: Re: [nfsv4] "Courtesy locks"

--b.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 14 Sep 2010 16:34:06 -0400

> operation (typically during the grace period).

Yes, agreed, and that's the sense which I meant in the above.

> conditions").  In general, getting that assurance is very difficult,

Well, if there's been no server reboot--if a client just expires and the
server hangs on to only those locks (and opens) that don't see
conflicts--that may be another case where may be able to selectively
grant reclaims without a whole lot of work.

--b.

> but

---

From: Rick Macklem <[email-masked]>
Date: Tue, 14 Sep 2010 19:59:38 -0400 (EDT)

> reconnected by the user).
Agreed that NFSv4 will not work well if the network connection is
intermittent (or for disconnected operation) at least in part because
recovery from NFS4ERR_EXPIRED can't be done well. (It really depends
on the server providing "courtesy locks" and a better understanding
of recovery for that case than I currently have.)

I had assumed that the client wouldn't need to do anything for
"courtesy lock" recovery (ie. the stateids would simply keep working).
Others seem to think that a "reclaim" operation is needed to recover
them so that they can be used.

It is also not obvious to me if/when a fresh SetClientID/SetClientIDConfirm
will be required to get past a NFS4ERR_EXPIRED error. (I had assumed it
would be necessary and do it whenever I see NFS4ERR_EXPIRED.)

It would be nice to hear how a server that does selective
"courtesy locks" (by selective, I mean expires some but not all StateIDs
for a ClientID when the client fails to Renew its lease within the lease
duration) expects a client to recover from this?
Does a Netapp filer do this? (or is there any current server that does
selective "courtesy locks"?)

rick

---

From: Mike Mackovitch <[email-masked]>
Date: Wed, 15 Sep 2010 00:25:37 -0700

> would be necessary and do it whenever I see NFS4ERR_EXPIRED.)

FWIW, we will do a SETCLIENTID every time we reconnect the socket - just
to make sure the server has the latest callback info (the client's address
may have changed).  And actually we will do a RENEW just before that - to
try to quickly flush out any errors like EXPIRED so that way we can start
recovery immediately.

--macko