Subject: [nfsv4] Lock reclaim improvements

From: David Noveck <[email-masked]>
Date: Fri, 4 Feb 2022 08:09:00 -0500

The following note explains the potential EXCHANGE_ID extension I mentioned
in discussions with Rick.  I will be delaying following up until
security-05 is out and possibly longer than that.

*Introduction*
All versions of NFS use the same approach to assuring lock state continuity
in the face of server reboot, even though the nature of the locking state
changed dramatically in Nfsv4.  Because opening a file is now required  to
access a file, necessarily obtaining a new lock or reclaiming one,  the
lock reclamation process has become more demanding and the consequent
interference with ongoing operations more troublesome.

This note will discuss a few ways of addressing this situation without
making major changes in the lock reclamation model:

   - providing helpful implementation guidance regarding the efficient use
   of lock persistence to limit the duration of lock reclamation.
   - defining potential extensions to NFSv4, that would allow lock
   reclamation to be less disruptive of client activity.

*Lock reclaim in NFS  v2 and v3*
Lock reclaim in v2 and v3 is, in practice, not very disruptive, for two
reasons:

   - a large portion of processes do not obtain byte-range locks and so are
   not affected by the reclamation process.
   - even for processes that do use byte-range locks, they can continue to
   do IO and only need to wait for lock reclamation if they need to obtain
   additional byte-range locks.

*Lock reclaim in NFSv4*
Because of necessary changes to make NFSv4 a stateful protocol, the demands
on lock reclamation have increased and many client activities cannot
proceed until lock reclamation is complete.

   - almost all processes need to obtain locks to access files causing most
   client activities to be suspended until lock reclamation is complete.
   - because of this fact, there are environments in which there are
   typically many locks to reclaim, making the time necessary to complete
   reclamation unnecessarily long.  Aa a result, the delay in resuming normal
   operation can become onerous.

*Lock Classification*
It will be helpful to divide locks that might be reclaimed into the
following classes:

   - byte-range locks.
   - true share reservations, i.e opens with a non-zero deny mode.

These are, <http://are.in> in many environments, quite rare since Posix
file access interfaces have no provisions to create them.

   - trivial share reservations, i.e. opens with zero specified as the deny
   mode.

These are often the most common locks and the need to reclaim them before
continuing to do IO is a potential trouble spot.

Other reclaimable locks, such as delegations can be similarly classified
into the above two levels based on whether multiple instances of the same
lock are compatible or conflict.  Thus, read delegations can be classified
with trivial share reservations and write delegations with non-trivial ones.
*Taking Note of and using Lock Classification in Reclamation*
Existing explanations of lock reclamation explain why new locks cannot be
granted until all locks are reclaimed. It is not noted, and should be, that
the following important exceptions exist:

   - Once all non-trivial share reservations are reclaimed, new trivial
   share reservations can be granted, since they cannot conflict with existing
   ones.
   - Once all byte-range locks are reclaimed, new byte-range locks can be
   granted.

*Using (Limited) Lock Persistence to streamline Lock Reclamation*
One possible approach to lock reclamation difficulties is to eliminate the
need for it to be done by providing, on the server, persistent storage of
lock state. However, doing such recording universally faces enough
implementation difficulties that it is worthwhile to consider alternatives.

One important element in providing alternatives is to limit the set of
locks that need to be persistently recorded.  Further reductions involve
limiting, for particular classes of locks, the information to be stored,
potentially all the way down to a single bit for each client, indicating
whether the client has any reclaimable lock of that class.

   - If only byte-range locks and non-trivial share reservations are
   persistently recorded, then new locks, other than non-trivial share
   reservations,  can be granted immediately, since any locks not persistently
   recorded cannot conflict with new lock.
   - If only non-trivial share reservations are persistently recorded, and
   the server supplements this by noting the clients that do possess
   byte-range locks then new requests for trivial share reservations from
   clients that do not possess byte-range locks can be granted immediately.
   Only clients that do possess byte-range locks or that try to obtain new
   non-trivial share reservations would have to wait until the end of lock
   reclamation.
   - If a server knows the set of clients who might possess byte-range
   locks or non-trivial share reservations it can consider the reclamation
   process partially completed once the union of these two sets of clients
   have signaled RECLAIM_COMPLETE. Once this happens, requests for new
   byte-range locks or trivial share reservations can be granted immediately
   while requests for non-trivial share reservations need to wait for
   global completion of the reclamation process.

*Possible Lock Reclamation Extensions.*

The above improvements take good advantage of many common situations but it
possible to provide extensions to provide better support for lock
reclamation, even where non-trivial lock reservations are common. Such
extensions would provide:
   - A way for the client, at EXCHANGE_ID time, to indicate the classes of
locks to be reclaimed.
  - New facilities allowing the client to indicate the completion of
reclamation of each class of locks.
The result would be to eliminate the negative consequences of combing
reclamation of byte-range locks and share reservations while taking
advantage of the asymmetry between trivial and non-trivial share
reservations.

*EXCHANGE_ID Extensions*

It is anticipated that four flags would be added to the flags field in the
EXCHANGE_ID arguments and response.

The first flag would be used by the client to indicate it's knowledge of
and intent to use the extension and by the server to indicate its support
of the extension.

The other three flags would each be assigned to one of the specific lock
classes described above and would be used as follows:

   - In the request, the flag would indicate that the client has locks of
   the corresponding class to reclaim.
   - In the response, the flag would indicate that the client needs to
   reclaim locks of that class. These flags would be a subset of those in the
   request and would be adjusted to reflect, for example, server knowledge
   based on lock persistence. When these flags are all zero, no lock
   reclamation by the client is required.

*RECLAIM_STATUS Operation*

This new operation would be used by clients to signal the completion of
reclamation of one of the class of locks.

The argument would consist of a set of flags indicating the classes of
locks whose reclamation has not yet completed.  When this operation is
called more than once by a single client-id, the argument for each
successive call is expected to be a subset of the previous one and the
server is able to conclude that reclamation for the classes of locks now
not in the argument has been completed, possibly allowing more types of new
locks to be granted.

The use of this operation is not obligatory and RECLAIM_COMPLETE should be
considered equivalent to a RECLAIM_STATUS with an argument of zero.

*Documents to be Changed/Written*

It is being assumed that:
   - An improved description of lock reclamation, including lock
classification outlined above and significant implementation guidance,
could be done as part of the rfc5661bis effort.
   - the extensions discussed above could be done as part of a NFSv4.2
extension document, normatively referencing rrc5661bis.

---

From: [email-masked] (J. Bruce Fields)
Date: Fri, 4 Feb 2022 11:26:57 -0500

Looks interesting, thanks.

>    non-trivial share reservations would have to wait until the end of lock

Why can't we immediately grant a trivial share reservation to a client
that posesses byte-range locks?

> not in the argument has been completed, possibly allowing more types of new

Just curious: what evidence do we have about how much that will help?

I mean, in theory, it could be that most of the time is typically spent
waiting for clients to notice that the server has rebooted, rather than
waiting for clients to actually do the reclaims.

--b.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Jeff Layton <[email-masked]>
Date: Fri, 04 Feb 2022 12:46:34 -0500

> waiting for clients to actually do the reclaims.

That probably depends on how onerous the underlying reopening process
is. If you're re-exporting a netfs of some sort (for instance), then
reclaim can take quite a while.

-- 
Jeff Layton <[email-masked]>

---

From: David Noveck <[email-masked]>
Date: Fri, 4 Feb 2022 15:21:50 -0500

> that posesses byte-range locks?

The problem is that when you grant the trivial share reservation to the
client, it assume the grace periods is over
and then when he tries to reclaim a byte-range lock you have to tell him he
is in the grace period, whereupon he
says "WTF, you tell me the grace period is over and then it isn't.  Are
you crazy?" or other words to that effect.

> Just curious: what evidence do we have about how much that will help?

Evidence requires an implementation and so far, this is just a notion.

> waiting for clients to notice that the server has rebooted,

It could be and that is usually the case in NFsv2/v3.

> rather than

I've definitely heard complaints from customers who have thousands of locks
to reclaim and are dissatisfied with NFSv4 because there is a long period
in which IO cannot happen, when there is a server reboot/failover.

---

From: Chuck Lever III <[email-masked]>
Date: Fri, 4 Feb 2022 20:27:54 +0000

> On Feb 4, 2022, at 3:21 PM, David Noveck <[email-masked]> wrote:
> 
> I've definitely heard complaints from customers who have thousands of locks to reclaim and are dissatisfied with NFSv4 because there is a long period in which IO cannot happen, when there is a server reboot/failover.

Me too. However, I always attributed that to servers that did not
implement grace period shortening after RECLAIM_COMPLETE. That
turns out to be hard to do in environments that have NFSv3 and
NFSv4.0 and NFSv4.1 clients accessing the same server.

I appreciate very much the new implementation guidance. But I
wonder whether we truly need new protocol here. I can't really
tell at this point.

--
Chuck Lever

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 4 Feb 2022 16:08:57 -0500

On Fri, Feb 04, 2022 at 03:21:50PM -0500, David Noveck wrote:
> The problem is that when you grant the trivial share reservation to
> the client, it assume the grace periods is over and then when he tries
> to reclaim a byte-range lock you have to tell him he is in the grace
> period, whereupon he says "WTF, you tell me the grace period is over
> and then it isn't.  Are you crazy?" or other words to that effect.

Oh, interesting, got it.

But I don't think the client's ever been able to assume that one
succesful non-reclaim guarantees that later non-reclaims won't get
NFS4ERR_GRACE.  See for example the bottom of:

	https://datatracker.ietf.org/doc/html/rfc7530#page-121

which does allow the server to grant non-reclaim lock requests based on
partial information, even while it is still hasn't ended the grace
period completely (and hence may still be returning NFS4ERR_GRACE to
other requests).

I've also always assumed it's OK for a clustered server to return
NFS4ERR_GRACE mid-operation if it needs to wait for another server's
clients to reclaim (even though its own clients don't need to).

I wouldn't expect a client to do anything more complicated than just
delay and retry.

--b.