Subject: [nfsv4] New reclaim-related operations for v4.1

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 1 Feb 2006 06:48:56 -0500

This note describes the RECLAIM_START and RECLAIM_COMPLETE operations,
proposed for inclusion in NFSv4.1.  It lays out the problems within 
v4.0 that are solved by these additions and explains how these problems
are addressed.  Note that this is slightly different from what was
mentioned on the call (two ops rather than one op and an attribute)
in order to allow this to be optional for both the client and the 
server.  Things could be simplified a bit if we decided this was OK
to make mandatory.     

Background:

Locking poses difficulties in the face of server failure in that it is
necessary to provide some way of continuing operation, in a semantically
appropriate way, after server reboot.  It is not acceptable for NFSv4
to simply have the clients lose their locks (as happens in CIFS).  Also,
in order to meet NFS's goals for wide deployment, solutions which 
require the server to maintain a large amount of rapidly changing
state such as the corpus of all locks in a persistent fashion are 
likewise not acceptable.

RFC 3530 deals with the situation by putting the burden on the client 
to "reclaim" his locks, i.e. to re-obtain them upon noticing a server 
reboot.  The client is trusted to only reclaim locks that the server
has granted the client, in a previous server reboot instance.  In order
to allow all clients to reclaim their locks, and thus establish a new
lock state baseline for the new server instance, a "grace period" is
defined, in which the server will allow reclaims but not allow new locks
to be obtained.  This period must be at least as long as the lease 
period.  This is so that the client, if he has any locks at all, will 
be forced to communicate with the server in a way which ensures that 
it will find about the server reboot (via a stale stateid or clientid 
error) before the grace period ends, and thus have an opportunity to 
reclaim its locks during the defined grace period.  

If a client fails to reclaim a lock during the defined grace period, 
the server cannot generally grant a later request to do so, even if no 
conflicting lock has been given out.  This is because the semantics 
that the client requires for a successful reclaim is not only that the 
lock is available, but that no conflicting lock has ever been granted 
to any other client within the duration of the original client's 
possession of the lock.  Once the grace period ends and normal lock 
requests are acted upon, a server, without maintaining a large amount 
of persistent lock state, cannot make such a determination, and so must 
not grant reclaim-type requests.

A scheme such as this, in which the server relies on the client to 
establish locking state normally within the province of the server to
determine poses the issue of when it is appropriate for the server to
trust the client to provide this state and thus act on it, and thus
treat the client as the definitive source of information about locks 
that the client asserts he owns.

One part of this question concerns trust and security.  Put simply, we
have the issue of a client which lies and asks for the reclaim of locks
that it does not possess.  RFC 3530 does not specifically address this
issue, but the basic requirements are that reclaim requests must be
properly authenticated by a suitable principal, i.e. one with the proper
access to the file in question.  At that point, the question becomes
one of trust in the client implementations who possess and pass on the
credentials.  Just as we have to trust them not to give out data 
obtained by one principal and cached to someone that does not have the
rights to it, we similarly have to trust the client to obey the reclaim
protocol.  Note that if clients are not behaving properly in this 
regard, the server can expect situations in which a reclaim, during the
grace period encounters a conflicting.  Such a situation indicates that 
at least one client, not necessarily the one encountering the error, is
not obeying the protocol.

The second part of this question concerns situations in which the client
faithfully reclaims only locks he has been granted, but in situations
in which, because of a lack of the proper information, it is not valid,
or at least not safe, to incorporate that lock within the corpus of
re-established lock state for the server.

Such situations can arise as a result of network partitions in which 
the lock state can be changed by the server in ways that the client,
because of the network partition, is unaware of.  The most difficult-
to-deal-with such situations combine network partitions with one or
more server failures.  

Section 8.6.3 of RFC3530 deals with such issues.  It presents an 
algorithm whereby a server may persistently store only a small set of 
state for each locking client  and determine whether one of a number
of "edge conditions" apply that make lock reclamation by a client not
valid.

Analysis of edge conditions:

These edge conditions arise when a client is granted a lock and records
that lock's existence but there is a later a change in locking state, 
made authoritatively by the server, which causes that lock to be invalid,
without any assurance that the client has been notified of this change
in locking state.  Under such conditions, the client's reclaim of that 
lock cannot be safely granted.

Section 8.6.3 deals with two cases in which such a situation can arise
and provides an algorithm whereby a server can safely determine whether
they might apply, thus inhibiting reclaims.

The first situation in which a lock obtained by a client may cease to
be validly held is revocation, either within or outside the lease 
period.  When a lock is revoked, the client may still believe it to be
held, and thus, after a server reboot, acting on a reclaim would not be
valid.  The algorithm described in section 8.6.3 maintains persistently
a single revocation bit for each client to indicate the potential 
existence of such revoked locks.  When it is set, a client will not 
be allowed to do reclaims.  While the server is free to maintain more 
detailed information about the locks revoked, it is reasonable to 
refuse all reclaims for that client in this situation, as described in 
the algorithm laid out in section 8.6.3.

A second situation in which a lock, validly held by a client, can be
lost is through server reboot during a network partition.  In this
case, the client may be granted a lock, there may be a server reboot,
and because of a network partition, other clients may reclaim locks, 
the grace period may end, and then conflicting locks may be granted,
all during a period in which the original client has no knowledge of
these events.  In such a case, a new server reboot after a healing
of the partition may result in a situation in which the client believes
he has a valid lock and proceeds to attempt to reclaim it, despite the
fact that it was lost during a previous server reboot instance, of
which the client had no knowledge.

The algorithm in section 8.6.3 deals with this situation by maintaining
a timestamp for each client that represents the point in time at which
the client first obtained any locking state.  When this is combined 
with timestamps for the last two server boot instances, the server can
determine whether it is possible for a client to have lost locks simply
by not being around to reclaim them, at the time of the last server 
reboot.

One similar edge condition not addressed by section 8.6.3 concerns 
a client which loses a lock, which it previously obtained, not by
being down throughout a server reboot, but rather by being partitioned
from the server after reclaiming one lock, but before reclaiming all
of them.  Any such lock not reclaimed can be lost, since, once the
grace period terminates, a conflicting lock can be granted, without
either a revocation which the server can note, or the absence of 
of a client throughout a server reboot instance.  When a further
reboot occurs, if the client attempts now to reclaim all of its locks,
it may reclaim a lock that it is not entitled to have and for which 
there may be a conflicting lock.  It is this newly-recognized edge
condition, not known at the time that RFC3530 was written, that 
motivates changes being proposed for v4.1.
  
Solving the new edge condition:

The basic issue that makes this situation hard to address within the
existing protocol, is that when a client fails to reclaim all of its
locks, the server has no basis to note that anything unusual has 
happened.  Given that it doesn't know how many locks the client has
to reclaim, the fact that the client had some unreclaimed at the time
of a conflicting lock is granted is something that the server is not
in a position to know.  The server simply sees the grace period end
and a new lock being granted, but there is no way to single out a
particular client as one who might lose a lock previously granted as
a result of that new lock.

It is therefore the case that for v4.0, it is the clients, rather than 
the servers who have to take action to prevent this situation, even 
though this action is not explicitly mandated by RFC3530.  In order to
prevent the inappropriate reclamation of locks, clients must not, when 
they have attempted to reclaim one lock but have not reclaimed all 
their locks and there is a server reboot, ever try to again reclaim 
those non-reclaimed locks on a subsequent reboot.  Those locks must be
considered to be at least potentially lost and so not subsequently 
reclaimable.

This solution, while it prevents inappropriate reclaims, is not really
satisfactory in that it can cause excessive locking failures.  If a 
server should have a hardware a problem, it is not unheard of for it 
to take more than a single reboot to get things back to normal and
it is highly undesirable for such a situation to result in clients
being unable to subsequently reclaim their locks.

Consider the case in which a client does a SETCLIENTID and associated
confirm and sends at least one reclaim request to the server and then
gets no response, and then a bit later sees another server reboot.  An 
omniscient observer might see that that reclaim was lost and never made
it to the server, making a subsequent reclaim perfectly valid.  Or,
the reclaim may have got to the server but the server rebooted or at 
least ceased to function long before the end of the grace period.  Note
that only the end of the grace period and the grant of a new (i.e. not
reclaimed) lock can trigger any difficulty.  The client, however, only
sees that it sent a request and didn't get a reply and it is perfectly
possible, from its point of view, that the server received the reclaim
request, processed it, continued through to the end of the grace period,
granted a lock which would conflict with other locks that the client
held, and only then rebooted.  The result is that by being safe, as we
have to be in v4.0, many client applications will be sorry, in that 
they will lose their locks.

New reclaim-related operations:

In order to cleanly correct this situation, the server needs better 
information about the client's state with regard to lock reclamation,
and more particularly it needs to be able to isolate those clients
that have begun and not yet completed lock reclamation so that, when
the grace period ends, and new locks are given out, it can mark those
clients that have not completed reclaim, and only those clients, as
having potentially lost locks.

To accommodate these requirements while allowing this handling to be
optional (albeit highly recommended), two new operations will be 
defined in NFSv4.1: RECLAIM_START and RECLAIM_COMPLETE.

Both operations are optional but if the server supports one, it must
support the other.  Similarly, clients may or may not use these 
operations, but if a client uses one, the server will expect that
the client will use the other, and a failure to do so will have 
severe consequences for lock recoverability.

Both operations have a single parameter, the current clientid and
can return NFS4ERR_STALE_CLIENTID to indicate that the previously 
obtained clientid is no longer valid, and thus that a server reboot 
has occurred.

The RECLAIM_START operation is used to indicate to the server that
the client issuing it is capable of and intends to use RECLAIM_COMPLETE
to delimit its reclaim operations.  By accepting RECLAIM_START (and
not returning NFS4ERR_NOTSUPP) the server indicates that it supports 
both RECLAIM_START and RECLAIM_COMPLETE and the client should reclaim 
its locks on that basis.  In particular, if a server indicates that it
does not support these operations, than the failure to complete 
reclaiming of locks means that the client should not attempt to 
reclaim those locks after a further server reboot.  They are lost.

An efficient way of providing execution of the RECLAIM_START function
is to add it to the compound performing the SETCLIENTID_CONFIRM.
After the completion of the compound, locks may be reclaimed by
the client.  Any use of RECLAIM_START after one or more locks have
been reclaimed will result in an NFS4ERR_LOCKS_HELD error.  Since a 
client may encounter a server reboot at any time, he should get 
information about lock recoverablity before proceeding to do any 
reclaims.

The RECLAIM_COMPLETE operation is used by the client in v4.1 to indicate
to the server that it has done all of its reclaims and will not do
any more.  If the client has indicated to the server that it is 
capable of doing RECLAIM_COMPLETE and the server indicates that it
supports these operations, the client needs to do a RECLAIM_COMPLETE
after completing his lock reclaims.  If it does not, any further 
server reboots may result in an inability of the client to reclaim
its locks since the server will think that the client was still in 
its lock reclamation phase.

Once a client issues a RECLAIM_COMPLETE, any subsequent reclaim-type
requests will result in an NFS4ERR_NO_GRACE error.  Servers, may, and 
are encouraged to, keep track of all clients which may validly reclaim 
locks, and note when all such have done a RECLAIM_COMPLETE.  When this 
situation occurs, servers may terminate the grace period and grant
normal lock requests even though the normal timer for the grace period
has not expired.  Implementation of such an abbreviated grace period
makes it more acceptable, from an availability viewpoint to err on 
the side of safety in establishing relatively long worst-case graced
period since this longer period will only be used when it is actually
necessary.

In v4.1, the RECLAIM_COMPLETE operation is optional.  A server which
does not support this operation should return the error NFS4ERR_NOSUPP, 
although most clients will typically note the absence of support when
doing a RECLAIM_START.

RECLAIM_COMPLETE may validly be issued after the grace period has ended.
When a client has so many locks that he cannot reclaim all of his locks
in the time available, he should still do the RECLAIM_COMPLETE when
he finds out that the grace period is over.  If he does not, the server
will typically not allow him to reclaim any of his locks after a 
subsequent reboot.  Therefore, the RECLAIM_COMPLETE should be done
as quickly as possible, to avoid a situation in which the server reboots
after the grace period is over, but before the RECLAIM_COMPLETE is
received from the client.  By sending the RECLAIM_COMPLETE, the client
assures that all the locks he has reclaimed can be reclaimed after a
subsequent reboot, while acknowledging the loss of those that it did 
have time to reclaim.

One technique that may be used to reduce the window between the last
reclaim and the RECLAIM_COMPLETE is to issue the RECLAIM_COMPLETE as
part of the COMPOUND which reclaims the last lock.

When a server supports the RECLAIM_COMPLETE operation, the client, if
it perceives a server reboot before completing the reclaiming of all of
its locks need not avoid future reclaims on those not yet reclaimed, 
in the case in which there is a server reboot.  The server has the job
of determining whether those reclaims can be done safely since 
RECLAIM_COMPLETE provides the necessary information.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: David Robinson <[email-masked]>
Date: Wed, 01 Feb 2006 17:08:29 -0600

Noveck, Dave wrote:
> proposed for inclusion in NFSv4.1.

...

> Things could be simplified a bit if we decided this was OK
> to make mandatory.     

I would make a strong request that we make this mandatory.

There is a clear distinction between adding a new "feature"
that some clients or server cannot, or may choose not
to implement, and one that is a refinement (well really a bug
fix) of an existing feature. Examples of the former that
make sense to be optional are things like new attributes or
pNFS. Existing systems work "just fine" without these new
options. In this case, we have discovered that the existing
specification is unclear and existing implementations may
be doing unsafe things.

Making the a mandatory part of 4.1 will allow us to improve
quality of all implementations.

I would be willing to support it being optional if there was
some clear argument that a class of implementations simply
could not build this.

	-David

---

From: Paul Mazzurana <[email-masked]>
Date: Thu, 2 Feb 2006 15:08:03 -0600

Dave, 
Comments below...

> to delimit its reclaim operations. 

Based on today's conference call, just need to cleanup this part of the 
write up to indicate the server should use the version info in the 
COMPOUND to determine how a client will be reclaiming state. 

> its lock reclamation phase.

In the case where a v4.1 client has reclaimed state, the grace period 
ends, then the client issues a non-reclaim request (say open with 
CLAIM_NULL) prior to issuing a RECLAIM_COMPLETE, for the server does that 
have any bearing (or should it have) on how the server "views" the 
client's reclamation? That is, may the server consider that the reclaims 
for the client are complete OR should it be MUST NOT consider it to be 
reclaim complete? 

> RECLAIM_COMPLETE provides the necessary information.

I'm not completely clear on this last para. What I think you are saying is 
that the server reboots BEFORE the grace period is over, correct?  Also, 
for example:

Given client BOB which has lock state on files X, Y, and Z on the server. 
Server reboots and goes into GP. BOB detects reboots and set client ID and 
begins reclaims. Reclaims all of state for X and Y. Server then reboots 
before exiting grace period and before BOB can initiate any reclaims for 
Z.
Server enters grace period, BOB detects reboot and initiates reclaim of 
state on X, Y. Should it [must it?] be allowed to reclaim state on Z? OR 
is it that BOB MUST NOT be allowed to reclaim any state since no 
RECLAIM_COMPLETE was done? 

Also, I am assuming that v4.1 client reclaims have no impact on v4.0 
client reclaims, even if they are against the same file(s) as v4.1. 
Correct? 

--
Paul D. Mazzurana
[email-masked]

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 2 Feb 2006 17:23:48 -0500

> for the client are complete OR should it be MUST NOT consider it to be 

The client is free to intermix regular opens and locks with reclaims
(and the server is free to reject them with ERR_GRACE); so the fact that
a client sends regular opens and locks says nothing about whether it's
done reclaiming.

> that the server reboots BEFORE the grace period is over, correct?

The problem is that if the client detects a reboot before it finishes
reclaiming, it doesn't know whether or not the server ended the grace
period (and granted other locks) before the server rebooted.

> Also, 
> state on X, Y. Should it [must it?] be allowed to reclaim state on Z?

If BOB is a 4.0 client, it shouldn't reclaim state on Z, because it
can't tell whether the server ended the grace period and granted a
conflicting lock before rebooting.

If BOB is a 4.1 client, it should go ahead and try reclaiming the state
on X, Y, and Z--if the server did end the grace period, then it will
reject all of BOB's reclaims.   If the server didn't get to the end of
the grace period, then it will grant all of BOB's reclaims.  (Actually,
the server may be more lenient in the former case if it happens to
remember more about the lock state before the reboot, but that's the
basic idea.)

> OR 

Right.

> client reclaims, even if they are against the same file(s) as v4.1. 

Right.

--b.