Subject: [nfsv4] RECLAIM_COMPLETE operation

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 12 Jan 2006 13:00:11 -0500

This message concerns the possibility of adding an operation to
v4.1.  This operation was discussed some at IETF-60.  The slides
at http://www3.ietf.org/proceedings/04aug/slides/nfsv4-6.pdf were
presented at that meeting.

At that time there was a lot complicated discussion about the 
underlying recovery problem and what could be done about it and it 
wasn't clear if we were reaching consensus, at least in the case 
of how to deal with the issue in v4.0.

However, I think there may be a consensus about the simpler issue 
of simply avoiding the problem by adding an op to v4.1 and I'd like 
to explore that alternative, and see if we can reach a consensus, or 
uncover a consensus already in existence (I'm an optimist :-).

So the simple statement of the proposal is that we add an operation
which clients would do, once they had reclaimed all of their 
locking state.  I think it would be best for this not to be optional
for clients in v4.1, despite the normal preference for optional 
features, since it is very easy for clients to implement such an 
operation and it much simplifies the server to not have to deal
with clients who don't do this.  If this is required, servers would
know that a client who didn't indicate that recovery was complete 
had not completedd recovery.  Servers would be allowed not to 
support it although I would expect that servers typically to do so.

One important (and I hope relatively non-controversial) benefit
of this is that it would allow a simple way to cut short the
grace period in the common case in which all clients quickly 
gather their locks.  We would expect that while severs would like 
the grace period to be as short as possible (for better availability),
the need to accommodate cases in which there are various delays 
plus the fact that the grace peiord and the lease time are interrrelated

will constrain our ability to make this really short.  We will wind 
up needing to set this based on worst-case considerations, while having
to bear that cost all the time.  It would be far better to allow long
lease times to be set while allowing us to pay the cost of a longer
grace period only when it is in fact needed.

This is where a reclaim-complete operation would come in.  Servers
need to keep a list of clients who may have locks (for reasons that
are discussed in section 8.6.3 of RFC 3530).  Given that, once there 
is a RECOVERY_COMPLETE, for each such client, the server can be 
sure that all client lock reclaimation is completed and can safely 
grant new locks, even where the grace period based on the lease
period would be considerably longer.

Now let's proceed to a second, more involved motivation for such 
an operation.  The problem here is that if a server reboots after a
client has reclaimed some of his locks, the server has no way of
determining whether the client has reclaimed all of them.  This
is a problem in the case in which a client has not recovered all
of his locks.  If there is a lock that the client has not reclaimed,
then you can have a situation in which a lock inconsistent with
that not-yet-reclaimed lock can be given out to another client.
The result is that after another server reboot both clients may
think they legitamately own the lock and try to reclaim it.  The
result of the reclaim not done is very similar to that of a lock
being revoked, but unlike the case of a lock being revoked, the
server has no good way to mark that revocation and prevent an
inconsistent situation.

Adding the RECOVERY_COMPLETE op allows this situation to be dealt
with fairly simply.  If a client has not recovered all of its locks
and we have the possiblity of having given out a lock inconsistent
with one of those (the normal realization of this would be that once 
we declare grace over with some client's reclaims not complete) we 
mark that client as essentially having had a lock effectively revoked
and thus it would not allowed to reclaim locks after a subsequent 
reboot since it could no longer vouch for all the locks it thinks
it had.

Anyway, I think we need to determine whether there is a consensus
on going ahead with this op for v4.1.  I'd argue that this is very
simple extension to implement that has two significant benefits. 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Marc Eshel <[email-masked]>
Date: Thu, 12 Jan 2006 11:10:21 -0800

I can not make todays conference call. I think that this a good idea that 
will improve high availability.
Marc.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 12 Jan 2006 14:52:01 -0500

On Thu, Jan 12, 2006 at 01:00:11PM -0500, Noveck, Dave wrote:
> Now let's proceed to a second, more involved motivation for such an
> operation.  The problem here is that if a server reboots after a
> determining whether the client has reclaimed all of them.  This is a
> problem in the case in which a client has not recovered all of his
> locks.  If there is a lock that the client has not reclaimed, then you
> can have a situation in which a lock inconsistent with that
> not-yet-reclaimed lock can be given out to another client.  The result
> is that after another server reboot both clients may think they
> legitamately own the lock and try to reclaim it.

I don't believe this can happen between correctly functioning v4.0
clients and servers.

I had some trouble understanding this situation, so here's my attempt to
make the sequence of events more concrete:

	Client A acquires two locks, x and y.
	The server reboots.
	Client A reclaims lock x.
	The server ends the grace period, and makes a record in stable
		storage indicating that client A successfully
		established some state.
	Client B gets a lock conflicting y' conflicting with y, and the
		server makes a record in stable storage indicating that
		client B also succesfully established some state.
	The server reboots again.

At this point, client A hasn't yet had the chance to attempt reclaim of
lock y, and hasn't seen any err_no_grace.  It does know that the
server rebooted again, because it gets stale errors on state it acquired
after the first reboot, but it doesn't know that the server actually
completed its grace period before rebooting.  So client A may think it's
still OK to attempt reclaiming lock x again.

And of course B should clearly attempt reclaim y' again.

And the server only knows that both are allowed to do reclaims, so has
no way to refuse the reclaims of either y or y'.

The bug here is on client A's part: it can detect that the server
rebooted before it had a chance to finish reclaiming everything.  So it
should know that it's not safe to attempt to reclaim locks again.

> Adding the RECOVERY_COMPLETE op allows this situation to be dealt with
> fairly simply.  If a client has not recovered all of its locks
> and we have the possiblity of having given out a lock inconsistent with
> one of those (the normal realization of this would be that once we
> declare grace over with some client's reclaims not complete) we mark
> that client as essentially having had a lock effectively revoked and
> thus it would not allowed to reclaim locks after a subsequent reboot
> since it could no longer vouch for all the locks it thinks it had.

So I think that what RECOVERY_COMPLETE actually does is permit a v4.1
client to attempt reclaim in some cases where normally it wouldn't be
safe to.  But, OK, I guess I've convinced myself that RECOVERY_COMPLETE
helps here.

--b.

---

From: David Robinson <[email-masked]>
Date: Thu, 12 Jan 2006 14:53:24 -0600

> should know that it's not safe to attempt to reclaim locks again.

But how can client A know that it is a bug or not? The client knows
how long the grace period is, but not when it started.  So it could
have reclaimed x at the very start or very end of the server's
reclaim. Until it attempts to reclaim y it won't know if
it is allowed.  If the server reboots during the middle of
the reclaim period, it would be perfectly fine for client A
to attempt to reclaim x and y as the server would not have
allowed y'. But if client B gets y' instantaneously after the
grace period ends and then the server reboots, client A
should not try to reclaim y. But it cannot know which case it is.

What really needs to happen is that the server tracks era's and
communicates to the client which era the locks that it has
acquired belong to.  An era is defined as the period from the
end of a grace period until the completion of the next grace
period.  Failure to complete a grace period due to a reboot
does not end that era.  Attempts to reclaim locks from a
previous era within the current grace period MUST fail.

Possible suggestion, change the "bool reclaim" in the LOCK4args
into an "int era", if non-zero the it is a reclaim from the
specified era. We could then extend the LOCK4resok to also return
the era.

During reclaim, on successful reclaim the era returned is of
the era that will exists after the grace period ends. This
introduces the possibility that if the server reboots before
the end of the grace period and the full establishment of the
new era, on next grace period the client may be trying to
reclaim locks from last established era and some number
of started but incomplete grace periods. This should not
be a problem as long as the server maintains all possible
era values that it has sent out on stable storage until
it has completed a grace period and established a new era.
Once that new era has been established it can discard
any information about previous eras.

The era can be an opaque value, as long as the server has a
method to ensure that the era is no reused for very long
time periods. Simple examples would be an incremented
integer or absolute boot time in UTC. In theory using a
32 bit field should be enough, as 4 billion server reboots
with a single client still holding an era value is virtually
impossible.

We could also reserve the values 0 and 1. The former means
it is not a reclaim, the latter would allow an easy way
to reuse the 4 bytes that were a bool and detect misuse.

> safe to.  But, OK, I guess I've convinced myself that RECOVERY_COMPLETE

I agree it is useful to allow faster recovery of attentive clients.

	-David

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 12 Jan 2006 17:12:10 -0500

On Thu, Jan 12, 2006 at 02:53:24PM -0600, David Robinson wrote:
> reclaim.  Until it attempts to reclaim y it won't know if it is
> allowed.  If the server reboots during the middle of the reclaim
> period, it would be perfectly fine for client A to attempt to reclaim
> x and y as the server would not have allowed y'.  But if client B gets
> y' instantaneously after the grace period ends and then the server
> reboots, client A should not try to reclaim y. But it cannot know
> which case it is.

Exactly.  And since the client can't detect which situation it's in, the
only safe (hence the only correct) thing for the client to do is to not
attempt the reclaims.  So a client that proceeds with the reclaims
anyway has a bug.  We agree?

After the addition of RECOVERY_COMPLETE, it becomes safe for a client to
proceed with reclaims in this situation, because it knows it can leave
detection of the problem above to the server.

This means that previously there were cases where a client would have to
give up and just issue SIGLOST's, but where now the client has a chance
of recovery.

> communicates to the client which era the locks that it has acquired
> belong to.  An era is defined as the period from the end of a grace
> period until the completion of the next grace period.  Failure to
> complete a grace period due to a reboot does not end that era.
> Attempts to reclaim locks from a previous era within the current grace
> period MUST fail.

That might work too.  Could you describe a case where that would work
better than v4.0 + RECOVERY_COMPLETE?

--b.

---

From: David Robinson <[email-masked]>
Date: Thu, 12 Jan 2006 17:42:12 -0600

> anyway has a bug.  We agree?

Agreed. I stumbled on what was the "bug".

> better than v4.0 + RECOVERY_COMPLETE?

I believe that RECOVERY_COMPLETE will work to prevent the
bug/silent corruption, the client will simply try to
recover all of the locks that it thinks it has and can
leave the server to arbitrate, but it is an awfully coarse hammer
to use.

If client A has sent RECOVERY_COMPLETE, regardless of when the server
reboots again it is safe to allow client A recover any locks
in the second grace period. However, if the server has not yet
received a RECOVERY_COMPLETE when it crashes, it cannot know if
the client has in fact done all of its recovery (e.g. server crashed
when the RECOVERY_COMPLETE was on the wire) or not. So its
only defense is to embargo that client and not allow it to
recover anything in the second grace period.

If I have a large client with a huge number of locks, and it
sucessfully recovers almost all of them, which would allow
most processes to continue, a lone straggler lock that did not
get recovered in time will cause the many successful
processes to get SIGLOST'd if the server reboots.

So it is server safe, but it just seems anti-social to
my client. The addition of an era will allow a server to
sort which recovery requests are safe and which are not.
If a process got all of its lock back but one, it will
still get SIGLOST'd, but the other unrelated processes
that got all their locks back are unharmed. The impact to
make this recovery more fine grained is very small, the
server only needs to save a few more words to persistent
storage during recovery and the client send a few more bits
when trying to reclaim.

	-David