Subject: RE: [nfsv4] RECLAIM_COMPLETE operation

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 12 Jan 2006 17:18:26 -0500

> The bug here is on client A's part: it can detect that the server 
> rebooted before it had a chance to finish reclaiming everything.  So 
> it should know that it's not safe to attempt to reclaim locks again. 

This is the kind of issue/assertion that caused us to fail to come to
a consensus at the IETF meeting.  But rather than argue right away
about this assertion (I will later because I don't think it is true),
let me note first that if this assertion is true and that clients
should or are requried to do this in v4.0, then this strengthens the
case for RECLAIM_COMPLETE in v4.1.

So let me restate what I understand Bruce's requirement to be.

    If a client sends at least one lock reclaim request (either for a
    byte-range lock or a reclaimed open), and the server reboots before
    the client has successfully reclaimed all of its locks, then that
    client MUST not attempt to reclaim its locks upon that further 
    server reboot (or any subsequent reboot) and all the applications
    must deal with the loss of their locks.

So it seems to me that if this is correct for v4.0, then it is something
we'd like to get rid of in v4.1.  And RECLAIM_COMPLETE would make this
unnecessary, since it would allow the server to determine whether in
fact
a client could have lost a lock due to an incomplete reclaim.  The
server
would be able to do this in a more precise way, so that for example, if
the server rebooted before the grace period was over, making it
impossible
for the client to have lost his lock at all, the original clients would
not be foreclosed from reclaiming their locks.

So now let me address Bruce's assertion that the client trying to
reclaim
his lock in this situation has a "bug".  The first thing to note is that
there is nothing in the spec that that says the client should not
reclaim
his lock, at least as far as I can see.  In other words, if it "should 
know that it's not safe to attempt to reclaim locks again", it isn't
because the spec tells it so.  If there is a bug here, it is the v4.0
spec that has a bug.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Thursday, January 12, 2006 2:52 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] RECLAIM_COMPLETE operation

On Thu, Jan 12, 2006 at 01:00:11PM -0500, Noveck, Dave wrote:
> Now let's proceed to a second, more involved motivation for such an 
> operation.  The problem here is that if a server reboots after a 
> client has reclaimed some of his locks, the server has no way of 
> determining whether the client has reclaimed all of them.  This is a 
> problem in the case in which a client has not recovered all of his 
> locks.  If there is a lock that the client has not reclaimed, then you

> can have a situation in which a lock inconsistent with that 
> not-yet-reclaimed lock can be given out to another client.  The result

> is that after another server reboot both clients may think they 
> legitamately own the lock and try to reclaim it.

I don't believe this can happen between correctly functioning v4.0
clients and servers.

I had some trouble understanding this situation, so here's my attempt to
make the sequence of events more concrete:

	Client A acquires two locks, x and y.
	The server reboots.
	Client A reclaims lock x.
	The server ends the grace period, and makes a record in stable
		storage indicating that client A successfully
		established some state.
	Client B gets a lock conflicting y' conflicting with y, and the
		server makes a record in stable storage indicating that
		client B also succesfully established some state.
	The server reboots again.

At this point, client A hasn't yet had the chance to attempt reclaim of
lock y, and hasn't seen any err_no_grace.  It does know that the server
rebooted again, because it gets stale errors on state it acquired after
the first reboot, but it doesn't know that the server actually completed
its grace period before rebooting.  So client A may think it's still OK
to attempt reclaiming lock x again.

And of course B should clearly attempt reclaim y' again.

And the server only knows that both are allowed to do reclaims, so has
no way to refuse the reclaims of either y or y'.

The bug here is on client A's part: it can detect that the server
rebooted before it had a chance to finish reclaiming everything.  So it
should know that it's not safe to attempt to reclaim locks again.

> Adding the RECOVERY_COMPLETE op allows this situation to be dealt with

> fairly simply.  If a client has not recovered all of its locks and we 
> have the possiblity of having given out a lock inconsistent with one 
> of those (the normal realization of this would be that once we declare

> grace over with some client's reclaims not complete) we mark that 
> client as essentially having had a lock effectively revoked and thus 
> it would not allowed to reclaim locks after a subsequent reboot since 
> it could no longer vouch for all the locks it thinks it had.

So I think that what RECOVERY_COMPLETE actually does is permit a v4.1
client to attempt reclaim in some cases where normally it wouldn't be
safe to.  But, OK, I guess I've convinced myself that RECOVERY_COMPLETE
helps here.

--b.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 12 Jan 2006 17:47:55 -0500

> case for RECLAIM_COMPLETE in v4.1.

Agreed.

>     byte-range lock or a reclaimed open), and the server reboots
>     before the client has successfully reclaimed all of its locks,
>     then that client MUST not attempt to reclaim its locks upon that
>     further server reboot (or any subsequent reboot) and all the
>     applications must deal with the loss of their locks.
> 
> So it seems to me that if this is correct for v4.0, then it is
> something we'd like to get rid of in v4.1.  And RECLAIM_COMPLETE would
> make this unnecessary, since it would allow the server to determine
> whether in fact a client could have lost a lock due to an incomplete
> reclaim.  The server would be able to do this in a more precise way,
> so that for example, if the server rebooted before the grace period
> was over, making it impossible for the client to have lost his lock at
> all, the original clients would not be foreclosed from reclaiming
> their locks.

I agree completely.  (And RECLAIM_COMPLETE sounds like a great idea.)

> reclaim his lock in this situation has a "bug".

Instead of saying that such a client has a bug, we could say that such a
client is willing to risk silent data corruption.  I find the
distinction between the two statements rather subtle.

> The first thing to note is that there is nothing in the spec that that
> says the client should not reclaim his lock, at least as far as I can
> see.  In other words, if it "should know that it's not safe to attempt
> to reclaim locks again", it isn't because the spec tells it so.  If
> there is a bug here, it is the v4.0 spec that has a bug.

The spec is very far from being a description of how to write a correct
client and server.  This certainly isn't the only case where writing a
correct implementation requres inferring behavioral requirements that
aren't in the spec.

I agree that such cases are bugs in the spec--at the very least, they
make it much more difficult to read--but they are also bugs in the
implementation.

If certain behavior is the only way that the protocol gives a client to
avoid data corruption, then a client that doesn't behave in that way has
a bug.

--b.