Subject: RE: [nfsv4] RECLAIM_COMPLETE operation

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 12 Jan 2006 20:20:17 -0500

> If client A has sent RECOVERY_COMPLETE, regardless of when the server
> reboots again it is safe to allow client A recover any locks
> in the second grace period. However, if the server has not yet
> received a RECOVERY_COMPLETE when it crashes, it cannot know if
> the client has in fact done all of its recovery (e.g. server crashed
> when the RECOVERY_COMPLETE was on the wire) or not. 

I think this ignores an important point.  If a server crashes before
having received the RECLAIM_COMPLETE from client A, there is not a 
problem with allowing client A to recover his locks, unless the 
grace period is over.  If the grace period is not over, then there 
is no chance of a lock being granted that conflicts with any client
A's not-yet-reclaimed locks.  So client A only has a problem if it
does not send a RECLAIM_COMPLETE by the time the grace period is 
over.

> So its
> only defense is to embargo that client and not allow it to
> recover anything in the second grace period.

That's one defense.  If we don't know that a client has reclaimed
all his locks then, once the grace period is over, any lock that
is newly granted should be treated as potentially revoking a
lock held by client A, and so marking A as unable to reclaim,
as in section 8.6.3, is reasonable.

> If I have a large client with a huge number of locks, and it
> sucessfully recovers almost all of them, which would allow
> most processes to continue, a lone straggler lock that did not
> get recovered in time will cause the many successful
> processes to get SIGLOST'd if the server reboots.

I think it is reasonable, if a client has very large number of
locks for a server to hold off ending the grace period as long
as the client is making reasonable progress reclaiming them.

Also, it is up to the server whether he wants to use the big 
hammer of preventing all lock reclaims.  Just as in the case of
a revoked lock, the server is free to keep more persistent state
and thus be more discriminating.

In the case you cite, the server could decide, when transitioning
from the grace period to normal operation, to write all of the 
locks successfully reclaimed by client A to stable storage, and
then after a crash allowing those and only those to be reclaimed
the second time.

-----Original Message-----
From: David Robinson [mailto:[email-masked]]
Sent: Thursday, January 12, 2006 6:42 PM
To: [email-masked]
Subject: Re: [nfsv4] RECLAIM_COMPLETE operation

J. Bruce Fields wrote:
> On Thu, Jan 12, 2006 at 02:53:24PM -0600, David Robinson wrote:
> 
>>But how can client A know that it is a bug or not? The client knows
>>how long the grace period is, but not when it started.  So it could
>>have reclaimed x at the very start or very end of the server's
>>reclaim.  Until it attempts to reclaim y it won't know if it is
>>allowed.  If the server reboots during the middle of the reclaim
>>period, it would be perfectly fine for client A to attempt to reclaim
>>x and y as the server would not have allowed y'.  But if client B gets
>>y' instantaneously after the grace period ends and then the server
>>reboots, client A should not try to reclaim y. But it cannot know
>>which case it is.

> Exactly.  And since the client can't detect which situation it's in, the
> only safe (hence the only correct) thing for the client to do is to not
> attempt the reclaims.  So a client that proceeds with the reclaims
> anyway has a bug.  We agree?

Agreed. I stumbled on what was the "bug".

> That might work too.  Could you describe a case where that would work
> better than v4.0 + RECOVERY_COMPLETE?

I believe that RECOVERY_COMPLETE will work to prevent the
bug/silent corruption, the client will simply try to
recover all of the locks that it thinks it has and can
leave the server to arbitrate, but it is an awfully coarse hammer
to use.

If client A has sent RECOVERY_COMPLETE, regardless of when the server
reboots again it is safe to allow client A recover any locks
in the second grace period. However, if the server has not yet
received a RECOVERY_COMPLETE when it crashes, it cannot know if
the client has in fact done all of its recovery (e.g. server crashed
when the RECOVERY_COMPLETE was on the wire) or not. So its
only defense is to embargo that client and not allow it to
recover anything in the second grace period.

If I have a large client with a huge number of locks, and it
sucessfully recovers almost all of them, which would allow
most processes to continue, a lone straggler lock that did not
get recovered in time will cause the many successful
processes to get SIGLOST'd if the server reboots.

So it is server safe, but it just seems anti-social to
my client. The addition of an era will allow a server to
sort which recovery requests are safe and which are not.
If a process got all of its lock back but one, it will
still get SIGLOST'd, but the other unrelated processes
that got all their locks back are unharmed. The impact to
make this recovery more fine grained is very small, the
server only needs to save a few more words to persistent
storage during recovery and the client send a few more bits
when trying to reclaim.

	-David

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: David Robinson <[email-masked]>
Date: Thu, 12 Jan 2006 20:23:45 -0600

> does not send a RECLAIM_COMPLETE by the time the grace period is 

Yes, you are correct that within a grace period everything is OK.
But the core problem is that the client has no way to ensure
that it will send a RECLAIM_COMPLETE within the grace period
window, even if it has actually recovered everything within
the grace period.

> as the client is making reasonable progress reclaiming them.

The trouble I have is that we have a protocol that requires
a server to create some ill defined heuristics on when
it should extend the grace period, with the negative side
effects of restricting access by new clients. The server
has to say "enough" after some point in time, but if
bad things happen, the consequences are pretty severe.

It just makes more sense to me that by adding a small amount of
information, the server exposing when it has transitioned
out of the grace period, the accuracy of recovery is increased
greatly.

> then after a crash allowing those and only those to be reclaimed

But the server also now needs to catch any other clients accessing
those file ranges in order to remove them from the list of
reclaimable locks.  In the extreme, a server can maintain on
stable storage the complete lock state and make recovery
trivial.

If we are willing to fix this "bug", why not do it in a manner that
adds a minimal amount of additional server state and maximizes
the amount of state the client can recover. By exposing state
in NFSv4 users are much more likely to hit lost/dead files
that would have been silently covered over in the past. While
it may be the right thing to do, no more silent corruption, it
is also important to only inflict that new pain when it is
absolutely necessary.  If not they are just going to stick with v3.

	-David