Subject: RE: [nfsv4] RECLAIM_COMPLETE operation

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 19 Jan 2006 08:48:24 -0500

First of all I don't have a fundamental objection to adding this
sort of knowledge to the locking/state model although I haven't 
yet been convinced that that is the way to go.  If there is a 
consensus that we should do that for v4.1, I would be OK with 
that.  My main concern is that the working group choose some way 
of dealing with this issue, that provides at least the benefits 
that RECLAIM_COMPLETE does.  More benefits are fine, as long as 
the complexity cost is within reason.

The problems I see with the scheme proposed, although they may just 
be a lack of understanding on my part, are:

      It solves the reboot-reclaim interaction problem as well as
      RECLAIM_COMPLETE, but it doesn't allow a grace period to be
      cut short.

      In terms of the amount of v4.1 change, it is a lot more complex.  
      First of all, in addition to reclaim of byte-range locks, we also 
      have reclaim of opens to deal with.  Also, I don't think that 
      changing a boolean to an int in the XDR, as you suggest, although 
      it can be made to work from a byte-stream interpretation point of 
      view, is in accord with minor versioning rules.  I don't think we 
      can change the XDR definition for existing ops.  Also, I don't see
      how we can extend the response for existing ops to include this 
      new information.  I think we are looking at new RECLAIM_LOCK and 
      RECLAIM_OPEN ops, and maybe more, rules for the reclaiming of v4.0
      locks on v4.1, etc.  If the feature is optional then you also have

      to deal with reclaiming across a transition between v4.1 servers 
      which don't and which do support this new feature.  I think there 
      is a lot of complexity here around the transition.  Of course if 
      we'd thought of this in the beginning, putting this into v4.0, it
      would have been pretty much  a no-brainer, but without a time 
      machine ...

So it comes down to a disagreement on whether there are significantly
greater benefits, and I'm having trouble with your arguments, in that
I think they overstate the size of the issue.

> > I think this ignores an important point.  If a server crashes before
> > having received the RECLAIM_COMPLETE from client A, there is not a 
> > problem with allowing client A to recover his locks, unless the 
> > grace period is over.  If the grace period is not over, then there 
> > is no chance of a lock being granted that conflicts with any client
> > A's not-yet-reclaimed locks.  So client A only has a problem if it
> > does not send a RECLAIM_COMPLETE by the time the grace period is 
> > over.

> Yes, you are correct that within a grace period everything is OK.
> But the core problem is that the client has no way to ensure
> that it will send a RECLAIM_COMPLETE within the grace period
> window, even if it has actually recovered everything within
> the grace period.

So there is a small window between the time that you have recovered
all your locks and when the server receives your RECLAIM_COMPLETE.
However I don't think that the client has to "ensure that it will send
a RECLAIM_COMPLETE within the grace period."  It is perfectly alright
to send the RECLAIM_COMPLETE after the grace period.  After the grace
period ends, you cannot reclaim additional locks, but there is no
reason that you can't tell the server that you are not going to 
reclaim any additional locks.

So now consider the case in which a client has a lot of locks to 
reclaim and simply doesn't get done in time.  When it gets NOGRACE,
it has to give up on reclaiming additonal locks.  By sending a
RECLAIM_COMPLETE, the client acknowledges that and tells the server
that he will not try to reclaim those additional locks on future
reboots and so the server is now OK with allowing this client to
do reclaims on subsequent reboots.  

It is true that there is a window here where the client will not be
able to reclaim any lock, but it is a small one.  What has to happen 
is that the client will have to run out of time to do reclaims and
then the server, before receiving the RECLAIM_COMPLETE, has to reboot.
If the server goes down before the end of the grace period or after
receiving the RECLAIM_COMPLETE, there is no issue. 

> > I think it is reasonable, if a client has very large number of
> > locks for a server to hold off ending the grace period as long
> > as the client is making reasonable progress reclaiming them.
>
> The trouble I have is that we have a protocol that requires
> a server to create some ill defined heuristics on when
> it should extend the grace period, with the negative side
> effects of restricting access by new clients. The server
> has to say "enough" after some point in time, but if
> bad things happen, the consequences are pretty severe.

NFSv4.0 and NFSv4.1 have a fixed recovery period.  If a client
cannot recover his locks in that time, then he will lose some.
The server can accommodate a client which is making progress,
at the cost of restricting access by others who want to get new
locks, or he can choose not to do so.  The protocol doesn't
require him to do that and most servers probably won't.  But
this is a problem that existed in v4.0 and there has been no
proposal to change the protocol in this respect in v4.1.

> It just makes more sense to me that by adding a small amount of
> information, the server exposing when it has transitioned
> out of the grace period, the accuracy of recovery is increased
> greatly.

It depends on how you measure it.  Just to use some semi-random
numbers as an example.  This might improve lock recovery accuracy
from 99.99% to 99.999% which you could say is is a .009% improvement
in accuracy and accuracy is only improved by one part in ten thousand,
Or you could say that inaccuracy goes from .01% to .001% for a 
ten-fold improvement.   Which of those is more appropriate is a
matter of judgment.

> > In the case you cite, the server could decide, when transitioning
> > from the grace period to normal operation, to write all of the 
> > locks successfully reclaimed by client A to stable storage, and
> > then after a crash allowing those and only those to be reclaimed
> > the second time.

> But the server also now needs to catch any other clients accessing
> those file ranges in order to remove them from the list of
> reclaimable locks.

I think this can be handled by mechanisms that are already described 
in section 8.6.3.  Any case in which any of these locks reclaimed
by the client would be taken by someone else would require a revoke
and we have to have to have a mechanism to mark a client with revoked 
locks as unable to reclaim, unless you go to the next level of 
sophistication, and record finer-grained information about revoked 
locks.  

> In the extreme, a server can maintain on
> stable storage the complete lock state and make recovery
> trivial.

Sure but my point is that this is not very many steps along the way 
on the path from the simplest reasonable implementation to one that
embodies the dream (or nightmare) of persistent storage of locks and
absolutely correct recovery.  You are only writing lock state when 
there is a major system state transition, rather than on every lock 
and free.

> If we are willing to fix this "bug", why not do it in a manner that
> adds a minimal amount of additional server state and maximizes
> the amount of state the client can recover. 

I agree that this does not add much server state but I think that in
its currently proposed form, it just adds too much complexity to the
protocol.  I've been toying with the idea of identifying eras with 
the client-id of the client which initially got or last reclaimed 
the lock in question.  That would not require any changes to the 
lock and open responses since you always know your client-id and 
thus your era.  It does leave open the issue of how you communicate
that to the server on reclaim without adding multiple new ops, but
maybe we could add a piece of per-session state with only a single
op to set it.  I think the basic issue is whether this problem is
important enough to merit the effort.  I think we need to spend
some time discussing this at the next v4.1 conference call.

> By exposing state
> in NFSv4 users are much more likely to hit lost/dead files
> that would have been silently covered over in the past. While
> it may be the right thing to do, no more silent corruption, it
> is also important to only inflict that new pain when it is
> absolutely necessary.  If not they are just going to stick with v3.

My concern is that we not add too much complexity to v4.1, and 
thus delay it.

-----Original Message-----
From: David Robinson [mailto:[email-masked]] 
Sent: Thursday, January 12, 2006 9:24 PM
To: [email-masked]
Subject: Re: [nfsv4] RECLAIM_COMPLETE operation

Noveck, Dave wrote:

> I think this ignores an important point.  If a server crashes before
> having received the RECLAIM_COMPLETE from client A, there is not a 
> problem with allowing client A to recover his locks, unless the 
> grace period is over.  If the grace period is not over, then there 
> is no chance of a lock being granted that conflicts with any client
> A's not-yet-reclaimed locks.  So client A only has a problem if it
> does not send a RECLAIM_COMPLETE by the time the grace period is 
> over.

Yes, you are correct that within a grace period everything is OK.
But the core problem is that the client has no way to ensure
that it will send a RECLAIM_COMPLETE within the grace period
window, even if it has actually recovered everything within
the grace period.

> I think it is reasonable, if a client has very large number of
> locks for a server to hold off ending the grace period as long
> as the client is making reasonable progress reclaiming them.

The trouble I have is that we have a protocol that requires
a server to create some ill defined heuristics on when
it should extend the grace period, with the negative side
effects of restricting access by new clients. The server
has to say "enough" after some point in time, but if
bad things happen, the consequences are pretty severe.

It just makes more sense to me that by adding a small amount of
information, the server exposing when it has transitioned
out of the grace period, the accuracy of recovery is increased
greatly.

> In the case you cite, the server could decide, when transitioning
> from the grace period to normal operation, to write all of the 
> locks successfully reclaimed by client A to stable storage, and
> then after a crash allowing those and only those to be reclaimed
> the second time.

But the server also now needs to catch any other clients accessing
those file ranges in order to remove them from the list of
reclaimable locks.  In the extreme, a server can maintain on
stable storage the complete lock state and make recovery
trivial.

If we are willing to fix this "bug", why not do it in a manner that
adds a minimal amount of additional server state and maximizes
the amount of state the client can recover. By exposing state
in NFSv4 users are much more likely to hit lost/dead files
that would have been silently covered over in the past. While
it may be the right thing to do, no more silent corruption, it
is also important to only inflict that new pain when it is
absolutely necessary.  If not they are just going to stick with v3.

	-David

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 19 Jan 2006 10:09:46 -0600

My personal tendency is to keep a solution to the issue of shortening
the grace period.  I agree with Dave's concern of adding too much
at this point of trying to complete a 4.1 specification.  If we find
that this particular portion of recovery becomes a significant issue
for recovery then we have 4.2 to fix it or server's will maintain
all NFSv4 state is stable storage to avoid the problem. :-)

Spencer

---

From: David Robinson <[email-masked]>
Date: Thu, 19 Jan 2006 11:51:05 -0600

Noveck, Dave wrote:
>  My main concern is that the working group choose some way 

I have no problems with RECLAIM_COMPLETE as specified, it solves
a real problem. But it does not solve the multiple reboot-reclaim
correctness problem, but it doesn't make it worse either.

>       new information.

I am agnostic about exactly how the changes get implemented in the
protocol.  The point of showing that converting a reclaim bit into
an era int, was mostly to indicate that it is not a big change.
 From an implementation perspective, it is hardly a rearchitecture.

>       I think we are looking at new RECLAIM_LOCK and 

I am not 100% happy with our minor version rules, but that is a
different story.  I would actually propose for cleanliness that
we create entirely new LOCK and OPEN ops that are supersets of
the existing ops. Adding auxilary or "helper" ops just seems real
messy. From an implementation side, supporting both ops would
likely share most of the same code path. If a client is trying to
reclaim with the old ops they get the old semantic that they get
only one shot at it. I would also want to make them mandatory to use.
Seems like not good reason to make something like this optional.

> receiving the RECLAIM_COMPLETE, there is no issue.

I agree that it is a small window, but the consequences are that
the client must SIGLOST all of the apps, including the ones
that were able to reclaim. That from a client vendor perspective
seems harsh, albeit "safe".

> proposal to change the protocol in this respect in v4.1.

I agree, my comments were simply in response to assertions
that the server can just keep extending the grace period
until the client is done. The protocol does not explicitly
allow that, although if there are no conflicting requests you
can get away with it.  This reliance on conditions that
may or may not be true, depending on uninvolved client
behavior, to try and minimize when the client has to
throw away all of its reclaimed locks, just seems like a
bad design resulting in users seeing seemingly inconsistent
behavior.

I don't think we need to shift to a variable sized recovery
period (that would be a lot of work), all I was proposing was
that the client and server have a way to communicate/determine
if a reclaim request was indeed one that had been previously
reclaimed or not. It would still be possible that a crash
after the recovery period ending would force the client
to destroy held locks, but only those locks.

> I think the basic issue is whether this problem is

I agree.

I think the RECLAIM_COMPLETE op has merit simply
based on the fact that it can allow a server to exit
recovery early. I have personally found it very annoying
to have to wait when I know recovery is done.

If we want to solve the multiple reboot-reclaim problem
that should be taken up as a separate issue.

	-David