Subject: RE: Some issues with stateid's

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 4 Nov 1999 06:30:51 -0800

> On Wednesday November 3, [email-masked] wrote:
> > 
> > The other case is lease expiration.  When a lease expires,
> > the server may let you keep the lock subject to revalidation,
> > if nobody else wants a conflicting lock.  I have been 
> > treating losing the lock after expiration, as a kind of
> > revocation.  In that case, you might lose one of your
> > locks due to a conflict but, but have others still 
> > available.  I agree this is of no value so maybe the
> > spec should say that once you lose one of your locks
> > due to expiration, the whole set of locks for that
> > file is revoked.  I think we would need to slightly 
> > change some definitions and do some rewording.
> > 
> > The most interesting case is if the leases for some of
> > your locks have expired but others have not.  Suppose
> > you have a lease interval of 2 minutes and do a lock
> > at 11:00, another at 11:01, and if there is no IO,
> > the lease for the lock will expire at 11:02, while the
> > lease for the second will expire at 11:03 and is subject
> > to indefintie renewal as long as anyone on the client
> > does IO.
> 
> This shows you have to read the spec carefully.... I was under the
> impression that "any successful use of a stateid" would renew all
> locks for all lockowners for the same client. But as you say, it
> isn't "any successful use" but is "any successful IO" (Section 8.4).
> Given that the I/O is seen as "a positive indication that the client
> is still alive", and that a lock request is also an indication that
> the client is still alive, should we change:
>  
>    Any I/O request that has been made with a valid stateid is 
> a positive
>    indication that the client is still alive and locks are being
>    maintained.  
> 
> to something like:
> 
>    Any I/O request or Locking or unlocking request with a valid
>    stateid, or any open request with a valid clientid, is a positive
>    indication that the client is still alive and locks are being
>    maintained.

I proposed something like this in:

http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1670.html

and Robinson responded:

http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1671.html

I think that locks and unlocks would be OK, but since that doesn't
get us all the way to a uniform expiration point, I don't
see that it's worth it.

> 
> or maybe:
> 
>    Any successful use of a valid stateid or a clientid is a positive
>    ..... 
> 
> That would neatly sidestep the different expiry times, though we still
> need a well defined resolution path from lock expiry.  Maybe bad
> (NFS4ERR_EXPIRED) stateids should still be allowed for LOCKU
> requests.  Afterall, the problem is that the lock is gone, so asking
> "please remove this lock" shouldn't be too onerous..

Maybe but since you have a lock which the server will take away
if anyone make a cnflicting request (whether you unlock it or not),
what's the point of allowing the unlock?  It doesn't really do 
anything.

> 
> One the topic of lease times though, something else (very small)
> occurred to me when I was reading through the debate about lease times
> (I think it was "locking errors" september last year).
> The issue was that some people thought that lease times should be
> long, because they had mission critical locks and frequent network
> partitions, and others thought they had to be short for reasonable
> recovery times.
> It occurred to me that this issue could be addressed by making lease
> times a statement about client behaviour rather than server
> behaviour.  It is a small change but it may help clarify things.
> 
> Rather than saying:
>    The server can revoke locks that have not been renewed within the
>    lease time - so the client better renew pretty often
> we could say:
>    The server can do whatever it likes (because we all know that it
>    can), but that if the client endevours to renew at least every
>    'leasetime', then the server will do it's best-effort to maintain
>    the locks for the client.

I don't see how this clarifies things. quite the reverse, it seems
to make things much more obscure.  Instead of focusing on what
actually happens, we are now dealing with intentions: what the
client endevours to do, and what the server tries to do.
 
> 
> The effect of this change was highlighted by a comment by Carl Beame 
> in 
>    http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/0692.html
> 
> that client should renew every 1/2 the lease time, just to be sure.
> Given that the server can hold on to a lease a little bit longer when
> it wants to (such as allowing rebooted clients to reclaim delegations)
> we have the situation where the client doesn't believe the leasetime,
> and halves it, and the server doesn't believe the leasetime and
> extends it.  The change would mean that the client should believe the
> leasetime, because it cannot gain by not doing so, and that the server
> is expressly permitted to treat the leasetime in a configurable way.
> 
> If, we tell the client precisely what to do (renew every leasetime -
> more often won't gain you anything), then the server admin can tune
> the server behaviour to conditions.
> A cautious admin with stable clients, dodgy networks, and plenty of
> spare time to chase up stray locks can tune the server to hold on to
> clients for hours or days.  A busy admin with stable networks
> and no real dependancy on locking could tune it right down so that
> locks expire moments after the leasetime.
> A default configuration would probably be to hold locks for 2 or 3
> times the leasetime.
> 
> This could certainly still be done with the current definition of
> "leasetime", but that would leave you wandering what exactly it really
> meant.
 
OK.  I see where you're going.  But in some contexts, the quality
of the network will not be controlled by the server.  It may be 
different for different clients.  I may be coming in over a 
DSL connection routed through a VPN over an ISP that the server administrator
knows nothing about.  My client can observe the 
latency for requests and get a good idea of the delays and 
adjust when he sends renewals so that they get there before 
the lease expires.  Since the client can determine real
network conditions based on actual results, it makes sense
for the burden to be on him, rather than having the server
essentially lie to him about lease times.

---

From: Neil Brown <[email-masked]>
Date: Mon, 8 Nov 1999 11:43:16 +1100 (EST)

> get us all the way to a uniform expiration point, I don't

Ah yes, I hadn't got up to there in the archive yet (it is heavy
going).  However....

  David: you said:

      On an OPEN, the client does not send a stateid so it cannot tell
      the server what its view of the world is.

  Yet the spec, when it talks about automatic lease renewal in section
  8.4  says that what is needed is
       "a positive indication that the client is still alive and locks
       are being maintained"

  So the issue is not "agreed view of world" but is "still alive and
  active".  Surely an OPEN call, which uses a clientid to gain a share
  reservation, is "a positive indication that the client is still
  allive and locks are being maintains".

> what's the point of allowing the unlock?  It doesn't really do 

What the LOCKU request achieves is that it says to the server "I
acknowledge that I have lost this lock".  This allows the client-state
to go through a state-transition, or series of transitions, from being
an EXPIRED state to being a non-EXPIRED state which can then be used
or closed like any other state.

> client endevours to do, and what the server tries to do.

As I understand the current protocol, it doesn't really say what
"actually happens".  The client 'actually' sends lease renewals more
frequently than the lease period, for safety.
The server 'actually' holds on to locks for longer than the lease
period (if possible) out of politeness.

> essentially lie to him about lease times.

Thanks for the example.  I think I have sorted out the issues a bit
more clearly in my mind now.

1/ There are three important numbers in the lease-renewal protocol.
     C: the maximum time that the client will wait after receiving a
        lease renewal confirmation from the server before sending another
        lease renewal request.
     S: The minimum time that the server will preserve locking state
        on behalf of a client from which it has not received a lease
        renewal.
     G: The length of the 'grace' period - the period after server
        restart during which only state-reclamation is allowed

   For reliable operation, C must be less than S, and S-C represents
   the longest network partition that client state will reliably
   survive while the server is up.  Also, C must be less than G, and
   G-C is an upper bound for the length of a network partition that is
   survivable immediately after server restart.

   The current protocol allows the server to set S, assumes that G==S,
   and leaves a choice of C up to the client.
   I suggests having the server set C, and leaving a choice of S and G
   up to the server. 

2/ There are three specific  situations that this protocol needs to
   cope with effectively: server restart, client failure, and network partition.

   A/ Server restart happens rarely (hopefully). 
      The effect of the grace period is very dramatic.  The server is
      effectively stalled during the grace period.

      In this case, the cost of a high G value is that it means a long
      time when then whole server is nearly useless, and this would
      weigh heavily against the value of surviving long network
      partitions. 
      Thus G would need to be
	   C + time-it-takes-to-reclaim-state + safety-margin
      where safety-margin is the sort of partition that we would want
      to survive even on server restart.
      If server restarts are uncommon and network partitions are
      uncommon, safety-margin might well be 0.

   B/ client failure and network partition are indistinguishable for
      the server, until which ever problem it was heals.  The purpose
      of the S value is to tell the server when it is allowed to
      assume that is was client failure. 
      The cost of a large S is much less than the cost of a large G,
      because only a small subset of the files will be affected, and
      so in many cases only a small subset of other clients will be
      affected (as opposes to the server restart which affects all
      files and all clients equally).
      The value of a large S is much the same as the value of a large
      G, it allows a long network partition to be tollerated safely.
      Thus S might be:
	   C + maximum-sustainable-network-partition
      and may be much more than G.

Thus it would seem that S and G need to be set independantly, as they
reflect very different situations with different tradeoffs, and that
the server needs at least an upper bound on C to be able to set S and
G.   Hence, I would suggest that the only number that appear in the
protocol be C and that the server set it.  S and G a configuration
options for the server administrator.

Looking back at Dave's example:
  While the client may have information that the server doesn't, by
  measuring typical trip times and loss rates, it doesn't have any
  where near the control that the server does.  It can at best reduce
  it's C value to (slightly more than) 0 and thereby survive a delay
  or partition of duration 'C'.  The server has more flexability and
  can be generous allowing very long values for S and making the
  appropriate tradeoffs when setting G.

One problem with the client not knowing S is made clear by David
Robinson in 
  http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1602.html

       [In one scenario,] The client knows that it hasn't gotten a response in
       [the lease period] so it knows the state of its locks is questionable. They
       may be valid because a pending I/O was received but the response was
       dropped, but they may not because the instant the lease expire the
       server granted the lock to another client. So even if a response
       delayed in transit gets through after the lease period, the client
       must assume locks are "unvalidated" (the draft unfortunately uses
       the term invalidated, I thought I used the former in my original e-mail).
       Odds are that most locks are still valid, but the client doesn't know
       which. Therefore it must probe each "unvalidated" file, the server will
       respond success if it has maintained the state and failure if it has
       not.

and 

  http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1605.html

       If there is not a renewal process the client may continue to act as if
       it had the lock. You are correct that on the next I/O those files
       that lost their locks will find this out. However, it is common that
       locks are used as a semaphore where no I/O is occuring. In this case
       the client will not detect that it lost a lock and corruption can occur.

so as the protocol stands, the client MUST know when the server MIGHT
have revoked locks, so that it can probe each file (for each
lockowner) and check that the locks are still valid - most of which
probably will be.

If the client didn't know S as I suggest, it would have to rely on the
server telling it when locks were lost.  This could be done be having
every use - except CLOSE, LOCKU,  DELEGRETURN and RENEW -  of every
stateid associated with a particular clientid return NFS4ERR_something
with the meaning "one of your locks has been revoked".
RENEW, as well as extending leases on un-revoked locks, could return a
list of revoked locks/shares/delegations.  CLOSE/LOCKU/DELEGRETURN
could then be used to acknowledge the loss of these locks.  Once all
revoked locks were released, RENEW would return an empty list and the
remaining valid locks etc would be usable again.

This seems to give a nice, precise, deterministic path for recovery
from lock-loss.

NeilBrown