Subject: Re: Section 7.6 ???

From: David Robinson <[email-masked]>
Date: Tue, 3 Aug 1999 17:05:05 -0700 (PDT)

> > Both sides will detect this state
> 
> No.  The server will detect this state and the client will
> deal with the consequences for each of its locks.  There is
> no way that I can see that a client rushing to renew using
> some IO that happens to come along is going to know if he
> was in time.  He may think he had 500ms to spare but some
> network problem means his message gets delayed.  He can't
> use the success or failure of the IO since the fact that 
> the IO succeeded doesn't mean that the lease was renewed;
> it just means that if it wasn't then no conflicting lock
> was given out corresponding to that stateid.  You may very
> well have success for the IO when the lease renewal failed.

If the lease has expired then the stateid held by the client is invalid,
with an invalid stateid the IO operation must fail or risk corruption
(ignoring possible server optimizations where it can track no change
and allow the IO anyways). Thus the server knows the lease has expired
(its timer has run out) and the client knows it because any IO using
its "current" stateid will fail. If the IO has succeeded then the
lease must have been renewed because the stateids matched.  If the stateids
don't match then the client must go thru the recovery process of
revalidating the held locks.

> > and can recover without data corruption.
> 
> Yes but not using the algorithm discussed below.
> 
> > The client must mark all locks held as "invalidated" and
> > then must issue an I/O request, either a pending I/O or 
> > zero length read to revalidate the lock.
> 
> There's no way he can know that he has to do this (see above).
> What he should do is use the stateid's he has.  When they
> are used in IO and corresponding locks have not been given 
> to conflicting requestors (and the lease renewal had failed)
> then they will be upgraded to valid on the server.  
> 
> > If the response is success, the lock is upgraded to valid,
> 
> No.  If the IO succeeds, the client does nothing special.
> His lock remains valid just like any other lock from his
> point of view.

This may not be worded correctly in the draft, but the stateid is
simply a short hand mechanism for the client and server to exchange
to indicate that they mutually agree on the files state (locks and shares).
If any operation that changes the state causes a new stateid is generated.
All operations that must respect the state of a file must use a stateid,
if the stateid used by the client does not match the server's stateid
(either the client is sending a bad stateid or the server has revoked it
due to a lease expiration).  So in your above scenerio the I/O must fail
or risk corruption.

I will agree that the current draft may not be well written on this.

	-David

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 9 Aug 1999 08:18:36 -0700

> revalidating the held locks.

It is true that if the lease has expired then the stateid is
invalid.  The converse is not true.  If the stateid is invalid,
you cannot assume that the lease has expired.  All you can assume
is that either the lease has expired or that the lock has been
revoked.

I'm really having trouble understanding what is intended here.
First, the basic assumption is that if the lease has expired
then IO with one of the associated stateid's will fail.  Leaving
aside the question of what the client may validly assume in 
that case, let's focus on the procedure which he is asked to
perform.  He now issues IO's using stateid's for all of his
locks.  The idea is that this will somehow "revalidate" them.
OK, so this means that the server is supposed to honor them
and not give back an error.  But these stateid's being 
revalidated are precisely in the same condition that the 
original stateid was.  So, on what basis is the server going
to honor those while giving back an error on the original
one?  Now, look at the case of doing two IO's to different
stateid's soon after a lease expiration.  Is one of them
going to fail and the other succeed (by being revalidated)?
What about an OPEN or LOCK that was in flight together with
the IO?  Do these have to be invalidated-revalidated also?

The only thing that seems to make sense is that if there is
server revalidation logic that it works for all of the stateid's
involved uniformly.  If that's the case, the client never finds
out that the stateid he used was ever invalid and he doesn't
go through any explicit revalidation procedure.  For any 
stateid that were either revoked or which were given away
to conflicting requests, the server is going to give back
an error which the client will then deal with.
> due to a lease expiration).  So in your above scenerio the 

As far as principles go, I agree.  

> I will agree that the current draft may not be well written on this.

Right now, I can't understand what is intended by that section
and exactly what logic the client and server should implement.
The thing I'm worried about is that the lack of clarity is hiding
a substantive problem behind it.  I don't think that's the case,
but until there is some clear agreement about how this is going
to work, I remain worried.

---

From: David Robinson <[email-masked]>
Date: Mon, 9 Aug 1999 22:04:52 -0700 (PDT)

> is that either the lease has expired or that the lock has been

If the stateid is invalid, this is a clear sign that the client
and server do not have the same view of the state of the file and
some sort of recovery is needed (possibly signalling the application).

> one?

Given a lease interval of say 60 seconds, and due to congestion
the client packets are stalled for 61 seconds so no I/O has gotten thru to
refresh the leases. The client knows that it hasn't gotten a response in
60 seconds so it knows the state of its locks is questionable.  They
may be valid because a pending I/O was received but the response was
dropped, but they may not because the instant the lease expire the
server granted the lock to another client. So even if a response
delayed in transit gets through after the lease period, the client
must assume locks are "unvalidated" (the draft unfortunately uses
the term invalidated, I thought I used the former in my original e-mail).
Odds are that most locks are still valid, but the client doesn't know
which.  Therefore it must probe each "unvalidated" file, the server will
respond success if it has maintained the state and failure if it has not.
In a trivial server, after 60 seconds the server revokes all locks and
every attempt to revalidate will fail.  A more reasonable server will
not revoke the lock until there is an actual conflict (lock or I/O)
so these revalidation attempts have a good chance at suceeding. The alternative
scenerio is the server revoking within the lease interval and that is
just treated as if the single file got its locks yanked since it is a rare
event.

> Now, look at the case of doing two IO's to different

The I/O operation will only fail if the server has revoked the lock,
hopefully smart servers will delay that as long as practicable. As I
stated above, if the client hasn't received a response within the
lease interval it must revalidate.

> an error which the client will then deal with.

My mental model of a server implementation is one where the stateid
references through to a per client data structure shared by all files.
Whenever I/O is performed on a file the common client lease expiration
time is updated, so in normal operation all files get refreshed. Once
the expiration time has expired, a conflicting request will see that
it can get through and be granted and the server will on the per file
data structure invalidate the stateid. Other files will still refer to
the underlying client data structure that has an expired lease interval
but their stateids won't have been invalidated (no conflicts).  Some
client I/O operation then arrives, if the stateid is invalid due to
a conflict it gets an error.  If it is still valid, the underlying
client expireation time is updated and the operation is granted
preventing any further conflicting operations from succeeding on all
other files.  The client may have gotten success back, but because it knows
that it didn't get a response through within the lease interval it
revalidates as described.

> to work, I remain worried.

In all honesty I haven't thought about the details in a few months
so your questions have been useful to me.  In fact in formulating
responses I have thought you had found an error, but so far I don't
think so. :-)  I did spend quite a bit of time thinking through these
scenerios when I wrote it up initially.

	-David

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 26 Aug 1999 13:36:15 -0700

> time is updated, so in normal operation all files get refreshed.  

This seems like a nice way to implement this stuff.  I think there's
one problem with doing it this way as the spec stands, but I think it
is fixable.

Suppose I open a file at noon and the lease interval is one minute.
The lease associated with that file open is set to expire at 12:01:00 PM.
Now suppose I open another file or get a lock on the same file at
12:00:30 PM.  I now have a lock whose lease is set to expire at
12:01:30 PM as well as the original open file whose lease is set
to expire at 12:01:00 PM.  As soon as you do any IO, you would be 
back at the desirable state that all leases associated with a 
given client are set to expire at the same time.  It would be nice
if that property were always the case and you didn't have to deal
with exceptions.

The problem is that the spec says that the server does an implicit
renewal of all leases when IO is done, but it doesn't says the same
thing when you do an operation (e.g. LOCK or OPEN) that gets a new
stateid (and associated lease).

I propose we change things so that such operations also are defined
as causing an implicit lease renewal, just as IO's do.  Then it
would always be the case, that all leases for a given client would be 
set to expire at the same time, avoiding implementation complexity.

---

From: David Robinson <[email-masked]>
Date: Thu, 26 Aug 1999 14:02:20 -0700 (PDT)

> set to expire at the same time, avoiding implementation complexity.

Good point, but I don't think that an OPEN will work to do lease renewal.
The stateid model is designed as a concise mechanism for client
and server to let the other know what they think the state is.  On
an OPEN, the client does not send a stateid so it cannot tell
the server what its view of the world is. If the server just
extends the lease it may not catch an error condition.

With the addition of RENEW, compounding it with an OPEN will accomplish
what you want.

	-David

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 27 Aug 1999 06:05:53 -0700

> extends the lease it may not catch an error condition.

OK, I see that.  I guess that means that the server is going
to have to have an expiration time in the per-stateid data
structure as well as one in the per-client data structure
with the actual expiration time being the max of the two.
When a lease expires, you have to be careful not to assume
that all the leases for that client have expired until you
check the specific expiration times for each of the
stateids.  RENEW (or an IO) would update the expiration
time in the per-client data structure while OPEN and 
LOCK would initialize the per-stateid expiration time
using the current time plus the lease interval.

> With the addition of RENEW, compounding it with an OPEN will 
 
What I want is for the server not to have to worry about the
case in which OPEN happens without RENEW and I'm not going to
get that :-(