Subject: RE: Section 7.6 ???

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 10 Aug 1999 09:07:11 -0700

> > It is true that if the lease has expired then the stateid is
> > invalid.  The converse is not true.  If the stateid is invalid,
> > you cannot assume that the lease has expired.  All you can assume
> > is that either the lease has expired or that the lock has been
> > revoked.
> 
> If the stateid is invalid, this is a clear sign that the client
> and server do not have the same view of the state of the file and
> some sort of recovery is needed (possibly signalling the application).

Yes.  The difference is that in the case of lease expiration we
have a situation in which it is clear that other files are
involved while in the case of other sorts of revocation, this
won't be the case.  Of course, if what you do is safe in either
case, then this may not matter, and that may the source of my 
confusion here.

> 
> > I'm really having trouble understanding what is intended here.
> > First, the basic assumption is that if the lease has expired
> > then IO with one of the associated stateid's will fail.  Leaving
> > aside the question of what the client may validly assume in 
> > that case, let's focus on the procedure which he is asked to
> > perform.  He now issues IO's using stateid's for all of his
> > locks.  The idea is that this will somehow "revalidate" them.
> > OK, so this means that the server is supposed to honor them
> > and not give back an error.  But these stateid's being 
> > revalidated are precisely in the same condition that the 
> > original stateid was.  So, on what basis is the server going
> > to honor those while giving back an error on the original
> > one?
> 
> Given a lease interval of say 60 seconds, and due to congestion
> the client packets are stalled for 61 seconds so no I/O has 
> gotten thru to
> refresh the leases. The client knows that it hasn't gotten a 
> response in
> 60 seconds so it knows the state of its locks is questionable.  They
> may be valid because a pending I/O was received but the response was
> dropped, but they may not because the instant the lease expire the
> server granted the lock to another client. So even if a response
> delayed in transit gets through after the lease period, the client
> must assume locks are "unvalidated" (the draft unfortunately uses
> the term invalidated, I thought I used the former in my 
> original e-mail).

So a client can determine when it is at least possible that lease
expiration may have occurred.  By saying, "Both sides will detect
this state", the current spec led me a little astray since I was 
looking for actual agreement between client and server which isn't 
necessary here.  What is necessary is that the server should never 
expire the lease without the client knowing that it *might* have been 
expired.

> Odds are that most locks are still valid, but the client doesn't know
> which.  Therefore it must probe each "unvalidated" file, the 
> server will
> respond success if it has maintained the state and failure if 
> it has not.

In the case of success, state may have been maintained 
because the lease never expired in the first place or because
there was lease expiration and the server kept the lock around in
the absence of a conflicting request.

The point I keep coming back to is: why must the client do this?
If he didn't, he would still find out about those that had become
invalid eventually (when he actually needed to do an IO) and 
for those that were still valid, what does he gain?  Is there an
issue with assumptions about implicit renewal?

Normally any IO I do on any file implicitly renews leases for all
of files.  Now if a lease expires, and I do an IO on file A, then
it revalidates the stateid for A.  I guess I was assuming that this
would revalidate all of the stateid's that were revalidatable (i.e.
had not been revoked due to conflicting requests during the time
the lease was expired).  If you are assuming otherwise, then I
now understand what the revalidation is for.  Is that it?

If that's the issue, why couldn't the server do the revalidation
that I was assuming it was doing?  It would save the client from
a lot of tricky timer code. (I have tendency to get that wrong
at times but since I'm not doing clients these days, maybe
that doesn't matter).

> In a trivial server, after 60 seconds the server revokes all locks and
> every attempt to revalidate will fail.  A more reasonable server will
> not revoke the lock until there is an actual conflict (lock or I/O)
> so these revalidation attempts have a good chance at 
> suceeding. The alternative
> scenerio is the server revoking within the lease interval and that is
> just treated as if the single file got its locks yanked since 
> it is a rare
> event.

As I understand what you've written above, if the client gets the
NFS4ERR_EXPIRED error, then he simply returns an error to the user.
He doesn't go off and revalidate unless timers show him that a lease
expiration may have occurred.  If lease expiration may have occurred,
and you go through revalidation, you will find that single file
go its locks yanked, and maybe a few others if lease expiration
really happened.

> 
> > Now, look at the case of doing two IO's to different
> > stateid's soon after a lease expiration.  Is one of them
> > going to fail and the other succeed (by being revalidated)?
> > What about an OPEN or LOCK that was in flight together with
> > the IO?  Do these have to be invalidated-revalidated also?
> 
> The I/O operation will only fail if the server has revoked the lock,
> hopefully smart servers will delay that as long as practicable. As I
> stated above, if the client hasn't received a response within the
> lease interval it must revalidate.

I was misunderstanding what triggered revalidation.  I was assuming
that it was an error response rather than the client making a 
judgement on the basis of the timing of requests and responses.

> 
> > The only thing that seems to make sense is that if there is
> > server revalidation logic that it works for all of the stateid's
> > involved uniformly.  If that's the case, the client never finds
> > out that the stateid he used was ever invalid and he doesn't
> > go through any explicit revalidation procedure.  For any 
> > stateid that were either revoked or which were given away
> > to conflicting requests, the server is going to give back
> > an error which the client will then deal with.
> 
> My mental model of a server implementation is one where the stateid
> references through to a per client data structure shared by all files.
> Whenever I/O is performed on a file the common client lease expiration
> time is updated, so in normal operation all files get refreshed. Once
> the expiration time has expired, a conflicting request will see that
> it can get through and be granted and the server will on the per file
> data structure invalidate the stateid. Other files will still refer to
> the underlying client data structure that has an expired 
> lease interval
> but their stateids won't have been invalidated (no conflicts).  Some
> client I/O operation then arrives, if the stateid is invalid due to
> a conflict it gets an error.  If it is still valid, the underlying
> client expireation time is updated and the operation is granted
> preventing any further conflicting operations from succeeding on all
> other files.  The client may have gotten success back, but 
> because it knows
> that it didn't get a response through within the lease interval it
> revalidates as described.

OK, this a good framework for discussion.

Within this framework, what happens expiration occurs?  Does the stateid
continue to point to the per-client data structure?  If so, wouldn't
an IO on any file extend the leases for all not-invalidated locks?
If that happened, what is the benefit of the revalidation procedure?
What am I missing?

> 
> > > I will agree that the current draft may not be well 
> written on this.
> > 
> > Right now, I can't understand what is intended by that section
> > and exactly what logic the client and server should implement.
> > The thing I'm worried about is that the lack of clarity is hiding
> > a substantive problem behind it.  I don't think that's the case,
> > but until there is some clear agreement about how this is going
> > to work, I remain worried.
> 
> In all honesty I haven't thought about the details in a few months
> so your questions have been useful to me.  In fact in formulating
> responses I have thought you had found an error, but so far I don't
> think so. :-)  I did spend quite a bit of time thinking through these
> scenerios when I wrote it up initially.
 
I don't doubt that.

I know that there are lots of reasons to boil down the spec (e.g. size 
vs. IETF approval time), but I think that if the guy who wrote a
section has trouble understanding it some months later, then I
can be forgiven my confusion and also we have definitely gone too far 
in appeasing the angry Gods of Conciseness.

---

From: David Robinson <[email-masked]>
Date: Tue, 10 Aug 1999 12:57:21 -0700 (PDT)

> expire the lease without the client knowing that it *might* have been 

Yes, the client and server MUST know what the lease interval is and it needs
to be fixed.

> issue with assumptions about implicit renewal?

If there is not a renewal process the client may continue to act as if
it had the lock.  You are correct that on the next I/O those files
that lost their locks will find this out.  However, it is common that
locks are used as a semaphore where no I/O is occuring.  In this case
the client will not detect that it lost a lock and corruption can occur.

> now understand what the revalidation is for.  Is that it?

When the first request succeeds then from the servers side all held
locks have a renewed lease on life.  However, if some were not
renewed the client still doesn't know which locks were lost so it
needs to check them all, primarily for the semaphore case above.

> go its locks yanked, and maybe a few others if lease expiration

Correct, loss of a lease within a lease interval is fatal and can't be
recovered.  The only way this can happen is if the server is broken
or forceably removed the lease.

> If that happened, what is the benefit of the revalidation procedure?

Because all leases are renewed when any file gets renewed, all files
expire at the same time. After expiration the stateid still needs to
point to the per client data structure so that another request looking
to grab a lock can check if it has expired or not.  I think the
misunderstanding is that revalidation is for the *client* benefit, not the
server.  Once one has gotten thru to the server it then knows the rest are
valid. But the client still needs to know what it may have lost.

> in appeasing the angry Gods of Conciseness.

Also the designer is often not the editor, so things can be skewed. I have
a meeting with the editor today to help this section.

	-David