Subject: Some issues with stateid's

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 27 Oct 1999 07:16:48 -0700

There are some issues with the use of stateid's that I wanted
to discuss.  I think they are interesting and exploring them 
may be helpful to those who are about to implement those
pieces of the protocol.

First I'd like to summarize the results of our discussion of
stateid's at the bake-off.  Here are the basic points, as I
remember them.  Correct me where I'm wrong.

   For every given file-lockowner pair, there is a temporally-
   ordered sequence of stateid's generated by the server as
   a result of locking and unlocking requests (in which classes
   OPEN and CLOSE are included, respectively).

   This temporal order is well-defined and the client and server
   agree on it.  In our discussion, I think we concluded that
   the client could detemine the order of generation on the
   server (which is definitive) by deriving the temporal order
   of two stateid's (for the same file-lockowner pair) from
   the (numerical) ordering of the sequence id's of the
   requests that generated each of the respective stateid's.
   I think that it is easier than that though; the client
   should never have two stateid-geenerating requests
   outstanding for the same lockowner on the same file.
   Thus the client-ordering and the server-ordering must 
   be the same.

   The desirability of changing the stateid on each locking
   and unlocking request derives from the observation that
   each change in the locking status marks a transition
   between one shared understanding (between client and 
   server) regarding the state of a file and another and
   that IO done under one such understanding may be suspect
   in another.

   A crucial example of the above would be if a retried WRITE
   were held in a router for a long period, so long that
   the orignal WRITE request and an ensuing unlock completed.
   If the delayed WRITE requests were disgorged into the
   network, and the server were to act on that write request,
   the file could be become corrupt since the old write 
   should no longer be acted on in the new locking regime.
   This is true even if the original client relocked the
   region in question.  

   One way to implement stateid's is to divide them into
   three fields: one identifies the server instance so
   that stateid's handed out by one server instance are
   not erroneouly recognized by another, the second field
   is an index into a table from which lock state information
   may be derived and the third is a sequence field which is
   changed for each stateid handed out with the same table
   index.  This sequence field would be different from the
   sequence id in that it is opaque to the client, but the
   mapping from the stateid sequence field to the sequence 
   id of the request which generated it would normally be an
   order-preserving injection.  Below, I'll refer to these
   three parts of the stateid as the server-instance field,
   the state-index field, and the state-sequence field.

Given the above, the issue I want to consider is whether 
there are situations in which changing the stateid in this
way isn't necessary (or helpful) and causes problems and 
whether those problems can be dealt with without changing 
the existing stateid framework.

My answers to the above are yes, yes, and yes.  That is,
there are cases in which this isn't necessary, they do 
pose some problems but the problems can be dealt with 
without a major change in the framework.

Consider the case where a client has opened a file, maybe
done a lock and written a fair amount of data which is
cached on the client.  The client may, but doesn't have 
to, write some of this data to the server before the
user does an unlock.  Let us say that because of memory
tightness or periodic syncing, some of this data is
sent to the server.  If the user locks an additional area
while some of WRITE requests are outstandiing, there is
an issue because doing the LOCK may cause the stateid
embedded in the WRITE requests to become invalid which
would result in otherwise-valid WRITE's being rejected.

The first way to deal with this is to hold up sending 
the LOCK until there are no conflicting WRITE's pending
(and you would also be unable to send new WRITE's while
a LOCK was in flight).  I think this would work but it 
is a bit of otherwise unnecessary serialization that 
might be troublesome in some environments.  

The second way to deal with this is to let the LOCK
go through and do any necessary recovery for WRITE's
that get rejected because of it.  This is desirable
because, almost all of the time, the WRITE's won't get
rejected.  On TCP, the requests are normally not 
going to be done in a different order than they were
sent and even in many UDP environments the frequency
of re-ordering is going to be fairly low.  This means
that retrying with the new stateid when this problem
occurs is going to be a win since it won't happen
very often.  (The other case mentioned above, doing
a WRITE while a LOCK is outstanding is probably best
dealt with by waiting since the stateid you would use 
would almost certainly be invalid).

The problem with this approach is that, as the spec 
stands, it is not clear what error the server would 
return in this case, and whether the client could 
distinguish this case from a revoked lock, for instance.  
It seems that right now, the only choice is to return 
NFS4ERR_EXPIRED.

So here's what I think needs to be done when the server
detects the following problems with a stateid.  I have
assumed the tripartite division suggested above although
the description of each of these cases could be reworked 
to not depend on implementation details. Each has a 
paraphrase that reflects the interpretation of a 
stateid as representing an agreement between client
and server.

The server-instance field doesn't match ("This agreement
wasn't with me.  It was with somebody else").

    Server returns NFS4ERR_DEADSTATE.  Client tries to
    reclaim locks and then reissues requests.

    Current answer is NFS4ERR_GRACE which is confusing 
    since it is also denotes a non-reclaim lock request
    made during the grace period.  Also, NFS4ERR_GRACE
    now only applies during the grace period but the 
    spec does allow a server to do reclaims outside
    the grace period if no conflicting request was ever
    granted.  But if is this is possible, there doesn't
    seem to be any way for the client to discover that
    his stateid applies to a previous server instance
    unless he finds it out during the grace period.  
    Having a separate NFS4ERR_DEADSTATE, independent of
    the grace period fixes this.

The server-instance field is OK but the state-index field
is not recognized ("There never was any such agreement,
you bleeding liar").

    Server returns NFS4ERR_BADSTATE and prints message
    about brain-dead client.  Client prints message
    about brain-dead server.  At least we know we have 
    a true protocol problem to debug.

The server-instance and state-index fields are OK but
the state-sequence field doesn't match.  ("There may have
been such an agreement at one time but there isn't 
now and I don't remember it").

    Server returns NFS4ERR_OLDSTATE.  Client sees if
    the stateid had changed and if so reissues request.
    If not, it checks if there is some outstanding 
    locking request that could be changing the stateid
    and if so, waits for it to complete and then reissues
    the request with a new stateid.  If neither of the 
    above, prints message about brain-dead server.

All of the stateid fields are OK but one of the locks
designated by the stateid has been revoked.  ("Yes,
that is our agreement, but I have decided it no longer
applies.  So sue me").

    Server returns NFS4ERR_EXPIRED.  Client (unless
    exceptionally litigious :-), reports revocation to 
    the lock owner.

All of the stateid fields are OK and no locks have been
revoked but the stateid (or rather the locks denoted
by the stateid) do not confer the right to do what the 
client is trying to do.  For example, you take the stateid 
which you got from closing a file and try to use that
do a WRITE.  ("We had an agreement but you aren't 
abiding by it").

     Server returns NFS4ERR_NOLOCK.  Client and server
     each report that the other is brain-dead.

Categorizing failures of stateid agreement in this way,
even if it does nothing else, is going to make things a 
lot easier trying to debug this stuff.  Over the internet,
while stepping through breakpoints, I particularly want
to separate time-based leasing issues from all of the 
other protocol elements (that are all going to work 
perfectly the first time, of course :-)

---

From: Neil Brown <[email-masked]>
Date: Thu, 28 Oct 1999 16:21:21 +1000 (EST)

On Wednesday October 27, [email-masked] wrote:

>    This sequence field would be different from the
>    order-preserving injection.
     ^^^^^^^^^^^^^^^^^^^^^^^^^^

Does this imply that the funciton of the sequenceid can be subsumed by
the stateid in its current form?
The sequenceid is there precisely to guarantee "at-most-once"
semantics.  It does this by defining an ordering that both the client
and the server must acknowledge to each other. (server returns
"next-in-order" number, client acknowledges it by using it in the next
request).   The stateid provides a similar ordering, and so guarantees
at-most-once semntics.

On the whole, I think your description is very clear.  The client
needs to be ready to write buffered data with whatever stateid is
current.  In the case of reclaiming state after a server restart, it
will presumably re-establish all the locking state first, and then
start flushing data, all with the same stateid (though it could write
data as soon as the relevant address range is locked).
Ofcourse, dirty data has to be flushed before a range is unlocked, but
I think that goes without saying.

Also, I like the idea of having error values which are impossible if
both sides follow the protocol, especially is an area of the protocol
that is relatively new.

NeilBrown