Subject: RE: Spec-related discussions at the Bake-off

From: Mike Eisler <[email-masked]>
Date: Fri, 3 Nov 2000 01:52:36 -0800 (PST)

> > > > The previous stateid is valid only with seqid X, not X+1. So that's
> > > > why we don't increment the seq-id on error.

> > A sequence consists of a series of
> >  { E0, T0 }, { E1, T1 } ... { En, Tn }
> >  
> >  where 
> >  	Ei is a sequence number for the i'th lock state changing operation
> >  	Ti is a state id fr i'th lock state changeing operation
> >  such that
> >  	E(i+1) = E(i) + 1
> 
> But we are saying precisely that E(i+1) = E(i), if the operation i fails.
> 
> >  	T(i+1) may not be equal to T(i)
> 
> Does the spec actually say that anywhere?

The example of how a stateid can be built includes a sequence value
which changes. Aside from that, because lock state modifying operations
return a stateid, it is implicit that T(i+1) is not necessarily equal
to T(i), otherwise why return T(i+1)?

> > 	the client cannot assume that T(i) is valid for use with E(i+1).
> 
> Well if I say that E(i+1) = E(i), if operation i+1 fails, then if T(i)
> isn't valid with E(i+1) = E(i), then the client is dead in the water.

Exactly. So don't increment i when you get a failure.

> But why can the client not assume that T(i) is valid for use with a new
> sequence id?  To maintain the {En, Tn} sequence?  But the spec doesn't

Because new stateids get returned on each operation.

> talk about that sequence or such a tie-in between seqid and state-id.

Section 8.1.3:

   The server must be able to distinguish the following situations and
   return the error as specified:
[...]
   o    The stateid was generated by the current server instance but the
        stateid no longer designates the current locking state for the
        lockowner-file pair in question (i.e. one or more locking
        operations has occurred).  The error NFS4ERR_OLD_STATEID should
        be returned.

        This error condition will only occur when the client issues a
        locking request which changes a stateid while an I/O request
        that uses that stateid is outstanding.

Despite the reference to "lockowner-file pair", given that lock
operations take a sequence number, and given that stateids are
associated with lock state, if follows that stateids and seqids have
a tie in. Of course, it is a permissable thing for a server to
not change the stateid on any or every lock state modifying operation,
but a client cannot assume that.

> It might be better if it did.  Remember that state-id's are also used
> in requests that don't have sequence id's.  When does the stateid become
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

No requests that change the lock state.

> invalid?  When you get a new one that replaces it, or actually when the
> server executes an operation that changes it.  Since we have a model

What's an example of an operation that returns a new stateid without
changing lock state?

> of when stateid become invalid (get OLD_STATEID when used), it makes
> sense for that to also apply when they are used with sequence id's.
> I don't see that we have a right to create this {En, Tn} rule which
> I don't see in the spec, although I might have missed it.

The rule is quite implicit, and in any case is the most conservative
interpretation. If a client assumes anything more, then it won't work
against some servers.
	
> > > > A stateid is associated with an nfs_lockowner, not an open
> > > > file. If there are multiple open files associated with the
> > > > nfs_lockowner, then removing the nfs_lockowner is not allowed.
> > > 
> > > Well there is some difference of opinion on this.  My understanding
> > > is that it designates a given lockowner-file pair.  I think I got this
> > > impression in a discussion with Robinson, but at this point I can't
> > > be sure.  Anyway, this is the way my server is implemented.
> 
> > Well, I think your server is wrong. Section 8.1.5:
> > 
> >    Note that for requests that contain a sequence number, the client,     |
> >    for each nfs_lockowner, there should be no more than one outstanding   |
> >    request.          
> >   
> > Awkwardly written, but clear.
> 
> Yes, but I'm not sure why that indicates my server is wrong.  As far
> as sequence id's, that is precisely my understanding and my server
> does nothing that contradicts that.

I don't see how you can implement a server that associates a sequence
of seqids with a given nfs_lockowner, but has stateids associated with
a particular nfs_lockowner-file pair. E.g.

r1: OPEN { seq # 0, lockowner 0, file1 } ->
<- fh1, stateid 1, OPEN4_RESULT_CONFIRM, confirm_verifier 

r2: OPEN_CONFIRM { fh1, seq # 1, confirm_verifier } ->
<- stateid 2

r3: LOCK { fh1, seq # 2, stateid 2 } ->
<- stateid 3

r4: OPEN { seq # 3, lockowner 0, file2 } ->
<- stateid 4

r5: UNLOCK { fh1, seq # 4, stateid 4 } ->
<- stateid 5

If your server doesn't return stateid errors for RPC r5,
then this is wonderful.

> > > > If this is the last open file associated with the nfs_lockowner, then
> > > > there are issues, which are discussed in section 8.1.7 of the i-d.
> > > > In a nutshell, removing an nfs_lockowner before a time period
> > > > corresponding to the lease time is not advisable.
> > > 
> > > Gievn stateid=lockowner, yes, although I don't see why the timeout
> > > is necessarily the lease time.
> 
> > A client cannot expect lock operations to work after the lease expires.
> 
> Who said a lease expired here?

The scenario in mind is the client sends the CLOSE, which is the CLOSE
for the last open file for this nfs_lockowner. The server returns
NFS4_Ok (and a stateid that is uselss sicne there are no open files ...
the server implementor would probably want to return a special stateid
for this occasion).

However, the client doesn't get the response; it re-sends its CLOSE.
The spec mandates that the server keep a lock state duplicate request
cache consisting of the last response sent for given nfs_lockowner. If
the client never gets a response after N seconds from the last
successful response it got for that nfs_lockowner, where N is the lease
time, then the client cannot reasonably expect his lease to be around,
nor the lock state duplicate request cache. Thus the client has to be
prepared to get back a stateid error on his retry, instead of the expected
NFS4_OK, when resending after N seconds.

Hence, section 8.1.7 suggests lease expiration as a heuristic for when
to delete nfs_lockowner state after close of the last file opened by
the nfs_lockowner.

BTW, fodder for the implementation RFC is what sequence number does an
nfs_lockowner use for an OPEN subsequent to a successful CLOSE of its
last OPEN file. I claim it should be the sequence number used in that
CLOSE, plus 1. If the OPEN is issued before the lease expires, the
server won't return the confirm flag.

> > So, what do NFS/TCP implementations do today when presented with a
> > record marking header that is say, one gigabyte? Surely they break the
> > connection?
> 
> Exactly.  The value is wrong.  What you do not do is adjust some

Today. 14 years from now, I can imagine there will be at least
as many desktop computers with 0.5 TB of ram, as there were desktops
with 4 MB in 1986. And many of those boxes then ran NFS with 8192
byte buffers, which happens to be 1/512 of 4 MB. 1/512 of 0.5TB is
one GByte. As a server implementor, I'd rather provide a knob to
allow the field to keep up with the times, versus forcing customers to 
spend big $$$ for an upgrade.
 
	-mre

---

From: David Robinson <[email-masked]>
Date: Fri, 03 Nov 2000 08:02:30 -0800

The original designed purpose of seqid was to create an at-most-one
semantic
for locking operations, thus if the stateid is changed then a seqid
must have changed and vice-versa.

Originally the seqid and stateid number space was supposed to be
maintained on a owner/file pair which allows concurrency and doesn't
force synchronization of I/O operations with state change on different
files.

What appears to have happened is in order to solve the problem of
at-most-once semantics the existing seqid was used across opens
and this forcing it to be a single number space for each owner. Because
of the tying to stateid it also becomes a single number space for
each owner.  This is unfortunate.

A possible resolution of this issue is to separate the seqid number
space for share locks from record locks. In other words, the seqid
space used to manage all OPENs is independant of the seqid
space used once the file has been successfully opened. This should
then also allow the stateid space to be maintained on a per owner/file
pair as well with the stateid returned from the OPEN being the initial
stateid for that file.

I believe that this change can be managed without changing the wire
protocol but by simply clarifying the semantics in the specification.

	-David