Subject: RE: leases revisited was RE: Some issues with stateid's

From: David Robinson <[email-masked]>
Date: Mon, 15 Nov 1999 20:24:39 -0800 (PST)

I got multiple weeks behind in my e-mail so I will try to answer
some of these.  If I have missed something, please gently point
me to the archive, then call me stupid...

>      Unbounded storage for lockowner data structures on the
>      server (for seqid's):
>      
>           http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1842.html

There have been multiple uses of the phrase "sequence id" which
is confusing. But I am not clear why the lockowner structure needs
to be held beyond the time that the last OPEN file held by that owner
is CLOSEd?  The necessity of ordering of locks is to insure that
state is maintained, once all files are CLOSEd all lock owners degenerate
into one common state, empty. As referenced in the above article,
you want a hierarchy of data structures of decreasing number, stateid ->
lockowner -> clientid.  Multiple stateids (different files) can refer to
the same lockowner, multiple of which can refer to the same clientid.  Once
one level has collapsed it can be removed.  So I don't understand the need.

>      My proposed clarification of Section 8.6:
> 
>           http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1847.html

I have no problems with the clarifications.

>      Issues of changing stateid's interacting with IO requests 
>      in-flight and the implications for the set of stateid-related
>      errors to be returned:
> 
>           http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1847.html

I don't think this is the right reference, but if this is really:

	http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1851.html

which is a good explainations of different scenerios where stateids
can cause errors.  I would not write the specification in terms of
the implementation of a stateid on the server as a triple, but the
types of errors described should have separate error codes which allows
the client to decide if it is worth doing some recovery or giving up.

>      Conflict between sequence mismatches being "ignored silently"
>      and COMPOUND:
> 
>           http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1843.html

I originally wrote to silently ignore certain sequence mismatches as
it didn't seem worth the effort to send back an error to a client that
has already seen the response from that request.  It will always just
toss the reply on the floor as a duplicate (current seqid is N, receiving
reply of n < N).  In the ugly case where a client gets N+1 before receiving
the reply to N, this is a log the braindead server error we have had
a warp in the space-time continum.

The real mess that COMPOUND implies is that there is no way to pass the
stateid between operations.  As a possible solution, we can extend the
concept of the current file handle (cfh) to include that all locking
state is maintained while it is currently active, otherwise the
sequence of LOCK, WRITE, UNLOCK would always require 3 round trips.
This implicit passing of arguments between operations will require some
sort of magic values to say use the current values. I would prefer it
to be explicit but it may be too late. (Many may know my desires with
respect to COMPOUND :-)

> There's also a few possible issues that have been discussed recently  
> that relate to the locking protocol for which David Robinson's input
> might help dispel my uneasy feelings about the details of what is 
> currently in the spec:
> 
>       The whole open-upgrade-downgrade vs. multiple opens for the
>       same lockowner-file pair confusion.

Since Unix has no concept of SHAREs we must look at the Windows
semantics.  If my recollection is correct, someone stated that it
is impossible to change a SHARE on an OPEN file, it must be CLOSEd first.
Unless someone can show an existing API that allows upgrade/downgrade
I would leave it as not possible without a CLOSE. (Don't site Solaris'
share API as an example, I wrote it and it is wrong and will kill it).

>       The issue of lockowners and how they are affected by UNIX
>       fork's.

As with the regular Unix semantics, locks are not preserved across fork()
so all new locks have new lockowners. Manditory locking and I/O will
behave as if it were two different lock owners.  I haven't thought
of what the semantics of a SHARE on fork are, since Windows doesn't
have fork() it isn't defined. I would state that like locks, SHAREs are
lost so that interaction reverts back to access_modes/DENY_NONE.

>       The semantics of revocation and how the spec should address
>       the issue of whether it is meaningful to revoke a one of
>       set of locks held on a file by a single lockowner.

In general most would expect that revoking a lock is a last resort
activity which the client (and application) has little hope of doing
anything useful besides exiting.  You could just revoke that one
lock and let the client attempt to make progress or more likely
exit and UNLOCK the rest.  This would be the simpliest semantic which
will virtually always result in all the locks being removed.  The trouble
with having the server revocation imply more is that it embeds a certain
understanding of how the client is managing and multiplexing lockowners.

	-David

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 16 Nov 1999 09:51:44 -0800

> one level has collapsed it can be removed.  So I don't 

Right now, the need is that the spec requires it.  I don't
know how to read Section 8.4.2 in any other way as saying
that you have to keep a sequence number for a given lockowner
which is montonically increasing (BTW, it should make clear
that it is incremented by one each time.  Monotonically
increasing, while true, is misleading because it is
insufficiently restrictive.  I think Neil Brown pointed
this out).  It says that the sequence is reset when the
client verifier changes, implying that it isn't reset at
any other time.

If the server has to know the proper sequence number (in
order to determine how and whether to process requests)
then it has to keep it somewhere.

If we think about changing things so that the sequence 
number is reset whenever a given lockowner has no locks,
we have to decide whether that exposes to the problems
that sequence numbers were intended to prevent in the
first place.  The answer turns how far we want to go
in implementing at-most-once semantics.

If the only issue is making sure that a given request 
does not run into an error while it is being retransmitted
(and thus executed twice), then resetting the sequence
number when you have no locks will work.  (But maybe
not; see below).

In section 8.4.2, re-ordering is mentioned.  If we assume
that a router may unboundedly delay an open request
(lets say it has sequence number zero) then suppose
the client retransmits that, does the corresponding 
close and resets the sequence.  If the router than 
disgorged the open, then it would be acted on (sequence 
zero would be the "right" one) and the clients subsequent 
open (also using sequence zero) would not be processed 
correctly.  If the two opens had different parameters,
the response to one is probably going to confuse the
hell out of the client.  Of course if when the spec
says,  "If a duplicate of last request (r == L) is 
received, the stored response is returned", the stored
respnse include the original request id, then client 
open is going to retry forver.

Oops!  If the server were to reset the sequence number
on the close that brought us to a no-lock state, then
even a retransmission of that close could not be handled
properly.

> I don't think this is the right reference, but if this is really:

You're right.

> 	http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1851.html

You've "gently pointed me to the archive".  I guess that
means that next time you can call me stupid :-)

> the client to decide if it is worth doing some recovery or giving up.

I agree that the specification of the error should not be 
written in terms of a stateid-as-a-triple.  But I do think
the spec should mention this somewhere as a possible 
implementation.  I think the spec currently discusses
how a volatile filehandle might be implemented, so this
kind of thing has been done before.

If we decide that all material like that should be an
informational RFC (and we actually do one), then OK, but
it should be somewhere.  This kind of description would
have helped me to understand stateid's a lot quicker and
I assume that that would be the case for some others as 
well.

> a warp in the space-time continum.

I'm going to describe the cases using r and L as is done
in section 8.4.2.

As far as the (r < L) case, there is a very small effort to send
back the error response which, if everyone is playng by the
rules, will get ignored.  On the other hand, returning an error
is something servers know how to do while silently ignoring
a request is fairly strange.  It's particularly strange
when you may have already executed a number of ops including
state-modifying ops.  It just seems that an error is cleaner
here.

I had thought that the (r == L) case was troublesome but I
think it's OK.  The current spec, by talking about "the request"
led me to suppose that you might have to save the response 
to the whole rpc.  I don't think this is necessary.  In the
context of COMPOUND, the saved response for the current op
becomes part of the response for the current op and if that 
is not an error, processing of the COMPOUND continues.

For (r > L+1) case, I still don't see the point of ignoring
the error.  I believe that the whole mechanism assumes that 
clients should not have more than one locking request for a 
given lockowner outstanding at a given time. (I think it 
would help to make this explicit).  In this case, if a 
client runs into the (r > L+1) case, it is because he is 
doing something wrong.  The best way to have him find the 
problem is to return an error.  You may want to log something
as well, but not returning an error is going to make the 
problem harder to find.

Also, if this section is to have some sort of statement 
about only having one locking request outstanding for a 
given lockowner at a time, it should include the case
of doing two locking ops for the same lockowner in the
same rpc.  Think about doing two OPEN's for the same
lockowner in the same rpc.  Let's say they have sequence 
numbers 3 and 4.  If that rpc is to be retransmitted
then on the second try the first open (sequence 3) 
will find that the last sequence received is four
so the request will be ignored (or get an error as 
I propose).  In either case, I don't think clients
should be doing that.  (Brent's now going to accuse
me of having a holier-than-thou attitude :-)
  
> to be explicit but it may be too late.

It may be too late.  I think if we do this, then stateid's
should always be specified implicitly, just as file handles
are.  Magic values to allow them to be specified either
explicitly or implicitly, is just too complicated.  Neil
Brown recently posted a proposal to do this. 

> (Many may know my desires with
> respect to COMPOUND :-)

I suspect mine are similar.  

> share API as an example, I wrote it and it is wrong and will kill it).

Unix has no concept of shares but it can be affected by the
reservations of others.  So, if a Windows client opens a file
ACCESS=READ DENY=WRITE, then my subsequent Unix OPEN for write 
should fail.  At least, I assume so.

So now suppose that a Unix process opens file A/B for reading
and then a Windows client opens that same file specifiying
ACCESS=READ DENY=WRITE.  Now if my original process were to
do an additional open of A/B for read-write, what do we do
now.  Can I just do an OPEN (with the same lockowner) and
have that fail because of the share reservation?  (In fact,
even without needing to the the denial for the share 
reservation, I probably need to do an OPEN to get a stateid 
that will allow me to write the file; also the server
needs to know that I am writing so that it can deny 
subsequent share reservation requests from Windows clients).

So I think the server needs in some way to support the
concept of upgrading downgrading a lockowners current 
access.  This might be by somehow allowing multiple
OPEN's with different access or in sme other way.
As the spec now stands you can't allow multiple OPEN's;
there is no way that you can specify to CLOSE that
you want to close one or the other of the two open's
you have outstanding.

> lost so that interaction reverts back to access_modes/DENY_NONE.

Suppose process 100 opens file X/Y and then forks process
101.  Unix thinks that process 101 has the file open and
can get record locks but the server doesn't.  He doesn't
have a stateid corresponding to process 101.  He could
use the process 100 stateid to do IO but if he wants to
get a record lock, then using this stateid would result 
in getting locks for process 100.  So if process 100 and
process 101 were to try to use locking to co-ordinate 
their access to the file, it wouldn't work.  Maybe
we should go back to having LOCK take a lockowner
rather than a stateid.

If this is done, the protocol is probably OK but there
are some tricky implementation issues that people have
to worry about.  In my example above, assume that 
process 100 exit'ed after doing the fork.  You wouldn't
have a real close of X/Y because of the open reeference
count so process 101 would wind up having to do IO
using the stateid that process 100 got.  Also, when 
he did a close that stateid would be used and the sequence
number associated with process 100 would have to be 
presented.  This information would all have to preserved
even though process 100 is long gone and then finally
freed when appropriate.  Gee, I'm glad I work on servers. :-)
 
> understanding of how the client is managing and multiplexing 

OK.  The client needs to unlock all his locks.  I thought
there was a problem doing that but since LOCKU returns
a stateid but doesn't require one as a parameter you can 
attempt to get rid of all the locks you think you have.

One question is what happens when you try to unlock the 
revoked one.  I think that should be OK and not an
error.

On the other hand, CLOSE does take a stateid so if you 
have a revoked lock, as things stand that stateid should
be invalid.  I think the spec should allow you to use
stateids that are invalid (because they include a revoked 
lock) if you are just giving up locks.

One other thing was the issue of whether 64 bits is enough
as a stateid.  Since this would be hard to change with
minor versioning (needs old and new versions of LOCK, LOCKU,
OPEN, CLOSE, READ, WRITE, DELEGRETURN), I'd like to make
sure we have enough room.

     http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1852.html

One other other thing is the issue of trying to make sure
that all leases for a given client always expire at the 
same time (as opposed to the current spec in which this is
almost always the case).  We discussed this a while back.

I proposed this in:

   http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1670.html

And you responded that it wouldn't work:

   http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1671.html

But recently Neil Brown and I have had a further discussion
of this leading me to conclude that maybe it would.  I can't
point you to all of this because the archive ends on 
Nov 5.  Take a look at:

   http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1900.html

I've quoted below from a recent (11/15) post on the subject 
that quotes some of the background.  I've avoided adding yet
another level of ">" so the text of that post below (as 
opposed to what it quotes) appears below without any leader:

> > > 
> > >   Yet the spec, when it talks about automatic lease 
> renewal in section
> > >   8.4  says that what is needed is
> > >        "a positive indication that the client is still 
> alive and locks
> > >        are being maintained"
> > > 
> > >   So the issue is not "agreed view of world" but is 
> "still alive and
> > >   active".  Surely an OPEN call, which uses a clientid to 
> gain a share
> > >   reservation, is "a positive indication that the client is still
> > >   allive and locks are being maintains".
> > 
> > It certainly proves the client is alive.  I guess the issue
> > is whether the client is still alive (i.e has not gone down 
> > in the interim).  With a stateid, that is should be assured
> > by the server putting some sort of instance id (boot time or
> > something).  I may be wrong, but I don't see that is assured 
> > for an nfs_lockowner.

I think I was wrong here.  I don't think that the client can 
have rebooted and then wind up doing an OPEN without the
the server knowing.  If the client is going to do an OPEN,
he is going to have to present a clientid.  Where does he 
get it?  He got it from the server, by doing a SETCLIENTID.
He isn't allowed to just do an OPEN with some made up 
clientid.  The SETCLIENTID would have told the server
to release all locks and thus all leases for that client.  

I think this means that it would be OK for all of the 
requests that create leases as well those that do IO
to renew all leases for the client.  This means that
all of the leases for a client would expire always
expire the same time (as opposed to now in which it
is almost always the case).  This would simplify both
the client and the server.

---

From: David Robinson <[email-masked]>
Date: Tue, 16 Nov 1999 12:08:27 -0800 (PST)

> client verifier changes, implying that it isn't reset at

Ok, I am stupid, I can't find 8.4.2 in any of the drafts on playground.
The only need to keep a sequence number between OPENs is if we have
required at-most-once semantics to OPEN operations. Otherwise
it is sufficient to maintain the seqid and lockowner for the duration
of the OPEN/CLOSE period. Do we care about at-most-once OPENs?
If we define multiple OPENs with the same lockowner as state
invariant then it shouldn't matter.

> kind of thing has been done before.

I actually preferred your descriptions in quotes: "There may have
been such an agreement at one time but there isn't 
now and I don't remember it".

> state-modifying ops.  It just seems that an error is cleaner

I have no trouble sending the error that should be ignored back, it
is likely a useful piece of information in some debugging cases.

> as well, but not returning an error is going to make the 

Again, from a theoretical perspective it is unnecessary, reality
would probably indicate it wise to error anyways.

> Brown recently posted a proposal to do this.

My fear is that we are in a rush to close the spec even though no one
has prototyped the locking portions yet. There were changes made
due to the simple ops being prototyped so we should not be in too
big a hurry to nail the lid shut.

> should fail.  At least, I assume so.

Yup.

> subsequent share reservation requests from Windows clients).

I would expect the subsequent OPEN (or CLOSE/OPEN pair) would fail
due to the DENY=WRITE.

> you want to close one or the other of the two open's

Why can't we require a COMPOUND op with a CLOSE/OPEN pair?

> we should go back to having LOCK take a lockowner

Since fork() loses all locks, process 101 effectively has a filehandle
as the result of a LOOKUP without OPEN.  If it wishes to participate
in locking it must reOPEN the file to get a new valid stateid.

> revoked one.  I think that should be OK and not an

Either the server doesn't error, or clients ignore the error.  I would
prefer the latter as that is how things mostly work today.  On close
everyone just ignores errors.

> lock) if you are just giving up locks.

If the stateid is invalid then the server may have no way to use the
value reliably.  So it is dubious to allow it.

> sure we have enough room.

In our earlier discussions we talked about how an implementation could
use pointers and needing a triple to represent it.  If you look at how
often a stateid will change then 64-bits is probably enough and a
good server can pack all the information in easily. So I am unmotivated
to expand the field when most of the bits will be static and compaction
is a well understood science.

> almost always the case).  We discussed this a while back.

All leases for a client should expire at once, to do otherwise unnecessarily
complicates the protocol and has a strange implication that the client/server
would be able to maintain some of its state but not other.  We should
tighten up the spec to remove the "almost always" for everything except
an adminstrative revocation.

	-David

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 16 Nov 1999 13:10:00 -0800

> Ok, I am stupid, I can't find 8.4.2 in any of the drafts on 

Duh.  I'm stupid.  I meant 8.2.4.

That's in:
 
   http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-02.txt

> invariant then it shouldn't matter.

I was assuming that OPEN and CLOSE are considered locking 
requests because share reservations are locks, etc.  OPEN
does have a seqid4 which I assumed was for that purpose.
On the other hand, I've just looked at CLOSE and it doesn't
have a sequence number so I don't know what to think (Other
than things need to be clarified).

If we decide that OPEN and CLOSE do not need to take part
in the sequence logic (and that might depend on how we
resolve the issue of whether there can be multiple open
for the same file and lockowner at the same time), then
I still have concerns about whether sequencing can be
reset when a lockowner has no locks.

I guess it depends on what the sequence logic is intended
to prevent.  It is intended only to deal with at-most-one
semantics for retries, then this resetting might work.
The problem is that in other cases (changng stateid's
for example) we have used something I'll call the 
Byzantine Router Test.  We assume that a nasty router 
has held onto a request for a long time (re-ordering
across a very large span of messages) and send the
delayed request just at the wrong time.  If we want
to be safe against this, then you can come up with
scenarios that violate at-most-once semantics.

    OPEN a file.

    Do a lock with sequence zero (Router holds onto some
    instance of that at is successfully retried).

    Do an unlock with sequence one.

    Do a CLOSE

    Do an open of the same file.

    Router disgorges saved LOCK request which is erroneously
    acted on.

Maybe this isn't an issue but we have made (and accepted)
arguments of this form in the past.  The logic that requires
stateid's to change so that a delayed WRITE is not erroneously
acted on past an unlock that invalidates it is exactly of
this form.
    
> would probably indicate it wise to error anyways.

In theory, people don't mistakes ;-)

> big a hurry to nail the lid shut.

I agree.

> Why can't we require a COMPOUND op with a CLOSE/OPEN pair?

What if the CLOSE succeeds and the OPEN fails?  Then you
have to try to do an OPEN to get back the original state.
The problem is that it may fail.  If the first OPEN, fails 
because the file has been renamed, for example, then the 
OPEN to try to get back the original read-only open is 
going to fail as well, which is not very nice.  I don't 
think this is workable.

>From the Unix point of view, it has the file open and it
is entitled to do the locking.  If the client file system
has to do an OPEN to effect this, this would be OK, except
that the OPEN can fail (file could have been renamed).

> everyone just ignores errors.

OK.

> value reliably.  So it is dubious to allow it.

If a stateid is invalid only because some embodied in the 
set of locks it covers has been revoked, then I don't see
any problem in allowing a CLOSE to indicate that the client
won't refer to this invalid stateid any more.

> is a well understood science.

OK.  But when you start running out of bits, just remember, you
had the chance to expand this.

> an adminstrative revocation.
 
I agree.

---

From: David Robinson <[email-masked]>
Date: Tue, 16 Nov 1999 15:03:21 -0800 (PST)

> acted on past an unlock that invalidates it is exactly of

Dealing with the Byzantine Router is a very hard problem.  The only way
to deal with it is to maintain state forever.  Consider the case where
it is the initial OPEN on the initial contact to a server, the server
has no state so it must accept the request.  If the BR retransmits this
request in the future it may be hard to defend it, probably can. Then
consider a server reboot with the same BR, it needs to defend against
all possible requests that ever got from that client which is an
unsolveable problem (run out of storage and seqid space).

I think we have to accept the possibility of rogue requests and recommend
that servers do their best to defend against them, however there will
be an upper bound similar to defending against duplicate XIDs.

I am starting to believe that multiple OPENs with the same lockowner
is going to be more trouble than it is worth.  If instead we allow
an OPEN to upgrade/downgrade an existing OPEN with a single CLOSE
terminating the state it pushes the multiplexing back to the client
where it is more easily handled. The seqid can be used to guarentee
ordering but with the above caveat, the server can timeout and saved
lockowner within some reasonable amount of time (N * lease interval?).

> that the OPEN can fail (file could have been renamed).

The client is responsible for recreating the OPEN, typically this
would be hid from the application.  If a failure occurs then that
may happen and the child gets and error.  The alternative is to have
the server try and handle multiple threads and handle one app doing
a CLOSE, or grabbing new locks (should it use the old or new lockowner,
how does it distinguish requests if it is the same...?) This just seems
too hard for a server to deal with and easier for the client.

> had the chance to expand this.

I'll open this question up to the rest of the WG, is an additional
32 bits a concern to anyone? I am really neutral on this.

	-David

---

From: David Robinson <[email-masked]>
Date: Wed, 17 Nov 1999 11:01:51 -0800 (PST)

> I would simply avoid the first case and say you have to
> get rid of the record locks before you do the CLOSE.
> I think the spec now says the the server may give you
> an error if you do a CLOSE while you still have locks.
> I would change this to having required to give you an
> error and require the client to do the unlocks first.
> This means that a LOCKU cannot be the final request
> in a sequence stream so it is ok as far as (non-BR)
> at-most-once semantics.

I agree. In the non-COMPOUND world it seemed nice to just let the CLOSE
free all the locks, like Unix close() does.  But with COMPOUND
it would not be hard to just make it a complex request (hey COMPOUND
is useful :-).

> This leaves CLOSE.  It doesn't have a seqid currently
> and if it did, the fact that the server would throw
> the information away would be a problem.  As it stands,
> a retransmitted CLOSE would be rejected with a bad
> stateid.  Most servers will have a reply cache so the
> question is whether we need an additional level of 
> protection against retransmission issues.  I'm inclined
> to say "no".

Agreed.

	-David

---

From: David Robinson <[email-masked]>
Date: Wed, 17 Nov 1999 11:01:51 -0800 (PST)

Agreed.

	-David