Subject: RE: Some issues with stateid's

From: Neil Brown <[email-masked]>
Date: Tue, 2 Nov 1999 10:35:45 +1100 (EST)

On Friday October 29, [email-masked] wrote:
> 
> I just thought of one other issue for this approach.  It is
> basically an extension of the issue I brought up a few days
> ago regarding spece management for per-lockowner structures.
> As things currently stand, per-lockowner structures need to
> hold a sequence id to compare to the sequence id in any
> forthcoming requests for that lockowner.  The problem is
> that the server has no way of knowing when a given lockowner
> is gone and dispensing with the data structure since it
> must keep the sequenceid to compare with a possible subsequent
> request for that lockowner.
> 
> If stateid take over the function of sequencid's than the 
> problem migrates and instead of having lockwoner structures
> which you have to keep around, you have the problem for 
> each lockowner-file pair.  This is much worse.  A given 
> lockowner can easily open millions of files (not all at the
> same time usually).  If stateid's stay as they are, then
> on the CLOSE, we can return some distinguished dummy stateid
> and get rid of the storage associated with that lockowner-
> fiel pair.  If stateid's take on the sequence function, we 
> can't.  You have to keep the stateid, so it can determine
> whether a subsequent OPEN of that same file is correct.
> 
> It's true that we don't have a solution to lockowner space
> management problem anyway, but solving it for lockowner-file
> pairs seems a lot harder.

I think there is an answer which is actually much the same either way.

Let me explain.  

The the way I picture client-state working, there is an "overall
state" for each client which is sub-divided into a number of
independent sub-states (A Commonwealth of independent states?:-) The
over-all state is created by SETCLIENTID and referred to by the
clientid.  The sub-states are created by the appearance of a new
nfs_lockowner and are referred to by  filehandle/stateid/sequenceid
triples.  As these states change over time, it is perhaps best to think
of the "overall state" as a main trunk, with the sub-states being
branches that fork off and have their own independent sequence of
state changes.

Given this image, it seems that what needs to happen when a sub-state
is no longer needed is that it needs to be joined back into the
main state trunk (the "overall state").  This overall state would then
need to show a state transition, either by having the clientid change,
or by having an associated "sequenceid"-like number change.

Whether the clientid changes, or whether there is a separate
client-sequence-id is pretty much a question of syntax, so I will
assume that the clientid changes each time a branch is joined back to
the "overall state" trunk.

Given this we need to introduce a new operation (or redefine an old
operation) to explicitly join a sub-state back into the "overall
state", and explicitly change the clientid.

With the current sequence-id scheme, this join operation would
probably look like:

  EXIT: clientid, (cfh), stateid, sequenceid -> clientid
or 
  EXIT: nfs_lockowner, sequenceid -> clientid

The second is possibly more natural.

I chose the name "EXIT" because it would effectively be used when an
nfs_lockowner ceased to exist, and nfs_lockowners sort of model
processes.

If state sequencing was changed to be per-lockowner-file rather than
per-lockowner, by the removal of the sequenceid, then the join
operation would look like:

   CLOSE: (cfh), stateid, clientid -> clientid

The "CLOSE" here is slightly different to the CLOSE currently in the
draft as the draft seems to allow byte-range-locks to remain active
after a close.

Ofcourse if we also merge stateid with filehandle, the CLOSE
operation becomes
   CLOSE: (cfh), clientid -> clientid

So, I think the space-management issue is solvable, that it will need
a change to the protocol, but that it is essentially the same
issue whether per-lockowner sequenceids are maintained or not.

>  
> > > 
> > > The biggest issue is OPEN.  OPEN now takes a seqid and a lockowner
> > > but not a stateid.  You are presumed not to have one with OPEN 
> > > being the way you get the first one.  This is why the seqid is
> > > per-lowckowner, while stateid's are sequenced
> > > per-lockowner-per-file.
> > 
> > Yes, I see.  Hadn't thought of that side of it.
> > But this begs the question:
> >   "Can a given lockowner have a given file open twice?"
> > 
> > If "no", then we could tell open to take a stateid, and give the first
> >    open of any file a stateid of 0.  Subsequent opens get the stateid
> >    returned from the previous close (or if the server rejects it, a
> >    stateid of 0... might need a bit of work)
> 
> If you can't have multiple simultaneously active OPENS for 
> the same lockowner-file pair, then I think the idea of having
> a special initial stateid (like zero) will work.
> 

> > 
> > If "yes", then...
> >    Would this mean that there might be two active stateids for one
> >    lockowner/file pair?
> 
> I think the answer is no, stateids are per lockowner/file pair.
> 
> >    If "yes" then we expose the "interesting" Posix semantic that locks
> >       are shared among all file descriptors for the one process.  I
> >       would rather not expose one API so blatently in the protocol.
> 
> I don't think having multiple stateid's for a given lockowner/file
> pair is a good idea.  That's not the way the protocol is now and
> I don;t see the benefit.
> 
> I don;t follow how this would expose any semantics however.  The
> semantics are up to the client.  The server assumes all locks for
> a give nfslockowner are shared.  It is up to the client to determine
> what constitutes an nfslockowner and set the lockowner value when
> making requests.
> 
> >    If "no", then we can require the open to take a stateid which must
> >       be the "current" stateid.
> > 
> > However, if "no", (that is, a lockowner can only have a file open
> > once), then:
> >    How does a client implement:
> >      fd1 = open("/a/file", O_RDONLY);
> >      fd2 = open("/a/file", O_RDWR);
> > 
> >    and still get the ACCESS_READ/ACCESS_WRITE/ACCESS_BOTH state right?
> >    It really needs to be able to "reopen", or "reshare" or something,
> >    which sounds a lot like calling open again, but with a 
> > non-zero stateid.
> 
> Or like calling open again with no stateid and the appropriate
> sequenceid, as would be the way to this as the protocol stands.

I guess we can forget the bit about exposing semantics.  If we are
agreed that "The server assumes all locks for a give nfslockowner are
shared.", then that is a good starting point (but see (*) below).

An nfslockowner owns share reservation as well as locks, so obviously
they will be shared too.  The image I get here is that each
nfslockowner has a single "connection" to the file on the server, and if
the lockowner internally thinks that it has multiple handles
(e.g. file descriptors in unix) then they are effectively multiplexed
over that "connection".

Anyway, there needs to be a way for an nfslockowner to "upgrade" (or
"downgrade") it's share reservation on a particular file.  It could be
that "open" can do that.  However I don't think it is quite up to it
at the moment.
Currently OPEN does not take the filehandle of a file, but rather the
filehandle of a directory, and a filename (except when reclaiming
after server restart).
Now when a process wants to upgrade the access mode on a file in the
POSIX API, the file name is always available.  I don't know that we
can guarantee that for every API.

Also, when thinking around this issue I came across the scenario:

  fd = open("/a/file");
  lockf(fd, F_LOCK, 20);
  if (fork()==0) {
     lockf(fd, F_LOCK, 20); /* block until parent releases lock */
  .....

It is currently very unclear how the second lock call would be
implemented.
Looking at section 12.2.10 of draft-02, the SYNOPSIS of LOCK contains
an "owner", but "struct LOCK4args" contains a stateid!! I assume that
using a stateid is the more current intention.

So we need a stateid to make the lock.  The only way to get a new
stateid associated with a new nfslockowner is to use the OPEN call.
But as noted, this needs a filename.  In general (given the
possibility of intervening renames), the client does not know what the
filename is, and so cannot use OPEN (as it currently stands).  This
actually applies to any "fork" call where there is a need to identify
a new nfslockowner with the new process.

(*)It was this uncertainty about how an share-reservation could be
upgraded that lead me to wonder if a file could have two stateids for
the one lockowner.  I'm glad that we are agreed that it cannot.
However this is separate from the question of whether sequenceids could,
or should, be merged with stateids.

>  
> > It seem to me that there are two choices:
> >   1/ Allow a lockowner to have two active stateids on the one file,
> >      and expose the details of posix locks in the protocol, (and
> >      thereby break WIN32 Locks? I don't know, but I guess so).
> > 
> >   2/ Require the Client(*) to do any intra-lockowner negotiation and
> >      appear to open each file just once (per lockowner), with the
> >      option of changing the "access" type without closing the file.
> > 
> > I like option 2 even if we keep sequenceids, and I think it means that
> > we don't need them.
> 
> I don't see why these are the only options.  What about explicitly
> allowing multiple OPEN's of a file for a given lockowner just
> as we allow multiple LOCK's for a given file-lockowner pair.
> 
We allow multiple LOCKs for a given file-lockowner, but locks apply
to file-lockowner-byterange.  We do not allow multiple locks for a
given file-lockowner-byteaddress (do we?).  For exactly the same
reason we cannot allow multiple OPENs for a given file-lockowner.

NeilBrown

---

From: Neil Brown <[email-masked]>
Date: Tue, 2 Nov 1999 10:45:55 +1100 (EST)

On Wednesday October 27, [email-masked] wrote:
> 
> All of the stateid fields are OK but one of the locks
                                       ^^^
> designated by the stateid has been revoked.  ("Yes,
> that is our agreement, but I have decided it no longer
> applies.  So sue me").

I think the above "one" should be "all".
Unless there is a reliable way for the client to determine which lock
(or DENYal, or delegation) was revoked, there seems little point in
revoking one lock without revoking them all, and I can't see a client
doing anything useful with the remaining locks when one is gone.

NeilBrown

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 15 Nov 1999 08:31:20 -0800

> Hi Dave,
>  Thanks for the ongoing discussion...
> 
> On Saturday November 13, [email-masked] wrote:
> > I'm not sure how changing the clientid interacts with
> > other operations, that are using the previous clientid
> > (e.g. OPENS of other files by other lockowners).  Also,
> > it seems like EXIT's have to be single-threaded with 
> > this approach.
> > 
> 
> Yes, EXITs would have to be single-threaded with respect to each
> other, and with respect to any OPENs.  I don't think that this should
> be a problem as EXIT should not be time-critical - assuming that all
> access/deny rights can be dropped independantly of the EXIT.
> It is true that the client might have to delay some OPEN calls while
> an EXIT is in-flight. I think there should be plenty of times 
> between OPEN
> calls to slip in EXIT calls.

Given that EXIT's are highly delayable, this probably won't
be a performance issue, but I worry that eliminating the
performance effect is going to add to complexity.  You need 
to have a list of EXIT's that need to be done, a count of
in-flight OPEN's, a flag to indicate whether an EXIT is
being done and code to wait when sending an OPEN or EXIT
for the appropriate conditions to .   This is all
an SMOP, but there is added opportunity to make mistakes.
Note that all these of these fields are per-server, not
per-client-mount, which provides another great opportunity
for this stuff to work most-of-the-time.

As EXIT is defined, you have to do a single RPC per EXIT,
which is single-threaded.  Maybe EXIT could take an
array of nfslockowner's?

> > I agree that a change to the protocol is required but I'm
> > hoping that we don't need a large one.  Freqently changing
> > clientid's does seem a fairly radical step.  I'm going to
> > propose something simililar to EXIT (I call it LOCKU_OWNER) 
> > which addresses the space issue and also allows recovery
> > from abandoned stateful requests (which was an issue I
> > talked about at the wg meeting in Washington).
> 
> I look forward to hearing the details - maybe in the minutes.

There are some cryptic words on a slide.  I think Brent
will be posting them somewhere.

The issue is that we are used to assuming we can give up on
nfs requests because of a soft or intr mount.  With NFSv4,
any requests that include one or more of the Stateful Four
ops (OPEN, CLOSE, LOCK, LOCKU), can't be given up on so easily.  
After giving up on such a request,

    You don't know the correct sequence id because you don't
    know if the requests was processed.

    The request may be processed after you have stopped 
    listening for replies and so you won't know the proper
    stateid returned in the reply and can never get it.

    The reply that you are not listening for may have given
    you a lock which you don't know about and which you
    can't reference (no stateid), but whose lease will
    continue to get implicitly renewed, until the client
    reboots or it is revoked by administrator action.

I was hoping that LOCKU_OWNER would solve the sequence
id space problem, but it won't :-(  and also allow,
by allowing you to release all locks for a given lockowner,
without a sequencid or stateid, give you some sort of
recovery path, in the event process termination while 
waiting for a stateful request to be processed.
    
> 
> I see clientid changes as very similar in concept and purpose to
> stateid changes.  I see the particular need being to cope with the
> possibility of out-of-order and delayed delivery.
> Does your LOCKU_OWNER deal with this situation?:
> 
>   Imagine the sequence
>    OPEN - first use of a new lockowner, so state gets created
>    READ
>    WRITE
>    CLOSE
>    LOCKU_OWNER - the process (lockowner) exits so client 
> discards state.
>   
>   Now suppose the first OPEN gets inadvertantly routed via 
> timbuktu - or
>   in some other way gets delayed but not lost - while the 
> other requests
>   and responses go direct.
>   The OPEN will be retransmitted and the whole sequence could be
>   complete in seconds.
>   Then the original OPEN arrives.  How is the server to know that this
>   isn't a valid new request for an OPEN by a new lockowner? 

Yup.  That's the problem that I was forgetting about.
When a lockowner that you haven't seen appears with a 
sequence id of zero, the server implicitly has to
create an object to represent the lockowner.  I now
think it is clear that you need some sort of sequence
(or other) constraint to prevent this from happening
when you don't want xit to.

Here's an alternative proposal.  The idea is that
a lockowner is an object that has to be explicitly 
created and destroyed is managed very much like locks 
are currently managed in the protocol, just one level
above.

You would have two ops, CREATE_OWNER and DESTROY_OWNER.
CREATE_OWNER would take a lockowner and ownerid,
which is very like a stateid, DESTROY_OWNER would 
take an ownerid and make it invalid.  Ops that currently
take lockowers would take ownerid's instead.  Presenting
an invalid ownerid would be an error.

All {CREATE,DESTROY}_OWNER would have a sequence number
which would be a global (per-client-per-server) sequence
stream.  This would not require that the client single-
thread requests.  The client could send multiple requests
with sequence numbers which would be used by the server 
to sequence processing.  An out-of-order request could be
immediately rejected or kept around for a short time to
allow network re-ordering to correct itself.  However,
this sequence number would not have to appear in OPEN
or any other requests so these could proceed unhindered 
once the appropriate ownerid is gotten.

This is not an enormous change to the protocol but I
think it is bigger than many people would like to be
making at this stage, but if there is an answer to this
problem that only changes the protocol in tiny ways,
I don't know what it is.

I think there are a range of opinions about exactly what
stage we are at and thus what kinds of changes are
appropriate for the current stage.  To indulge in a
somewhat extended metaphor, with the inclusion of minor
versioning, we are done with major construction.  No more
hammers, saws or nails.  I would say we are not 
purely at the stage of polishing (dealing with small
inconsistencies and obscurities in the spec).  There are
a number of areas where coarse sandpapering is necessary
first and we just need to get it done.  
  
> > Wehn you say "For exactly the same reason", I'm trying to 
> > figure out precisely what that reason is.  As far as I can
> > see, we don't allow multiple locks for the same byterange
> > because,
> > 
> >     a) It adds complexity
> > 
> >     b) The locking packages that servers will use to 
> >        implement this typically don't support such a
> >        notion.
> > 
> >     c) There is no benefit.
> > 
> I think my "same reason" was
>       d) It doesn't have a well defined meaning.
> 
> Suppose that lockowner X has a shared lock on file F at offset O.
> Then X asks for an exclusive lock on file F at offset O.  What does
> that mean? What is supposed to happen?
>    1/ Should the request block/fail until the first lock is
>       released?  No. That would happen if it was a different
>       lockowner, but not if it was the same.
>    2/ Should the status of the lock be "upgraded" from share to
>       exclusive?  Probably, that that isn't "another lock" it is
>       upgrading the first lock.
> 
> In the same way, if a lockowner has a ACCESS_READ/DENY_WRITE open on a
> file, and asks for an ACCESS_BOTH/DENY_BOTH open, what should happen?
> I think the answer has to be that this upgrades the access/deny mode
> on the original open.   Anything else doesn't seem to make sense.

That appears to be saying it does have a clear meaning.

If you did close of the first open, you would have accomplished
an upgrade without an explicit upgrade request.  That's 
where I was going.

> 
> 
> 
> > I was hoping that allowing multiple opens on the same file
> > would allow upgrade/downgrade to be done without adding 
> > new operations.  You could just do the the open for the new
> > level and close of the old.  That was my answer to c).
> > Also, servers are used to doing multiple opens for the same
> > file for the same process so b) doesn't apply and the
> > complexity is less since all the overlapping byte range
> > stuff doesn't apply so a) is not as strong.
> > 
> > Because of the problems with OPEN requiring the name, I
> > think we will need explicit upgrade/downgrade operations,
> > so I'm no longer pursuing the multiple opens approach.
> 
> I think we need explicit upgrade/downgrade too, though I don't think
> the need for names in open should be the reason.  My feeling is that
> OPEN should be able to take a file handle to open, only requiring a
> name when a create/open is needed.
> 
> (Vaguley) Related to that, there is one aspect of delegations that I
> haven't managed to figure out yet - maybe you could help me.
> 
>    A rename requires that all delegations on the file be recalled. Why
>    is that? 
 
It's definitely related.  The delegated client can perform
opens on its own on the delegegated file.  Opens are in terms
of a name so if the delegation were not recalled at this
point, an open under the old name could erroneously succeed.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 15 Nov 1999 09:12:03 -0800

> On Monday November 8, [email-masked] wrote:
> > > 
> > > Ah yes, I hadn't got up to there in the archive yet (it is heavy
> > > going).  However....
> > > 
> > >   David: you said:
> > > 
> > >       On an OPEN, the client does not send a stateid so 
> it cannot tell
> > >       the server what its view of the world is.
> > 
> > I think I was paraphrasing Dave Robinson's explanation of why this
> > won't work.
> >   
> 
> The "David: you said" was addressed to David Robinson - I was assuming
> that the difference between Dave and David would disambiguate.  Sorry
> for the confusion.  The words were David Robinson's.
> 
> > > 
> > >   Yet the spec, when it talks about automatic lease 
> renewal in section
> > >   8.4  says that what is needed is
> > >        "a positive indication that the client is still 
> alive and locks
> > >        are being maintained"
> > > 
> > >   So the issue is not "agreed view of world" but is 
> "still alive and
> > >   active".  Surely an OPEN call, which uses a clientid to 
> gain a share
> > >   reservation, is "a positive indication that the client is still
> > >   allive and locks are being maintains".
> > 
> > It certainly proves the client is alive.  I guess the issue
> > is whether the client is still alive (i.e has not gone down 
> > in the interim).  With a stateid, that is should be assured
> > by the server putting some sort of instance id (boot time or
> > something).  I may be wrong, but I don't see that is assured 
> > for an nfs_lockowner.

I think I was wrong here.  I don't think that the client can 
have rebooted and then wind up doing an OPEN without the
the server knowing.  If the client is going to do an OPEN,
he is going to have to present a clientid.  Where does he 
get it?  He got it from the server, by doing a SETCLIENTID.
He isn't allowed to just do an OPEN with some made up 
clientid.  The SETCLIENTID would have told the server
to release all locks and thus all leases for that client.  

I think this means that it would be OK for all of the 
requests that create leases as well those that do IO
to renew all leases for the client.  This means that
all of the leases for a client would expire always
expire the same time (as opposed to now in which it
is almost always the case).  This would simplify both
the client and the server.

> 
> I think the clientid returned by SETCLIENTID needs to be unique in
> space and time for essentialy the same reasons that I think it needs
> to change on CLOSE/EXIT/LOCKU_OWNER (see previous mail) - to guard
> against delayed delivery.

I think we definitely need to be safe against delayed 
delivery.  I think the current protocol isn't safe
against delayed delivery of an open request with 
sequence id zero.  You could solve that problem
by making clientid changeable and unique across
time (even across client and server reboots), but
I'm leaning to the view that the basic problem is
that you have an operation that gets a lock whose
sequence id checking is not very strong.  Zero is
a very common value to be hanging around is delayed
requests and since zero with a random out-of-date
nfslockowner is certain to be accepted you have a
problem.  I think the concept of stateid's makes
the protocol strong once you get them.  The problem
is in the part of the protocol before you a stateid.
I'm hoping ownerid's will make that part of the prorotol
just as strong.
 
> 
> Actually, I would really like it if clients could get the same
> clientid between server restarts, as then it would be a perfect
> candidate for the create verifier!  But I don't think it can work.