Subject: RE: Some issues with stateid's

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 13 Nov 1999 07:13:37 -0800

On Monday, November 01, Neil Brown wrote:
> 
> On Friday October 29, [email-masked] wrote:
> > 
> > I just thought of one other issue for this approach.  It is
> > basically an extension of the issue I brought up a few days
> > ago regarding spece management for per-lockowner structures.
> > As things currently stand, per-lockowner structures need to
> > hold a sequence id to compare to the sequence id in any
> > forthcoming requests for that lockowner.  The problem is
> > that the server has no way of knowing when a given lockowner
> > is gone and dispensing with the data structure since it
> > must keep the sequenceid to compare with a possible subsequent
> > request for that lockowner.
> > 
> > If stateid take over the function of sequencid's than the 
> > problem migrates and instead of having lockwoner structures
> > which you have to keep around, you have the problem for 
> > each lockowner-file pair.  This is much worse.  A given 
> > lockowner can easily open millions of files (not all at the
> > same time usually).  If stateid's stay as they are, then
> > on the CLOSE, we can return some distinguished dummy stateid
> > and get rid of the storage associated with that lockowner-
> > fiel pair.  If stateid's take on the sequence function, we 
> > can't.  You have to keep the stateid, so it can determine
> > whether a subsequent OPEN of that same file is correct.
> > 
> > It's true that we don't have a solution to lockowner space
> > management problem anyway, but solving it for lockowner-file
> > pairs seems a lot harder.
> 
> I think there is an answer which is actually much the same either way.
> 
> Let me explain.  
> 
> The the way I picture client-state working, there is an "overall
> state" for each client which is sub-divided into a number of
> independent sub-states (A Commonwealth of independent states?:-) The
> over-all state is created by SETCLIENTID and referred to by the
> clientid.  The sub-states are created by the appearance of a new
> nfs_lockowner and are referred to by  filehandle/stateid/sequenceid
> triples.  As these states change over time, it is perhaps 
> best to think
> of the "overall state" as a main trunk, with the sub-states being
> branches that fork off and have their own independent sequence of
> state changes.
> 
> Given this image, it seems that what needs to happen when a sub-state
> is no longer needed is that it needs to be joined back into the
> main state trunk (the "overall state").  This overall state would then
> need to show a state transition, either by having the clientid change,
> or by having an associated "sequenceid"-like number change.
> 
> Whether the clientid changes, or whether there is a separate
> client-sequence-id is pretty much a question of syntax, so I will
> assume that the clientid changes each time a branch is joined back to
> the "overall state" trunk.
> 
> Given this we need to introduce a new operation (or redefine an old
> operation) to explicitly join a sub-state back into the "overall
> state", and explicitly change the clientid.
> 
> With the current sequence-id scheme, this join operation would
> probably look like:
> 
>   EXIT: clientid, (cfh), stateid, sequenceid -> clientid
> or 
>   EXIT: nfs_lockowner, sequenceid -> clientid
> 
> The second is possibly more natural.
> 
> I chose the name "EXIT" because it would effectively be used when an
> nfs_lockowner ceased to exist, and nfs_lockowners sort of model
> processes.

I'm not sure how changing the clientid interacts with
other operations, that are using the previous clientid
(e.g. OPENS of other files by other lockowners).  Also,
it seems like EXIT's have to be single-threaded with 
this approach.

> 
> If state sequencing was changed to be per-lockowner-file rather than
> per-lockowner, by the removal of the sequenceid, then the join
> operation would look like:
> 
>    CLOSE: (cfh), stateid, clientid -> clientid
> 
> The "CLOSE" here is slightly different to the CLOSE currently in the
> draft as the draft seems to allow byte-range-locks to remain active
> after a close.

I've proposed previously that byte-range lock not be allowed at
CLOSE.  The spec already (or did at one time) say that servers
may return an error if there are byte range locks, so clients
can't take advantage of this (dubious) feature.  Requiring
the server to make this an error wouldn't be a burden.

> 
> Ofcourse if we also merge stateid with filehandle, the CLOSE
> operation becomes
>    CLOSE: (cfh), clientid -> clientid
> 
> 
> So, I think the space-management issue is solvable, that it will need
> a change to the protocol, but that it is essentially the same
> issue whether per-lockowner sequenceids are maintained or not.

I agree that a change to the protocol is required but I'm
hoping that we don't need a large one.  Freqently changing
clientid's does seem a fairly radical step.  I'm going to
propose something simililar to EXIT (I call it LOCKU_OWNER) 
which addresses the space issue and also allows recovery
from abandoned stateful requests (which was an issue I
talked about at the wg meeting in Washington).

>
> >  
> > > > 
> > > > The biggest issue is OPEN.  OPEN now takes a seqid and 
> a lockowner
> > > > but not a stateid.  You are presumed not to have one with OPEN 
> > > > being the way you get the first one.  This is why the seqid is
> > > > per-lowckowner, while stateid's are sequenced
> > > > per-lockowner-per-file.
> > > 
> > > Yes, I see.  Hadn't thought of that side of it.
> > > But this begs the question:
> > >   "Can a given lockowner have a given file open twice?"
> > > 
> > > If "no", then we could tell open to take a stateid, and 
> give the first
> > >    open of any file a stateid of 0.  Subsequent opens get 
> the stateid
> > >    returned from the previous close (or if the server 
> rejects it, a
> > >    stateid of 0... might need a bit of work)
> > 
> > If you can't have multiple simultaneously active OPENS for 
> > the same lockowner-file pair, then I think the idea of having
> > a special initial stateid (like zero) will work.
> > 
> 
> 
> > > 
> > > If "yes", then...
> > >    Would this mean that there might be two active stateids for one
> > >    lockowner/file pair?
> > 
> > I think the answer is no, stateids are per lockowner/file pair.
> > 
> > >    If "yes" then we expose the "interesting" Posix 
> semantic that locks
> > >       are shared among all file descriptors for the one 
> process.  I
> > >       would rather not expose one API so blatently in the 
> protocol.
> > 
> > I don't think having multiple stateid's for a given lockowner/file
> > pair is a good idea.  That's not the way the protocol is now and
> > I don;t see the benefit.
> > 
> > I don;t follow how this would expose any semantics however.  The
> > semantics are up to the client.  The server assumes all locks for
> > a give nfslockowner are shared.  It is up to the client to determine
> > what constitutes an nfslockowner and set the lockowner value when
> > making requests.
> > 
> > >    If "no", then we can require the open to take a 
> stateid which must
> > >       be the "current" stateid.
> > > 
> > > However, if "no", (that is, a lockowner can only have a file open
> > > once), then:
> > >    How does a client implement:
> > >      fd1 = open("/a/file", O_RDONLY);
> > >      fd2 = open("/a/file", O_RDWR);
> > > 
> > >    and still get the ACCESS_READ/ACCESS_WRITE/ACCESS_BOTH 
> state right?
> > >    It really needs to be able to "reopen", or "reshare" 
> or something,
> > >    which sounds a lot like calling open again, but with a 
> > > non-zero stateid.
> > 
> > Or like calling open again with no stateid and the appropriate
> > sequenceid, as would be the way to this as the protocol stands.
> 
> I guess we can forget the bit about exposing semantics.  If we are
> agreed that "The server assumes all locks for a give nfslockowner are
> shared.", then that is a good starting point (but see (*) below).
> 
> An nfslockowner owns share reservation as well as locks, so obviously
> they will be shared too.  The image I get here is that each
> nfslockowner has a single "connection" to the file on the 
> server, and if
> the lockowner internally thinks that it has multiple handles
> (e.g. file descriptors in unix) then they are effectively multiplexed
> over that "connection".
> 
> Anyway, there needs to be a way for an nfslockowner to "upgrade" (or
> "downgrade") it's share reservation on a particular file.  It could be
> that "open" can do that.  However I don't think it is quite up to it
> at the moment.
> Currently OPEN does not take the filehandle of a file, but rather the
> filehandle of a directory, and a filename (except when reclaiming
> after server restart).
> Now when a process wants to upgrade the access mode on a file in the
> POSIX API, the file name is always available.  I don't know that we
> can guarantee that for every API.

Making that a requirement makes me unconfortable.

> 
> Also, when thinking around this issue I came across the scenario:
> 
>   fd = open("/a/file");
>   lockf(fd, F_LOCK, 20);
>   if (fork()==0) {
>      lockf(fd, F_LOCK, 20); /* block until parent releases lock */
>   .....
> 
> It is currently very unclear how the second lock call would be
> implemented.
> Looking at section 12.2.10 of draft-02, the SYNOPSIS of LOCK contains
> an "owner", but "struct LOCK4args" contains a stateid!! I assume that
> using a stateid is the more current intention.
> 
> So we need a stateid to make the lock.  The only way to get a new
> stateid associated with a new nfslockowner is to use the OPEN call.
> But as noted, this needs a filename.  In general (given the
> possibility of intervening renames), the client does not know what the
> filename is, and so cannot use OPEN (as it currently stands).  This
> actually applies to any "fork" call where there is a need to identify
> a new nfslockowner with the new process.
> 
> (*)It was this uncertainty about how an share-reservation could be
> upgraded that lead me to wonder if a file could have two stateids for
> the one lockowner.  I'm glad that we are agreed that it cannot.
> However this is separate from the question of whether 
> sequenceids could,
> or should, be merged with stateids.
> 
> >  
> > > It seem to me that there are two choices:
> > >   1/ Allow a lockowner to have two active stateids on the 
> one file,
> > >      and expose the details of posix locks in the protocol, (and
> > >      thereby break WIN32 Locks? I don't know, but I guess so).
> > > 
> > >   2/ Require the Client(*) to do any intra-lockowner 
> negotiation and
> > >      appear to open each file just once (per lockowner), with the
> > >      option of changing the "access" type without closing 
> the file.
> > > 
> > > I like option 2 even if we keep sequenceids, and I think 
> it means that
> > > we don't need them.
> > 
> > I don't see why these are the only options.  What about explicitly
> > allowing multiple OPEN's of a file for a given lockowner just
> > as we allow multiple LOCK's for a given file-lockowner pair.
> > 
> We allow multiple LOCKs for a given file-lockowner, but locks apply
> to file-lockowner-byterange.  We do not allow multiple locks for a
> given file-lockowner-byteaddress (do we?).  For exactly the same
> reason we cannot allow multiple OPENs for a given file-lockowner.
 
Wehn you say "For exactly the same reason", I'm trying to 
figure out precisely what that reason is.  As far as I can
see, we don't allow multiple locks for the same byterange
because,

    a) It adds complexity

    b) The locking packages that servers will use to 
       implement this typically don't support such a
       notion.

    c) There is no benefit.

I was hoping that allowing multiple opens on the same file
would allow upgrade/downgrade to be done without adding 
new operations.  You could just do the the open for the new
level and close of the old.  That was my answer to c).
Also, servers are used to doing multiple opens for the same
file for the same process so b) doesn't apply and the
complexity is less since all the overlapping byte range
stuff doesn't apply so a) is not as strong.

Because of the problems with OPEN requiring the name, I
think we will need explicit upgrade/downgrade operations,
so I'm no longer pursuing the multiple opens approach.

---

From: Neil Brown <[email-masked]>
Date: Mon, 15 Nov 1999 12:01:58 +1100 (EST)

Hi Dave,
 Thanks for the ongoing discussion...

> it seems like EXIT's have to be single-threaded with 

Yes, EXITs would have to be single-threaded with respect to each
other, and with respect to any OPENs.  I don't think that this should
be a problem as EXIT should not be time-critical - assuming that all
access/deny rights can be dropped independantly of the EXIT.
It is true that the client might have to delay some OPEN calls while
an EXIT is in-flight. I think there should be plenty of times between OPEN
calls to slip in EXIT calls.

> talked about at the wg meeting in Washington).

I look forward to hearing the details - maybe in the minutes.

I see clientid changes as very similar in concept and purpose to
stateid changes.  I see the particular need being to cope with the
possibility of out-of-order and delayed delivery.
Does your LOCKU_OWNER deal with this situation?:

  Imagine the sequence
   OPEN - first use of a new lockowner, so state gets created
   READ
   WRITE
   CLOSE
   LOCKU_OWNER - the process (lockowner) exits so client discards state.
  
  Now suppose the first OPEN gets inadvertantly routed via timbuktu - or
  in some other way gets delayed but not lost - while the other requests
  and responses go direct.
  The OPEN will be retransmitted and the whole sequence could be
  complete in seconds.
  Then the original OPEN arrives.  How is the server to know that this
  isn't a valid new request for an OPEN by a new lockowner?  

>        implement this typically don't support such a
I think my "same reason" was
      d) It doesn't have a well defined meaning.

Suppose that lockowner X has a shared lock on file F at offset O.
Then X asks for an exclusive lock on file F at offset O.  What does
that mean? What is supposed to happen?
   1/ Should the request block/fail until the first lock is
      released?  No. That would happen if it was a different
      lockowner, but not if it was the same.
   2/ Should the status of the lock be "upgraded" from share to
      exclusive?  Probably, that that isn't "another lock" it is
      upgrading the first lock.

In the same way, if a lockowner has a ACCESS_READ/DENY_WRITE open on a
file, and asks for an ACCESS_BOTH/DENY_BOTH open, what should happen?
I think the answer has to be that this upgrades the access/deny mode
on the original open.   Anything else doesn't seem to make sense.

> so I'm no longer pursuing the multiple opens approach.

I think we need explicit upgrade/downgrade too, though I don't think
the need for names in open should be the reason.  My feeling is that
OPEN should be able to take a file handle to open, only requiring a
name when a create/open is needed.

(Vaguley) Related to that, there is one aspect of delegations that I
haven't managed to figure out yet - maybe you could help me.

   A rename requires that all delegations on the file be recalled. Why
   is that? 

NeilBrown