Subject: RE: unix file descriptors and nfsv4 stateids

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 19 Jun 2001 11:27:43 -0700

Spencer Shepler wrote: 
> On Mon, Kendrick M. Smith wrote:
> > 
> > Here is an issue that I have run into on our Linux client recently, which
> > was actually first brought to my attention by Spencer, who is having
> 
> To give due credit, Eric Kustarz and Bill Ricker here at Sun initially
> raised this issue.

Is that credit or blame?  Gee, everything was fine until you guys started
pointing out problems :-)

> > similar issues in the Solaris client.  Our client currently allocates
> > lockowners by
> >    (1) creating one lockowner per pid
> >
> > The issue here is that the mapping between unix file descriptors and nfsv4
> > open files (=stateids) is as bad as it could possibly be.  On the one
> > hand, multiple file descriptors can correspond to the same nfsv4 open
> > file.  This can happen if a process opens a file more than once, since it
> > is impossible for an nfsv4 lockowner to open a file twice.  

I thought so, but when Kendrick brought up the issue of different
credentials with the same stateid, there was at least some support
for the idea that the same process could do another v4 open if it
had changed to a different identity.  See

http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2001/0342.html et seq.

> > On the other
> > hand, a single file descriptor can correspond to multiple nfsv4 open
> > files.  This can happen if a process fork()'s, and the child tries to use
> > an old file descriptor.  Since the child is a new lockowner, it will have
> > to use a new nfsv4 open file.  So the mapping between unix file
> > descriptors and nfsv4 open files is as bad as it could possibly be, in the
> > sense that it is a many->many mapping.  Implementing this mapping so that
> > it is correct in all cases, and interacts correctly with other parts of
> > the client, such as mmap(), has ended up being a major headache.

I can imagine.

>    In addition to the implementation complexity, I think that there are
> other problems associated with the one-lockowner-per-pid approach.  For
> one thing, it means that when a process fork()'s, a new OPEN is needed
> for each of its file descriptors (of course, in practice, the OPEN would
> be deferred until the child actually did I/O on the descriptor).  Since
> files must be opened by name, this new OPEN might fail if another client
> has renamed the file.

Worse, it might succeed and you get the wrong file.  I think Mike Eisler 
proposed an open by file handle in the above thread to deal with that issue.  
In wierd cases, there may be further credentials issues since an identity 
change may mean you don't have the right to OPEN the file.  Outside NFS-v4, 
you don't have to.  This is not an open, it's essentially a DUP.  It seems
wrong to try to shoehorn this into being an OPEN.
  
> >   For these reasons, I am considering an alternative to (1), namely
> >     (2) creating one lockowner per _inode_.
> > 
> > The client would emit an OPEN when the first process opens the file, and
> > emit the matching CLOSE when the last process closes the file.
> > Throughout the lifetime of the OPEN, it would present the server with the
> > appearance that the file is being accessed by a single lockowner.

Are you saying per-inode or per-(UNIX)-open-file (i.e. DUPs and forks 
excluded)?  

> > This
> > simplifies the client-side bookkeeping of open file data enormously (the
> > seqid and stateid can just be stored in the inode, without need for any
> > auxiliary data structures).  The catch, of course, is that the client
> > must assume responsibility for part of the locking logic, since only the
> > client knows which byterange locks are held by which processes (although
> > the server does know which locks are held by the client as a whole).  For
> > example, when a process locks a file, the client kernel would have check
> > whether another local process has set a conflicting lock, before sending
> > the LOCK request to the server (which will check whether another client
> > has set a conflicting lock).  So there is a certain amount of
> > implementation complexity associated with choice (2) as well.  However, I
> > think that the extra complexity of choice (1) is significantly higher, and
> > the overall tradeoff favors choice (2).  Furthermore, choice (2) has the
> > advantage of confining the extra complexity to the locking subsystem,
> > which makes it easier to audit for bugs.

I'm assuming that you have concurrency control to prevent multiple 
locking/unlocking requests from being handled at the same time.  You
need this or else sequence number logic won't work properly, but it
also simplifies the logic you are proposing to do lock arbitration.

I think even with this there may be a lot of hard-to-manage comlexity
due to the POSIX locking semantics that we all know and despise.
Consider one of the processes with an exclusive lock and suppose
the other tries to do a shared lock for a partially overlapping 
range.  You have to wait for the orginal process to unlock (or
reduce shared) the overlapping bytes and then try to get the 
lcok you want which may have in the interim gone to someone else.
I think this approach my be difficult to get right.

> > In addition to implementation complexity, there are also performance
> > tradeoffs to be considered.  I imagine that choice (2) will slow things
> > down if many processes are participating in byte-range locking on the same
> > file, because of the extra serialization (since LOCK/LOCKU requests are
> > serialized on a per-lockowner basis). 

If you really mean per-inode, then I think this will be a problem. 

> > On the other hand, choice (2) may
> > actually be faster if many processes are participating in ordinary file
> > I/O, since it avoids the overhead of all the extra
> > OPEN..OPEN_CONFIRM..CLOSE sequences (although you'd probably still want
> > some ACCESS calls).  Choice (2) is clearly faster if a process is doing
> 
> ACCESS would be required so that the client has the "right" answer
> before allowing an application to open() the file.

Yes if the client is doing an open(), but not if he is just doing a
dup() or a fork().

> > lots of fork()'s and sharing file descriptors, since the extra
> > OPEN..OPEN_CONFIRM..CLOSE sequences are eliminated completely.  So the
> > performance tradeoffs paint sort of an unclear picture, but I don't see a
> > strong performance-related argument against choice (2).

It depends on what (2) means.  I'm not clear exactly what you are
proposing.  I think we are getting to the point where e-mail is not
the best way to continue this discussion, at least for a while.

We have a number of related issues about the lockowner/stateid model,
including:

    The upgrade/downgrade issue for NT servers.

    The issue of the relation of credentials to lockowners.

    The issue of fork/dup and how those relate to locking.

Luckily, many of the interested parties are going to be at the bakeoff.
I propose we reserve some time to explore these inter-related issues
and come up with something that addresses them all.  I know there may be 
some who are interested who are not going to be at the bakeoff but we
can accommodate them on a conference call.

Can people who would like to participate but will not be at the bakeoff
let me know about their requirements for scheduling.  We'll try to 
accommodate everyone if we can.

> > So my question is, does anyone see any reason that I haven't thought of
> > why choice (2) won't work?  I'm asking because it seems to be "abusing"
> > the protocol (or at least using it in a way that wasn't intended), so
> > I wanted to get some feedback before diving in and implementing (2).

I think (2) can be made to work but I don't think you should have to
do that.  It seems to me a sign of some problem in the protocol, if
something that complex is required.

I think this is not a case of you abusing the protocol but of the
protocol abusing you.  I think we need to fix the problem.  Now if
we just figure out what the problem is.

> Actually, choice (2) aligns more closely with the delegation
> mechanisms.  As you point out, the client has to do this bookkeeping
> for recovery purposes anyway.

Bookkeeping is one thing.

Doing conflict resolution among your processes when you are the only 
client (for write delegation) is another.

Try to resolve conflicts among your processes when there are otehr
clients active seems a lot more difficult than either of those.

> > Finally, I'd like to add a "philosophical" remark about the source of
> > the difficulty.  It seems to me that nfsv4's concept of a lockowner is
> > really overloaded with 2 meanings:
> >   a) it has the obvious meaning, the unit of lock contention
> >   b) it is the unit of open file ownership

The assumption was that those were the same, but I think they aren't.

> > In our Linux client, we can define our nfs_lockowner4's so that we get
> > a clean fit with either of these meanings, but not both (choice (1)
> > matches cleanly with (a); choice (2) matches cleanly (sort of) with (b)).
> > Our problems would be avoided if there was a way for us to seperate these,
> > e.g. by using a lockowner in a LOCK request which differed from the one
> > used to originally OPEN the file.  However, this is touching on delicate
> > issues of replay protection, so it's not clear to me whether this could
> > be done without an impratically large amount of change to the protocol.

It will have to be thought about carefully but my inclination is
to fix the model if it wrong.  I'm not so worried about the replay
protection issue.  I think what we want to do about that is to
leave the sequence logic with the openowner and let the lockowner
be different. 

> Not sure how large of a change it would be but we could introduce
> something like a LOCKOWNER_CONFIRM to be employed at the first use of
> a lockowner.  OPEN would then be used, as you suggest, when the client
> first instantiates the file and for the entire client.  The associated
> stateid would represent the initial OPEN and any upgrade/downgrade
> sequences along with the CLOSE.  It would also be used for the
> delegation piece as well.  I haven't thought about it much but it may
> not be to unreasonable a change to clarify/simplify the model.

That's a possible approach as well.

> If we choose your option (2), the specification will need to clarify
> when CLOSE is to be used.  As you point out it needs to be done on
> last application close().  The client should not defer use of CLOSE in
> the hopes of caching the file reference/data.  A delegation would
> signify to the client the appropriate behavior with respect to
> caching.

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 19 Jun 2001 13:58:54 -0500

> pointing out problems :-)

Just trying to deflect the arrows but you caught me.

> wrong to try to shoehorn this into being an OPEN.

However, the client should keep the credentials of the initial OPEN
around so that subsequent OPENs will present themselves correctly.  If
I understand the point correctly.

> I think this approach my be difficult to get right.

We have this issue in the current model for file locking because there
are no blocking locks, right?  Two procs on the same client can end up
with this issue today.  Are you saying that it is just difficult to
manage at the client because the client needs to be careful about what
lock requests are passed to the server vs. what can be kept at the
client?

> If you really mean per-inode, then I think this will be a problem. 

If Kendrick meant per-lockowner and this lockowner was in the "new"
model where there is a single OPEN and a single lockowner for all
processes accessing/locking the file, then the serialization would
handle the sequence id issue you raised earlier and should be similar
to the book keeping that needs to be done at the client for the
recovery scenarios.

> Yes if the client is doing an open(), but not if he is just doing a

Right.  Sorry that I was not complete in the comment.

> clients active seems a lot more difficult than either of those.

So to refer back to your earlier comment, the issues revolve around
how the client can track when it should answer locally with respect to
a lock request and when it should go to the server for that answer.

-- 

- Spencer -

---

From: "Kendrick M. Smith" <[email-masked]>
Date: Fri, 22 Jun 2001 11:10:52 -0400 (EDT)

> wrong to try to shoehorn this into being an OPEN.

Actually, I've always privately thought that open-by-filehandle would be
a good idea.  So far, no client implementation has needed it; they all
implement API's which call for open-by-name.  But it doesn't seem
farfetched at all to think that there might one day be clients whose
API's call for open-by-inode or the like (or worse yet, that an unforseen
special case might turn up on an existing client, which makes
open-by-filehandle necessary).  Furthermore, open-by-filehandle could
be introduced without changing the on-the-wire protocol at all; one
could just say that length 0 pathnames are allowed in OPEN.  As a proof
of concept, I tried implementing this in our Linux server this morning
and found that it only took 8 extra lines of code.  I personally like this
idea, since it can be implemented so straightforwardly and avoids a
potential major client issue, but this is not a live-or-die issue at
all for me, since Linux doesn't seem to need it.

> leave the sequence logic with the openowner and let the lockowner

I think that the replay protection has to be associated with the
lockowner, to prevent race conditions where (because of packet
reordering or replay) client & server disagree about which parts
of a file are locked by a particular lockowner.

- k

---

From: Eric Werme USG <[email-masked]>
Date: Fri, 22 Jun 2001 11:41:30 -0400

Actually, I've always privately thought that open-by-filehandle would be
   a good idea.  So far, no client implementation has needed it; they all
   implement API's which call for open-by-name.

Thos of us who cling to user level lock daemons need such a beast:

	fa.fh = a->lck.fileid.fhandle.n_bytes;
	fa.filemode = O_RDWR;
	fa.fd = -1;
	if (fcntl(0, F_CNVT, &fa) == -1) {

The lock request comes with a file handle, this call returns a file
descriptor to that file.  Then lockd does (nearly) conventional fcntl
lock/unlock calls to manipulate server state.