Subject: unix file descriptors and nfsv4 stateids

From: "Kendrick M. Smith" <[email-masked]>
Date: Mon, 18 Jun 2001 16:20:47 -0400 (EDT)

Here is an issue that I have run into on our Linux client recently, which
was actually first brought to my attention by Spencer, who is having
similar issues in the Solaris client.  Our client currently allocates
lockowners by
   (1) creating one lockowner per pid

The issue here is that the mapping between unix file descriptors and nfsv4
open files (=stateids) is as bad as it could possibly be.  On the one
hand, multiple file descriptors can correspond to the same nfsv4 open
file.  This can happen if a process opens a file more than once, since it
is impossible for an nfsv4 lockowner to open a file twice.  On the other
hand, a single file descriptor can correspond to multiple nfsv4 open
files.  This can happen if a process fork()'s, and the child tries to use
an old file descriptor.  Since the child is a new lockowner, it will have
to use a new nfsv4 open file.  So the mapping between unix file
descriptors and nfsv4 open files is as bad as it could possibly be, in the
sense that it is a many->many mapping.  Implementing this mapping so that
it is correct in all cases, and interacts correctly with other parts of
the client, such as mmap(), has ended up being a major headache.
   In addition to the implementation complexity, I think that there are
other problems associated with the one-lockowner-per-pid approach.  For
one thing, it means that when a process fork()'s, a new OPEN is needed
for each of its file descriptors (of course, in practice, the OPEN would
be deferred until the child actually did I/O on the descriptor).  Since
files must be opened by name, this new OPEN might fail if another client
has renamed the file.
  For these reasons, I am considering an alternative to (1), namely
    (2) creating one lockowner per _inode_.

The client would emit an OPEN when the first process opens the file, and
emit the matching CLOSE when the last process closes the file.
Throughout the lifetime of the OPEN, it would present the server with the
appearance that the file is being accessed by a single lockowner.  This
simplifies the client-side bookkeeping of open file data enormously (the
seqid and stateid can just be stored in the inode, without need for any
auxiliary data structures).  The catch, of course, is that the client
must assume responsibility for part of the locking logic, since only the
client knows which byterange locks are held by which processes (although
the server does know which locks are held by the client as a whole).  For
example, when a process locks a file, the client kernel would have check
whether another local process has set a conflicting lock, before sending
the LOCK request to the server (which will check whether another client
has set a conflicting lock).  So there is a certain amount of
implementation complexity associated with choice (2) as well.  However, I
think that the extra complexity of choice (1) is significantly higher, and
the overall tradeoff favors choice (2).  Furthermore, choice (2) has the
advantage of confining the extra complexity to the locking subsystem,
which makes it easier to audit for bugs.

In addition to implementation complexity, there are also performance
tradeoffs to be considered.  I imagine that choice (2) will slow things
down if many processes are participating in byte-range locking on the same
file, because of the extra serialization (since LOCK/LOCKU requests are
serialized on a per-lockowner basis).  On the other hand, choice (2) may
actually be faster if many processes are participating in ordinary file
I/O, since it avoids the overhead of all the extra
OPEN..OPEN_CONFIRM..CLOSE sequences (although you'd probably still want
some ACCESS calls).  Choice (2) is clearly faster if a process is doing
lots of fork()'s and sharing file descriptors, since the extra
OPEN..OPEN_CONFIRM..CLOSE sequences are eliminated completely.  So the
performance tradeoffs paint sort of an unclear picture, but I don't see a
strong performance-related argument against choice (2).

So my question is, does anyone see any reason that I haven't thought of
why choice (2) won't work?  I'm asking because it seems to be "abusing"
the protocol (or at least using it in a way that wasn't intended), so
I wanted to get some feedback before diving in and implementing (2).

Finally, I'd like to add a "philosophical" remark about the source of
the difficulty.  It seems to me that nfsv4's concept of a lockowner is
really overloaded with 2 meanings:
  a) it has the obvious meaning, the unit of lock contention
  b) it is the unit of open file ownership
In our Linux client, we can define our nfs_lockowner4's so that we get
a clean fit with either of these meanings, but not both (choice (1)
matches cleanly with (a); choice (2) matches cleanly (sort of) with (b)).
Our problems would be avoided if there was a way for us to seperate these,
e.g. by using a lockowner in a LOCK request which differed from the one
used to originally OPEN the file.  However, this is touching on delicate
issues of replay protection, so it's not clear to me whether this could
be done without an impratically large amount of change to the protocol.

- k

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 19 Jun 2001 11:38:43 -0500

> was actually first brought to my attention by Spencer, who is having

To give due credit, Eric Kustarz and Bill Ricker here at Sun initially
raised this issue.

> some ACCESS calls).  Choice (2) is clearly faster if a process is doing

ACCESS would be required so that the client has the "right" answer
before allowing an application to open() the file.

> I wanted to get some feedback before diving in and implementing (2).

Actually, choice (2) aligns more closely with the delegation
mechanisms.  As you point out, the client has to do this bookkeeping
for recovery purposes anyway.

> be done without an impratically large amount of change to the protocol.

Not sure how large of a change it would be but we could introduce
something like a LOCKOWNER_CONFIRM to be employed at the first use of
a lockowner.  OPEN would then be used, as you suggest, when the client
first instantiates the file and for the entire client.  The associated
stateid would represent the initial OPEN and any upgrade/downgrade
sequences along with the CLOSE.  It would also be used for the
delegation piece as well.  I haven't thought about it much but it may
not be to unreasonable a change to clarify/simplify the model.

If we choose your option (2), the specification will need to clarify
when CLOSE is to be used.  As you point out it needs to be done on
last application close().  The client should not defer use of CLOSE in
the hopes of caching the file reference/data.  A delegation would
signify to the client the appropriate behavior with respect to
caching.

Spencer