Subject: new state model

From: Jim Rees <[email-masked]>
Date: Fri, 19 Oct 2001 14:40:40 -0400

I have been trying to understand the new state model and feel that I am
approaching enlightenment.  But I still need some help.

For the OpenBSD implementation I am assuming that an open_owner corresponds
to exactly one call to open().  I am also assuming that a lock_owner
corresponds to all file descriptors open on a particular file in a
particular process.  Note that because each open uses a new open_owner, each
open_owner has just one open_stateid.

A consequence of this is that there is a many-to-many correspondence between
open_owners and lock_owners.  I believe Posix locking rules require this.

Consider the following example.  P1 and P2 are related processes, fd1, fd2,
and fd3 are file descriptors, and there is only one file.

P1 calls open() and gets fd1, associated with open_owner_1 and lock_owner_1.

P2 is related to P1, and inherits fd2 from P1's fd1.  fd2 is associated with
open_owner_1 and lock_owner_2.

P2 now calls open() again and gets fd3.  fd3 is associated with open_owner_2
and lock_owner_2.

Note that the nfsv4 file system code (vnode op) operating on P2's behalf is
now unable to distinguish between calls made on fd2 and calls made on fd3.
This is a consequence of not having access to the file table, and is the
same reason Solaris had trouble implementing the old state model.  Since fd2
and fd3 share a lock_owner, the only time the distinction is important is
when an open_owner is required.

So now I'm trying to figure out whether this is important.  I think the only
time an open_stateid is used is in lock and close.

For lock, the case I would worry about is where one open_owner has read-only
access, the other has write access, and the client wants to acquire a write
lock.  For close, I would worry about closing the one with the write access,
leaving the client unable to acquire a write lock using the other
open_stateid.

I think it's a mistake that BSD file systems generally have no access to the
file table.  They have a reference count, and know which process they are
acting on behalf of, but have no way to associate a particular open with a
particular close.  Linux doesn't have this problem.

I was also confused by Spencer's remark that "...the client may generate two
OPEN requests to the server that use the same open_owner.  This would be the
case where hardlinks are present..."  I would appreciate an example.

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 19 Oct 2001 16:26:16 -0500

> approaching enlightenment.  But I still need some help.

Thanks to you and others for putting up with this ill-defined state
model and the attempts to interpret what I started.

> open_owner has just one open_stateid.

Well, when we were discussing this and other ideas in July, what I
kept telling myself was that the protocol's state model had to be
flexible enough to take into account various uses at the client side.
This not only refers to the differences between *nix and Windows but
even within the *nix implementations as you have pointed out later
with reference to Linux.

As long as the client presents the correct usage of open_owner and
lock_owner (and the various stateids) over the wire, the client should
be free to employ the usage as it sees fit (or is limited to).

For some *nix implemtations, the open_owner will correspond to an
open() system call.  As you point out, this requires a method for the
client to tie the file table entry to the the open_owner.  At the
moment, Solaris is unable to do this.

For some *nix implementations that don't provide a method of
associating file table entry to open_owner (i.e. Solaris), the client
can use open_owner to represent all open instances for that file based
on the credential used for the open().  For Solaris, this works out
okay because the credential used for the initial open() is passed as
part of all file system API calls.  This means that a process that
open()s a file and then amplifies its credentials in some way and
open()s the file again will have two open_owners associated with it.

This allows a child sharing a file descriptor the ability to "find" an
appropriate open_owner to use for its operations.  This is possible
based on using the credential as the key.  It is appropriate to share
the open_owner because the server doesn't "care" how it is used as
long as it is used within the correct "user" or security context.

This usage requires the client NFS implementation to track open and
close correctly to do the right thing with over-the-wire OPENs and
CLOSEs.  Solaris provides for this as well by providing the reference
count of the file table entry on the close vnode operation.

(note: this approach run counter to a mail message I sent yesterday
stating that a lock_owner could be associated with only a single
open_owner).

> open_owners and lock_owners.  I believe Posix locking rules require this.

Right.  In my running example of Solaris, a single process can have
two open_owners and will need a single lock_owner since it needs to
coalesce the locking state into single stream of state.

> when an open_owner is required.

Note that in my suggestion of usage, the Solaris 

> time an open_stateid is used is in lock and close.

Right.

> leaving the client unable to acquire a write lock using the other

In my usage for Solaris, this would not be a problem since the
open_owner represents all open()s and would therefore upgrade access
as the various open()s were done.  The issue of locking the file for
read/write with respect to how it was open()ed is actually tracked at
a higher level in Solaris and therefore the NFS client doesn't need to
worry about validating the appropriate access.

> particular close.  Linux doesn't have this problem.

Does BSD pass credentials similarly to Solaris?

I should mention that the Solaris approach I describe above is a short
term solution given that we are not modifying the vnode/filesystem
APIs at the moment.  We eventually would like to do something as you
suggest where the NFS client can store a cookie in the file table that
is then passed on subsequent calls (much like the credential) and this
can be used to track usage and better manage the open_owner state more
closely to the process usage.

> case where hardlinks are present..."  I would appreciate an example.

A multi-threaded application does two open()s on different paths and
those paths are hardlinks.  The client will use the same open_owner
(presumably since it is the same process and uses the same
credentials).  The server will process both OPENs for the same file as
if the client were OPENing and then OPENing-for-upgrade.  The client
will need to determine which stateid is appropriate for use and
inspect the stateid->seqid returned from the server to determine which
is the most recent.

-- 

- Spencer -