Subject: RE: [nfsv4] Parallel open support

From: [email-masked]
Date: Tue, 19 Sep 2006 11:55:23 -0400 (EDT)

[good stuff snipped]
> If one thread opens a file for read and a second thread opens a file for
> write with the same openowner and file (either the same directory and
> component or hard links that link to the same file), you don't have an 
> upgrade.  The first thread gets back a stateid for a file opened for 
> read and the second thread gets a stateid for a file open for write.
> Each
> of those distinct stateids can be closed independently.  Since you
> did these thing in parallel, you don't care what order they happened 
> in and you can go about your business without knowing.

Can't a client do that now, just by using different openowner names?
(ie 1 per thread or whatever) I think all this does is replace increased
storage usage for more openowners with increased storage usage for more
Open stateids?

> This gets back to Ric's question about why v4.0 has the upgrade logic
> since it add complexity, expecially with parallel opens.  So far I 
> haven't found a good reason so I'm wondering if we can get rid of it.   

If this is referring to something I said, I disavow any knowledge of it:-)

[Trond wrote]
> One consequence would be that the server could no longer check if the
> client has cleared all the byte range locks prior to closing the file.
> 
> You still have to be able to support POSIX constructs such as
> 
>  (on the server)
>  echo "foo" > /mnt/foo
>  ln /mnt/foo /mnt/bar
> 
> 	(process on the client)
> 	fd1 = open("/mnt/foo", O_RDWR);
> 	fcntl(fd1, F_SETLKW, { F_WRLCK, SEEK_SET, 0, 0 });
> 
> 	fd2 = open("/mnt/bar", O_RDONLY);
> 	fcntl(fd2, F_SETLKW, { F_RDLCK, SEEK_SET, 0, 0});
> 
> without causing a deadlock, so lock_owner state now needs to be
> upgradeable _independently_ of the open_owner that was used to open the
> file.
> 
> However since the locks were created using file stateids that now have
> different open_owners, then the server has no way of knowing that
> close(fd1) requires it to check for locks that were created using fd2
> and vice versa, so it can no longer help the client if the two get out
> of sync.

I admit that I don't feel strongly that the client needs help in this
regard. In other words, personally, I wouldn't be upset with the requirement
for Close to check for associated locks, being dropped. However, since that
is currently in RFC3530, can dropping that requirement be considered a
"minor revision"?

[Mike wrote]
> Let's say that for a given open_owner/file handle pair,
> the server assumes the open_owner won't have more than 2^32 - 1
> simultaneous opens. Then the server reserves 32 bits of the stateid
> to uniquify each simultaneous open (there just happens to be
> an explicit 32 bit field in the stateid for this). The remaining bits
> are the same for each simultaneous open of the same
> open_owner/file_handle pair. The server bumps the uniqifier for
> each simultaneous open.

Yep, but doesn't the server have to still maintain some sort of list of
these (admittedly maybe only a bitmap) to track which ones have been already
closed? And then I'm not sure I know how to compare this with the increased
storage required if a client uses more openowners?

For example, suppose the extreme case of a client that uses a unique
open_owner for every Open. Those open_owners only need to persist on
the server until the Close, which often is a fraction of a second.
As such, although the client would be using lots'o'open_owners, the
server only needs to store the ones that currently have Opens. (Unlike 4.0,
for 4.1 no OpenConfirm is required and there is no benefit keeping an
open_owner once all the Opens on it are Closed.)

rick

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 19 Sep 2006 12:09:59 -0400

> storage usage for more openowners with increased storage usage for more

No! That screws up byte range locking semantics. Byte range lock state
is currently a triplet: <filehandle,open_owner,lock_owner> because of
the open_to_lockowner stuff. If you have two different open_owners, then
their locks will always conflict, even if the lock_owner is the same.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 19 Sep 2006 09:36:32 -0700 (PDT)

--- [email-masked] wrote:

> closed? And then I'm not sure I know how to compare this with the

To be clear, I'm talking about a stateid with say 96 bits
that map map to the open_owner and lock_owner, and 32 bits to count
up to 2^32 simultaneous open owners. This model then requires
the server to share the following storage among all simultaneous
OPENers of the 96 bit value:

- the 96 bit value
- the count of the number of outstanding OPENers (increment on each
  OPEN, decrement on each CLOSE)
- the lowest outstanding uniquifier
- the highest outstanding uniquifier

96 + 3 * 32 = 192 bits, not including the full open_owner
and the file handle. 

If the range of outstanding uniquifiers ever equals 0 to 2^32 - 1,
then the next OPEN of the open_owner, file_handle pair requires a
a new 96 bit value in the stateid. So the next OPEN cannot share
the memory the previous set of simultaneous OPENers. 

Alternatively you could have a bitmap, but that has potentially
higher memory requirements, and of course depends on the ratio of 
of the count of current OPENs to the difference between the highest
and lowest outstanding uniquifiers.

> increased
> server only needs to store the ones that currently have Opens.

If the client has a unique open_owner for every OPEN then the
parallel OPEN issue Dave is trying to solve doesn't apply.

> (Unlike 4.0,

---

From: Mike Eisler <[email-masked]>
Date: Tue, 19 Sep 2006 16:44:54 -0700 (PDT)

> their locks will always conflict, even if the lock_owner is the same.

Over NFSv3 and local files I wrote program that
opens a file in a parent process, forks, then in the child:
- opens the file again
- gets an exclusive byte  range lock on both file descriptors. 

The inherited file descriptor from parent to child process is one open
owner (per RFC3530), and the file descriptor opened in the child is a
second open owner distinct from the first open owner (despite the
fact client implementations don't produce separate open owners
for each process that opens a file, that is nonetheless
a legal thing for clients to do). The
lock owner in the child for each lock attempt is the same.

This succeeds on Linux and Solaris, NFSv3 and v4 and local files, as
expected, and in contrast to: "If you have two different 
open_owners, then their locks will always conflict, even if the
lock_owner is the same."

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 19 Sep 2006 20:05:41 -0400

> open_owners, then their locks will always conflict, even if the

In the V3 case we have known ordering issues w.r.t. LOCK/UNLOCK. NFSv4
was supposed to fix those.

In the NFSv4 case, your testcase as described above will not result in
the creation of 2 different open_owners on either Linux or Solaris
afaik. It will result in the creation of 2 different file descriptors,
that will both refer to the same open_owner at the filesystem level.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 19 Sep 2006 18:16:41 -0700 (PDT)

--- Trond Myklebust <[email-masked]> wrote:

> afaik. It will result in the creation of 2 different file

Yes, but if did result in different open_owners it, as long as the
lock_owner is the same, the locks on the file cannot conflict. It is
permitted by RFC3530 for the client to produce different open_owners,

      The state model has added an open_owner4 identifier.  This was
      done to accommodate Posix based clients and the model they use
for
      file locking.  For Posix clients, an open_owner4 would correspond
      to a file descriptor potentially shared amongst a set of
processes
      and the lock_owner4 identifier would correspond to a process that
      is locking a file.

So fd1 shared between a parent and child process must have the same
open_owner, and fd2 known only to the child process may have
a distinct open_owner.

> descriptors,

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 19 Sep 2006 23:07:14 -0400

> open_owner, and fd2 known only to the child process may have

Stupid, 'cos it is completely impractical. As I have repeatedly said, if
you have more than 1 sequence stream, then the LOCK/LOCKU/CLOSE are not
ordered, and you are allowing the client to screw itself.