Subject: RE: [nfsv4] Parallel open support

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 18 Sep 2006 18:41:52 -0400

> So, I must be very confused.  Does the second option you present
remove 
> the ability to upgrade an OPEN (read -> read/write)? 

Yes.

If one thread opens a file for read and a second thread opens a file for
write with the same openowner and file (either the same directory and
component or hard links that link to the same file), you don't have an 
upgrade.  The first thread gets back a stateid for a file opened for 
read and the second thread gets a stateid for a file open for write.
Each
of those distinct stateids can be closed independently.  Since you
did these thing in parallel, you don't care what order they happened 
in and you can go about your business without knowing.

This gets back to Ric's question about why v4.0 has the upgrade logic
since it add complexity, expecially with parallel opens.  So far I 
haven't found a good reason so I'm wondering if we can get rid of it.   

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Monday, September 18, 2006 6:33 PM
To: [email-masked]
Subject: Re: [nfsv4] Parallel open support

On Mon, Noveck, Dave wrote:
> Two ways of supporting this have been proposed:
> 
> 	Maintaining the current open upgrade logic and mandating
incrementing 
> of successive seqids for successively return stateids for the same 
> file-handle/openowner pair.
> 	 
> 	Getting rid of the v4.0 upgrade logic. and therefore getting rid
of 
> successive stateid that would have to be sorted using seqids.
> 
> The second seems to me to produce significantly simpler 
> implementations in dealing with parallel opens.
>  
> It has been claimed that POSIX somehow requires the v4.0 upgrade 
> logic, but I don't see this. The current implementations of open 
> upgrade/downgrade do not conform to the POSIX semantics of releasing 
> all locks for a given file, no matter what the file descriptor, 
> whenever a close on any of those file descriptors is done.  This is 
> because when you do a close() you will either locally drop a reference

> to file (and do nothing over the wire) or do an OPEN_DOWNGRADE and so 
> if all these locks are released, the client has to do it itself. Given

> that, I don't see that we have any reason to maintain the upgrade
logic.
>  
> If someone does really need to implement this semantic (and so far 
> nobody has indicated a real need for this, i.e. there have been no 
> people complaining about a bug), I think the right thing to do would 
> be to create an UNLOCK_ALL op which releases all locks on a given file

> for a given lockowner (no matter what the openowner), and let anyone 
> who is interested in having this  semantics issue it whenever
appropriate, i.e.
> whenever the client has multiple opens for the same file where there 
> are or might be unreleased byte-range locks.

So, I must be very confused.  Does the second option you present remove
the ability to upgrade an OPEN (read -> read/write)?  Or does it say
that an upgrade can occur but it doesn't really matter in what order it
is done since if there is a deny requested that the deny will sort
itself out with respect to ordering?

Spencer

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 19 Sep 2006 09:04:00 -0400

> haven't found a good reason so I'm wondering if we can get rid of it.   

One consequence would be that the server could no longer check if the
client has cleared all the byte range locks prior to closing the file.

You still have to be able to support POSIX constructs such as

 (on the server)
 echo "foo" > /mnt/foo
 ln /mnt/foo /mnt/bar

	(process on the client)
	fd1 = open("/mnt/foo", O_RDWR);
	fcntl(fd1, F_SETLKW, { F_WRLCK, SEEK_SET, 0, 0 });

	fd2 = open("/mnt/bar", O_RDONLY);
	fcntl(fd2, F_SETLKW, { F_RDLCK, SEEK_SET, 0, 0});

without causing a deadlock, so lock_owner state now needs to be
upgradeable _independently_ of the open_owner that was used to open the
file.

However since the locks were created using file stateids that now have
different open_owners, then the server has no way of knowing that
close(fd1) requires it to check for locks that were created using fd2
and vice versa, so it can no longer help the client if the two get out
of sync.

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Tue, 19 Sep 2006 08:03:12 -0700 (PDT)

> in and you can go about your business without knowing.

"can be independently" -> "must be indepedently" ?

If so, then stateids from OPEN are effectively file descriptors,
and there are some interesting dividends:

- READ and WRITE (as well as CLOSE, LOCK, and LOCKU) don't need to take
  the current file handle as an argument (except when we use those
  special stateids ... can we kill those please?). The stateid is
  sufficient to reference the file. We still want file handles, because
  they can be used to tell if a file is cached or not.

- It is not the end of the world if the client doesn't follow OPEN
  with GETFH.

However, dividends aside, I do wonder about the additional
space required on the server, and to a lesser extent, on the client
for maintaining independent stateids. But perhaps this isn't a big
deal for the server that is interested in economy.

Let's say that for a given open_owner/file handle pair,
the server assumes the open_owner won't have more than 2^32 - 1
simultaneous opens. Then the server reserves 32 bits of the stateid
to uniquify each simultaneous open (there just happens to be
an explicit 32 bit field in the stateid for this). The remaining bits
are the same for each simultaneous open of the same
open_owner/file_handle pair. The server bumps the uniqifier for
each simultaneous open.

This is very close to Spencer's original proposal, except
that the client doesn't have to worry about checking the sequence
number field, the server has the option of doing this, and 
each OPEN does have to have a corresponding CLOSE.

We might want to add a variant of CLOSE that says for this
open_owner (as identified by a stateid), close all OPENs it
has for the filer.

Regarding:

> I think the best path would be to get rid
> of the upgrade path and give us an operation that said
> "free all locks for this fh associated with this openowner"

LOCKU should be able to that now; just pass an offset of zero and
length of zero. My read of the code Eric pointed us to was that
the OpenSolaris client is assuming that works.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 19 Sep 2006 11:11:50 -0400

> the OpenSolaris client is assuming that works.

That will free all locks associated with this lock_owner. There will be
no association between locks and open_owners.