Subject: RE: [nfsv4] Parallel open support

From: Peter Varga <[email-masked]>
Date: Mon, 18 Sep 2006 23:29:45 -0400

I believe that I originally proposed the idea that all OPENs return unique
stateids on Aug 22 after you had suggested doing it for OPENs to different
file names in order to avoid the hard link problem.
 
As the instigator of the idea, I will also instigate the only downside I can
think of:
There is extra state on the server. Essentially, every file descriptor on
the client will have a corresponding stateid. This is fine for sane
applications, but you could have some app open up a file once per thread,
with a few dozen threads running. In this example, a stateid per file
descriptor will result in a few dozen stateids for each user that runs this
app. With 4.0, there would be 1 stateid per user.
 
Do we care about such applications? Is it worth it if it simplifies the
protocol? If it's of any relevance, CIFS works this way (unique descriptor
for each CREATE_ANDX request returns a unique identifier) and they're gotten
away with it since inception.
 
Peter

-----Original Message-----
From: Noveck, Dave [mailto:[email-masked]] 
Sent: Monday, September 18, 2006 6:17 PM
To: [email-masked]
Subject: [nfsv4] Parallel open support

Two ways of supporting this have been proposed:

Maintaining the current open upgrade logic and mandating incrementing of
successive seqids for successively return stateids for the same
file-handle/openowner pair.
 
Getting rid of the v4.0 upgrade logic. and therefore getting rid of
successive stateid that would have to be sorted using seqids.

The second seems to me to produce significantly simpler implementations in
dealing with parallel opens.
 
It has been claimed that POSIX somehow requires the v4.0 upgrade logic, but
I don't see this. The current implementations of open upgrade/downgrade do
not conform to the POSIX semantics of releasing all locks for a given file,
no matter what the file descriptor, whenever a close on any of those file
descriptors is done.  This is because when you do a close() you will either
locally drop a reference to file (and do nothing over the wire) or do an
OPEN_DOWNGRADE and so if all these locks are released, the client has to do
it itself. Given that, I don't see that we have any reason to maintain the
upgrade logic.
 
If someone does really need to implement this semantic (and so far nobody
has indicated a real need for this, i.e. there have been no people
complaining about a bug), I think the right thing to do would be to create
an UNLOCK_ALL op which releases all locks on a given file for a given
lockowner (no matter what the openowner), and let anyone who is interested
in having this  semantics issue it whenever appropriate, i.e. whenever the
client has multiple opens for the same file where there are or might be
unreleased byte-range locks. 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 20 Sep 2006 09:46:36 -0400

Note that even if you eliminated upgrade from the protocol, a client
would be free, if it did an open for the same openowner, mode, and
name-directory pair to simply increment a local reference count
(intead of an over-the-wire OPEN) and decrement the count on
close (instead of doing an over-the-wire CLOSE). 

If a client did this the additional storage would be limited to the
number of different modes that you opened with times the number of
links (usually one).

I think that clients would be motivated to implement such an
optimization, given that it would reduce latency and given that,
you probably would rarely be in operating conditions where a 
bit map was a net win.

A related question.  Whether we do this or not, we still have the
problem of client trying to do too many opens (with the same openowner
or different ones), but servers have to be able to impose a limit,
to deal with DOS attacks.  What is the right error to return for 
this case?

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Tuesday, September 19, 2006 4:48 PM
To: [email-masked]
Subject: RE: [nfsv4] Parallel open support

--- [email-masked] wrote:

> > To be clear, I'm talking about a stateid with say 96 bits that map 
> > map to the open_owner and lock_owner, and 32 bits to count up to 
> > 2^32 simultaneous open owners. This model then requires the server 
> > to share the following storage among all simultaneous OPENers of the

> > 96 bit value:
> > 
> > - the 96 bit value
> > - the count of the number of outstanding OPENers (increment on each
> >   OPEN, decrement on each CLOSE)
> > - the lowest outstanding uniquifier
> > - the highest outstanding uniquifier
> 
> And if a client uses a Stateid mid way between lowest and highest 
> uniquifier after it has Closed that Stateid, how does the server know 
> to return NFS4ERR_BADSTATID?

Good point, you're correct, the server doesn't know. In the new world of
exactly once semantics, I didn't imagine a client would do that. If it
does, then the server would need to keep the reference count as a signed
value, and if there is an attempt to lower the reference count to below
zero (or if it goes to zero and there is a period of idleness) the
server tosses the 96 bit value of the stateid, and the client has harmed
itself (which it will see when it uses stateid with the same upper 96
bits). I can live with that.
 
> You can add a bitmap for the range of lowest<->highest outstanding 
> uniquifier, but how do you set a bound on the size of that bitmap?
> (other than 2^32 bits long)

If you wanted to use a bit map (to save the client from
itself) instead of a 32 bit reference
count you'd use something with fewer bits. A 7 bit bit reference counts
uses 128 bits, equal to the memory for one stateid.
As you detected an open_owner opening the same file_handle twice, the
bit map could be allocated then. At two openers, you are using twice as
much memory as NFSv4.0, but at 3 or more you are amortizing costs. Once
the 128th simulateous OPEN is reached, you allocate a fresh stateid, and
fresh 96 bit value.

I'm sure there are other ways; I mainly wanted to be convinced economy
was possible if we got rid of upgrade/downgrade.