Subject: [nfsv4] POSIX exclusive open

From: Rick Macklem <[email-masked]>
Date: Wed, 4 Mar 2009 16:02:12 -0500 (EST)

Hmm, Nico brought up another point I'll admit I'm not completely clear
on, and that is whether the EXCLUSIVE4 option on Open is sufficient to
implement a POSIX compliant exclusive open?

I had thought that the problem was that, since EXCLUSIVE4 doesn't take
any attributes, if another client does a Setattr on the file between
the Open4 with EXCLUSIVE4 and the subsequent Setattr (that needs to be
done as part of the POSIX open to set owner/group/mode bits), then this
subsequent Setattr could fail, leaving the client in an awkward spot.
(ie: It had exclusively created the file, but couldn't set the 
owner/group/mode, so the POSIX open has to fail.)

Anyone else have thoughts w.r.t. whether or not the NFSv4 Open will
always achieve POSIX O_EXCL open semantics?

rick
ps: I should have used the O_APPEND example and avoided the above, but
     I'd be interesyed in hearing what the consensus on the above is:-)

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 4 Mar 2009 15:13:46 -0600

> owner/group/mode, so the POSIX open has to fail.)

NFSv4.1 definitely fixes this:

   /// struct creatverfattr {
   ///          verifier4      cva_verf;
   ///          fattr4         cva_attrs;
   /// };
   ///
   /// union createhow4 switch (createmode4 mode) {
   ///  case UNCHECKED4:
   ///  case GUARDED4:
   ///          fattr4         createattrs;
   ///  case EXCLUSIVE4:
   ///          verifier4      createverf;
   ///  case EXCLUSIVE4_1:
   ///          creatverfattr  ch_createboth;
   /// };

As for v4.0, I think it's odd for the client to cause a file to be
created yet return EEXIST, but better than to create the file with
different attributes than requested.  On the other hand, I don't think
this will happen frequently at all, and if it does it will be another
process running as the same user (or you have inherittable ACL entries
that grant other users the right to setattr) that will be racing to
SETATTR.  I suppose there might be an attack in the case where you have
inherittable ACL entries that grant other users the right to setattr,
but, that strikes me as also not likely to happen often -- as long as
there's no way (haven't thought enough about how) for the attacker to
misdirect a client app/user to try an exclusive create in such a
directory then there's no real issue.

> always achieve POSIX O_EXCL open semantics?

For v4, see above.

>     I'd be interesyed in hearing what the consensus on the above is:-)

:)

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 5 Mar 2009 11:28:46 -0500

Rick Macklem wrote:
> Anyone else have thoughts w.r.t. whether or not the NFSv4 
> Open will always achieve POSIX O_EXCL open semantics?

I think the answer to that precise question is "No", although Nico gives
some good reasons not to expect that to be a large problem in practice.

Let me give you another one.  A server could very well look past the
OPEN and, if there is a SETATTR following, do the two operations
essentially as one, avoiding the possiblity of a troublesome SETATTR in
the middle.  While this kind of thing is kludgy and you would not like
to have your server littered with this sort of special case, your server
can still do it and be compliant with RFC3530 as far as I can see.
There could not be a pyNFS test that checked for the race condition and
considered you non-compliant if you didn't have it.

So I think a compliant server could have the race but a server that
didn't have the race would be compliant as well.  Up to you to choose
which you implement.

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Wednesday, March 04, 2009 4:14 PM
To: Rick Macklem
Cc: [email-masked]
Subject: Re: [nfsv4] POSIX exclusive open

For v4, see above.

>     I'd be interesyed in hearing what the consensus on the above is:-)

:)
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 5 Mar 2009 10:41:08 -0600

> didn't have the race would be compliant as well.  Up to you to choose

Though, of course, it'd be the client that gets hurt if there is a race.
But as I said, it's not that big a deal, I think, and v4.1 fixes it.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 5 Mar 2009 11:54:34 -0500

Well yes, but a "don't-bother-if-only-the-client-gets-screwed"
philosophy might result in a server that very few people would want to
run, and I'd prefer not to deal with the consequences of that :-)

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 5 Mar 2009 10:56:39 -0600

> run, and I'd prefer not to deal with the consequences of that :-)

Only because you're aware of the issue.  (I.e., it ought to be
documented.)

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 5 Mar 2009 09:17:45 -0800 (PST)

> always achieve POSIX O_EXCL open semantics?

Yes. NFSv4.1 will achieve complete POSIX O_EXCL semantics.

If the session has a persistent reply cache, then the client
used GUARDED to achieve O_EXCL. GUARDED allows attributes to be specified
in the same OPEN operation.

If the session does not have a persistent reply cache, then the client
uses EXCLUSIVE4_1 to achieve O_EXCL. EXCLUSIVE4_1 allows attributes
to be specified in the same OPEN operation, however the
"allowable attributes
   for EXCLUSIVE4_1 are indicated in the suppattr_exclcreat
   (Section 5.8.1.14) attribute."

EXCLUSIVE4 is deprecated. NFSv4.1 supports it grudingly.
I expect if there is an NFSv4.2, EXCLUSIVE4 will be history.

>      I'd be interesyed in hearing what the consensus on the above is:-)

Yes, indeed, start the annual O_APPEND thread. :-)

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 5 Mar 2009 11:31:37 -0600

> Yes, indeed, start the annual O_APPEND thread. :-)

No need.

The current status quo w.r.t. O_APPEND is that clients that really care
to can do it, and that's probably good enough, even if it's really
sub-optimal in the case of multiple clients appending to the same file.

Clients can implement O_APPEND trivially when the have a delegation, and
by locking the entire file around any append write when they don't.  The
latter means that multiple clients racing to append will get lousy
performance.  There are better ways to handle append in that case, but
not if you're striping files in a pNFS case (or, at least, I've not yet
figured out an efficient way to handle racing appends when striping is
involved).

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 05 Mar 2009 12:52:43 -0500

> figured out an efficient way to handle racing appends when striping is

The above assumes mandatory byte range locking, which isn't implemented
on most servers.

Trond