Subject: Re: [nfsv4] Asking for details about OPEN_DOWNGRADE

From: Mike Mackovitch <[email-masked]>
Date: Fri, 13 Jun 2008 10:33:15 -0700

On Fri, Jun 13, 2008 at 10:03:29AM -0700, eric kustarz wrote:
>
> On Jun 13, 2008, at 9:49 AM, Mike Mackovitch wrote:
>
>> On Fri, Jun 13, 2008 at 08:01:03AM -0400, Trond Myklebust wrote:
>>> On Fri, 2008-06-13 at 11:23 +0200, [email-masked] wrote:
>>>>     The
>>>>    share_access and share_deny bits specified must be exactly equal to
>>>>    the union of the share_access and share_deny bits specified for some
>>>>    subset of the OPENs in effect for current openowner on the current
>>>>    file.
>>>
>>> It basically means that in order to be allowed to downgrade to
>>> OPEN4_SHARE_ACCESS_READ|OPEN4_SHARE_DENY_NONE, then at least one of  
>>> the
>>> OPENs that made up the previous stateid must have been of that type.
>>
>> What's strange is that it appears that only Linux seems to think
>> this is the case.  I can't figure out (1) why they think it must
>> work this restrictively (my best guess is that the phrase "OPENs in
>> effect" is being misinterpreted as "OPENs sent to the server"), and
>> (2) why they haven't bothered/managed to convince the rest of the
>> NFSv4 community of this (and get the spec updated).
>>
>>> IOW: you cannot just do
>>>
>>>        OPEN(OPEN4_SHARE_ACCESS_BOTH|OPEN4_SHARE_DENY_NONE)
>>>        OPEN_DOWNGRADE(OPEN4_SHARE_ACCESS_READ|OPEN4_SHARE_DENY_NONE) = NFS4ERR_INVAL
>>
>> This works fine on Solaris server - and the Solaris client
>> even behaves exactly this way.
>
> Yeah, we've got some bugs to fix in this area:
> NFSv4 client can send a bogus OPEN_DOWNGRADE
> http://bugs.opensolaris.org/view_bug.do?bug_id=4877567
>
> NFSv4 server should deny bogus OPEN_DOWNGRADE
> http://bugs.opensolaris.org/view_bug.do?bug_id=4877573

Hmmm...if that's really the agreed upon behavior, I must say it's quite
a shame that nobody bothered to fix the spec - not even in NFSv4.1.

Also, the fact that the Solaris bugs have been around for five years
makes it seem like they'll never be fixed.

I feel compelled to ask the folks on the IETF list what the right
behavior should be.  Any comments from [email-masked] about what
the right behavior is supposed to be here?

THanks
--macko
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 13 Jun 2008 14:52:51 -0400

> a shame that nobody bothered to fix the spec - not even in NFSv4.1.

How should it be 'fixed' in your opinion?

The spec reflects the fact that all userland open() requests are
supposed to be checked by the server, and your client should not be
requesting an access mode that does not correspond to an
application-specified open() mode.
Furthermore, when the application calls close(), you want the resulting
state on the server to be the union of the remaining application open()
modes on your client.

There is no POSIX or Windows sanctioned syscalls corresponding to an
operation "change this read-write file descriptor and make it
write-only", and so the spec doesn't provide support for one.

   Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 13 Jun 2008 15:20:19 -0400

> > http://bugs.opensolaris.org/view_bug.do?bug_id=4877567

Where are the comments, by the way?  (How hard is it to make the client
do this?)

> a shame that nobody bothered to fix the spec - not even in NFSv4.1.

NFSv4.1 actually expands on this a bit:

   After checking for valid values of share_access and share_deny, the
   server replaces the current access and deny modes on the file with
   share_access and share_deny subject to the following constraints:

	o  The bits in share_access SHOULD equal the union of the
	share_access bits (not including OPEN4_SHARE_WANT_* bits)
	specified for some subset of the OPENs in effect for the current
	open-owner on the current file.

	o  The bits in share_deny SHOULD equal the union of the
	share_deny bits specified for some subset of the OPENs in effect
	for the current open-owner on the current file.

   If the above constraints are not respected, the server SHOULD return
   the error NFS4ERR_INVAL.
  
> makes it seem like they'll never be fixed.

But, yeah, it's unfortunate if a server has to ignore the final SHOULD
above in order to interoperating with any existing Solaris client.

--b.

> NFSv4 mailing list
> [email-masked]
> http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 13 Jun 2008 14:38:16 -0500

> above in order to interoperating with any existing Solaris client.

Go ahead and have the server enforce the correct behavior and
maybe the Solaris client will eventually be fixed.

The scenario that produces the behavior is a little off the beaten
path so I doubt it will arise that much but putting pressure
on the implementations to correct themselves is a good thing.

Spencer

---

From: eric kustarz <[email-masked]>
Date: Fri, 13 Jun 2008 12:52:40 -0700

> Where are the comments, by the way?  (How hard is it to make the  

Yeah sorry about that, i put the comments in the public viewable  
section - should take a day or so to show up there.

I don't think it will be hard to make the client do this, just wasn't  
motivation.

eric

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 13 Jun 2008 16:10:01 -0400

> maybe the Solaris client will eventually be fixed.

But the server vendor's customers may be rather annoyed in the interim.

> on the implementations to correct themselves is a good thing.

With v4.1, the pressure can be put on the client implementations from
the beginning.  If the servers all check this, then v4.1 clients will
have to correct themselves as part of implementing v4.1, which would be
a good thing.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, June 13, 2008 3:38 PM
To: J. Bruce Fields
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Asking for details about OPEN_DOWNGRADE

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 13 Jun 2008 15:18:41 -0500

> But the server vendor's customers may be rather annoyed in the  

Sure but without a little pressure it won't get fixed.  We can fix
it in the development release but the changes will likely not be pushed
to previous releases without the pressure....

> have to correct themselves as part of implementing v4.1, which  

I agree -- good intentions and hope for the best...

Spencer

---

From: Mike Mackovitch <[email-masked]>
Date: Fri, 13 Jun 2008 13:29:55 -0700

> How should it be 'fixed' in your opinion?

By explicitly specifying that the behavior described above should not
be allowed.  It is clear that the spec is NOT clear about that....
otherwise, the spec wouldn't be getting "misinterpreted" by many people.

The text in the OPEN and OPEN_DOWNGRADE descriptions, as well as "Open
Upgrade and Downgrade" sections, leads me (and others) to believe that
the server only keeps one "OR"ed-together status for that owner and
thus sending multiple, redundant OPEN requests (for that owner) was
unnecessary.

(Well, that and a VFS implementation that doesn't pass a
directory/filename pair to its OPEN call. [grrr])

--macko

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 13 Jun 2008 16:34:49 -0400

> thus sending multiple, redundant OPEN requests (for that owner) was

The text of rfc 3530 ("exactly equal to the union of...") doesn't seem
ambiguous to me.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 13 Jun 2008 16:43:34 -0400

> to previous releases without the pressure....

The linux server has been enforcing this for a long time, for what it's
worth.  I vaguely recall this catching bugs in a few clients at testing
events before (don't remember which clients), but assumed they were
fixed....

Should the linux server turn this checking *off* for 4.0?  There's
really no other reason for it than "maybe we'll need this restriction
some day, so let's try to make sure we complain to clients violating
it."

--b.

---

From: Mike Mackovitch <[email-masked]>
Date: Fri, 13 Jun 2008 13:50:34 -0700

Well, the whole text there is:

> The share_access and share_deny bits specified in this operation
> replace the current ones for the specified open file.  The

And, to me at least, the qualifier "subset of the OPENs in effect"
doesn't clearly say "the subset of the OPEN ops sent to the server".

So, if an implementation didn't send an OPEN op to the server because
that OPEN op wasn't going to change/upgrade the open state for that
owner, then the server doesn't know about that open even though it is
"in effect".

--macko

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 13 Jun 2008 15:53:13 -0500

> Should the linux server turn this checking *off* for 4.0?  There's

No. It should remain as is.

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 13 Jun 2008 16:58:17 -0400

> thus sending multiple, redundant OPEN requests (for that owner) was

I don't see how it can be interpreted that way. The texts in section
8.11 and in 14.2.19 both take care to talk about a _union_ of the OPEN
share and deny modes and proper subsets of that union.

The pynfs test suite will even check for this condition, so there is not
much excuse for claiming ignorance.

Trond

---

From: Mike Mackovitch <[email-masked]>
Date: Fri, 13 Jun 2008 14:43:19 -0700

On Fri, Jun 13, 2008 at 04:58:17PM -0400, Trond Myklebust wrote:
> I don't see how it can be interpreted that way.

Perhaps that's because you now have a different perspective?  :-)

Apparently the Linux NFS client also interpreted it incorrectly up 
until a couple years ago (2.6.15).

> The texts in section

See my other responses regarding "in effect" vs. "particular ops sent
to the server".

> much excuse for claiming ignorance.

Uhh, OK then ... I guess I'll just have to claim stupidity?
After all, everyone knows you should never trust the Solaris NFS
implementation to do things correctly.

Silly me.
--macko

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 13 Jun 2008 17:47:18 -0400

> doesn't clearly say "the subset of the OPEN ops sent to the server".

That's "OPEN", not "open", and can only refer to the on-the-wire op, not
to whatever notion the client vfs has of an open file descriptor (which
isn't really the protocol spec's business anyway).

Do you find the 4.1 language clearer?

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 13 Jun 2008 17:55:08 -0400

> See my other responses regarding "in effect" vs. "particular ops sent

I agree with Bruce's reply: if the text had said open, then there might
be ambiguity, but it says OPEN, clearly referring to the op.

> implementation to do things correctly.

That may have been the case back in the case of v2 and possibly v3, when
the "spec" was basically a description of the Solaris implementation,
but that was never the case for NFSv4. Neither is it the case for
NFSv4.1.

Trond

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 13 Jun 2008 17:03:19 -0500

> but that was never the case for NFSv4. Neither is it the case for

Maybe a better way to phrase this is that no implementation should
be assumed correct until there is a consensus of interpretation within
the working group.

Spencer

---

From: Mike Mackovitch <[email-masked]>
Date: Fri, 13 Jun 2008 15:11:35 -0700

> That's "OPEN", not "open", and can only refer to the on-the-wire op, not

The OPEN description also uses "lockowner" where it should be "open owner".
So, clearly, details like this can't always be taken verbatim.
And with the other text talking about upgrades and how the state on
the server is an "OR"ed-together status, I (obviously incorrectly)
concluded that it was not referring to distinct OPEN ops.

> Do you find the 4.1 language clearer?

Not really... but it's really too late to affect it.

My hope is that this information will at least be in the NFSv4 archives
for other implementers to find.... which is why I brought it up on the
list.

Note, that I am not arguing that it behave the way I (and others)
interpreted it.  I'm merely trying to describe the confusion and
get the consensus regarding the correct operation recorded somewhere.

Thanks
--macko

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 13 Jun 2008 18:13:06 -0400

> Not really... but it's really too late to affect it.

It may be too late to actually change the requirement, if that's your
goal, but if it needs a little wordsmithing to make it clearer, I was
under the impression that was still possible?

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 13 Jun 2008 18:29:26 -0400

> be assumed correct until there is a consensus of interpretation within

There's a reason why I will rarely if ever volunteer to edit
anything. :-) ...but yes, that was the essence of what I meant to say.

My other point was that over the years, Fred has put a lot of effort
into distilling various consensus results from the the IETF discussions
into the pynfs test suite. If your server is failing those tests, then
at this point you are extremely likely to be facing an uphill battle if
you seek a consensus for your interpretation of the rules.

Trond

---

From: Steven Huntington <[email-masked]>
Date: Mon, 16 Jun 2008 16:50:08 -0700

Question to the gallery:

Does all this imply that for open reclaim, the client needs to send each
individual OPEN reclaim OP as opposed to a single composite OPEN for the
openowner/FH pair, so that the new instance of the server can know what
unions are valid and what aren't?  In short, does this apply to reclaimed
open share/deny flags as well as non-reclaimed ("normal") ones?

Cheers,

Steve Huntington, IBM San Jose

Phone:       (408) 463-2716 (T/L 543-2716)
Fax:         (408) 463-4727
Office:      SVL 90-D215
Internet:    [email-masked]
Lotus Notes: Steven Huntington/San Jose/IBM@IBMUS

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 16 Jun 2008 20:28:41 -0400

On Mon, 2008-06-16 at 16:50 -0700, Steven Huntington wrote:
> Question to the gallery:
> 
> Does all this imply that for open reclaim, the client needs to send
> each individual OPEN reclaim OP as opposed to a single composite OPEN
> for the openowner/FH pair, so that the new instance of the server can
> know what unions are valid and what aren't? In short, does this apply
> to reclaimed open share/deny flags as well as non-reclaimed ("normal")
> ones?
> 
> Cheers,
> 

Yes, but assuming that you have applications that are all opening the
same file with a mixture of all three open modes (I'm also assuming you
don't care about the POSIX-incompatible DENY modes), you should be able
to get away with just sending a maximum of 3 OPEN calls per file
+open_owner combination.
i.e.
  - OPEN(OPEN4_SHARE_ACCESS_READ, OPEN4_SHARE_DENY_NONE)
  - OPEN(OPEN4_SHARE_ACCESS_WRITE, OPEN4_SHARE_DENY_NONE)
  - OPEN(OPEN4_SHARE_ACCESS_BOTH, OPEN4_SHARE_DENY_NONE)

That will allow you to downgrade to a READ-only mode once all the O_RDWR
and O_WRONLY descriptors have been closed, or to a WRITE-only mode once
the O_RDWR and O_RDONLY descriptors have been closed.

Cheers
  Trond

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Wed, 25 Jun 2008 11:49:01 -0400 (EDT)

> the O_RDWR and O_RDONLY descriptors have been closed.
Flame suit on:-)... Personally, I can't see much point in a posix client
(one always using deny_none) in doing open_downgrades at all. As the 
recent discussion has shown, it's a pain to get right and what benefit
does it have?

- Since all opens are deny_none, other clients aren't restricted against
   opening the file.
- Unless another client had the file open, the open probably provided a
   delegation, so the client is doing opens locally anyhow and just using
   Access to check permissions, etc.
- If the file is being opened by other clients, holding the open with
   the most access permissions acquired will prevent the other client
   being given a delegation. (The case where rw is downgraded to ro and
   then another client opens it for reading and gets a read delegation.)
   Some might argue this is a case where the open_downgrade is a good thing
   to do, but I haven't found read delegations particularily useful and,
   if this client wants to "open upgrade" to rw again, the read delegation
   just results in extra overhead. As such, I'd argue that just holding
   onto the rw open and preventing another client from getting a read
   delegation is preferable, or at least "as acceptable".

In summary, I can't see an advantage for the client in doing an
open_downgrade and I can't see what advantage it provides to the server
or other clients, either? (Assuming it's always using deny_none, of 
course.)

Comments? rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 25 Jun 2008 12:31:57 -0400

> - Since all opens are deny_none, other clients aren't restricted against

Local users may be unnecessarily prohibited from executing it, though,
right?

--b.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 25 Jun 2008 16:17:23 -0400 (EDT)

> Local users may be unnecessarily prohibited from executing it, though,
Are you saying that servers will say NFS4ERR_ACCESS or NFS4ERR_IO in
reply to an Access Op for ACCESS4_EXECUTE? Or, are you simply noting
that the client will have to locally decide if a file still opened
against the server rw can be executed? (The latter seems to be a
client side implementation issue. In my client code, the relationship
between local opens and Open state on the server might be descibed as
"loose" and this is not an issue. As you might have guessed, my client
never does Open_downgrades.)

rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 25 Jun 2008 16:16:55 -0400

> never does Open_downgrades.)

At least in the case of the linux server, if an nfsv4 client has a file
open for write, then the server holds that file open.  So if a client
has a file open for write on the server, then a user on the server that
tries to execute that file will get ETXTBUSY.

So in theory failing to open_downgrade could lead to spurious
ETXTBUSY's.

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 25 Jun 2008 16:19:14 -0400

> never does Open_downgrades.)

No. He is saying that it may make sense for a POSIX machine to add a
DENY_WRITE while a file is being executed in order to enforce the
ETXTBSY error that most people would expect if another process attempts
an open for write.
If you don't do an open_downgrade once you are done writing the
executable then the above scheme would fail.

The other user of DENY_* would be Microsoft machines. If you don't plan
to ever have your client run against a Windows-based server or work with
other Windows-based clients, then you are fine, but if not, you are
likely to be preventing the correct operation of many of their
applications.

Cheers
  Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 25 Jun 2008 16:26:33 -0400

> executable then the above scheme would fail.

I was actually thinking of the case of a user on the server trying to
execute a file open for write from a client--in which case the user on
the server might get ETXTBUSY.

But, yeah, it does seem reasonable for a posix client to set DENY_WRITE
when they execute, in which case the problem would extend to users
executing the file on those clients.

(BTW, does anyone know of a client that actually does this?  I've heard
it discussed as an idea, but don't think I've personally seen it.)

--b.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 25 Jun 2008 16:43:16 -0400 (EDT)

> So in theory failing to open_downgrade could lead to spurious
Ok, now I understand. To be honest, I think I can live with it, since 
about the only time an executable file will be open for writing is when
it's being compiled and it should be closed when the compile is done.

rick, who is too lazy to implement the open downgrade stuff correctly.

---

From: "Halevy, Benny" <[email-masked]>
Date: Wed, 25 Jun 2008 16:38:44 -0400

Well it could be a script too.  These might be rewritten more frequently.
 
Benny

________________________________

From: [email-masked] on behalf of Rick Macklem
Sent: Wed 2008-06-25 23:43
To: J. Bruce Fields
Cc: Steven Huntington; [email-masked]; Trond Myklebust
Subject: Re: [nfsv4] Asking for details about OPEN_DOWNGRADE

---

From: Rick Macklem <[email-masked]>
Date: Wed, 25 Jun 2008 16:51:16 -0400 (EDT)

> executable then the above scheme would fail.

I can see why you might choose to do that and then, of course, open 
downgrades would be appropriate. (You could limit your implementation
to doing the downgrade for that specific case.)

> likely to be preventing the correct operation of many of their
Lost me here. I can see why a WIN32 API client would do DENY_*, but I
don't see why having a server implemented under windows makes any
difference. Ditto w.r.t. other Windows clients. If your POSIX client
does DENY_WRITE, then another POSIX client won't be able to open the
file for writing and that will confuse it. Ditto for POSIX clients w.r.t
DENY_READ.

rick

---

From: Rick Macklem <[email-masked]>
Date: Wed, 25 Jun 2008 16:51:16 -0400 (EDT)

rick

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 25 Jun 2008 17:35:23 -0400

> file for writing and that will confuse it. Ditto for POSIX clients w.r.t

As stated earlier, there is absolutely a precedence for DENY_WRITE (viz
ETXTBSY) on posix systems.

As for Windows apps what matters isn't _why_ they should use DENY_*, but
the fact that they _do_. In fact, the default share mode when you create
a file is exclusive access (DENY_READ+DENY_WRITE).

  http://msdn.microsoft.com/en-us/library/aa363874(VS.85).aspx
  http://msdn.microsoft.com/en-us/library/bb540534(VS.85).aspx

and yes, Windows application writers do rely on this:

  http://www.linuxforums.org/forum/linux-programming-scripting/117514-there-file-share-mode-linux.html

so by not downgrading your READ_WRITE share mode to READ (or WRITE), you
are creating situations where you unnecessarily impede access to those
applications that do specify the conflicting share mode of DENY_WRITE
(or DENY_READ).

To summarise: the problem is that you are holding on to a mandatory lock
after your application is done with it.

Trond

---

From: Rick Macklem <[email-masked]>
Date: Thu, 26 Jun 2008 11:16:05 -0400 (EDT)

[stuff snipped]
> after your application is done with it.
Ok, now I understand what you are referring to. I still don't believe it 
to be a likely problem, but I see your point.

Here's my view of it (assuming an open_owner4 granularity of a posix 
process):
1 - Few posix processes open the same file multiple times with different
     modes. Most will just open with whatever mode is needed and close when
     done. Some might open->close, open->close. (To be honest, a lot of
     posix processes don't even bother to do explicit closes and everything
     just gets closed upon exit().)
2 - For the rare posix process that does do open_for_reading,
     open_again_for_writing, close_for_writing, sometime later
     close_for_reading, what is the likelyhood that a Windows app running
     on a different client will want to open the file during the
     "sometime later" time period.

I just don't believe Windows apps will be sharing files with posix
apps running on different clients in this way, but I can see your
argument, rick.