Subject: [nfsv4] Share reservations and special stateid

From: "Frank Filz" <[email-masked]>
Date: Mon, 13 Sep 2021 16:53:35 -0700

Section 9 documents that when a special stateid is in use, byte range locks
that conflict with it should not be granted. Presumably this would result in
the client re-issuing the lock request.

What about share reservations? If we get an OPEN with a deny mode that
conflicts with an in progress I/O using a special stateid (or an NFSv3 I/O),
do we reject with a conflict (which would result in an error on the client
since share reservations aren't blocking or retried), allow it anyway, or
delay the response until the I/O is complete? We could return NFS4ERR_DELAY
to make the client retry, but I'm also thinking about NLM requests where
there is no such option.

Thanks

Frank

---

From: Rick Macklem <[email-masked]>
Date: Thu, 16 Sep 2021 00:12:11 +0000

Frank Filz wrote:
>Section 9 documents that when a special stateid is in use, byte range locks that >conflict with it should not be granted. Presumably this would result in the client re->issuing the lock request.
>
>What about share reservations? If we get an OPEN with a deny mode that conflicts >with an in progress I/O using a special stateid (or an NFSv3 I/O), do we reject with >a conflict (which would result in an error on the client since share reservations >aren’t blocking or retried), allow it anyway, or delay the response until the I/O is >complete? We could return NFS4ERR_DELAY to make the client retry, but I’m also >thinking about NLM requests where there is no such option.
Not sure why the NLM is relevant to Open share reservations?
If an Open with a SHARE_DENY that conflicts with an I/O operation using a
special stateid is in progress, I think replying NFS4ERR_DELAY is probably
the technically correct response w.r.t. Sec. 9 of RFC5661, but read on...

However, the whole business of mandatory locking is a mess that I doubt
will ever be fixed.
Here's my take from a client implementer's perspective:
(byte range locking first)
- If a server can switch a file between mandatory and advisory byte range
  locking at any time (which appears to be allowed by the RFCs, except
  for RFC3010), a client must acquire byte range locks for all byte ranges
  that the client is data caching (so that a conflicting byte range lock cannot
  be acquired by another client, blocking I/O on the cached data, particularly
  dirty bytes that need to be written back to the server at some time).
  Doing this would be inconsistent with POSIX advisory byte range locking,
  since the locks would not be ones done explicitly by POSIX applications and
  could conflict with them.
  Short version...
  - The FreeBSD (and maybe the Linux?) client does not do this and, therefore,
     can only function correctly for servers that only implement advisory byte
     range locking.
RFC3010 actually got this right, by returning a flag in the Open reply that
indicated whether the server was implementing "mandatory" vs "advisory"
byte range locking. The problem was, to implement this implied that the
server could not switch between the two while there was an outstanding
Open (the property a client required to decide if it needed to acquire byte
range locks for the byte ranges being data cached in the client).
--> Since Solaris liked to switch at any time, the flag was deleted for
      RFC3530 and this has never been fixed.

As for Open shares with Deny values other than Deny_none, which are
always mandatory...
- Although, in theory, a client can try I/O operations with special stateids,
  it makes no sense in practice, since they can fail at "random" times, when
  a client chooses to do an Open with a conflicting Deny.
The FreeBSD (and, again, maybe the Linux?) client, never does I/O with
special stateids (there is code that falls back on special stateids in the
FreeBSD client, but that code should *never* happen. Or, only happens
if there is a bug in the implementation, if you prefer).

As for NFSv3, if a server chooses to implement NFSv4 Open share denial
for NFSv3 I/O RPCs, things just break horribly for the NFSv3 client.
--> As such, I doubt any NFS server that supports NFSv3 and NFSv4 will
      do this.

Put another way, the NFSv4 client should probably not assume that any
Share_deny is being enforced by the NFSv4 server for NFSv3 I/O.
--> The client needs to always get an appropriate Open for I/O and
      always use it.
      --> Never use a special stateid for I/O.

Also, does "in progress" include RPCs in flight to the server (including the
RPC layer receive queue)?
- In general, making a client wait until some "random moment" when no
  NFSv4 I/O using special stateids or NFSv3 I/O is "in progress" (whatever
  that actually means) proves nothing, imho. (And, as you noted, is not
  explicitly required by RFC5661, as far as I can see.)

rick

Thanks

Frank

---

From: "Frank Filz" <[email-masked]>
Date: Fri, 17 Sep 2021 11:58:42 -0700

> -----Original Message-----
> From: Rick Macklem [mailto:[email-masked]]
> Sent: Wednesday, September 15, 2021 5:12 PM
> To: Frank Filz <[email-masked]>; [email-masked]
> Subject: Re: [nfsv4] Share reservations and special stateid
> 
> Frank Filz wrote:
> >Section 9 documents that when a special stateid is in use, byte range
locks that
> >conflict with it should not be granted. Presumably this would result in
the client
> re->issuing the lock request.
> >
> conflicts >with an in progress I/O using a special stateid (or an NFSv3
I/O), do we
> reject with >a conflict (which would result in an error on the client
since share
> reservations >aren't blocking or retried), allow it anyway, or delay the
response
> until the I/O is >complete? We could return NFS4ERR_DELAY to make the
client
> retry, but I'm also >thinking about NLM requests where there is no such
option.
> Not sure why the NLM is relevant to Open share reservations?

NLM has share reservations also. The Windows NFSv3 client uses them...

> special stateid is in progress, I think replying NFS4ERR_DELAY is probably
the
> technically correct response w.r.t. Sec. 9 of RFC5661, but read on...
> 
will
> ever be fixed.
server
> could not switch between the two while there was an outstanding Open (the
> property a client required to decide if it needed to acquire byte range
locks for
> the byte ranges being data cached in the client).
> As for Open shares with Deny values other than Deny_none, which are always
> mandatory...
special
> stateids (there is code that falls back on special stateids in the FreeBSD
client,
> but that code should *never* happen. Or, only happens if there is a bug in
the
> implementation, if you prefer).
> 
> As for NFSv3, if a server chooses to implement NFSv4 Open share denial for
> NFSv3 I/O RPCs, things just break horribly for the NFSv3 client.

Ganesha does enforce NFSv4 Deny Write against NFSv3 WRITE. We haven't had
any complaints and I believe we do have users who have Windows clients
(pretty much the only way you will ever get Deny). I'm not sure what the
state of Windows NFSv4 client is though...

We can't enforce NLM_SHARE against NFSv3 clients because there is no way to
determine if the NFSv3 I/O is associated with an active NLM_SHARE.

NFSv3 READ operations are always allowed, treated the same as the NFSv4
Special State ID that allows READ bypass.

NFSv4 WRITE with Special State ID will be blocked by any active share
reservation.

What breaks if an NFSv3 client can't WRITE to a file with an NFSv4
DENY_WRITE share reservation? This would imply mixing Windows and
non-Windows users of the same file. I would tend to expect things to not go
well in that case... And really, if two applications are trying to use the
same file and at least one is writing to the file, there should be some kind
of locking protecting the file and the two applications (or application
instances) should be prepared.

Since non-Windows READs are either blocked from OPEN (NFSv4) or allowed
(NFSv3 READ or NFSv4 READ with Special State ID) a non-Windows application
that doesn't care about data integrity and just wants to read the file
(maybe tail for an example?) will be OK.

I get that it's annoying if your non-Windows application using NFSv3
succeeds in opening the file for write (which won't be blocked since there's
no protocol operation for the open) and then failing when it does an actual
WRITE it gets a failure like EIO.

If this actually presents itself as a real issue to our users, it would be
easy enough to either remove it, or make the behavior configurable (I could
see a primarily Windows shop not caring that some Linux application breaks
when it tries to use a file that is also in use by a Windows client, the
more radical folks who care think there should be no cross-protocol use at
all since NFS can't enforce all the mandatory locking of Windows).

>       --> Never use a special stateid for I/O.

And yea, if NFSv4 clients in actuality never use the special stateid then
deny enforcement is really no problem for NFSv4 clients.

Hmm, do all SETATTR come with an open stateid? But that only matters for
truncate which I suppose does imply an open file for *ix clients.

> Also, does "in progress" include RPCs in flight to the server (including
the RPC
> layer receive queue)?

In progress would not include in flight. The primary issue is that as a user
space server, I need to make sure file descriptors don't go bad while in
use, especially if an actual I/O system call is not in progress. Getting
EBADF because a CLOSE closed an fd that was about to be used by I/O would be
bad. At the moment, I'm just going to resolve by blocking closing fds while
I/O is in progress in our FSAL (file system layer), but new I/O will be
prevented from starting (blocked) if a thread is blocked wanting to close
the fd. But that doesn't prevent a new NFSv4 OPEN because that will open a
new fd associated with the new stateid, nor will I/O on one stateid block a
CLOSE on a different stateid.

I expect in a well behaved client the conflicts will be close to zero,
though there could be some delay if the NFSv4 client wants to do an OPEN
upgrade or downgrade because it is sharing the stateid between multiple
actual client side file descriptors that are opening and closing. I suppose
the most likely scenario there is a fork where the child inherits some fds
which the parent then wants to close but didn't manage to close before the
child process started I/O on an inherited fd. So the parent process lingers
for a bit...

> =

---

From: David Noveck <[email-masked]>
Date: Sat, 18 Sep 2021 07:11:58 -0400

On Wed, Sep 15, 2021, 8:12 PM Rick Macklem <[email-masked]> wrote:

> locks that >conflict with it should not be granted. Presumably this would
> result in the client re->issuing the lock request.
> >
> I/O), do we reject with >a conflict (which would result in an error on the
> client since share reservations >aren’t blocking or retried), allow it
> anyway, or delay the response until the I/O is >complete? We could return
> NFS4ERR_DELAY to make the client retry, but I’m also >thinking about NLM
> requests where there is no such option.

I think it also technically correct to fail the open, although DELAY might
be preferable.  If there were a stream of such IOs, you don't want to
return DELAY indefinitely.

> However, the whole business of mandatory locking is a mess that I doubt

I was unaware of the issue but think it is possible to fix it via an
extension in v4.2.

Here's my take from a client implementer's perspective:
>   for RFC3010),

I wasn't aware of that.  I have never heard of  such a server and would
never knowingly use one.

a client must acquire byte range locks for all byte ranges
>   dirty bytes that need to be written back to the server at some time).

Sounds *UGLY.*

> --> Since Solaris liked to switch at any time, the flag was deleted for

I can't imagine reasons for actually switching or how a client might be
expected to respond to such a behavior shift.

Putting the flag back would require an OPEN_PLUS and is not worth the
trouble.

It would be possible to add an fs attribute that tells you if the server
implements mandatory byte-range locking on that fs.

>   a client chooses to do an Open with a conflicting Deny.

Agree.

> for NFSv3 I/O RPCs, things just break horribly for the NFSv3 client.

I'm not sure what you mean by "break horribly".

>       do this.

I will check on what ontap does.

> Share_deny is being enforced by the NFSv4 server for NFSv3 I/O.

The specs cannot deal with the issue so I suppose so.

>       --> Never use a special stateid for I/O.

Rfc5661bis cannot go that far but it possible to revise the text to
indicate some the problems with using special state ids together with open
deny modes.

> RPC layer receive queue)?

It really can't.

>   explicitly required by RFC5661, as far as I can see.)

I will look at what 5661 says to make sure the bis says the right thing
which I think would be that an when an open denying X is granted no X
operations with special stateids on that file can be in the process of
execution.

> rick
>
> Thanks
>
> Frank
>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Rick Macklem <[email-masked]>
Date: Sat, 18 Sep 2021 21:36:09 +0000

David Noveck wrote:
>On Wed, Sep 15, 2021, 8:12 PM Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:
Frank Filz wrote:
>>What about share reservations? If we get an OPEN with a deny mode that conflicts 
>>with an in progress I/O using a special stateid (or an NFSv3 I/O), do we reject with 
>>a conflict (which would result in an error on the client since share reservations >aren’t blocking or retried), allow it anyway, or delay the response until the I/O is 
>>complete? We could return NFS4ERR_DELAY to make the client retry, but I’m also 
>>thinking about NLM requests where there is no such option.
>I think it also technically correct to fail the open, although DELAY might be preferable.  If there were a stream of >such IOs, you don't want to return DELAY indefinitely.
>
>
>I was unaware of the issue but think it is possible to fix it via an extension in v4.2.
If an agreement that "mandatory vs advisory" locking applies for at least the duration
of an Open for the file, then, yes, a new attribute or Operation(s) to indicate
mandatory vs advisory locking would be useful. (See below.)

>I wasn't aware of that.  I have never heard of  such a server and would never knowingly use one.
Well, here's what RFC3530 says (page 56):
   Note that in UNIX, if a file has the MODE4_SGID bit set and no
   MODE4_XGRP bit set, then READ and WRITE must use mandatory file
   locking.
and there is nothing stopping a Setattr of Mode at any time, changing the above,
as far as I know.
(I'm guessing UNIX is referring to something like the SVID, or Solaris if you prefer?)

>Sounds UGLY.
Not necessarily, if the "remains advisory or mandatory while open" rule applies,
and there is a way (new operation or attribute) for the client to know which
it is.
For example, FreeBSD has a "nolockd" mount option (I think Linux has a similar
one called "nolock") which specifies that POSIX byte range locking be done locally
within the client.
If the mount specifies "nolockd", then the FreeBSD/POSIX like client could:
- Do a check for "mandatory" after Open in the same compound.
  - if mandatory
        - lock all bytes of file, so that the data can be cached
For example, if there were a MandatoryLock attribute, the compound could be:
Open
Verify MandatoryLock
Lock (all bytes of file)

--> if the lock operation fails, disable data caching for the file while Open.
      This should be relatively rare and implies file sharing across multiple
      clients and, as such, disabling data caching may be preferable.

The "if mandatory" test could be done via Verify/Nverify if there was a new
attribute to indicate "mandatory vs advisory". If there were a new operation
that returned NFS4_OK if mandatory and some other status if advisory, that
would work, as well.
--> An operation that returned NFS4_OK plus a boolean to indicate "mandatory"
      would not work for the above.
      However, an operation that returns NFS4_OK if mandatory and another
      operation that indicates NFS4_OK if advisory, would be fine.

The client could issue a warning if the mount does not specify "nolockd" and
a file using mandatory locking is encountered.
I would like to avoid the need for an extra compound RPC on every open,
but if the "mandatory vs advisory" locking rule applied for the duration of
a clientID's existence, then the client could simply do an RPC after ExchangeID
to find out, then either always/never to a Lock operation in the same compound
as the Open.

>I can't imagine reasons for actually switching or how a client might be expected to respond to such a behavior shift.
It would be nice to know if any NFSv4 server does follow the "switch at any time" rule.
If there are no NFSv4.2 servers doing this, then maybe there is time to fix it
for NFSv4.2?

>Putting the flag back would require an OPEN_PLUS and is not worth the trouble.
>
>It would be possible to add an fs attribute that tells you if the server implements mandatory byte-range locking on >that fs.
Yes, or new operations. It could also be "per file", "per fs" or "per server".
The key is the guarantee that it will not switch (during Open for "per file", or while a ClientID
exists for all Opens on that ClientID for the "per fs" or "per server" cases, I think?)

There is also another way that this can be handled for the "don't switch for a ClientID case":
The "mandatory" and "advisory" can be two sandboxes on the server, where the server can
support either or both of the sandboxes.
For the case where the server supports both sandboxes, the "mandatory" clients would only
see NFS4ERR_LOCKED replies if other client(s) in the "mandatory" sandbox had conflicting
locks and the "advisory" clients would never see NFS4ERR_LOCKED.
--> Yes, if a file were concurrently Opened in both sandboxes, the semantics would be "odd",
      but that would probably be rare and just need to be documented (or a server would choose
      to not allow a file to be accessed from both sandboxes).

A new Operation negotiates which sandbox the client is in.
--> Client asks for advisory.
    --> Server allows that or only supports "mandatory".

>I'm not sure what you mean by "break horribly".
I was referring to an NFSv3 I/O operation failing due to a conflicting lock.
Not sure what error would be replied, since there is no NFS3ERR_LOCKED?
Then, what does the client do. Fail the syscall with EIO or wait and try it
again?
Maybe the server could reply NFS3ERR_JUKEBOX for this case?

>Rfc5661bis cannot go that far but it possible to revise the text to indicate some the problems with using special >state ids together with open deny modes.
Yes, I would consider this "client implementation advise". I think it already states that servers can
choose to not allow I/O operations with special stateids.

>I will look at what 5661 says to make sure the bis says the right thing which I think would be that an when an open >denying X is granted no X operations with special stateids on that file can be in the process of execution.

rick

Thanks

Frank

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4