Subject: [nfsv4] Forked process locking question

From: Saul Tamari <[email-masked]>
Date: Thu, 18 Apr 2013 10:44:13 -0400

Hi,

I've stumbled on a case with a cthon byte range locking test running
on a Linux client where I'm not sure what is the expected behavior.

The sequence that I see is:
1. A file is opened and stateid si1 is returned
2. The same file is then exclusively locked using si1 and a new
lock-owner, resulting in stateid si2 (which is totally different from
si1)
3. The file is now being written using si1 (the stateid from step 1)

Does an NFSv4 server need to allow the write on step 3 or fail it?

Thanks,
Saul

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 18 Apr 2013 15:18:05 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Saul Tamari
> Sent: Thursday, April 18, 2013 10:44 AM
> To: nfsv4
> Subject: [nfsv4] Forked process locking question
> 
> Hi,
> 
> I've stumbled on a case with a cthon byte range locking test running on a
> Linux client where I'm not sure what is the expected behavior.
> 
> 1. A file is opened and stateid si1 is returned 2. The same file is then
> exclusively locked using si1 and a new lock-owner, resulting in stateid si2
> (which is totally different from
> si1)

By "exclusively locked", do you mean a mandatory lock? If so, the right to do is return NFS4ERR_LOCKED.
If, on the other hand, it is an ordinary advisory lock, then of course you should allow it.

Cheers
  Trond

---

From: Saul Tamari <[email-masked]>
Date: Thu, 18 Apr 2013 11:24:01 -0400

Hi,

I meant a write lock using WRITE_LT nfs_lock_type4.
I'm testing NFSv4 so I assume its always a mandatory lock.

I'll have to figure out why cthon lock test #12 runs such a flow and
complains about it.

Thanks,
Saul

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 18 Apr 2013 15:28:28 +0000

> Sent: Thursday, April 18, 2013 11:24 AM
> To: Myklebust, Trond
> Cc: nfsv4
> Subject: Re: [nfsv4] Forked process locking question
> 
> Hi,
> 

Why? In NFSv4, it is up to the server vendor to decide if they will implement mandatory or advisory byte range locks. Both are supported.

> I'll have to figure out why cthon lock test #12 runs such a flow and complains
> about it.

Normally I would expect the client to use the lock stateid if the thread doing the WRITE is the same as the one that took the lock.

> >> From: [email-masked] [mailto:[email-masked]] On
> >> Behalf Of Saul Tamari
> >> 1. A file is opened and stateid si1 is returned 2. The same file is
> >> then exclusively locked using si1 and a new lock-owner, resulting in
> >> stateid si2 (which is totally different from
> >> si1)
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Saul Tamari <[email-masked]>
Date: Thu, 18 Apr 2013 11:58:29 -0400

On Thu, Apr 18, 2013 at 11:28 AM, Myklebust, Trond
<[email-masked]> wrote:
[snip]
> Why? In NFSv4, it is up to the server vendor to decide if they will implement mandatory or advisory byte range locks. Both are supported.

We implemented mandatory byte-range locking so I was mistakenly
assuming its always required. Thanks for this clarification.

> Normally I would expect the client to use the lock stateid if the thread doing the WRITE is the same as the one that took the lock.

The LOCK call is done from a child process (not a thread) while the
WRITE is done from its parent process.

Thanks,
Saul

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 18 Apr 2013 16:36:25 +0000

> WRITE is done from its parent process.

Then the the LOCK will be owned by the child process, and the client is
correct to be using the OPEN stateid for the parent process' WRITE.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Rick Macklem <[email-masked]>
Date: Thu, 18 Apr 2013 16:54:23 -0400 (EDT)

Saul Tamari wrote:
> > Why? In NFSv4, it is up to the server vendor to decide if they will
> > implement mandatory or advisory byte range locks. Both are
> > supported.
> 
Long ago I tried to argue that mandatory locking was broken, but never
seemed to convince others.

The problem is that, for mandatory locking, a client must hold byte
range locks for all byte ranges cached. This would be fine, if the
client knew that the server was implementing mandatory locking.
However, the client doesn't know this. (RFC3010 had this correct, in
that there was an open reply flag that indicated whether or not
mandatory locking was being applied to the file. This flag got dropped
for RFC3530, I think because it required the server to retain the
mandatory vs advisory locking for the file while it was opened.)
At the time, someone suggested that the client could test for
mandatory locking. This is messy to do and since there is no
guarantee that it will not change mandatory vs advisory at any
time, it isn't sufficient. (There is some junk w.r.t. mode bits
that might indicate mandatory vs advisory, but that wasn't required
for servers to implement.)

So, clients don't know if a server is implementing mandatory vs advisory
so they don't know if they need to lock all byte ranges cached.
The FreeBSD client doesn't lock byte ranges it caches, so it is broken
for servers that implement mandatory locking and I don't know how to
fix it, except to always lock all bytes ranges and I consider that
unacceptable.

Maybe Trond can explain how the Linux client deals with this?

Then there is the problem related to your discussion, which is that
the RFC underspecified what stateid is to be used for I/O operations.
Since clients typically do this I/O asynchronously and the RFC doesn't
require that the stateid be one for a lock owner that holds a byte
range lock for the correct byte range/fh, it is hard to say what the
mandatory lock requirement should be? (A client that holds a delegation
can do byte range locally in the client. Does that mean a delegation
implies client can do I/O on any byte range in the file. I think that
is the case.)

I'd argue that the server should allow the I/O if the clientid (related
to the stateid) holds any byte range lock of the correct type (ro/rw,
fh, byte range).
> > Normally I would expect the client to use the lock stateid if the
> > thread doing the WRITE is the same as the one that took the lock.
> 
In summary, there are issues related to mandatory locking that I don't
think have even been resolved. As such, I know the FreeBSD client is
broken for them and I suspect other clients might be, as well.

rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 18 Apr 2013 21:18:04 +0000

> Maybe Trond can explain how the Linux client deals with this?

Locks are application-driven. They _should_not_ be driven by the
filesystem.

The whole text in RFC3530 about how the client should handle
NFS4ERR_LOCKED is utter bunk: the correct thing to do is either retry
the read/write until it succeeds, or return an error to the application.

> the RFC underspecified what stateid is to be used for I/O operations.

It is fully specified in RFC5661 and RFC3530bis.

> to the stateid) holds any byte range lock of the correct type (ro/rw,

Please read RFC5661 and RFC3530bis...

> broken for them and I suspect other clients might be, as well.

??????????????????????????????????????????????????????

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Thu, 18 Apr 2013 18:28:54 -0400 (EDT)

> the read/write until it succeeds, or return an error to the
Well, here's basically the same words from RFC5661. What it says makes
sense to me. For example, if a client is doing write back caching against
a server that implements mandatory locking and does not have a write lock
on the byte range, the write could fail. This error will occur after the
application's write syscall has returned success. (Same goes for writing
of modified pages when mmap'd.) Yes, the client could keep retrying the
write, but there is no guarantee it will ever succeed.

10.3.3.  Data Caching and Mandatory File Locking

   Client-side data caching needs to respect mandatory byte-range
   locking when it is in effect.  The presence of mandatory byte-range
   locking for a given file is indicated when the client gets back
   NFS4ERR_LOCKED from a READ or WRITE operation on a file for which it
   has an appropriate share reservation.  When mandatory locking is in
   effect for a file, the client must check for an appropriate byte-
   range lock for data being read or written.  If a byte-range lock
   exists for the range being read or written, the client may satisfy
   the request using the client's validated cache.  If an appropriate
   byte-range lock is not held for the range of the read or write, the
   read or write request must not be satisfied by the client's cache and
   the request must be sent to the server for processing.  When a read
   or write request partially overlaps a locked byte-range, the request
   should be subdivided into multiple pieces with each byte-range
   (locked or not) treated appropriately.

The part of the above that I do not like is the second sentence.
Mandatory locking could be in effect for some time before the client
sees NFS4ERR_LOCKED and might be switched to advisory locking without
the client knowing the change. (I reiterate that RFC3010 got it right,
but the open reply flag was dropped for some reason for subsequent RFCs.)

> It is fully specified in RFC5661 and RFC3530bis.
Yes, but with SHOULDs and not MUSTs. Also, for the lock stateid case,
it does state that it is based on the lockowner entity (process or ???)
being the same.

Does this imply that the server should return NFS4ERR_LOCKED if the client
holds an appropriate byte range lock but the client hasn't presented the
stateid for that lock?

The problem is that there are RFC3530 based clients out there (and they
will be out there for a long time) which don't follow the above rules.
As such, if a server assumes the client will do the above, interoperability
between such clients and servers will be broken for a long time.
I'd suggest server implementers be as flexible as possible when handling
the stateid in I/O operations.
> Please read RFC5661 and RFC3530bis...
As above, just because RFC3530bis may soon become an RFC doesn't
imply that extant NFSv4.0 clients will follow those rules, at least
for years.

I don't think the purpose of RFC3530bis was to break interoperability
with RFC3530 based implementations.

rick

> > > >> I'll have to figure out why cthon lock test #12 runs such a
> > > >> flow
> > > >> and complains
> > > >> about it.
> > > >
Like I've already said, a client can't easily know whether or not
mandatory locking is being implemented for a file. If it doesn't
know that, it can't follow the above rules in 10.3.3. You called them
bunk, but they're in RFC5661 as well as RFC3530.

Implement a test app that runs on two clients and run it against a
server that implements mandatory locking:
- On client1, it writes to a file in small non-block sized chunks.
  As soon as a write syscall replies success, it tells the other client
  to write lock the byte range just written, which client2 does.
- Client1 should never get NFS4ERR_LOCKED, but I suspect that will
  happen for some clients. (I know it will happen for the FreeBSD one
  and I don't know of a good way to fix it. If the client knew that
  mandatory locking is being applied, the client could lock the byte ranges
  during the write syscall and retain the lock until the write-back is
  completed.)

rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 18 Apr 2013 23:50:08 +0000

> but the open reply flag was dropped for some reason for subsequent RFCs.)

If the application is writing to a file that is being synchronised via
locking, without first taking a lock, then it is broken! It has no way
to ensure that it doesn't clobber other data, and it has no way to
ensure that its data won't be clobbered.

The NFS client can't fix that on behalf of the application.

> holds an appropriate byte range lock but the client hasn't presented the

The cases where the _server_ should return NFS4ERR_LOCKED are clearly
listed.

The use of SHOULD in the description of client behaviour is 100%
intentional.

> will be out there for a long time) which don't follow the above rules.

Why not? We debated them on the lists almost a decade ago.

> the stateid in I/O operations.

I see no conflict with that statement and anything I said above.

> with RFC3530 based implementations.

It doesn't.

> bunk, but they're in RFC5661 as well as RFC3530.

See above.

>   during the write syscall and retain the lock until the write-back is

There is no close-to-open cache consistency above. There is no
lock-mediated consistency.

It is a junk example...

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Fri, 19 Apr 2013 10:31:35 -0400 (EDT)

Trond Myklebust wrote:
> > > > > > Why? In NFSv4, it is up to the server vendor to decide if
> > > > > > they
> > > > > > will
Well, since BSD doesn't implement mandatory locking and there isn't any
POSIX spec for it afaik, I suppose you can define what is correct semantics.

However, what little I just read never stated that an application always needed
to lock all byte ranges being read/written when mandatory locking was in effect.
It did indicate that the write(2) syscall should either block until the lock is
released and it can complete the write or return EAGAIN if opened with O_NOBLOCK.
(Asynchronous write-back will result in the above semantics unless it locks all
 byte ranges being cached for write-back.)

I suspect we will simply remain in disagreement over what the correct client
handling of mandatory locking is. Since I can't tell when mandatory locking
is being applied to a file by a server, I'm not going to worry about it, rick

> > > > > > Normally I would expect the client to use the lock stateid
> > > > > > if
> > > > > > the
> > > > > The LOCK call is done from a child process (not a thread)
> > > > > while
> > > > > the