Subject: [nfsv4] Read delegations and LOCK operation

From: "Sandeep Joshi" <[email-masked]>
Date: Thu, 15 Oct 2009 10:38:22 -0700

>From V4.1 protocol specification section 18.10.4. :
 
"When one or more clients hold read delegations, any LOCK request 
  where the server is implementing mandatory locking semantics, MUST
  result in the recall of all such delegations."

I have following question for above statement.
 
Why is the recall of read delegations necessary if the LOCK requested is
a READ 
lock?

regards,
 
Sandeep

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 15 Oct 2009 17:58:32 -0400

It sounds to me like that is an oversight in the spec.  Am I (and
Sandeep) missing anything here?
 
I'm not sure exactly what can be fixed at this stage without triggering
a re-entry into review, to be avoided.  I would expect that changing
that paragraph to make it explicit that mandatory read locks are OK,
would be allowed.  I'd like Mike's and Spencer's opinion.
 
I think the argument to be made is that when you have read delegations,
you have already prevented writes and in that case a mandatory READ lock
is a no-op in that it doesn't prevent any IO's that would not be
prevented otherwise.  That is, it only would prevent writes and writes
are already prevented by the fact that you have a read delegation.  Any
writes or opens for write would already recall the delegations.  So I
think you could argue that in that case, for such locks, the server is
not implementing "mandatory lock semantics" because in the case of such
locks you can't get to any situation in which the mandatory lock
semantics' behavior is different from that of the non-mandatory
semantics.  In that somewhat strange context, a change in the text would
only have to make explicit what is already implicit, but might well
confuse people into recalling delegations when they don't need to be
recalled.
 
So my proposed text is:
 
   When one or more clients hold read delegations, any LOCK request
   where the server is implementing mandatory locking semantics, MUST
   result in the recall of all such delegations.  In this context, a 
   read lock is not considered to be implementing mandatory lock
semantics
   since the only operations that such locks prevents are WRITE's and 
   no such WRITE's can be done so long as the read delegations are 
   held.  Where mandatory lock semantics are being implemented for a
   WRITE lock, the lock must
   not be granted until all such delegations are return or revoked.
   Except where this happens very quickly, one or more NFS4ERR_DELAY
   errors will be returned to requests made while the delegation remains
   outstanding.

Opinions?

	-----Original Message-----
	From: Sandeep Joshi [mailto:[email-masked]] 
	Sent: Thursday, October 15, 2009 1:38 PM
	To: [email-masked]
	Subject: [nfsv4] Read delegations and LOCK operation

	"When one or more clients hold read delegations, any LOCK
request 
	  where the server is implementing mandatory locking semantics,
MUST
	  result in the recall of all such delegations."

	Why is the recall of read delegations necessary if the LOCK
requested is a READ 
	lock?

	regards,
	 
	Sandeep

---

From: Rick Macklem <[email-masked]>
Date: Fri, 16 Oct 2009 10:51:21 -0400 (EDT)

> confuse people into recalling delegations when they don't need to be
I'm not particularily familiar with mandatory lock semantics.
- If client X wants to read a byte range of a file, when it doesn't
   hold a read lock on that range, but client Y does hold a read lock
   on that byte range, is the Read allowed?
   (Yes, I realize that client X could acquire a read lock on the byte
    range, but if it hasn't bothered to do so...)

If the answer to this "Yes, the read is allowed", then I'd agree
in that I can't think of why the Read delegations would need to be 
recalled. If the answer is "No, the read is not allowed", then I'd say 
that the current text is correct, since the server can only know what byte 
range(s) are locked by which clients by recalling the delegations.

rick, who doesn't believe read delegations are much use anyhow.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 16 Oct 2009 10:59:35 -0400

Agree with your statements.

I'm not working on the basis of any authoritative knowledge of mandatory
lock semantics.  I'm assuming that a READ lock prevents WRITEs and not
READs because nothing else seems to make sense.

That is, a WRITE lock prevents both READs and WRITEs, presumably because
you are changing the area and you don't want anyone to interfere with
your changes or see the that area of the file without the changes being
completed.  If that's the case, then if a READ lock prevents READs then
it would almost certainly have to prevent WRITEs as well and so it would
be exactly the same as a WRITE lock and so be pointless.  On the other
hand, a READ lock that prevented WRITE locks and WRITEs in that area
would make perfect sense.  You want to see a consistent state of that
area for your set of read and don't want anyone changing things while
your locks is held.  That would be consistent with N clients each hold
their own READ lock for the same or overlapping areas.

-----Original Message-----
From: Rick Macklem [mailto:[email-masked]] 
Sent: Friday, October 16, 2009 10:51 AM
To: Noveck, Dave
Cc: Sandeep Joshi; [email-masked]
Subject: Re: [nfsv4] Read delegations and LOCK operation

that the current text is correct, since the server can only know what
byte

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 28 Oct 2009 14:10:40 -0700

I can't remember why the spec is the way it is, but ...

> Sandeep) missing anything here?

... possibly Sandeep and you are missing something.

Earlier in that section, the spec says:

"For example, Windows 2000 allows a write
   lock of a file open for READ, while a POSIX-compliant system does
   not.
"

Elsewhere the spec also says that:

"When a client holds a read open delegation, lock operations are not
   performed locally.  All lock operations, including those requesting
   non-exclusive locks, are sent to the server for resolution.
"

So I am wondering if these two statements provide the motivation for
the spec requiring a recall of delegations, even if a shared byte range
mandatory lock is sent.

Mandatory byte range locking requests that there always be a byte
range lock that encompasses the byte range a process or client
reads or writes.

Client A has a read delegation.

Client B has gets a mandatory byte range read lock.

If we don't recall the delegation, A does not know it mandatory
byte range locking is on. So its reads out of cache will not be
protected with byte range locks. This seems like a minor
problem because the lack of a byte range lock does not result in
file corruption. OTOH, it might be case that the server is required
to account for mandatory byte range locks. E.g. if client B tries to
atomically upgrade its lock from shared to exclusive, that attempt
would require A's delegation to be recalled. Perhaps the thinking
was (and this dates to RFC3530 and perhaps RFC3010) that an "attempt"
to atomically upgrade should not wait for a recall.

In thinking about this, rather than recalling a delegation because
of shared or exclusive byte range mandatory lock on the file,
it would be better to have a callback that informs the client that
byte range mandatory locking is in effect, and so all its reads from
cache MUST have explicit byte range locks sent to the server. Then, if
a client ever obtains a conflicting exclusive byte mandatory lock, the
client's read from cache will be properly denied. The read delegation
would continue to be in force. Of course, the value of a read delegation
when every read from cache has to be bracketed with a byte range LOCK/LOCKU
to the server is questionable.

So perhaps when we wrote that text, we were invoking the KISS principle.
It is also the case that in some operating systems mmap() and exclusive
byte
range locking are mutually exclusive. Read delegations are the moral
equivalent of mmap().

Regardless, I am not inclined to make any changes in NFSv4.1; too late. If
mandatory byte range locking is becoming popular, then I suggest
addressing the issue in v4.2.

On Thu, October 15, 2009 2:58 pm, Noveck, Dave wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 28 Oct 2009 14:10:28 -0700

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 28 Oct 2009 18:04:52 -0400

FWIW the Linux client will completely ignore this rule, and will cache
read locks as long as it holds a read or a write delegation.

The Windows 2000 client will just have to wait if it decides to try to
set a write lock on a file which it opened for read only.

Cheers
  Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 28 Oct 2009 19:03:53 -0400

> ... possibly Sandeep and you are missing something.

I find that hard to believe :-).

> Mandatory byte range locking requests that there 
> always be a byte range lock that encompasses the 
> byte range a process or client reads or writes.

Can you point me to something that says that?  I've been googling
looking for some clear specification of what mandatory byte range locks
actually do, but this seems hard to believe.  It means that if you set
the mode bits indicating mandatory locking, you can't do an ordinary
read of the file; you would need a special program that did locks around
the reads.

It seems to make more sense for it to allow IO unless mandatory locks
held to prevent it.  I thought the mandatoriness of mandatory locks were
that they had to be to be obeyed when they were used, not that it was
mandatory to use them.

But I have no direct experience here and I wasn't able to find any clear
documentation.  I'm just going by what makes sense, which is not always
a perfect guide.
 
> OTOH, it might be case that the server is required 
> to account for mandatory byte range locks. E.g. if 
> client B tries to atomically upgrade its lock from 
> shared to exclusive, that attempt would require A's 
> delegation to be recalled. Perhaps the thinking was 
> (and this dates to RFC3530 and perhaps RFC3010) that 
> an "attempt" to atomically upgrade should not wait 
> for a recall.

Perhaps but what B is doing is getting a write lock and to me it makes
most sense to require that (i.e. getting a write lock) to cause a
recall, which it does.

> So perhaps when we wrote that text, we were invoking 

Perhaps, or this could have been an oversight.  We don't know.

> It is also the case that in some operating systems mmap() 
> and exclusive byte range locking are mutually exclusive. 
> Read delegations are the moral equivalent of mmap().

Right but exclusive byte range locks are not the equivalent of mandatory
byte-range read locks.  They are non-exclusive.

> Regardless, I am not inclined to make any changes in NFSv4.1; 
> too late. 

Agreed.

> If mandatory byte range locking is becoming popular, then 
> I suggest addressing the issue in v4.2.

OK.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Wednesday, October 28, 2009 5:11 PM
To: [email-masked]
Subject: Re: [nfsv4] Read delegations and LOCK operation

So I am wondering if these two statements provide the motivation for the
spec requiring a recall of delegations, even if a shared byte range
mandatory lock is sent.

Mandatory byte range locking requests that there always be a byte range
lock that encompasses the byte range a process or client reads or
writes.

If we don't recall the delegation, A does not know it mandatory byte
range locking is on. So its reads out of cache will not be protected
with byte range locks. This seems like a minor problem because the lack
of a byte range lock does not result in file corruption. OTOH, it might
be case that the server is required to account for mandatory byte range
locks. E.g. if client B tries to atomically upgrade its lock from shared
to exclusive, that attempt would require A's delegation to be recalled.
Perhaps the thinking was (and this dates to RFC3530 and perhaps RFC3010)
that an "attempt"
to atomically upgrade should not wait for a recall.

In thinking about this, rather than recalling a delegation because of
shared or exclusive byte range mandatory lock on the file, it would be
better to have a callback that informs the client that byte range
mandatory locking is in effect, and so all its reads from cache MUST
have explicit byte range locks sent to the server. Then, if a client
ever obtains a conflicting exclusive byte mandatory lock, the client's
read from cache will be properly denied. The read delegation would
continue to be in force. Of course, the value of a read delegation when
every read from cache has to be bracketed with a byte range LOCK/LOCKU
to the server is questionable.

So perhaps when we wrote that text, we were invoking the KISS principle.
It is also the case that in some operating systems mmap() and exclusive
byte range locking are mutually exclusive. Read delegations are the
moral equivalent of mmap().

Regardless, I am not inclined to make any changes in NFSv4.1; too late.
If mandatory byte range locking is becoming popular, then I suggest
addressing the issue in v4.2.

On Thu, October 15, 2009 2:58 pm, Noveck, Dave wrote:
> I'm not sure exactly what can be fixed at this stage without 
> triggering a re-entry into review, to be avoided.  I would expect that

> changing that paragraph to make it explicit that mandatory read locks 
> are OK, would be allowed.  I'd like Mike's and Spencer's opinion.
>
> I think the argument to be made is that when you have read 
> delegations, you have already prevented writes and in that case a 
> mandatory READ lock is a no-op in that it doesn't prevent any IO's 
> that would not be prevented otherwise.  That is, it only would prevent

> writes and writes are already prevented by the fact that you have a 
> read delegation.  Any writes or opens for write would already recall 
> the delegations.  So I think you could argue that in that case, for 
> such locks, the server is not implementing "mandatory lock semantics" 
> because in the case of such locks you can't get to any situation in 
> which the mandatory lock semantics' behavior is different from that of

> the non-mandatory semantics.  In that somewhat strange context, a 
> change in the text would only have to make explicit what is already 
> implicit, but might well confuse people into recalling delegations 
> when they don't need to be recalled.
>
> So my proposed text is:
>
request
MUST
> 	Why is the recall of read delegations necessary if the LOCK
requested 
> is a READ

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 28 Oct 2009 16:06:59 -0700

> set a write lock on a file which it opened for read only.

It won't necessarily have to wait.

If the non-Linux/non-Windows client and server are following
a different semantic such as an advisory exclusive byte lock
is allowed on a file open for READ, then the server will grant the lock
despite the delegation on the Linux client. Then later, when the
Linux client flushes its conflicting byte range shared lock on the file,
the LOCKU will fail because the non-Linux/non-Windows client has the

>>    non-exclusive locks, are sent to the server for resolution.

:-)

This is somewhat theoretical, but there are lots of systems that
don't follow POSIX or Windows (such as the various IBM operating systems),
and possibly their semantics are such that clients and servers that
don't follow the above will get burned.

On Wed, October 28, 2009 3:04 pm, Trond Myklebust wrote:
>> If
>> triggering
>> delegations,

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 28 Oct 2009 19:15:05 -0400

Yes, but that's really because we also ignore the 'rflags' OPEN result.
I should really have us fail all file locking attempts on servers that
do not return the OPEN4_RESULT_LOCKTYPE_POSIX flag on OPEN. We're simply
not prepared to deal with non-posix locking semantics...

Cheers
  Trond

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 28 Oct 2009 16:18:54 -0700

> read of the file; you would need a special program that did locks around

Or the system implicitly locks the range on behalf of the process,
and unlocks it after the I/O is done. That's how System V did it, and
I suspect Windows does it. The NFSv4.0 and NFSv4.1
specs discuss this.

> that they had to be to be obeyed when they were used, not that it was

Yes, we are saying the same thing, provided you add that while the
IO is in progress, another process or client cannot get a conflicting
mandatory lock on the byte range of the I/O.

> Perhaps, or this could have been an oversight.  We don't know.

Agreed. I don't claim to know.

---

From: sfaibish <[email-masked]>
Date: Wed, 28 Oct 2009 20:03:18 -0400

Mike,

> to the server is questionable.
I was looking at the file delegation draft and I have similar concerns. In
fact I calle Peter Honeyman and ask him to take a look and see if he has
same concerns. My intention was just to add this item to 4.2. It is
definitely too big alteration for 4.1.

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Network Storage Group

         EMC²
where information lives

Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Rick Macklem <[email-masked]>
Date: Thu, 29 Oct 2009 11:52:55 -0400 (EDT)

> read of the file; you would need a special program that did locks around
I believe Mike was referring to Sec. 9.3.3 of rfc3530, which requires
clients to hold byte range locks for byte ranges being cached.
--> The I/O is allowed, but can't be cached unless the range is locked
     by the client.

As I noted before, since I client has no way of knowing if a server
is enforcing mandatory locking on a given file at any point in time,
this is all kinda theoretical.

I cover my b... by saying that any mount done against a server that
might be doing mandatory locking needs to be mounted with caching
disabled. (No one will actually do this, but...)

rick

---

From: "Sandeep Joshi" <[email-masked]>
Date: Thu, 29 Oct 2009 09:57:47 -0700

Sandeep>> I had considered Windows locking semantics point but I wasn't sure
about it. Hence, I raised the question.

Sandeep>> I had missed this case.

Sandeep>> Fine with me.

regards
Sandeep

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 29 Oct 2009 13:28:27 -0400

> --> The I/O is allowed, but can't be cached unless the range is locked

There is absolutely no reason for a posix based client to obey that if
it holds a read delegation.

So what if a non-posix client can grab an exclusive lock on the file? It
still can't actually modify the data without causing the delegation to
be recalled first.

Cheers
   Trond

---

From: Rick Macklem <[email-masked]>
Date: Thu, 29 Oct 2009 14:49:48 -0400 (EDT)

> still can't actually modify the data without causing the delegation to
Yep, this makes sense to me. (ie. The server will recall the delegation
before the data is modified/write locked by another client.) If you go
this route, it does imply that the cached data must be invalidated
before the DelegReturn, if mandatory locking is being done by the
server for the file.

Oh, and that isn't what Sec. 9.3.3 says. It makes no mention of
delegations and has "must" in it, so I could simply say, that's
just the way it is.

rick

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 30 Oct 2009 14:56:51 -0400

> lock. That's what the Linux client gets for failing to honor:

I'm not sure this is true...

AFAICS, given that the effect of the LOCK(WRITE_LT) request is to deny
clients the right to read the file, then such a request really MUST
cause an immediate recall of any outstanding read delegations.

If it doesn't, then there is no way for clients that are holding
delegations and caching the data, but that aren't using locking to know
whether or not a lock is in effect. Ignorance being bliss, they will
happily ignore the lock.
While applications that are written for a POSIX environment do not care
one way or the other, the Windows applications running on clients that
hold a read delegation might.

Cheers
  Trond