Subject: [nfsv4] cancelling blocking locks

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 9 Mar 2006 15:00:27 -0500

The following recommendation from rfc3530 is problematic:

	"The server should maintain an ordered list of pending blocking
	locks.  When the conflicting lock is released, the server may
	wait the lease period for the first waiting client to re-request
	the lock.  After the lease period expires the next waiting
	client request is allowed the lock."

There are two problems for servers implementing this recommendation.

	1. When the conflicting lock is released, the server assumes all
	clients that it has denied conflicting blocking locks to are
	still polling for that lock, even when some may not be.  This
	may cause unnecessary delays of up to a lease period.

	2. Most NFSv4 implementations depend on a general-purpose
	backend filesystem which may be accessible by other protocols
	and by applications on the server, and it may be difficult to
	block those other users without actually applying the blocking
	lock in question.  But applying such a lock may be an
	irreversible process if, for example, it downgrades an existing
	lock.

A new "cancel" operation might solve problem 1, but not problem 2.

Instead for 4.1 I'm proposing that we allow a server to use receipt of a
non-blocking LOCK operation as an indication that the client is no
longer polling for the given lock.  We would also require clients to end
polling in this way, so that we can distinguish clients that are still
polling from those that aren't.

Aside from telling the server that the client is no longer polling, this
LOCK request wouldn't be special in any way, and a server would process
it normally, possibly granting it in the process.

By using a regular LOCK operation for this purpose we'd solve the
problems above without introducing a new operation with new semantics,
avoiding the sort of races common to NLM implementations.

In more detail:  If an NFSv4.1 client performs a LOCK request of type
READW_LT or WRITEW_LT, it MUST continue to poll for the lock, using
subsequent LOCK requests with the same offset, range, and owner, and
type, until either:
 
	1. it recieves a succesful response, or any response with status
	other than NFS4ERR_DENIED or NFS4ERR_DELAY; or

	2. it sends a LOCK request of the same offset, range, and owner,
	and with type READ_LT (for a lock originally issued with
	READW_LT) or WRITE_LT (for a lock originally issued with
	WRITEW_LT), and receives any response other than NFS4ERR_DELAY;
	or

	3. It performs a succesful RELEASE_LOCKOWNER for the lockowner in
	question.

(Actually, non-sequenceid-modifying errors should, like DELAY, be
treated as having no effect; but they probably shouldn't occur between
correctly functioning clients and servers, with the exception of STALE
errors, which the client should take to mean that the client has
forgotten all blocking locks.)

The server may maintain a list of pending blocked locks as described in
the excerpt from rfc3530 above.  When it recieves a matching lock
request of type READ_LT or WRITE_LT from a polling client, if the lock
is still not granted, it may discard the pending blocked lock and return
DENIED.  If the lock has been granted, the server should process the
request normally and return success.  If the client no longer wants the
lock, the client must follow up with a LOCKU or RELEASE_LOCKOWNER.

As in rfc3530, the server is still not *required* to maintain the list
of pending blocked locks.

Opinions?--b.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Thu, 9 Mar 2006 12:33:29 -0800 (PST)

> 	may cause unnecessary delays of up to a lease period.

True but this isn't a new problem. Blocking locks caused so many problems
in NLM, that we've decided this problem is acceptable. 

> 	lock in question.  But applying such a lock may be an

Why is it difficult to do this without applying the blocking lock?
It just needs a new style of lock in the server's lock manager.

Are you are saying that the server as a different lock manager for each
file system type? 

> it normally, possibly granting it in the process.

Won't granting the lock have possible side effects? 

> READW_LT or WRITEW_LT, it MUST continue to poll for the lock, using

This is what kills us in NLM-land.
Clients that don't cancel pending blocking lock requests. By adding that
MUST you put NFSv4 locking back into the same problem area.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 09 Mar 2006 15:45:01 -0500

> MUST you put NFSv4 locking back into the same problem area.

It ought still to be possible for the server to fall back on the current
algorithm, which states that if the client doesn't claim the lock within
1 lease period, then it will fall to the next entry on the blocking
list.
Would changing the above MUST to a SHOULD, and then adding some
explanatory text help?

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Thu, 9 Mar 2006 13:00:53 -0800 (PST)

> 1 lease period, then it will fall to the next entry on the blocking

BTW, RFC 3530 does not mandate a claim within the current lease period.
It suggests the lease period, but there's no contract that the lease
period is required by the client or enforced by the server.

> Would changing the above MUST to a SHOULD, and then adding some

OPTIONAL at best since there's no operational environmental reason why a server
couldn't implement this, and that is usually why MUSTs become SHOULDs. 
It is just a bad idea to depend on clients to cancel locks.

Also it should be clear that the lock owners 
need to match. Process B on the client should not get 
to cancel process A's pending lock on the same client.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 09 Mar 2006 16:37:36 -0500

>From section 8.4:

                                         When the conflicting lock
   is released, the server may wait the lease period for the first
   waiting client to re-request the lock.  After the lease period
   expires the next waiting client request is allowed the lock.

That may not be a legally binding contract, but it is a rather strong
suggestion to the ears of this client developer.

> It is just a bad idea to depend on clients to cancel locks.

Agreed, but allowing them to do so might make the whole fair locking
scenario feasible instead of being an option that no server vendor ever
plans to implement.
In a mixed NFS environment, being the only kid with no option to block
will seriously bias the chances against you ever getting a lock.

Cheers,
  Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 9 Mar 2006 15:54:18 -0500

> It just needs a new style of lock in the server's lock manager.

Maybe so.

> Are you are saying that the server as a different lock manager for each

Well, distributed filesystems, for example, may manage locks on their
own--so changes to the vfs locking interface can require filesystem
changes.  Not that that's impossible.

> Won't granting the lock have possible side effects?

Like what?

> MUST you put NFSv4 locking back into the same problem area.

Is there some technical reason clients can't be counted on to do that?
It strikes me that this MUST is if anything quite a bit easier than
dealing with the requirement that every LOCK by UNLOCK'd and every OPEN
by CLOSE'd....

My impression was that the main problem with NLM was rather that it was
underspecified--e.g. do you unlock on CANCEL?  And what does the error
return from CANCEL mean?  By using a non-blocking LOCK as the way to
indicate the end of polling we make the semantics pretty clear.

In any case I'm not preventing servers from cancelling locks on their
own, any time they want to.  I could add text recommending that they
continue to do so whenever a client doesn't poll within a lease period,
as in 4.0, but I assumed that was obvious.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 13 Mar 2006 09:31:50 -0500

OK, so I don't seem to have convinced anyone of the need for mandatory
lock cancelling, so I'll give up and change the MUST to a SHOULD.

The resulting algorithm has the advantage that it's completely possible
for a v4.0 client or server to implement--if the other side has done the
same, it'll help, if not, it's harmless.

So the basic idea is still to allow a client to signal that it is done
polling by sending a non-blocking lock with the same parameters as the
original blocking lock.

The overly detailed version:

If a client performs a LOCK request of type READW_LT or WRITEW_LT, it
SHOULD continue to poll for the lock, using subsequent LOCK requests
with the same offset, range, and owner, and type, until either:

	3. it performs a succesful RELEASE_LOCKOWNER for the lockowner in
	question, or a CLOSE for the open from which the lock state is
	derived.

The server may maintain a list of pending blocked locks as described in
the excerpt from rfc3530 above.  When it recieves a matching lock
request of type READ_LT or WRITE_LT from a polling client, if the lock
is still not granted, it may discard the pending blocked lock and return
DENIED.  If the lock has been granted, the server should process the
request normally and return success, and should not assume that the
client intends to unlock--the client can follow up with a LOCKU if that
is the case.

As in rfc3530, the server is still not *required* to maintain the list
of pending blocked locks.  It also cannot assume that the client will
faithfully cancel all blocking locks, so should be prepared to cancel
on its own any lock that is not polled for within a lease period.

Opinions?--b.

---

From: Marc Eshel <[email-masked]>
Date: Mon, 13 Mar 2006 09:27:31 -0800

This is a reasonable suggestion since we are about to finalize NFSv4.1 and 
we can not make any major changes at this point. I have a question about 
the current version of NFSv4. Why did we not use callback (when possible) 
to notify the client that the lock was granted. NFSv4 added callbacks for 
delegation but for locking which uses call back in its NLM implementation 
it was dropped for some reason. Maybe we can add it in 4.2.
Marc.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 14 Mar 2006 07:54:02 -0500

Ow ow ow. Moving lock grants to the asynchronous callback channel would
explode server-side resources, introduce cancellation races, and unless we
moved all locking, would mean two ways to complete each lock op.

Much as I dislike the blocking lock operations, they do behave in a consistent
manner. Let's think through the consequences of callback locking, before
going there. Going "back" the dark ages of NLM does not sound to me like a
good example, in any case.

Tom.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 14 Mar 2006 10:44:33 -0500

> moved all locking, would mean two ways to complete each lock op.

One dead-simple way to avoid all these problems would be to make the new
grant callback a (possibly optional) hint: "I'm calling back to let you
know that someone just released this lock, and that I'd probably give it
to you if you polled for it again soon."  Maybe something like
NOTIFY_LOCK_AVAILABLE would be a better name than GRANT.

That might be slightly less efficient, but it would avoid all the
complicated races, since LOCK operations would still be the only
operations that actually changed the lock state.

--b.

---

From: Eric Werme USG <[email-masked]>
Date: Tue, 14 Mar 2006 10:54:31 -0500

There are a lot of hardware and software systems that work quite well
with a "doorbell" interrupt that says little more than "Hey, there's something
you can do."  In this case, it could mean "poll me - it might be worth your
time."  There are lotsa races, but they're all the sort of "Sorry, you already
got it" and hence have no material impact other than a little wasted cycles
and bandwidth.  That could be made up for by increasing the poll timeout a
little, but the doorbell could allow a big increase in the poll timeout.

	-Ric Werme

(who really doesn't know what he's replying to, but what the heck.)

---

From: Marc Eshel <[email-masked]>
Date: Tue, 14 Mar 2006 09:15:21 -0800

Although I raised a more controversial issue I did not mean to divert the 
discussion from Bruce's suggestion which should be considered for NFSv4.1. 
The callback option should wait for 4.2, I hope that we will be able to 
increase the speed on new versions. Like Bruce and Eric suggested there 
are way to avoid the race condition. I think that it is unacceptable that 
an available lock will not be granted immediately.
Marc. 

"Talpey, Thomas" <[email-masked]> wrote on 03/14/2006 04:54:02

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 20 Mar 2006 17:37:11 -0500

> polling by sending a non-blocking lock with the same parameters as the

Any objections to the basic idea?  If not, I assume we'd need a revised
"Blocking Locks" section?

Also, is there any interest at all in the optional "notify lock is
available" callback?  (Which would be only a hint, not a lock grant)

--b.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 20 Mar 2006 15:11:27 -0800 (PST)

> "Blocking Locks" section?

Bruce, 

Personally I see rough consensus to do this, as a SHOULD or MAY.

I've created an issue for it:

http://www.nfsv4-editor.org/cgi-bin/roundup/nfsv4/issue11

I suggest assigning  yourself the owner, declare consensus, and draft some text
for Spencer, and then re-assign the issue to Spencer.

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 27 Mar 2006 17:23:20 -0600

> "Blocking Locks" section?

Follow Mike's suggestions.

> available" callback?  (Which would be only a hint, not a lock grant)

I have seen a lack in interest to pursue this however.

Spencer