Subject: RE: [nfsv4] CB_NOTIFY_LOCK

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 21 Aug 2006 14:35:03 -0400

I agree that this is brilliant but ...

I think it needs some sort of indication by the server that it does
support this.  It doesn't have to be an absolute promise (so the 
client does have to poll), but clients are going to want to poll
at different frequencies for a server that does have this support
vs. one that has no such support.

In the absence of support for this (in v4.0) a client is going to 
have to poll quite frequently or else the latency to get contended 
locks is liable to be unacceptable.  If this is in v4.1, with no
indication that particular server supports it, the client would 
still have to poll with that high frequency, in which case the 
benefit is limited.  On the other habnd, if you knew there was 
support, the poll cycle could be made considerably longer.  You
would still have a polling as a backup (since the server is not
promising the callback under every possible circumstance) but the
fact that you had support would allow infrequent polling with 
good average latency, although not necessarily with good 
absolutely-worst-case latency.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Wednesday, August 16, 2006 7:04 PM
To: [email-masked]; [email-masked]
Subject: [nfsv4] CB_NOTIFY_LOCK

And, while we're at it, just for fun:  Here's a CB_NOTIFY_LOCK that's a
hint to the client that now might be a good time to retry a lock that
it's blocking on.

Why is this so brilliant?  Well!:

	- It gives cooperating clients the chance to acquire locks
immediately
	  after they're released, without having to set ridiculously
small
	  polling intervals.
	- It doesn't actually modify any locking state on its own, so
	  it has none of the raciness and annoying sequencing
requirements of
	  a real grant callback.
	- It's entirely optional--servers can omit it, clients can
	  ignore it.
	- It's simple to specify and to implement.

Neat-o.

--b.

22.11.  Operation 13: CB_NOTIFY_LOCK - Notify of possible lock
        availability

22.11.1.  SYNOPSIS

   fh, lockowner -> ()

22.11.2.  ARGUMENT

   struct CB_NOTIFY_LOCK4args {
       nfs_fh4     file;
       lock_owner4 lock_owner;
   }

22.11.3.  RESULT

   struct CB_NOTIFY_LOCK4res {
           nfsstat4        status;
   };

22.11.4.  DESCRIPTION

   The server may use this operation to indicate that a lock for the
   given file and lockowner may have become available.

   This callback is meant to be used by servers to help reduce the
   latency of blocking locks in the case where they recognize that a
   client which has been polling for a blocking lock may now be able to
   acquire the lock.  The notification is purely a hint, provided as a
   possible performance optimization, and is not required for
   correctness.

22.11.5.  IMPLEMENTATION

   The server must not grant the lock to the client unless and until it
   recieves an actual lock request from the client.  Similarly, the
   client receiving this callback cannot assume that it now has the
   lock, or that a subsequent request for the lock will be succesful.

   The server is not required to implement this callback, and even if it
   does, it is not required to use it in any particular case.  Therefore
   the client must still rely on polling for blocking locks, as
   described in the "Blocking Locks" section.

   Similarly, the client is not required to implement this callback, and
   even it does, is still free to ignore it.  Therefore the server must
   not assume that the client will act based on the callback.

---
 index.html                          |    1 +
 nfsv41_middle_op_cb_notify_lock.xml |   61
+++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+), 0 deletions(-)

diff --git a/index.html b/index.html
new file mode 100644
index 0000000..b9322ff
--- /dev/null
+++ b/nfsv41_middle_op_cb_notify_lock.xml
@@ -0,0 +1,61 @@
+<!-- Copyright (C) The Internet Society (2006) -->
+<section anchor="OP_CB_NOTIFY_LOCK" title="Operation 13: CB_NOTIFY_LOCK
- Notify of possible lock availability" >
+  <section toc="exclude" anchor="OP_CB_NOTIFY_LOCK_SYNOPSIS"
title="SYNOPSIS">
+    <figure>
+      <artwork>
+fh, lockowner -> ()
+      </artwork>
+    </figure>
+  </section>
+  <section toc="exclude" anchor="OP_CB_NOTIFY_LOCK_ARGUMENT"
title="ARGUMENT">
+    <figure>
+      <artwork>
+struct CB_NOTIFY_LOCK4args {
+    nfs_fh4     file;
+    lock_owner4 lock_owner;
+}
+      </artwork>
+    </figure>
+  </section>
+  <section toc="exclude" anchor="OP_CB_NOTIFY_LOCK_RESULT"
title="RESULT">
+    <figure>
+      <artwork>
+struct CB_NOTIFY_LOCK4res {
+        nfsstat4        status;
+};
+      </artwork>
+    </figure>
+  </section>
+  <section toc="exclude" anchor="OP_CB_NOTIFY_LOCK_DESCRIPTION"
title="DESCRIPTION">
+    <t>
+     The server may use this operation to indicate that a lock for the
given
+     file and lockowner may have become available.
+    </t>
+    <t>
+     This callback is meant to be used by servers to help reduce the
latency of
+     blocking locks in the case where they recognize that a client
which has
+     been polling for a blocking lock may now be able to acquire the
lock.
+     The notification is purely a hint, provided as a possible
performance
+     optimization, and is not required for correctness.
+    </t>
+  </section>
+  <section toc="exclude" anchor="OP_CB_NOTIFY_LOCK_IMPLEMENTATION"
title="IMPLEMENTATION">
+    <t>
+     The server must not grant the lock to the client unless and until
it
+     recieves an actual lock request from the client.  Similarly, the
client
+     receiving this callback cannot assume that it now has the lock, or
that a
+     subsequent request for the lock will be succesful.
+    </t>
+    <t>
+     The server is not required to implement this callback, and even if
it
+     does, it is not required to use it in any particular case.
Therefore the
+     client must still rely on polling for blocking locks, as described
in the
+     "Blocking Locks" section.
+    </t>
+    <t>
+     Similarly, the client is not required to implement this callback,
and even
+     it does, is still free to ignore it.  Therefore the server must
not assume
+     that the client will act based on the callback.
+    </t>
+  </section>
+</section>
-- 
1.4.1.rc2

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 21 Aug 2006 14:31:01 -0500

Is highly coordinated file-locking a goal for NFS or NFSv4.1 specifically.

It has been my impression that application developers and sys admins
have been sufficiently trained by the NFS locking ghosts of the past
that NFS file locking is avoided at most if not all costs.  Do we need
to fix that now or do we believe that the application developer
or sys admin will be sufficiently convinced that things have changed
that they can count on NFSv4.1 file locking support?

Spencer

---

From: Marc Eshel <[email-masked]>
Date: Mon, 21 Aug 2006 16:07:21 -0400

As far as I can tell there are clients/application that are dependent on 
NFS locking (with all its problems) and we must fix to the best of our 
ability as soon as possible.
Marc.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 21 Aug 2006 16:49:02 -0400

> Is highly coordinated file-locking a goal for NFS or NFSv4.1 specifically.

I'm not sure what highly coordinated file-locking is.  I think it'd be
nice to support locking without requiring long delays to acquire
contended locks, though, yes.

> that NFS file locking is avoided at most if not all costs.
> Do we need to fix that now or do we believe that the application
			     ^^
I'm assuming you mean "and" there.

> developer or sys admin will be sufficiently convinced that things have
> changed that they can count on NFSv4.1 file locking support?

So far, we've:

	- enabled local caching of locks using delegations,
	- added support for windows share locks,
	- added sequencing, etc., to fix locking races, and
	- enabled mandatory locking by adding stateid's to every read and
	  write.

That suggests a certain level of interest both in making life easier for
existing users of locking and in expanding the sorts of locking users we
can support.

Compared to that stuff, CB_NOTIFY_LOCK is trivial to implement.  And it
affects a core locking feature (advisory byte-range locking) that people
do seem to use.

--b.

---

From: Eric Werme USG <[email-masked]>
Date: Mon, 21 Aug 2006 16:51:34 -0400

Spencer asked:

I don't know about "highly coordinated," but I think "working" was a
major goal given the amount time spent on that part of the protocol.
IIRC, the first attempts were concerned about efficiency, but people
retreated from that due to problems of issuing callbacks through
firewalls.  I'm not entirely sure why no one wrings their hands about
CB_NOTIFY_LOCK and firewalls, but I assume that FTP and HTTP remain
the way most files are transferred between DNS domains and firewalls
aren't a huge concern.  (We did have a customer who recently enabled
a firewall on a server while some clients were using it.  Sigh.)

I disagree.  Most of us may still be haunted, but we had customers who
thought it worked ten years ago!  In a local web page, I wrote:

  Late in 1996 locking problems at a large university convinced me that it
  was time for me to fix rpc.lockd. The majority of the work dealt with
  recovering from lost NLM requests or responses. Soon after the bug report
  was closed, AOL reported locking problems and I was very, very, glad 
  we spent the time we did when we did.

At the time, AOL was growing and DEC's business with them was growing faster.
Times change, huh?

Internally we used to use NFS locking for all our mail files, but with
the advent of Tru64 V5.0 and "real" clusters, sendmail only uses NFS for
.forward files and whatnot.  Mail clients still use NFS locking for
reading mail files, but most people use IMAP these days.  I still use
xmh here.  Hey, I use the thoroughly modern exmh at home.  :-)

   ... Do we need

You guys know more about the V4 market than I do, but I think
the majority of our customers assume that locking works with NFS
V2, V3, or our cluster FS.

	-Ric Werme

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 21 Aug 2006 15:44:43 -0400

> that they can count on NFSv4.1 file locking support?

Can we drop support for write delegations too? One of their few real
performance advantages lies in allowing us to cache byte range locks.
You appear to be saying that this is all unnecessary now...

   Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 21 Aug 2006 16:26:39 -0400

> vs. one that has no such support.

They could just default to their 4.0 policy until they actually see a
NOTIFY_LOCK callback, and then adjust accordingly.

It'd be trivial to add a single-bit server-wide "NOTIFY_LOCK support"
attribute (or even an additional bit for the OPEN rflags result, to give
the server the ability to hint differently depending on client,
filesystem, etc.?), but we might be getting to the point of diminishing
returns--all you'd be preventing would be slightly over-agressive
polling the first time a new client hit a contended lock.

--b.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 21 Aug 2006 15:41:48 -0700 (PDT)

> > that they can count on NFSv4.1 file locking support?

We know that adding Kerberos to NFSv4 caused
customers to trust NFSv4 security, so we know that we can
fix reputations. So if people aren't using blocking locking
because it is broken on most NLM setups, we can fix that.

The question I have is what has changed recently to require this
optimization? Late breaking features will keep coming, that's OK,
but what's the justification?

I'd always been under the impression that people who wanted to
do high contention apps used databases instead of POSIX byte range
locking, whether to local or remote file systems.

> Can we drop support for write delegations too? One of their few real

This is interesting. Why aren't write delegations advantageous for
WRITEs?

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 21 Aug 2006 18:27:32 -0500

> contended locks, though, yes.

I meant "highly coordinated" to refer to a class of applications where
byte-range locks were in heavy use and there is a reasonable probability
that there will be contention resulting in the kind of polling and
delay to which you refer.

> affects a core locking feature (advisory byte-range locking) that people

Mike captured my intent; I am trying to have the dialog about requirements
given where we are with the development of NFSv4.1.  Attempting to
elicit the discussion around the addition of new features: what is the
impact of solving a particular issue? is it required? nice to have?
is is something that everyone "just wants", etc.

My apologies if the intent come across as I had hoped.

Spencer

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 22 Aug 2006 12:01:01 -0400

> delay to which you refer.
....
> is is something that everyone "just wants", etc.

Sure, fair enough.

I don't have the data to prove that this would be widely implemented.
My motivations are just that it clearly *could* make a noticeable
improvement for certain workloads, and that it's simple and optional
enough to make adding it to the protocol a reasonable risk.

I figured grant callbacks had been avoided in the past because of the
obvious difficulties involved in getting them right, but something a
little more light-weight might hit a better balance.

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 22 Aug 2006 15:45:12 -0400

> This is interesting. Why aren't write delegations advantageous for

Write delegations don't really allow you to go beyond the existing
close-to-open caching model. Pretty much all they do is to allow the
caching of OPENs, and locks.
If we don't need to cache locks, then a read delegation will suffice to
allow you to cache the OPENs...

Cheers,
  Trond