Subject: Re: [nfsv4] [3530bis] Item #57

From: Trond Myklebust <[email-masked]>
Date: Tue, 01 Mar 2011 07:35:53 -0500

On Mon, 2011-02-28 at 18:36 -0600, Thomas Haynes wrote: 
> Trond presented this issue at the Interim meeting:
> 
> 57) How do we deal with open owners for which some stateids are migrated and some aren't?  Do the sequence numbers fork? 
> 
> Action: Consensus must be reached
> Owner: Tom
> State Accepted, Low
> Target: IETF 80
> 5661: Yes

5661 doesn't have sequence ids on the open owner, since only-once
semantics are managed through the use of sessions instead.

> Anyone have any comments or implementation experience to share here?

I'm thinking that the sequence numbers should probably fork, and that
the server probably wants to make forking and migrating the sequence
numbers atomic with switching on LEASE_MOVED.

IOW: as soon as the original server starts returning LEASE_MOVED, then
the client needs to fork its view of the open owner, and track sequences
on the original server separately from the migrated sequences on the new
server. So for instance, it must assume that each LEASE_MOVED return
will cause it to bump the seqid on the original server without bumping
the corresponding seqid for the same open owner on the new server.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 01 Mar 2011 08:09:26 -0500

> the corresponding seqid for the same open owner on the new server.

That said, while forking semantics can be worked out, I'm not sure that
'merging' semantics can. If the client holds state on 2 filesystems,
sharing open owners on both filesystems, and the server migrates those 2
filesystems one at a time to a new server, how does one deal with
reconciling the two resulting sequence id streams?

IOW: when the second filesystem gets migrated a couple of days after the
migration of the first filesystem completed, what does one do about the
fact that due to forking, one now apparently has 2 sets of sequence ids
for the same open owner and clientid on the new server?

I'm wondering if the conclusion isn't just that a client that supports
migration MUST ensure that any open owner is confined to a single
filesystem. If the server finds itself in the position above, then it
MUST not migrate the state for the conflicting open owners on the second
filesystem: instead it returns NFS4ERR_STALE_STATEID if the client tries
to use those stateids.

Cheers
  Trond

NetApp

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 1 Mar 2011 22:55:22 -0600

Item #35 actually encompasses two issues.

The first issue is allowing OPEN_DOWNGRADE to return NFS4ERR_LOCKS_HELD.

I've read the thread (see below) and made those changes.

The second issue was raised by Dave and concerns OPEN_DOWNGRADE
being able to act like CLOSE and be able to release byte-range locks.

I'm not as willing to make those changes. There was no follow-up
on list and unless there is a need to incorporate those semantics,
I'd prefer not to add them.

Any opposition?

> 35) OPEN_DOWNGRADE and posix byte range
> 
> Trond started this discussion here:
> http://www.ietf.org/mail-archive/web/nfsv4/current/msg08169.html
> 
>   Neither RFC3530, nor RFC5661 appear to list NFS4ERR_LOCKS_HELD as a
>   valid response when the client calls OPEN_DOWNGRADE.
> 
>   The question is: what should the server then do if the NFS client holds
>   a WRITE_LT lock, but then asks for an OPEN_DOWNGRADE to
>   OPEN4_SHARE_ACCESS_READ. I understand that this is sanctioned in Windows
>   server environments, but it should definitely be forbidden in a POSIX
>   environment, and NFS4ERR_LOCKS_HELD would appear to fit the bill...
> 
> Dave's reply:
> 
>   Makes sense to me.
> 
>   18.10.4 of RFC 5661 says:
> 
>      LOCK operations are subject to permission checks and to checks
>      against the access type of the associated file.  However, the
>      specific right and modes required for various types of locks reflect
>      the semantics of the server-exported file system, and are not
>      specified by the protocol.
> 
>   Given that LOCK may exclude locks for write on files not open for write,
>   it makes sense that downgrade whereby a file can become
>   not-open-for-write when it previously was, has to reject a request which
>   would violate the invariant.
> 
>   Sounds like we need a paragraph in DOWNGRADE explaining the issue as
>   well as adding the error to the list.
> 
>   I think there is a another potential issue here.  CLOSE is allowed to
>   (but not required to) release byte-range locks.  I think windows does
>   that.
> 
>   The issue I'm wondering about is whether a server is allowed to release
>   locks for write as a valid response to DOWNGRADE.  This is separate from
>   the issue of LOCKS_HELD, which I agree it needs to be able to do.  I
>   think we also agree that a server may allow the write locks to stay.
>   The interesting question is whether there is a third choice: silently
>   getting rid of the locks that have become inappropriate to the new open
>   mode.
> 
>   When we fix this, we should be clear on what is and isn't allowed.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 02 Mar 2011 00:28:58 -0500

> I'd prefer not to add them.

So, the difficulty with allowing OPEN_DOWNGRADE to release byte range
locks is that (as discussed below) the protocol supports both servers
with POSIX semantics, that require a open for write in order to hold a
WRITE_LT lock, and servers with Windows semantics that don't.

How does the client distinguish between the two semantics so that it
knows whether or not locks have been freed? Clearly, setting the
OPEN4_RESULT_LOCKTYPE_POSIX flag would suffice, since it indicates "the
server's file locking behavior supports the complete set of Posix
locking techniques". So if the server fails to return LOCKS_HELD in this
case, the client knows that there are no remaining WRITE_LT locks held
if it downgraded to SHARE_ACCESS_READ or vice versa.

The problem is that a server that fails to set LOCKTYPE_POSIX could
still fail to support holding READ_LT locks in SHARE_ACCESS_WRITE mode:
it just doesn't set the flag, because it doesn't support the full POSIX
behaviour of upgrading/downgrading locks, and the merging of lock
requests with the same lockowner and overlapping bytes ranges. Such a
server would be forced to return LOCKS_HELD rather than freeing them
automatically so that the client knows what is going on.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 02 Mar 2011 00:30:29 -0500

> automatically so that the client knows what is going on.

Not really. I'm just pointing out that it is possible to set up a server
with these semantics should we so desire...
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 2 Mar 2011 14:05:28 -0600

I'm trying to capture the Third Edge condition,
and I don't see how the client can get in trouble.

The problem statement is:

   The third edge condition needs to be described.  Basically,
   this is the situation in which a client thinks he has
   locks because he rebooted in the middle of reclaim without
   reclaiming those locks.  If the client comes back and
   those locks were taken by another client and then
   released before that final reboot, the client could
   reclaim the locks unaware that it had lost them in
   the interim and could not be guaranteed to continuous
   holding of the locks.

When the client comes back, it has no knowledge of the locks
it held. The whole point of the courtesy lock approach
is that during a network partition, the client remembers
the locks it thinks it has established. If the client
forgets them and the server forgets them, how can the
client try to reclaim them?

Here is the model I see:

1 Client A acquires a lock.

2 Server reboots.

3 Client A issues a RENEW operation, and gets back a NFS4ERR_STALE_CLIENTID.

4 Client A and server experience mutual network partition,
  such that client A is unable to reclaim its lock within
  the grace period.

5 Client A reboots.

6 Server's reclaim grace period ends.  Client A has no locks
  recorded on server.

7 Client B acquires a lock that would have conflicted with
  that of Client A. 

8 Client B releases the lock.

9 Server reboots a second time.

10 Network partition between client A and server heals.

11 Client A issues bleah and gets NFS4ERR_STALE_CLIENTID.

And by bleah, this is the whole problem. It can't issue a RENEW
because the client knows it needs to do a SETCLIENTID. And once
it does that, the server knows it is free to nuke any and all
locks.

What am I missing from the description?

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 2 Mar 2011 14:06:47 -0600

Correction, item 17.

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 2 Mar 2011 14:17:35 -0600

>>  the interim and could not be guaranteed to continuous

Looking at the notes Andy took from the Interim meeting, I see
a difference.

The problem statement from the meeting seems to be:

1 Client A acquires a lock 1.

2 Client A acquires a lock 2.

3 Server reboots.

4 Client A issues a RENEW operation, and gets back a NFS4ERR_STALE_CLIENTID.

5 Client A reclaims lock 1.

6 Client A and server experience mutual network partition,
  such that client A is unable to reclaim its lock 2 within
  the grace period.

7 Server's reclaim grace period ends.  Client A has no locks
  recorded on server.

8 Server reboots a second time.

9 Network partition between client A and server heals.

10 Client A issues a RENEW opertion and gets back a NFS4ERR_STALE_CLIENTID.

11 Client A reclaims its lock within the server's grace period.

Trond claims that Linux detects it has reclaimed some locks and not all of them.
It assumes that those not reclaimed have been revoked.

So the point here is that if you only reclaim some, you have to be conservative
and assume the remaining are revoked.

I.e., at step 11, the client could have easily reclaimed both locks in this case
(no reuse from client B), but the safest thing to do is assume that a lack of
communication here means it no longer can try to reclaim lock 2.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 2 Mar 2011 19:52:23 -0500

> 7 Server's reclaim grace period ends.  Client A has no locks

I'm not sure what "Client A has no locks recorded on server" means.

At this point, as far as the server can tell, client A has recovered
just fine.  So it should have recorded in stable storage that client A
still holds some state.  (It may not record exactly which state.)

> 8 Server reboots a second time.

And this happens before the server revokes client A's state, so the
server still has an unexpired record for client A in stable storage.

> communication here means it no longer can try to reclaim lock 2.

Yes, the server's required to know whether the client can reclaim
anything at all (to solve the original "edge conditions"), so if enough
happened between steps 8 and 9 that client A had *all* its state
revoked, then the server can take care of preventing the reclaims.

But the server isn't required to know exactly which locks the client can
safely reclaim.  So in cases like the above it's up to the client to
only attempt reclaim of those locks that it knows it held the last time
it talked to the server.

--b.

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 2 Mar 2011 22:10:10 -0600

> I'm not sure what "Client A has no locks recorded on server" means.

It means the same as it does in the other two edge cases. If the
server had not rebooted, then the server would be keeping state
as a courtesy lock.

> still holds some state.  (It may not record exactly which state.)

I think you are jumping the gun with the talk about stable storage,
which admittedly is the point from the other two edge conditions.

> only attempt reclaim of those locks that it knows it held the last time

Which is the whole point of this third edge condition.

I've pushed the changes to github, so anyone can read this in draft-08,
section 9.6.3, pages 125 and 126.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 4 Mar 2011 22:01:24 -0500

> server had not rebooted, then the server would be keeping state

Apologies, I still don't understand.  I mean, I literally don't
understand what the words "Client A has no locks recorded on the server"
mean.

I thought maybe they meant "the server does not believe that client A
holds any locks".  But at this point, that's clearly not true, since
client A just reclaimed lock 1, and the grace period only just ended so
there hasn't been time for it to expire yet.

Help!

--b.

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 4 Mar 2011 23:25:46 -0600

> understand what the words "Client A has no locks recorded on the server"

In the context of that section, it means that the server has no
stable storage record of the locks from Client A. 

The point of the first two edge cases is that if the server wants to
provide "Courtesy Locks" and survive a server reboot and network
partition, then it can use stable storage to record the locks.

The three examples are working up to a best practices for
implementation.

a) If you don't want to do courtesy locks, when the lease expires, throw them away,

b) If you do want to do courtesy locks, when the lease expires, mark them as
courtesy. If another client tries to access such a lock, allow it to do so. When
the original tries to access it, report an error then.

c) If you do want to do courtesy locks and survive a server reboot coupled with
a network partition, then record some state for the locks in stable storage.

So in the edge cases, b applies and during the reboot/partition, the server
looses the fact that client A had locks.

Once we embrace c, the server would not lose the fact that client A had locks.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 9 Mar 2011 17:00:30 -0500

> stable storage record of the locks from Client A. 

That might be clearer if it was moved to the "server reboots" step.

Also, the first two both provide examples of what could go *wrong* if a
client or server is implemented incorrectly.  This one seems to show
only safe behavior.

I think it would be clearer to stick with the convention of the previous
to edge conditions and show what can actually go wrong (as in the
situation you describe in the last paragraph of the this section).
Something like this?:

--b.

diff --git a/nfsv4_middle_file_locking.xml b/nfsv4_middle_file_locking.xml
index 42bd331..4690b02 100644
--- a/nfsv4_middle_file_locking.xml
+++ b/nfsv4_middle_file_locking.xml
@@ -1790,10 +1790,14 @@
               </t>
 
               <t>
-                Server's reclaim grace period ends.  Client A has no locks
-                recorded on server.
+                Server's reclaim grace period ends.
               </t>
 
+	      <t>
+	        Client B acquires and releases a lock that would have
+		conflicted with Client A's lock 2.
+	      </t>
+
               <t>
                 Server reboots a second time.
               </t>
@@ -1807,25 +1811,40 @@
                 NFS4ERR_STALE_CLIENTID.
               </t>
 
+	      <t>
+	        Client A issues a new SETCLIENTID and
+		SETCLIENTID_CONFIRM.
+	      </t>
+
               <t>
-                Client A reclaims its lock 1 within the server's grace period.
+                Client A reclaims both lock 1 and lock 2 within the server's
+		grace period.
               </t>
             </list>
           </t>
 
           <t>
-            During the partition, client A decided that the server had revoked
-            lock 2. After the partition, it was able to reclaim lock 1, but
-            made no attempt to reclaim lock 2. After the grace period, it is
-            free to try to reestablish lock 2 via LOCK operations.
-          </t>
+	    At the last step, the client reclaims a lock as if it had
+	    held that lock continuously, when in fact a conflicting lock
+	    was granted to client B.
+	  </t>
+
+	  <t>
+	    A server could avoid this situation by rejecting the reclaim
+	    of lock 2.  However, to do so accurately it would have to
+	    ensure that additional information about individual locks
+	    held survives reboot.  Server implementations are not
+	    required to do that, so the client must not assume that the
+	    server will.
+	  </t>
+
+	  <t>
+            Instead, a client MUST reclaim only those locks which it
+	    succesfully acquired from the previous server, omitting any
+	    that it failed to reclaim before a new reboot.  Thus, in the
+	    last step above, client A should reclaim only lock 1.
+	  </t>
 
-          <t>
-            Note that the other two edge conditions are able to interact with this
-            third edge condition. Another client B may have established a conflicting
-            lock during the partition, made some changes, and the released the
-            lock before the second server reboot.
-          </t>
         </section>
 
         <section title="Client's Handling of NFS4ERR_NO_GRACE">