Subject: [nfsv4] Additional migration related issues in 3530

From: Bill Baker <[email-masked]>
Date: Mon, 21 Mar 2011 18:13:51 -0500

Greetings,

As I presented during Connectathon, we have a prototype implementation
of server (and client) migration.  As our implementation proceeds, we
we continue to discover interesting issues with 3530, which may warrant
clarification in the 3530bis.

I've attached two files describing the troublesome scenarios.  Comments
welcome.

Regards,
-- 
Bill Baker - Oracle Solaris NFS development

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 21 Mar 2011 20:14:05 -0500

> My initial reaction to this was that DELAY ought to be in the list of
> errors which did not increment the sequence ID.  However, some off-list
> discussion with Trond and Bruce suggested this had been previously rejected.
> Can anyone provide some rationale why DELAY error, returned to, say, a
> LOCK request, should also bump the lock owner's sequence ID?

Page 122 of 3530bis, Section 9.1.6 states:

   It is critical the server maintain the last response sent to the
   client to provide a more reliable cache of duplicate non-idempotent
   requests than that of the traditional cache described in [34].  The
   traditional duplicate request cache uses a least recently used
   algorithm for removing unneeded requests.  However, the last lock
   request and response on a given lock_owner must be cached as long as
   the lock state exists on the server.

   The client MUST monotonically increment the sequence number for the
   CLOSE, LOCK, LOCKU, OPEN, OPEN_CONFIRM, and OPEN_DOWNGRADE
   operations.  This is true even in the event that the previous
   operation that used the sequence number received an error.  The only
   exception to this rule is if the previous operation received one of
   the following errors: NFS4ERR_STALE_CLIENTID, NFS4ERR_STALE_STATEID,
   NFS4ERR_BAD_STATEID, NFS4ERR_BAD_SEQID, NFS4ERR_BADXDR,
   NFS4ERR_RESOURCE, NFS4ERR_NOFILEHANDLE, or NFS4ERR_MOVED.

I.e., the reason you have to bump the sequence ID on NFS4ERR_DELAY is that
in the case migration is not occurring, the server might have a quality of service
constraint that it will reply to a COMPOUND within a set amount of time. It does
so, with the NFS4ERR_DELAY. As soon as that occurs, the operation finishes
(say it was a slow disk IO) and the system decides to record that fact in the
duplicate non-idempotent cache. I.e., if it was slow this time, it will be slow the
next time.

If the client reuses the seqid because it was not bumped, what happens? The
cache is no longer reliable.

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 21 Mar 2011 20:44:15 -0500

> The issue we ran into was ambiguity in the server in interpreting
> SETCLIENTID4args after multiple migrations and server reboots.  Here
> is the troublesome scenario.
> 
> Client mounts server S1
> Client accesses FS1 and FS2 at S1
> FS1 is migrated to server S2
> Client updates callback info
> 
> S1 reboots
> Client re-establishes client id to S1
>     (same long form, same verifier)
> S1 generates new short form client ID
> FS2 is migrated to to S2
> Client updates callback info on S2
> 
> At this point, the server cannot differentiate between the two
> different short forms present for a single long form.  At this
> point, our server has 2 distinct leases with 2 different short
> form client IDs in play, in part due to forking of open owners and
> lock owners as described in bis issue #57.

The spec does provide some guidance here, if you squint your eyes
a little.

What happens in this scenario:

Client mounts server S1
Client access FS1 at S1
FS1 is migrated to server S2
Both long form and verifier match, but the short form client IDs clash?

I.e., S2 already is holding state for the client and the verifiers match.
Bad design? Sure. Possible? Sure.

See Section 7.7.6 on page 89 of 3530bis for how the spec wants us
to handle this scenario.

   In a file system transition, the client needs to handle cases in
   which the two servers have cooperated in state management and in
   which they have not.  Cooperation by two servers in state management
   requires coordination of client IDs.  Before the client attempts to
   use a client ID associated with one server in a request to the server
   of the other file system, it must eliminate the possibility that two
   non-cooperating servers have assigned the same client ID by accident.

Using the intent of that section, if during the migration you take care to
transfer the client ID (long, short, and verifier) as part of the state of
the migration event, can't S2 identify the conflict and throw away the
state associated with FS2?

---

From: Bill Baker <[email-masked]>
Date: Tue, 22 Mar 2011 12:42:40 -0500

> cache is no longer reliable.

Ok, so the server is returning DELAY and continuing to execute the OP
asynchronously, but then caching the (eventual) response in the open
owner or lock owner reply cache?

Let's say it's a lock request.  The client gets DELAY and increments the
lock owner sequence ID.  It's still interested in the lock and re-sends
the lock request, but with a sequence ID of N+1.  But, N+1 doesn't look
like a replay of N, it looks like a new request.  If the server did
cache the (eventual) response with sequence ID of N, then it cannot
reply with it here.  Are you saying the server pre-caches this result
with a sequence ID of N+1?

It seems to me that if a server acted as you described, you want
the client to NOT increment the sequence ID, so that when it sent the
request again (N), and the server had the completed response (also N),
the server would reply with the asynchronously formed response, as if
it were a replay.

-- 
Bill Baker, Oracle Solaris NFS development

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 22 Mar 2011 14:08:44 -0500

> owner or lock owner reply cache?

I'm not saying that. The server has to cache that it sent back the NFS4ERR_DELAY:

   The server maintains the last sequence number (L) received and the response that
   was returned. 

> with a sequence ID of N+1?

I'm detecting serious hand-waving in all the responses I want to give
here, so reader beware.

If a server allowed a LOCK to be granted after it sent the NFS4ERR_DELAY,
then it would need to record the client did get the byte-range. As shown
above, it would also record that at N, it sent back the NFS4ERR_DELAY.

Then when it got the same request at N+1, it would say, I already did this,
did not tell you, and so now I will tell you.

I doubt any implementation does it this way.

We should depart this hand-waving and skip down to a potential solution.

> the server would reply with the asynchronously formed response, as if

So the real issue here is that NFS4ERR_DELAY is overloaded - you want
it to behave differently if the intent is migration versus any other valid reason.
If we were implementing migration back when we were still crafting the
spec, we could add NFS4ERR_IN_MIGRATION and the client would know
not to bump the seqid. Since we can't do that, can we help the client detect
that the NFS4ERR_DELAY is in response to a migration event?

If the client gets NFS4ERR_DELAY on one of these operations, it could start
caching the original seqid. Once the server comes back with either a succesful
operation or a non-NFS4ERR_MOVED error, it could throw away that
seqid.

If the server came back for NFS4ERR_MOVED for that filesystem, the client
could then deduce the earlier NFS4ERR_DELAY were due to a migration
event and once it started connecting to the new server, it could rewind the
seqid to the cached value.

For NFSv4.2, we can add NFS4ERR_IN_MIGRATION to get rid of this extra
burden we have placed on the client.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 22 Mar 2011 15:49:38 -0400

> operation or a non-NFS4ERR_MOVED error, it could throw away that

I really don't like this kind of fuzzy semantics.

> seqid to the cached value.

Quite frankly, if we must have a 'wait, I'm in the process of migrating'
error, then I'd prefer to add a new error value for it (or possibly make
the case for returning NFS4ERR_RESOURCE - distasteful though that may
be).

The above proposal will just end up breaking existing NFSv4 clients that
make use of the (specwise legitimate) property that open owners can be
reused across filesystem boundaries.
IOW: if I assume that NFS4ERR_DELAY has the semantics that it has today,
and I do OPEN on a volume that is being migrated, then I may not be able
to deal with the final NFS4ERR_MOVED, but that should not break my other
OPEN state on any other filesystems.

> burden we have placed on the client.

Where is the need? The only sequence ids we have in NFSv4.1 and newer
are the sequence ids in the session which I can't see why we would want
to migrate anyway: the session is just a replay cache.

Trond

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 22 Mar 2011 15:31:27 -0500

> reused across filesystem boundaries.

I thought we already addressed this issue for migration as a solution for item 57:

   A problem exists if a client allows an open owner to have state on
   multiple filesystems on a server.  If one of those filesystems is
   migrated, what happens to the sequence numbers?  A client can avoid
   such a situation with the stipulation that any client which supports
   migration MUST ensure that any open owner is confined to a single
   filesystem.  If the server finds itself migrating open owners that
   span multiple filesystems, then it MUST not migrate the state for the
   conflicting open owners on the non-migrated filesystems; instead it
   MUST return NFS4ERR_STALE_STATEID if the client tries to use those
   stateids.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 22 Mar 2011 16:51:05 -0400

>    MUST return NFS4ERR_STALE_STATEID if the client tries to use those

OK, but I'm still not going to accept this bizarre sequence number
proposal.

The suggestion that the client can figure out a posteriori what an
NFS4ERR_DELAY meant is completely unreasonable: what if the first DELAY
meant 'wait I'm recalling a delegation' and the second one meant 'wait
I've initiated a move'?

Trond

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 22 Mar 2011 16:08:15 -0500

> OK, but I'm still not going to accept this bizarre sequence number

Okay, as long as we are debating bizarre proposals.

1) The source server keeps track of how many error delays it accumulates
per seqid and sends that over to the destination server after
it starts handing out NFS4ERR_MOVED.

The destination server does not start replying to clients until it gets
this final information from the source server.

2) When the state gets migrated, the seqid is always reset to a
known initial value. The destination server expects these seqids
and the client, once it is aware of the migration, knows to start
using it.

The one catch here could be that the seqids had just used those
values.

---

From: Robert Thurlow <[email-masked]>
Date: Tue, 22 Mar 2011 16:29:56 -0600

> it to behave differently if the intent is migration versus any other valid reason.

I sure didn't hear Bill say that.  Perhaps we could double back
to establish why it's important for sequence IDs to be incremented
in the non-migration case in the face of DELAY?  I don't get that
part yet.

Rob T

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 22 Mar 2011 18:56:33 -0400

> in the non-migration case in the face of DELAY?  I don't get that

Again, it is a question of only-once semantics.

If I get an NFS4ERR_DELAY reply to my OPEN request, then I'm not
actually obliged to resend that request; my application may have decided
to abort its open() syscall, in which case I'll stop trying to OPEN the
file.
So iff I do indeed decide to resend, the server needs to know that this
is indeed a new RPC call and not just a replay of my old request.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 22 Mar 2011 18:16:57 -0500

> in the non-migration case in the face of DELAY?  I don't get that

Another problem is whether changing 3530bis has any effect on 5661.

If we look at how it defines NFS4ERR_DELAY, we see it would:

15.1.1.3.  NFS4ERR_DELAY (Error Code 10008)

   For any of a number of reasons, the replier could not process this
   operation in what was deemed a reasonable time.  The replier should
   wait and then try the request with a new slot and sequence value.

   Some examples of scenarios that might lead to this situation:

   o  A server that supports hierarchical storage receives a request to
      process a file that had been migrated.

   o  An operation requires a delegation recall to proceed, and waiting
      for this delegation recall makes processing this request in a
      timely fashion impossible.

   In such cases, the error NFS4ERR_DELAY allows these preparatory
   operations to proceed without holding up requester resources such as
   a session slot.  After delaying for period of time, the requester can
   then re-send the operation in question.  If the operation that
   returned NFS4ERR_DELAY was not a Sequence operation, the initial,
   preceding Sequence operation of the Compound request MUST NOT be re-
   sent with same slot ID and sequence ID; one or both MUST be different
   on the re-send.  If the operation that returned NFS4ERR_DELAY was a
   Sequence operation, then the Sequence MUST be re-sent with the same
   slot ID and sequence ID.

   Note that without the ability to return NFS4ERR_DELAY and the
   requester's willingness to re-send when receiving it, deadlock might
   result.  For example, if a recall is done, and if the delegation
   return or operations preparatory to delegation return are held up by
   other operations that need the delegation to be returned, session
   slots might not be available.  The result could be deadlock.

I.e., we were able to add NFS4ERR_MOVED to the list of error codes
which would not bump the seqid because it had no real impact on
any existing NFSv4.0 or NFSv4.1 implementations. We can't say the
same for NFS4ERR_DELAY.