Subject: [nfsv4] Preventing an NFSv4.1 client from destroying a migrated lease after TSM

From: Chuck Lever <[email-masked]>
Date: Wed, 8 Mar 2017 16:19:37 -0500

One of Oracle's engineers, Xuan Qi, has been experimenting
with NFSv4.1 Transparent State Migration. The
implementations under test are a Solaris-based prototype
NFS server, and a recent vintage migration-capable Linux
NFS client. Xuan observed the following:

+ While a simple workload is running, there is an FS
  migration event from server A to server B. Transparent
  State Migration is done as part of this migration. The
  session that had been established between the client and
  server A is NOT migrated.

+ The client has never interacted with server B before the
  migration event. After the client retrieves fs_locations
  from server A, it begins trunking discovery with server B.

+ In the initial EXCHANGE_ID request with server B, the
  client presents the same client owner and boot verifier
  it previously presented to server A.

+ In the initial EXCHANGE_ID reply, server B reports the
  presence of the migrated lease by returning the same
  clientID the client got from server A and asserting
  EXCHGID4_FLAG_CONFIRMED_R.

+ The client thus believes this lease is an old one it
  forgot to clean up. It purges the migrated lease by
  sending a second EXCHANGE_ID with the same client owner
  but an arbitrary boot verifier. Server B returns a
  second distinct client ID.

+ The client issues a CREATE_SESSION with the second
  client ID. This client ID and session is not used for
  subsequent operations.

+ The client issues a DESTROY_SESSION and DESTROY_CLIENTID
  to server A, using the original session and client ID.
  How polite!

+ The client then establishes a fresh lease with server
  B by sending a third EXCHANGE_ID with the same client
  owner and the original boot verifier. Server B returns
  a third distinct client ID. The client then uses this
  new client ID for a CREATE_SESSION. This session is used
  to issue RECLAIM_COMPLETE with no OPEN state recovery.

+ The first WRITE operation to server B fails with
  NFS4ERR_BAD_STATEID. The client performs no-grace
  stateid recovery, and the workload continues normally.

+ Both servers report the same server scope, but different
  server owners. The scope string is

    "Solaris NFSv4.1 Server Scope"

RFC 5661 and draft-ietf-nfsv4-migration-issues are not
clear about how servers and clients are supposed to
interact after Transparent State Migration. For instance:

1. Since the client has had no previous contact with
server B, should the migrated lease be considered
confirmed or not confirmed on server B?

2. How should the client determine whether or not
Transparent State Migration has occurred?

3. If the session was migrated (which is not the case
in the current example), how should the client determine
it must perform BIND_CONN_TO_SESSION instead of
CREATE_SESSION? (I guess just try BC2S first?)

4. If TSM is detected, the client performs no state
recovery. Should it avoid sending RECLAIM_COMPLETE in
this case?

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 9 Mar 2017 05:52:40 -0500

>   migration event.

In that case, there is no possibility that the clientid used to
access server A will be merged with one used to access
client B.  So it is likely that the clientid used to access
client A, together with the stateid's used to access the
migrated fs are now usable on B.

> After the client retrieves fs_locations

I assume that it compares the server owners it got
from each server. Correct?

>  it previously presented to server A.

Makes sense.

>  forgot to clean up.

I don't see why it should believe that.  It is aware of the
migration from A to B and thus the possibility that
a clientid has been migrated.  The fact that it is the
same id makes it even more likely that the lease is
migrated, although a coincidence is possible.

> It purges the migrated lease by
>  but an arbitrary boot verifier.

OUCH!!. That sounds to me like a client bug.

>Server B returns a

Which is basically ignored I take it.

>  client ID. This client ID and session is not used for

OK.

>  to server A, using the original session and client ID.

Would it still do that if there was another non-migrated fs on A using
the same session?  I hope not.

>  a third distinct client ID.

This clientid will have no stateids associated with it.

> The client then uses this

Not clear why it is doing this, but it is not relevant.  The
damage was done when the existing migrated lock state
was thrown away.

.> + The first WRITE operation to server B fails with
 > NFS4ERR_BAD_STATEID.

Not surprising, since the session is associated with a client for
which there are no valid stateids.

I don't understand why it thinks that stateid was valid, since it
previously
invalidated it by doing an EXCHANGE_I'D with a new boot verifier.

> The client performs no-grace

Sigh!

> interact after Transparent State Migration.

probably not.  The basic assumption is that there needs
to be an update to bring rfc5661 into line with rfc7931, but
that hasn't been done yet.

With regard to this specific situation, it doesn't seem that anything
4.1-specific
has gone on.  In particular, I haven't noticed any particular consquences
due
to sessions or server name or server scope.  There has been an instance of
RECLAIM_COMPLETE but by that point, the damage had been done. It looks
to me that if this had been handled essentially as it would have been in
4.0,
things would have worked better. Maybe I've missed something.

>  For instance:

> confirmed or not confirmed on server B?

I'd say that if is confirmed on A and migrated to B,
then it should be confirmed on B as well.

> Transparent State Migration has occurred?

I think one should follow the approach in rfc7931.  If the stateids
are  valid on the new server, then transparent state
migfation has occurred.

> CREATE_SESSION? (I guess just try BC2S first?)

I think your guess is right.

> recovery. Should it avoid sending
> RECLAIM_COMPLETE in this case?

I don't see why it should send RECLAIM_COMPLETE
in a situation in which there is no indication that
 state was lost.  Do we understand why it was sent in this case?

On Wed, Mar 8, 2017 at 4:19 PM, Chuck Lever <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Chuck Lever <[email-masked]>
Date: Thu, 9 Mar 2017 11:02:57 -0500

> On Mar 9, 2017, at 5:52 AM, David Noveck <[email-masked]> wrote:
> 

I'm told that if the client has already mounted and used
an FS that exists on server B before the migration, lease
merging appears to work as expected.

> from each server. Correct?

It compares server owners to look for session trunking.

> OUCH!!. That sounds to me like a client bug.

In the sense that it hasn't been explained to the client
how to deal with NFSv4.1 migration recovery, this is
probably a "missing functionality" bug.

> damage was done when the existing migrated lock state 

From this we can assume that the client's state recovery
logic is aware that it did just purge its lease.

However, since the client had OPEN state and just purged
the lease on the server, why didn't client send any OPEN
operations? State recovery is obviously needed here, but
wasn't done. That's probably a client bug.

> I don't understand why it thinks that stateid was valid, since it previously 

The client's write path assumes that migration/state
recovery has worked properly, and resumes the workload
without checking.

> With regard to this specific situation, it doesn't seem that anything 4.1-specific 
> has gone on.  In particular, I haven't noticed any particular consquences due 
> to me that if this had been handled essentially as it would have been in 4.0, 

I'm not certain what you mean by "handled ... as it
would have been in 4.0". Server trunking discovery in
NFSv4.0 requires an elaborate dance. In NFSv4.1,
trunking discovery can be done by a single operation
(EXCHANGE_ID).

I'm speculating that if the server hadn't asserted
CONFIRMED_R, this client could have recovered
correctly from the migration event. Is that what you
mean?

> then it should be confirmed on B as well.

I ask because that's not clear from RFC 5661, IMO.

5661 implies that a lease is confirmed by a
subsequent CREATE_SESSION. In this case, the client
has done no prior CREATE_SESSION against server B.
No session exists at all when the client first
contacts that server, because no session was
migrated.

I think you are saying that the CREATE_SESSION on
server A confirms the lease until it expires or
is destroyed, no matter where it resides.
Confirmation of a lease is independent of the
particular server that handles that confirmation.

> are  valid on the new server, then transparent state 

If CONFIRMED_R is correctly asserted when the lease
is migrated, couldn't the client use the setting of
that flag to determine that TSM had been attempted,
and then look at the SEQ4_STATUS flags to determine
if any problems had occurred?

> I think your guess is right.

This would be only when CONFIRMED_R was set and the
client knows it is doing migration recovery?

Also, session migration promises EOS during migration
events. But I'm not sure what happens if a client is
still sending operations to the source server after
the session is moved (for example, the server would
have to return DELAY or MOVED, and the state of the
session there would be different than the state of
the session on the destination, wouldn't it). Don't
we have a situation similar to lock state mutations
after migration in NFSv4.0?

Well, this is speculative, since we don't have an
implementation that migrates sessions, so let's put
it aside for now.

>  state was lost.  Do we understand why it was sent in this case?

Probably because a final RECLAIM_COMPLETE is always
part of the Linux client's state recovery process.
But, state was lost in this case, when the client
purges its lease.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 9 Mar 2017 14:00:50 -0500

> trunking discovery can be done by a single operation

What I meant was that, apart from the greater
calories expended in v4.0 case, the result should have
been the same, i.e. that there is no trunking.

> correctly from the migration event. Is that what you

It wasn't what I meant, but, now that you say it, I think is
probably right.

> I ask because that's not clear from RFC 5661, IMO.

it's one of a number of things that is not clear in RFC5661.

> subsequent CREATE_SESSION.

I think it explicitly says that in the EXCHANGE_ID case; the
CREATE_SESSION is needed to to confirm creation of the
clientid.

Creation of a clientid through migration, without use of
EXCHANGE_ID was one of the things that were not dreamt
of in the RFC5661 philosophy.

> In this case, the client

or EXCHANGE_ID.

> contacts that server, because no session was

I can see how that puts things in an ambiguous
state.

I don't see rfc5661 getting fixed anytime soon, but
I can mention this as an issue to be resolved when
I produce migration-issues-12.

> particular server that handles that confirmation.

That's what I think.

> that flag to determine that TSM had been attempted,

If it was confirmed on A and the client sees it confirmed on
B, the only way it could have gotten to B is via migration.

> if any problems had occurred?

I hadn't thought of that but it sounds workable.

> > >how should the client determine

> client knows it is doing migration recovery?

I think I should mention this in migration-issues-12.

> implementation that migrates sessions, so let's put

We don't have to solve this, but it should be mentioned
as an open issue in migration-issues-12.