Subject: [nfsv4] MDS reboot and DS errors to the client

From: Rich Brown <[email-masked]>
Date: Wed, 06 May 2009 17:30:02 -0500

Consider this scenario:  A client sends on I/O operation to a dataserver
(DS) just as the MDS reboots.  For simplicity, let's say that the MDS
has completed its restart by the time the DS begins to process the I/O
operation.  In this particular implementation, the DS contacts the MDS
to validate the stateID for the I/O operation and discovers that the MDS
has rebooted.  The DS knows that the stateID used in the I/O operation
is from a previous boot instance of the MDS (i.e., the client's stateID
is "stale") but the client doesn't know that the MDS has rebooted.

In that case, what error should the DS return to inform the client
that the MDS has rebooted?

Note that this information could be used by the client to start MDS
recovery rather than wait for the periodic lease renewal operation to
the MDS to indicate server reboot.

The obvious error is NFS4ERR_STALE_STATEID, which is no longer
a legal error for READ and WRITE because...

   15.1.16.5. NFS4ERR_STALE_STATEID (Error Code 10023)

    A stateid generated by an earlier server instance was used.  This
    error is moot in NFSv4.1 because all operations that take a stateid
    MUST be preceded by the SEQUENCE operation, and the earlier server
    instance is detected by the session infrastructure that supports
    SEQUENCE.

Of course, errors on the SEQUENCE op to the DS would detect DS
reboot (as well as other session-related errors), but they have
nothing to do with the state of the MDS.

I discussed this with a few folks at Connecathon earlier this year.
One suggestion was for the DS to return NFS4ERR_FHEXPIRED to the
client.  However...

   15.1.2.2. NFS4ERR_FHEXPIRED (Error Code 10014)

    A current or saved filehandle which is an argument to the current
    operation is volatile and has expired at the server.

This works for volitile file handles, but we need a more general error.

I ran through the list of possible errors and found two that might
make sense:

   15.1.5.2. NFS4ERR_BAD_STATEID (Error Code 10026)

    A stateid does not properly designate any valid state.  See
    Section 8.2.4 and Section 8.2.3 for a discussion of how stateids are
    validated.

In the past, I've interpreted "bad stateid" to mean "never valid", but
my current reading of the spec doesn't necessarily support that.  This
seems like the logical choice.

Another possibility is:

   15.1.10.7. NFS4ERR_PNFS_NO_LAYOUT (Error Code 10080)

    The pNFS client has attempted to read from or write to a file (using
    a request to a data server) without holding a valid layout.  This
    includes the case where the client had a layout, but the iomode does
    not allow a WRITE.

Since the MDS rebooted, then the layout must be invalid.  But the
client may not know it yet.

So if the client sees that the DS thinks that the layout is
invalid then the client knows it needs to return the layout
and should find (from the SEQUENCE op) that the MDS rebooted.
At that point, we can recover the MDS based on information from
the DS.

Is it reasonable that the client interpret both NFS4ERR_BAD_STATEID
and NFS4ERR_PNFS_NO_LAYOUT as a strong hint to check the state of
the MDS (with the idea of kicking off recovery as soon as possible)?

Are there other errors that make sense in this case?

Thanks much,

	Rich

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Thu, 7 May 2009 10:49:09 -0400

On Wed, May 6, 2009 at 6:30 PM, Rich Brown <[email-masked]> wrote:
> operation.

What will happen when the client detects that the MDS has rebooted?
This will answer what the data servers need to do when they detect an
MDS reboot.

12.2.10.  Device IDs
  Clients cannot expect the mapping between a device ID and its storage
  device address(es) to persist across metadata server restart.

This says that the session between the client and the data server can
not expect to
persist across MDS restart.

12.7.4
 The client MUST
  stop using layouts and delete the device ID to device address
  mappings it previously received from the metadata server.

This says that the client MUST stop using the session to the data server.

I think that when the MDS reboots, all state on the DS' s should be
placed in grace and/or revoked - e.g. the DS session/clientid, and all
stateids from the rebooted MDS.

How about returning

SEQ4_STATUS_RESTART_RECLAIM_NEEDED

     When set indicates that due to server restart the client must
     reclaim locking state.  Until the client sends a global
     RECLAIM_COMPLETE (Section 18.51), every SEQUENCE operation will
     return SEQ4_STATUS_RESTART_RECLAIM_NEEDED.

Since the client is not allowed to actually create or reclaim any
state on the DS, it should be clear to the client that the state
refered to is MDS state.

-->Andy

> In this particular implementation, the DS contacts the MDS
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Rich Brown <[email-masked]>
Date: Thu, 07 May 2009 11:36:44 -0500

Hi Andy,

> What will happen when the client detects that the MDS has rebooted?

Ideally, a client would start MDS recovery as soon as it can.  (That's
what ours will do.)

We have a "heartbeat thread" on the client which will renew the lease
every 1/2 lease period, but we'd really like to start recovery as soon
as possible (such as on error from the DS.)

>   device address(es) to persist across metadata server restart.

Yes, our client will clean those up during MDS recovery.

> This says that the client MUST stop using the session to the data server.

That statement seems a bit strong.  The common case will likely be
that the client cleans up the layouts and device IDs (and their mappings)
during MDS recovery, then proceeds to get (new) layouts and device information.

The client could choose to keep the session to the DS alive for some
small amount of time in hopes that the new device IDs map to the same
DSes.

> stateids from the rebooted MDS.

This also sounds too strong.  A DS could be serving two communities.
That is, a client could have file systems mounted from two different
MDSes each of which have supplied device IDs that map to the same DS.
If the client has a single clientID and session to this DS, then we
don't want to tear it down in that case.

> state on the DS, it should be clear to the client that the state

This could work, but I'd much rather see an error on the operation
that failed (with the invalid stateID).

Thanks,

	Rich

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Thu, 7 May 2009 16:50:12 -0400

> as possible (such as on error from the DS.)

Maybe a heartbeat between MDS and DS would help - it's a lot less
traffic to the MDS compared to doubling the lease renewal from all the
clients, and you therefore do it at a lot smaller time interval.

Do your data servers verify stateid's with a message to the MDS on
every I/0, or do they only verify stateid's they haven't seen?
> That statement seems a bit strong.

I see your point.  The client can't use any layouts and deletes all
device address
mappings so the client can only send session level operations to the
data server on behalf of the file system represented by the MDS mount.

> The common case will likely be

True.

> small amount of time in hopes that the new device IDs map to the same

That's why I like the SEQ4_STATUS_RESTART_RECLAIM_NEEDED. The client
keeps the session alive using the session inactivity timer, and the DS
keeps returning the status until the client reclaims state on the MDS
and sends the RECLAIM_COMPLETE to the MDS. Then the MDS communicates
with all the DS's and the client's DS sessions are back on line.

> MDSes each of which have supplied device IDs that map to the same DS.

Do you use the same session for I/O from both MDs? If so, I can see
why you want a per operation error rather than a session level error.

-->Andy

---

From: Rich Brown <[email-masked]>
Date: Thu, 07 May 2009 18:20:05 -0500

> clients, and you therefore do it at a lot smaller time interval.

I'm looking at this from a client perspective (at the moment), hence my
question about what the DS might give me back if I send an I/O operation
with state from a previous MDS instance (and the DS notices that the MDS
has since rebooted).

Either way, we'll still do the lease renewals.

> every I/0, or do they only verify stateid's they haven't seen?

I have to defer to Piyush on this but I'm pretty sure it's the latter with
a side order of caching.

> data server on behalf of the file system represented by the MDS mount.

Yes, the sessions level operations should be fine , but I really
want any operations with invalid state to fail with distinct errors.

> with all the DS's and the client's DS sessions are back on line.

My concern is that this is ambiguous to a client that has file systems
mounted from multiple MDSes which have device IDs that map to a common DS.

> why you want a per operation error rather than a session level error.

Yes, we would effectively find the session that corresponds to the
address of the DS.  It looks like we're on the same page.

Thanks Andy,

	Rich

P.S.  I'll be out of the office tomorrow and next week, but there are
a few others here who understand the issues and can respond to comments.

---

From: Piyush Shivam <[email-masked]>
Date: Fri, 08 May 2009 12:22:09 -0500

<snip>
> I have to defer to Piyush on this but I'm pretty sure it's the latter 

Yes. The DS is designed such that we validate the stateID with MDS, and 
cache the validated stateID at the DS (among other state items).  If 
required, the stateID is explicitly invalidated by the MDS.  Also, if 
the DS detects MDS reboot, it drops all state related to the previous 
instance of the MDS.

-Piyush

<snip>

---

From: Robert Gordon <[email-masked]>
Date: Fri, 8 May 2009 10:00:15 -0500

> that the MDS has rebooted?

NFS4ERR_LAYOUT_MDS_HAS_RESTARTED... :) -- clearly there isn't one,  
however I think the DS would be perfectly valid to return [1]  
NFS4ERR_ADMIN_REVOKED (sort of works) or [2] NFS4ERR_OLD_STATEID (more  
appropriate).

The intent is to trigger the client to validate state with the  
associated MDS, so getting OLD_STATEID from a DS might be that  
indication, the client could then turn around and perform TEST_STATEID  
to the MDS, and via that operation learn it has restarted.

Robert.

--

[1]  15.1.5.1.  NFS4ERR_ADMIN_REVOKED (Error Code 10047)

    A stateid designates locking state of any type that has been revoked
    due to administrative interaction, possibly while the lease is  
valid.

[2] 15.1.5.5.  NFS4ERR_OLD_STATEID (Error Code 10024)

    A stateid with a non-zero seqid value does match the current seqid
    for the state designated by the user.

---

From: "Mike Eisler" <[email-masked]>
Date: Sat, 9 May 2009 02:01:58 -0700 (PDT)

> This works for volitile file handles, but we need a more general error.

DS filehandles are not necessarily MDS filehandles.

The spec recommends they be different, and uses the term "stale" in
conjunction with actions where the data server might invalidate
the data server filehandle.

So I think NFS4ERR_STALE is the way to go. It is how the NFSv4.1 layout
type fences pNFS clients from data servers. A rebooted MDS is a good
reason to fence.

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Robert Gordon <[email-masked]>
Date: Sun, 10 May 2009 09:47:12 -0500

> type fences pNFS clients from data servers. A rebooted MDS is a good

I agree STALE also works..

Robert.