Subject: Re: [nfsv4] Delegation Recall Problem

From: [email-masked]
Date: Fri, 18 Aug 2006 10:37:44 -0400 (EDT)

[Spencer wrote]
> The server should revoke the delegation after onc lease period
> has passed.  It should also send a CB_PATH_DOWN error to the client
> as a response to the RENEW requests.

[Trond wrote]
> No it shouldn't. It should indeed revoke the delegation, but the
> CB_PATH_DOWN error is bogus: the problem isn't that the client's
> callback path is inaccessible, but rather that it is failing to complete
> the delegation recall in a timely fashion.
> 
> It would be more correct to just return NFS4ERR_OLD_STATEID the next
> time the client tries to use that delegation stateid.

Actually, I believe that rfc3530 is vague enough that several errors are
allowed. (Since it only occurs for a broken client, I'm not sure it has to be
clarified?)

Here's what I believe the server should do when a client replies OK to a
CB_RECALL, but doesn't get around to doing a DELEGRETURN:
- I agree with Spencer that it should reply NFS4ERR_DELAY to the SETATTR,
  if the DELEGRETURN doesn't show up quickly (a few msecs?).
- In some finite time (>= lease duration), the server must give up on
  waiting for the DELEGRETURN. I believe that duration should be extended
  while the client is doing Opens (and Writes for WRITE DELEGATIONS) on the
  file, but not extended indefintely.
  - Now it must invalidate the delegation stateid. Depending on how it has
    implemented its stable storage for recovery, that may imply invalidating
    all state for that client or just the delegation Stateid. To avoid
    invalidating all state for the client requires a much more complex
    stable storage for recovery implementation than that descibed in
    Sec. 8.6.3. This is required so the client can't recover the delegation
    after a server crash when the edge conditions described in 8.6.3 occur.
    A - Invalidates all state for that client. For this case, it seems it
    can return any of NFS4ERR_EXPIRED, NFS4ERR_BADSTATEID or
    NFS4ERR_ADMIN_REVOKED for the clientid or any stateid issued to that
    client until the client does a SetClientID/SetClientIDConfirm.
    (Since 8.6.3 only mentions NFS4ERR_EXPIRED, I think that one is preferred
     for this case, but that might change in 4.1?)
    B - Invalidate the delegation stateid only. I think that NFS4ERR_BADSTATEID
    should be returned for any subsequent uses of that stateid.

Just my version of reality, rick

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 21 Aug 2006 10:38:20 -0400

I think this question was about v4.0 but I think it will clarify
some things to also discuss it with regard to v4.1 (as of draft-05).

I think  there seems to be consensus that the server here is remiss 
in not giving the SETATTR a DELAY error.  I think the issue for the 
spec is that it currently isn't clear that the server needs
to do that.  The definition of the NFS4ERR_DELAY does refer to this
situation but simply says the NFS4ERR_DELAY "may occur".  It doesn't
convey that this is not simply a question of server resources but 
that if the server does not do this, things may break.  I've created
issue 109 for this (and it is marked 3530bis since this applies to 
v4.0 as well).

There is also general agreement that the sever has to revoke the
delegation in this situation, even when the client properly renews
its lease.  I think this is explained OK in draft-05 but rfc3530 
doesn't really address this specifically, considering it part of 
"administrative revocation".  I've created issue 110 for this.  It 
is resolved-no-change since draft-05 handles it but  I've marked it 
3530bis for the v4.0 part.

Where we have differences of opinion about v4.0 and also differences 
with v4.1 is with regard to how the error is reported, and acknowledged.
Let me start with v4.1.

In v4.1 (as of draft-05), the client will see the status bit
SEQ4_STATUS_RECALLABLE_STATE_REVOKED being returned on subsequent
SEQUENCE operations for all sessions associated with the client
in question.  This bit will remain on until the client acknowledges
the revocation by doing a FREE_STATEID on the revoked stateid.
The client must be marked as unable to do reclaims until it 
acknowledges the loss of all revoked state.  If the revoked
stateid is used (other than with FREE_STATEID or TEST_STATEID),
NFS4ERR_DELEG_REVOKED will be returned.

So Spencer argues that NFS4ERR_ADMIN_REVOKED should be returned on 
RENEW.  I guess I think that is reasonable since it is listed in 
the error list for RENEW and if you don't return it in that case,
it is hard to understand how it would ever be returned for renew.

I will note that the definition of NFS4ERR_ADMIN_REVOKED does not
not really fit RENEW at all.  It refers to the "The lockowner's"
locks but renew does not have an associated lockowner (and also 
delegations are not associated with lockowners).  Despite this,
I think we need some way of indicating to the client that there 
has a been a loss of state and returning othe error on RENEW
seems the like tha only real way to do that, despite the fact the
spec isn't really clear here.

I think Trond is right that we also need to have some way of 
indicating the specific loss and I think that when the specific
stateid is used, NFS4ERR)_ADMIN_REVOKED should be return as well.
I don't believe NFS4ERR_OLD_STATEID is right nor that.  It can
be argued that NFS4ERR_BAD_STATEID is OK, and seems unavoidable 
unless there is some way to free the revoked state.

This leads to the issue of how long RENEW should return 
NFS4ERR_ADMIN_REVOKED and whether there is a way for the client to
acknowledge the revocation and get out of no-reclaim mode.  My
suggestion is that then DELEGRETURN is done with such a revoked
stateid (in v4.0), the error NFS4ERR_ADMIN_REVOKED is returned,
but the server may recognize this as acknowledging loss of the
stateid, allowing NFS4ERR_BAD_STATEID to be returned on any
subsequent use of the stateid, turning off the return of 
NFS4ERR_ADMIN_REVOKED on RENEW and allowing locks to be reclaimed
by this client in the event of server reboot. 

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]
Sent: Friday, August 18, 2006 10:38 AM
To: [email-masked]
Cc: [email-masked]; [email-masked];
[email-masked]
Subject: Re: [nfsv4] Delegation Recall Problem

[Spencer wrote]

[Trond wrote]

Actually, I believe that rfc3530 is vague enough that several errors are
allowed. (Since it only occurs for a broken client, I'm not sure it has
to be
clarified?)

Here's what I believe the server should do when a client replies OK to a
CB_RECALL, but doesn't get around to doing a DELEGRETURN:
- I agree with Spencer that it should reply NFS4ERR_DELAY to the
SETATTR,
  if the DELEGRETURN doesn't show up quickly (a few msecs?).
- In some finite time (>= lease duration), the server must give up on
  waiting for the DELEGRETURN. I believe that duration should be
extended
  while the client is doing Opens (and Writes for WRITE DELEGATIONS) on
the
  file, but not extended indefintely.
  - Now it must invalidate the delegation stateid. Depending on how it
has
    implemented its stable storage for recovery, that may imply
invalidating
    all state for that client or just the delegation Stateid. To avoid
    invalidating all state for the client requires a much more complex
    stable storage for recovery implementation than that descibed in
    Sec. 8.6.3. This is required so the client can't recover the
delegation
    after a server crash when the edge conditions described in 8.6.3
occur.
    A - Invalidates all state for that client. For this case, it seems
it
    can return any of NFS4ERR_EXPIRED, NFS4ERR_BADSTATEID or
    NFS4ERR_ADMIN_REVOKED for the clientid or any stateid issued to that
    client until the client does a SetClientID/SetClientIDConfirm.
    (Since 8.6.3 only mentions NFS4ERR_EXPIRED, I think that one is
preferred
     for this case, but that might change in 4.1?)
    B - Invalidate the delegation stateid only. I think that
NFS4ERR_BADSTATEID
    should be returned for any subsequent uses of that stateid.

---

From: [email-masked]
Date: Mon, 21 Aug 2006 18:08:58 -0400 (EDT)

[good stuff snipped for the sake of brevity]
> by this client in the event of server reboot. 

I don't think you can require a server to turn off
the NFS4ERR_ADMIN_REVOKED reply, since there may be cases where a server
can't keep all the revoked delegation state around. (Similar to the
NFS4ERR_BAD_STATEID vs NFS4ERR_EXPIRED argument.) However, if the server
knows that the client has DELEGRETURNed all the revoked delegation(s),
switching back to NFS4_OK replies to Renew tells the client that it can again
reclaim, without needing to do a fresh SetClientId/confirm.

Looks good to me, rick