Subject: [nfsv4] stats during grace

From: "J. Bruce Fields" <[email-masked]>
Date: Sun, 7 Dec 2008 15:47:31 -0500

Miscellaneous question:

I don't see mention of this in 3530 or the 4.1 drafts, but shouldn't a
server that gives out write delegations, without recording such
delegations to stable storage, reject getattr's during the grace period?

At a minimum, getattr's from v2/v3 clients can't be answered.  (They
might represent cache revalidations on open, and we don't know the
latest mtime or ctime till write delegations are all reclaimed.)

(And are there any v4 clients that would attempt to do a getattr on a
file before attempting to reclaim their write delegation on it?)

--b.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mike Eisler" <[email-masked]>
Date: Tue, 9 Dec 2008 12:24:28 -0800 (PST)

Bruce,

I agree this is an issue if the server implementation recalls
delegations in response to GETATTRs or if the server sends CB_GETATTRs
in response to v2/v3 GETATTRs. For NFSv4 GETATTRs on WRITE-delegated
files, I suspect many server implementors have concluded it isn't worth it
to send a callback (CB_RECALL or CB_GETATTR); if the client really wants
to validate a cache it should send an OPEN.

For NFSv4, shouldn't the client first have an OPEN and/or delegation
(in NFSv4.1 "or" is possible) on the file before
re-validating a cache via a GETATTR? If so, the client would not
send a GETATTR during grace period.

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Rick Macklem <[email-masked]>
Date: Tue, 9 Dec 2008 16:24:18 -0500 (EST)

On Tue, 9 Dec 2008, Mike Eisler wrote:

[good stuff snipped]
> send a GETATTR during grace period.
The client might just be doing an "ls -l" and may not have even seen an
NFS4_STALExxx error at that point in time.

I think Bruce is right, in that when clients do a Getattr for time_modify
and/or size for regular files in a volume that issues write delegations
and the server is in Grace, it should reply NFS4ERR_DELAY. (It may not
yet know if the file has been write delegated to a client.)

If a server is issuing write delegations and not bothering to do 
CB_GETATTR callbacks, this wouldn't apply, but is such server behaviour
permitted?

rick
ps: I suggested NFS4ERR_DELAY, since NFS4ERR_GRACE is related to state
     recovery and is not listed as an error for Getattr in rfc3530.

---

From: "Mike Eisler" <[email-masked]>
Date: Tue, 9 Dec 2008 14:29:41 -0800 (PST)

> NFS4_STALExxx error at that point in time.

Right. So the ls -l is not for cache revalidation. So in that case,
why the server need to, or more importantly want to, send a CB_GETATTR
and/or CB_RECALL. I.e., why should a random ls -l cause a WRITE
delegation to be recalled?

A client that is re-validating a cache as part of close-to-open semantics
will do an OPEN anyway (do an OPEN first), and
that will cause the write delegation to
be recalled (unless DENY modes conflict; in which case the
OPEN will be refused ... if an OPEN is going to be refused, the
server for sure does not want to be sending CB_RECALLs and
CB_GETATTRs ... remember that the CB_GETATTR only supports
the "change" and "size" attributes, and is only an optimization
to allow the server to decide whether it can satisfy the
GETATTR from its local version of the file's attributes.

> CB_GETATTR callbacks, this wouldn't apply, but is such server behaviour

AFAIK, the specs (v4.0 and v4.1) do not mandate the use of
CB_GETATTR or CB_RECALL in response to GETATTRs, and if so,
this is for reasons as I noted above.

I would be inclined to allow the
NFSv4 GETATTRs during grace. For NFSv3, Bruce technically Bruce
is right, but that leads to the logical conclusion that
NFSv3 can be use a trivial method to make NFSv4 write delegations
useless. So I would be inclined to by default not allow
NFSv3 GETATTRs to cause the recall of NFSv4 delegations. And thus
by extension, allow NFSv3 GETATTRs to succeed during the grace period.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 9 Dec 2008 18:15:46 -0500

> delegation to be recalled?

The current draft is pretty emphatic about this:

	"The GETATTR MUST NOT proceed until one of the following occurs:

	  o  The requested attribute values are returned in the response to
	     CB_GETATTR.

	  o  The write delegation is returned.

	  o  The write delegation is revoked."

And I'm a little wary of second-guessing what the client is doing with
the GETATTR result.

If we want to allow "ls -l"'s that don't interfere with write
delegations, I think we'll need some way for the client to indicate that
it doesn't care how up-to-date a given getattr result is.

> useless.
> So I would be inclined to by default not allow

Violating close-to-open is a pretty big step.

I think I'd rather say that if servers care about write leases and
faster recovery times, then they should record write leases to stable
storage somehow (so at least unrelated files aren't affected).

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 9 Dec 2008 18:18:42 -0500

> send a GETATTR during grace period.

Yeah, it shouldn't need to.  I just had a vague fear some client might
be sticking a routine (if senseless) getattr in before its open reclaim,
causing a deadlock on recovery against a server that was being careful
about getattrs during the grace period.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 10 Dec 2008 11:18:32 -0500 (EST)

> delegation to be recalled?
That's what a CB_GETATTR is for, as I understand it. To allow a client
to get up-to-date attributes (size and change, then change also allows
the server to concoct time_metadata and time_modify). What the client
is using the attributes for is not known by the server.

The last sentence in rfc3530 sec. 9.4.3 seems to clearly state that the
server has to do one of the two. (CB_GETATTR seems the more desirable
alternative to me?)

Since the server might not know if there is a write delegation issued
to a different client during Grace, replying with NFS4ERR_DELAY seems
appropriate. (If another client has already recovered a write delegation,
then I still think returning NFS4ERR_DELAY might be a good idea, since
I'm not sure clients will be ready to handle CB_GETATTR while they are
doing recovery, but I can't see anything in the RFC that restricts a
server from trying CB_GETATTR in this situation.)

> GETATTR from its local version of the file's attributes.
I'm not sure why you have focused on cache validation, although
that is one use of the Change attribute.

> this is for reasons as I noted above.
Last sentence of Sec 9.4.3 rfc3530:

    It should be noted that the server is under no obligation to use
    CB_GETATTR and therefore the server MAY simply recall the delegation
    to avoid its use.

This seems to say the server should do one or the other? rick