Subject: [nfsv4] Optional reply caching

From: Daniel Picken <[email-masked]>
Date: Wed, 2 Sep 2009 16:42:56 +0100

I'm trying to understand what exactly a server has to cache, for COMPOUND requests that don't set sa_cachethis in the SEQUENCE operation.

2.10.6.1.3 offers two choices, cache the entire original reply, or:

   o  The replier enters into its reply cache a reply consisting of the
      original results to the SEQUENCE or CB_SEQUENCE operation, and
      with the next operation in COMPOUND or CB_COMPOUND having the
      error NFS4ERR_RETRY_UNCACHED_REP.  Thus if the requester later
      retries the request, it will get NFS4ERR_RETRY_UNCACHED_REP.

There are several ways this could be interpreted (at least by me!):

  1. Cache the results of the first two operations, but change the second (cached) operation's status to NFS4ERR_RETRY_UNCACHED_REP.

  2. Cache the results of the first operation, and partial results for the second operation (its opcode and a status of NFS4ERR_RETRY_UNCACHED_REP).

  3. Cache the results of the first operation (i.e. status == NFS4_OK), set a duplicate COMPOUND's status to NFS4ERR_RETRY_UNCACHED_REP.

The third seems like the most logical, does anyone have a different/another opinion?

Regards,

Dan Picken

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 2 Sep 2009 16:09:51 -0400

I think that if you do:

COMPOUND (3 ops)
SEQUENCE (sa_cachethis = 0)
PUTFH (fh)
READ (0 for 65536, stateid, etc.)

and your response was:

COMPOUND (3 ops, OK)
SEQUENCE (OK)
PUTFH (OK)
READ (OK, data buffer with, for example +start of draft-29.txt)

the duplicate response could be

COMPOUND (2 ops, RETRY_UNCACHED_REP)
SEQUENCE (OK)
PUTFH (RETRY_UNCACHED_REP)

I think I understand what is written in the spec and I'm making a guess
about 1 and 2.  I don't understand what 3 is.

In the case that you had an op which returned anything besides the
result in the operation response, you cannot just change the status and
make something that doesn't match the XDR spec, if that is what is
suggested by option 1.  Normally, an error indicates that there is no
other response data and none should be returned in the
RETRY_UNCACHED_REP case, although I don't normally see how any could be
if you didn't have a current fh.  You would be changing one error
another.

The one really interesting case is SETATTR.  I suppose that if you issue
SEQUENCE, SETATTR, the response would have NOFILEHANDLE as the error and
then the response contains a mask of attributes changed, which better be
none.  A case could be made that the text as written requires you to
save the length and return a number of zero words which matches the
number in the original response.  However, since almost every server
would a use fixed length in this case (or always use a zero-length for
the error case), I don't see any real problem.

One interesting case is if you issue a COMPOUND that only one op (i.e.
SEQUENCE).  I guess in that case, the second choice listed is not open
to you and you have to cache "whole reply", i.e. just the SEQUENCE and
OK.

-----Original Message-----
From: Daniel Picken [mailto:[email-masked]] 
Sent: Wednesday, September 02, 2009 11:43 AM
To: [email-masked]
Subject: [nfsv4] Optional reply caching

I'm trying to understand what exactly a server has to cache, for
COMPOUND requests that don't set sa_cachethis in the SEQUENCE operation.

  1. Cache the results of the first two operations, but change the
second (cached) operation's status to NFS4ERR_RETRY_UNCACHED_REP.

  2. Cache the results of the first operation, and partial results for
the second operation (its opcode and a status of
NFS4ERR_RETRY_UNCACHED_REP).

  3. Cache the results of the first operation (i.e. status == NFS4_OK),
set a duplicate COMPOUND's status to NFS4ERR_RETRY_UNCACHED_REP.

The third seems like the most logical, does anyone have a
different/another opinion?

Regards,

Dan Picken
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Daniel Picken <[email-masked]>
Date: Thu, 3 Sep 2009 11:48:18 +0100

> PUTFH (RETRY_UNCACHED_REP)

I'd disagree - the only operation for which RETRY_UNCACHED_REP is a valid error is SEQUENCE/CB_SEQUENCE(?).

I can't change the status of the cached SEQUENCE result to RETRY_UNCACHED_REP however, because that contradicts "The replier enters into its reply cache a reply consisting of the original results to the SEQUENCE", so I arrived at 3...

> I don't understand what 3 is.

Using your example, the duplicate response would be:

COMPOUND (1 op, RETRY_UNCACHED_REP)
SEQUENCE (OK)

i.e. valid XDR and the (only) op has a valid status.  This still doesn't seem quite right though, because the spec says the COMPOUND's status "must" (although not "MUST") be the same as the final operation's status:

   Contained within the COMPOUND results is a "status" field.  If the
   results array length is non-zero, this status must be equivalent to
   the status of the last operation that was executed within the
   COMPOUND procedure.  Therefore, if an operation incurred an error
   then the "status" value will be the same error value as is being
   returned for the operation that failed.

-----Original Message-----
From: Noveck, Dave [mailto:[email-masked]] 
Sent: 02 September 2009 21:10
To: Daniel Picken; [email-masked]
Subject: RE: [nfsv4] Optional reply caching

I think that if you do:

COMPOUND (3 ops)

and your response was:

COMPOUND (3 ops, OK)
SEQUENCE (OK)
PUTFH (OK)

-----Original Message-----
From: Daniel Picken [mailto:[email-masked]] 
Sent: Wednesday, September 02, 2009 11:43 AM
To: [email-masked]
Subject: [nfsv4] Optional reply caching

Regards,

Dan Picken

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 4 Sep 2009 16:54:05 -0400

I still think #1 is right.  I think the error table is wrong.  It's
wrong in any case since there doesn't seem to be any way that SEQUENCE
is the only op that can return this.  

For #1, SEQUENCE can return this but only in the case of a request
beginning SEQUENCE-SEQUENCE.  But in that case, all ops could return
RETRY_UNCACHED_REP.

For #3, no op would be able to return this error.

It seems to me that the spec should say that no op can return
RETRY_UNCACHED_REP per se, but that in dealing with retries, the server
can return replies in which the response appears as though the op (for
all ops, including ILLEGAL_OP) had returned RETRY_UNCACHED_REP.

Other opinions?  I'm particularly interested in what Mike Eisler, who I
believe wrote the original text, intended.  Or at least thinks it means
now.

-----Original Message-----
From: Daniel Picken [mailto:[email-masked]] 
Sent: Thursday, September 03, 2009 6:48 AM
To: Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] Optional reply caching

> PUTFH (RETRY_UNCACHED_REP)

I'd disagree - the only operation for which RETRY_UNCACHED_REP is a
valid error is SEQUENCE/CB_SEQUENCE(?).

I can't change the status of the cached SEQUENCE result to
RETRY_UNCACHED_REP however, because that contradicts "The replier enters
into its reply cache a reply consisting of the original results to the
SEQUENCE", so I arrived at 3...

i.e. valid XDR and the (only) op has a valid status.  This still doesn't
seem quite right though, because the spec says the COMPOUND's status
"must" (although not "MUST") be the same as the final operation's
status:

-----Original Message-----
From: Noveck, Dave [mailto:[email-masked]] 
Sent: 02 September 2009 21:10
To: Daniel Picken; [email-masked]
Subject: RE: [nfsv4] Optional reply caching

I think that if you do:

COMPOUND (3 ops)

and your response was:

COMPOUND (3 ops, OK)
SEQUENCE (OK)
PUTFH (OK)

-----Original Message-----
From: Daniel Picken [mailto:[email-masked]] 
Sent: Wednesday, September 02, 2009 11:43 AM
To: [email-masked]
Subject: [nfsv4] Optional reply caching

Regards,

Dan Picken

---

From: "Mike Eisler" <[email-masked]>
Date: Fri, 4 Sep 2009 14:07:57 -0700

> is the only op that can return this.

Yep, good catch; the error table is wrong. The spec says:

 The replier enters into its reply cache a reply consisting of the

Indeed, in case the caller sends an illegal COMPOUND like:

  COMPOUND { SEQUENCE (cache_this == FALSE) ; SEQUENCE }

The second SEQUENCE has to be allowed to return
NFS4ERR_RETRY_UNCACHED_REP.

I'll fix the error table.

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 4 Sep 2009 17:36:15 -0400

Note that the spec does currently say that SEQUENCE can return
NFS4ERR_RETRY_UNCACHED_REP.  The problem is that it says that the only
ops that can return NFS4ERR_RETRY_UNCACHED_REP are SEQUENCE and
CB_SEQUENCE.  I believe that all ops can or at least that the requester
can receive responses that look as if any of the ops have returned
NFS4ERR_RETRY_UNCACHED_REP

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 8 Sep 2009 13:14:22 -0400

> NFS4ERR_RETRY_UNCACHED_REP

SEQUENCE and CB_SEQUENCE actually can't return RETRY_UNCACHED_REP, can
they?

Which would make the set of operations that can return
RETRY_UNCACHED_REP the complement of the currently specified set....

--b.

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 8 Sep 2009 14:14:35 -0400

I think they can, in the case in which you have SEQUENCE (cache_this =
FALSE) followed by SEQUENCE.

I don't this there would be any specific harm in excluding it from the
list that can return NFS4ERR_RETRY_UNCACHED_REP and treat it like
SETCLIENTID which also can't return NFS4ERR_RETRY_UNCACHED_REP.

If you do SEQUENCE (cache_this = FALSE) followed by SETCLIENTID, your
get NFS4ERR_NOTSUPP on both the initial request and any retry.

So the spec could require SEQUENCE (cache_this = FALSE) followed by
SEQUENCE to return NFS4ERR_SEQUENCE_POS on both the initial request and
any retry.  But rather than have a big discussion on the list of ops
where NFS4ERR_RETRY_UNCACHED_REP should or shouldn't be allowed and
getting ourselves confused and having some text not matching other text,
I would prefer that we stayed with what Mike has put in PAv5, unless
there is some strong reason to do otherwise.

This group is capable of arguing whether such things as SEQUENCE
(cache_this = FALSE) followed by EXCHANGE_ID should return
NFS4ERR_RETRY_UNCACHED_REP on a retry or only NFS4ERR_NOT_ONLY_OP, while
shedding light on the issues of protocol definition style involved.
However, given where are in the process, I'd say that Mike's choice
which is to include everything except ILLEGAL and obsolete ops doesn't
have anything wrong with it and we should stick with it, any stylistic
reasons to make another choice notwithstanding (or to make the same
choices after extensive debate). 

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Tuesday, September 08, 2009 1:14 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Optional reply caching

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 8 Sep 2009 14:19:19 -0400

> there is some strong reason to do otherwise.

Fine by me.--b.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 08 Sep 2009 14:21:45 -0400

> FALSE) followed by SEQUENCE.

Wouldn't that result in a NFS4ERR_SEQUENCE_POS instead?

Cheers
  Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 8 Sep 2009 14:35:35 -0400

That's the question.  Clearly the first time you issue this, you get
NFS4ERR_SEQUENCE_POS.  The question is what you get if you re-issue this
so that it finds that it is a retry.  You are allowed to return the
original response which includes NFS4ERR_SEQUENCE_POS, but the spec also
allows you to return OK for the first op and NFS4ERR_RETRY_UNCACHED_REP
for the second, or at least that is how I read PAv5.

I don't have any problem allowing that or not but I'd like to avoid
gratuitous changes at this point, so if that is what PAv5 says, I'd like
to keep it that way, unless people can show that it leads to a major
issue.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Tuesday, September 08, 2009 2:22 PM
To: Noveck, Dave
Cc: J. Bruce Fields; [email-masked]
Subject: Re: [nfsv4] Optional reply caching

Cheers
  Trond

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 08 Sep 2009 14:45:44 -0400

So you are basically saying that the replay cache error is allowed to
(but does not have to) take precedence over other errors? That sounds
reasonable.

It would in any case not exactly be a major shortcoming of the protocol:
a client that sets up this kind of compound is clearly broken...

Cheers
  Trond

---

From: "Mike Eisler" <[email-masked]>
Date: Tue, 8 Sep 2009 12:06:45 -0700

As I noted in the other email thread, unless specified otherwise,
if there are multiple errors that apply, then the replier can return
any of them. (And I'm having trouble thinking of a sitation
where we require precedence).