Subject: RE: COMPOUND and the nfs reply cache

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 1 Oct 1999 09:27:13 -0700

> >Working on COMPOUND for the bake-off, I've run into some 
> interesting issues
> >that I wanted to bring to everybody's attention.  I'm not 
> sure that they
> affect
> >the spec too much if at all, but I wanted to warn anybody 
> who is tempted to
> >use COMPOUND too creatively.  The server, at least my 
> server, might not
> >be able to cope and return NFS4ERR_RESOURCE.
> >
> It's not clear from protocol where NFS4ERR_RESOURCE will be returned:
> as a status of COMPOUND4res or status of a procedure inside COMPOUND
> request or both. The protocol says that the only error for COMPOUND
> procedure
> (except NFS4_OK) could be NFS4ERR_RESOURCE, but this error 
> never show up
> in any COMPOUND procedure.
> I believe NFS4ERR_RESOURCE should be returned in both places 
> so that client
> will know about this error in advance before processing 
> respond from server.

The second paragraph of the DESCRIPTION for COMPOUND indicates
that this is returned for the specific op in which the resource exhaustion
occurred.

As I understand the error model, any error returned by an op is also returned
in front as the error associated with the compound as a whole.

---

From: "Sergey Klyushin" <[email-masked]>
Date: Fri, 1 Oct 1999 13:55:45 -0400

>that this is returned for the specific op in which the resource exhaustion

I believe that NFS4ERR_RESOURCE should be included in error list for all
compound operations.

>in front as the error associated with the compound as a whole.

I would understand it the same way, but it's better to specify it clearly in
protocol.
(or may be I just can't find where it's specified).

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 1 Oct 1999 11:02:37 -0700

> I believe that NFS4ERR_RESOURCE should be included in error 

Or else there should be a list of errors somewhere that can be returned
by all ops so that we can avoid useless repetition.  I think NFS4ERR_RESOURCE
and NFS4ERR_SERVERFAULT can be retunred by any op.

> (or may be I just can't find where it's specified).

I looked in the obvious places and couldn't find it.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 4 Oct 1999 08:16:15 -0700

> "Noveck, Dave" wrote:
> sure that they affect
> Right.  The client's willingness to generate arcane compound requests
> must be checked by the client's responsibility to recover from
> retransmissions or incompletely executed requests. Client beware.

I want to make sure I understand you.  It appears that you are saying
that the server may return NFSERR4_RESOURCE when the client puts
together a COMPOUND that is troublesome, and that it is the client's
responsibility to recover from that.  That is my understanding but 
it seems to contradict pretty directly what you say below.

You mention "retransmissions or incompletely executed requests".
In the paragraph above, it's the "incompletely executed requests"
case that I'm dealing with.  The server indicates how far things have
proceeded and it's up to the client to reissue what wasn't executed
by the server the first time.

The case of retransmissions is entirely different.  In that case, the
client doesn't know if any part of the request has been executed.
The only thing he can do is re-issue the entire request.  The client
does not try to figure out if the original request (or part of it)
was executed.  That's just too hard.  It reissues it and the server,
using the reply cache, makes sure that the right thing happens 
(i.e. non-idempotent requests are issued only once and the results
returned faithfully reflect that single execution).
 
> 
> > In the good old days, requests such as READ (and also READLINK and
> > READDIR) did not cache their results to be returned in the case of
> > retransmitted requests, although they might be dealt with 
> in the reply
> > cache to avoid executing a retransmitted request while the 
> original is
> > in the process of being retransmitted.  One doesn't need to 
> cache the
> > results because no fs state changes (everybody, by common consent,
> > ignoring atime issues) so that if a retransmission gets 
> different data
> > from the orignal request, the new data is just as right as 
> the original.
> > Caching the result for these procs would be extremently onerous
> > because a very large amount of data would have to be saved so
> > people don't do it.
> 
> Right, caching of idempotent requests has no effect on correctness.
> 
> 
> > Now, back to the future.  With COMPOUND, you can have a READ
> > op, combined with ops that change fs state.  There are two cases
> > to consider.  The state-modiying op may be either before or
> > after the READ.  If it is before, you have a chance.  You can avoid
> > saving the results.  You do have to save the results of the 
> state-modifying
> > op, and the current file handle for the READ.  This does add some
> > complexity since you now have to deal with retransmission in
> > which you partially replay and partially re-execute the 
> request.  In the
> > bake-off time frame, if you do a READ, READLINK or READDIR that
> > follows a state-modifying op, I'm going to return NFS4ERR_RESOURCE.
> 
> That's a mistake.  Your server should not be making judgment on 
> whether a request is sufficiently idempotent.  Your duplicate
> request cache should be transparent to the client.  If you
> get a request that's burdensome for your server to cache, then
> don't cache it.  But don't return an error to the client for
> what is otherwise a perfectly valid request.

I disagree.  Servers have always made judgements about whether
requests are idempotent.  This has not been necessary for
correctness, in that you could, as a theoretical matter, simply
cache everything, but in practical terms this would be an
insupportable burden on the server, and thus on use of the
protocol.

As you say, the duplicate request cache should be transparent 
to the client.  The only way that it can be transparent is if
every non-idempotent request is stored in it and used to mask
the problems of transmission from the client.  If the server
were to simply not cache certain requests (silently), then any
retranmission of that request would wind up getting erroneous
results.  The client would start "seeing" the reply cache in
that its characteristics (i.e of not being able to cache
certain request) would be all too visible.

> > Maybe the spec should somehow stigmatize this particular use
> > of compound as seriously anti-social behavior for which
> > NFS4ERR_RESOURCE is an appropriate sanction.
> 
> Yes, the spec should certainly include some text that guides
> implementors on the construction of compound requests, i.e.
> do's and don'ts.  However, I'm very much against the server
> returning errors for valid compound requests that it doesn't
> like.  I still subscribe to the "smart client, dumb server"
> model.
 
If there are going to be do's and don'ts, what are they going
to be?  The case I'm talking about would seem to apply to all 
of the servers I have ever worked on.  I can't imagine a server that
would not have problems with this.  So if there are going to 
be don'ts, and this case isn't among them, I don't userstand
what (other than a limit on raw length) they could be.

The spec currently says that the server can return NFS4ERR_RESOURCE
if it cannot finish processing a COMPOUND.  It is up to the
client to recover.  You agree with that above.  Since all
of those are resumably valid requests, I don't understand
wh at you mean by being "very much against the server
returning errors for valid compound requests that it doesn't
like."  Being extremely literal, I guess I could conclude 
that the server could only return NFS4ERR_RESOURCE for valid
requests that it *did* like :-).

As to the client-server IQ competition, I think the server
has to be smart enough to maintain a reply cache that works, too
dumb to figure out if non-idempotent COMPOUND's as used by
the client will really turn out to be a problem, too smart
to try to cache all reply data including that for READ, too
dumb to do anything else but give up (by returning NFS4ERR_RESOURCE)
when it can't handle request properly.  The client should 
be smart enough to recover from NFS4ERR_RESOURCE when he gets
it and too dumb to function properly when the server does not
properly handle retransmissions of non-idempotent requests
(by sending back the original results).

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 4 Oct 1999 14:17:04 -0700

> Several things, some random, have occured to me throughout 
> this discussion:
> 
> (1) With nfs/tcp, dup requests are a rarity:
> 
> jurassic% nfsstat -s     
> 
> Server rpc:
> Connection oriented:
> calls         badcalls      nullrecv      badlen        xdrcall       
> 270034517     1             0             0             1             
> dupchecks     dupreqs       
> 21106029      20            
> Connectionless:
> calls         badcalls      nullrecv      badlen        xdrcall       
> 768197        0             0             0             0             
> dupchecks     dupreqs       
> 9422          94            
> 
> Server nfs:
> calls         badcalls      
> 270762326     8606          
> 
> 1% of non-idempotent UDP calls hit the dup request cache, and 
> 0.0001 % of TCP calls hit the cache. It is possible that my server's
> cache is too small, but even if it were, I'd expect a higher hit rate.
> 
> The dup request cache is for caching responses to executed requests.
> If we add the require that NFS V4 servers over TCP must not drop the
> response to any executed request, except when the connection breaks,
> then dup requests from the client requiring a hit in the dup
> req cache are very rare.
> 
> (2) delegations further increase the rarity of duplicate requests.
> 
> (3) Server's don't have to return all N bytes requested in a 
> READ or READDIR.
> This would let a server return fewer bytes than asked for if 
> the COMPOUND
> contains a non-idempotent operation. For example,  the server 
> could return the
> full results in the initial request, and cache an abbreviated 
> result in the
> dup req cache.

It's kind of an ugly special case but this will work.  

> 
> (4) Not all non-state modifying ops result big results ... 
> LOOKUP for example
> has simple a 32 bit error code. True it is likely to have a 
> file handle and
> attributes tacked on in other operations, but that's often 
> going to be about
> the size of NFS V3's post op attributes for a WRITE responses.

Yes.

> 
> (5) The problem is now reduced to the situation of state 
> modifying ops (SMOs)
> mixed with non-state-modifying op with big, non-truncatable results
> (NSMOWBNTRs). An example of an NSMOWBNTR might be READLINK, 
> or a GETATTR
> with a big ACL.
> 
> While Dave proposed returning NFS4ERR_RESOURCE for each SMO 
> that appeared in
> the same COMPOUND as non-state modifying op (NSMO), it seems 
> to me that
> limiting this approach to SMOs mixed with NSMOWBNTRs is both 
> sufficient and
> rare enough to not be a significant issue.
 
Actually I only had a problem with non-state-modifying op's
with big results (NSMOWBRs).  I had thought about READ,
READLINK and READDIR.  I hadn't thought about very big
READDIR's but that is an issue.

What I'm currently planning to do for the bake-off is:

     1) NSMOWNBR after SMO gets NFS4ERR_RESOURCE.

     2) SMO after NSMOWBR gets NFS4ERR_RESOURCE

Case 1) was just an expedient since I think that can
be dealt with without major difficulty.

I think case 2) can be dealt with in the truncatable
sub-case as you propose leaving NFS4ERR_RESOURCE for
the non-truncatable subcase, but not for the bake-off.

I think all of the sub-cases of 2) are going to be 
rare.  It's just a question of how to deal with them
if someone does send you a request of that form.  My
preference would be to simply say this is a bad thing
for the client to do.

---

From: Brent Callaghan <[email-masked]>
Date: Mon, 4 Oct 1999 14:35:30 -0700 (PDT)

> the non-truncatable subcase, but not for the bake-off.

Almost all state-modifying ops will be mixed with
non-state-modifying ops, e.g.

	GETATTR
	WRITE
	GETATTR

equivalent to the pre-op and post-op attributes of NFSv3.

You're saying your server will allow these as long as
the results of the GETATTR's are not "big".

How big is "big" ?

How is the client to know the size of the returned
result, e.g. the size of a symbolic link or an ACL ?

Again, I think it's a mistake for the server to 
adopt a holier-than-thou attitude in returning
errors for mixes of ops that it doesn't like.

	Brent

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 4 Oct 1999 14:38:45 -0700

> > I am not proposing returning different things at the initial
> > request and at the the retransmission.
> >
> > Suppose someone issues: PUTFH, READ, WRITE.
> >
> > I would return (on the initial request): NFS_OK, NFS_OK, 
> NFS4ERR_RESOURCE
> > for each of the respective ops.
> >
> 
> > If there were a retransmission of that, the same form of response
> > would be returned each time.  Since the PUTFH-READ combination
> > is idempotent, that wouldn't be through caching the response
> > but though re-executing the same request.  You might get
> > different data each time if somebody else was writing the
> > file, but that's OK, since the client gets accurate data
> > as of some time that the request is outstanding.
> >
> > I'm assuming the client will recover from the NFS4ERR_RESOURCE
> > be issuing a new request PUTFH-WRITE which will have a
> > separate xid and whose response will be cached.  The response
> > to that will normally be NFS_OK & NFS_OK.
> >
> 
> 
> I don't like the idea to let to every server the freedom to 
> decide what it
> doesn't like. This means that a fully COMPLIANT V4 server can 
> implement some
> specific procedures and returns NFS4ERR_RESOURCE for 
> everything that it
> doesn't implement.

Sure but nobody will use it.

A compliant server can respond to every READ request with a 
very short (one-byte?) response leaving the client to re-isuse
again and again.  But nobody would use that server.

Similarly, v2 and v3 don't require a server reply cache but 
people object to using one without it.
 
> This one will kill the idea of Compound 
> operation.

I'm not sure I would be among the mourners.
  
> Think
> at the poor client. The client that implement a 100 ops in a 
> COMPOUND,  has
> to handle and to recover from 99 different NFS4ERR_RESOURCE response.

I think you need a general recovery procedure.  It has to take into
account not only NFS4ERR_RESOURCE, but things like the fact that 
some requests are truncatable (and have to be partially re-issued
out-of-order).  Also, it may have to deal with semantic errors, or
IO errors on any of the ops.  If you assume that they are all
going to work and that you don't have to deal with recovery then
you are going to have problems, NFS4ERR_RESOURCE or no.

> In my opinion we have to decide and to include in the protocol what
> operation doesn't work together and to answer from a 
> duplicate request cache
> for non-idempotent operations.

I have proposed (using Mike Eisler's terminology) that an SMO
after an NSMOWBR should result in NFS4ERR_RESOURCE.

Mike Eisler (I think) has proposed that an SMO after an NSMOWBNTR
should result in NFS4ERR_RESOURCE.  I could live with this as
well.

You'd also need some sort of limit on how many op's a compound
could have.  This might be hard to get agreement on.

With these specified, I still think servers may still sometimes be
unable to complete a compound.  Letting them return NFS4ERR_RESOURCE
seems reasonable to me.  Clients are going to have to deal with
recovery from these situations, but if they are doing big compounds
they are going to have to have general recovery support in any
case.

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 5 Oct 1999 09:41:51 -0700

> NFS4ERR_RESOURCE  and NFS4ERR_EJUKEBOX are errors that
> can occur at any time, not just the mix of idempotent and 
> non-idempotent
> operations that the server might want to handle reasonably.
 
My impression is that JUKEBOX should be able to happen
(almost) anytime but right now the spec has a fair number 
of procedures for which it isn't listed: ACCESS, CLOSE,
COMMIT, CREATE, GETFH, LINK, LOCK, LOCKT, LOCKU, LOOKUP,
LOOKUPP, NVERIFY, OPEN, PUTFH, PUTPUBFH, PUTROOTFH, 
REMOVE, RENAME, RENEW, RESTOREFH, SECINFO, SETCLIENTID.

Some of these probably shouldn't allow JUKEBOX (e.g.
GETFH because recovery from failure of a GETFH after
a create is hopeless), but a lot are probably
oversights: VERIFY can return JUKEBOX but NVERIFY
can't. ???

Almost all of the ops are listed as allowing MOVED
and SERVERFAULT.  RESOURCE is only listed under
COMPOUND but the text implies it can be returned for
any op.

Maybe MOVED, RESOURCE, JUKEBOX need to be discussed
together (under COMPOUND) together with a discussion
of the client recovery issues.  It can just state
that they can be returned for any op (with any
exceptions, such as GETFH, that are appropriate).
Having these listed under each of the inidividual
ops seems like its just an opportunity to get it
wrong through clerical error.  SERVERFAULT probably 
should be handled in the same way although the 
recovery issues are different (throw up your hands).