Subject: RE: COMPOUND and the nfs reply cache

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 4 Oct 1999 10:05:24 -0700

> >
> > "Noveck, Dave" wrote:
> > >
> > > Working on COMPOUND for the bake-off, I've run into some
> > interesting issues
> > > that I wanted to bring to everybody's attention.  I'm not sure
> > that they affect
> > > the spec too much if at all, but I wanted to warn anybody who
> > is tempted to
> > > use COMPOUND too creatively.  The server, at least my 
> server, might not
> > > be able to cope and return NFS4ERR_RESOURCE.
> >
> > Right.  The client's willingness to generate arcane 
> compound requests
> > must be checked by the client's responsibility to recover from
> > retransmissions or incompletely executed requests. Client beware.
> >
> >
> > > In the good old days, requests such as READ (and also READLINK and
> > > READDIR) did not cache their results to be returned in the case of
> > > retransmitted requests, although they might be dealt with 
> in the reply
> > > cache to avoid executing a retransmitted request while 
> the original is
> > > in the process of being retransmitted.  One doesn't need 
> to cache the
> > > results because no fs state changes (everybody, by common consent,
> > > ignoring atime issues) so that if a retransmission gets 
> different data
> > > from the orignal request, the new data is just as right 
> as the original.
> > > Caching the result for these procs would be extremently onerous
> > > because a very large amount of data would have to be saved so
> > > people don't do it.
> >
> > Right, caching of idempotent requests has no effect on correctness.
> >
> >
> > > Now, back to the future.  With COMPOUND, you can have a READ
> > > op, combined with ops that change fs state.  There are two cases
> > > to consider.  The state-modiying op may be either before or
> > > after the READ.  If it is before, you have a chance.  You 
> can avoid
> > > saving the results.  You do have to save the results of the
> > state-modifying
> > > op, and the current file handle for the READ.  This does add some
> > > complexity since you now have to deal with retransmission in
> > > which you partially replay and partially re-execute the 
> request.  In the
> > > bake-off time frame, if you do a READ, READLINK or READDIR that
> > > follows a state-modifying op, I'm going to return 
> NFS4ERR_RESOURCE.
> >
> > That's a mistake.  Your server should not be making judgment on
> > whether a request is sufficiently idempotent.  Your duplicate
> > request cache should be transparent to the client.  If you
> > get a request that's burdensome for your server to cache, then
> > don't cache it.  But don't return an error to the client for
> > what is otherwise a perfectly valid request.
> >
> >
> >
> > > Maybe the spec should somehow stigmatize this particular use
> > > of compound as seriously anti-social behavior for which
> > > NFS4ERR_RESOURCE is an appropriate sanction.
> >
> > Yes, the spec should certainly include some text that guides
> > implementors on the construction of compound requests, i.e.
> > do's and don'ts.  However, I'm very much against the server
> > returning errors for valid compound requests that it doesn't
> > like.  I still subscribe to the "smart client, dumb server"
> > model.
> >
> > 	Brent
> >
> 
> I agree with Brent. I don't know any NFS V3 server that have 
> a problem with
> retransmission. The NFS Version 4 remove some burdensome from 
> the server,
> but this doesn't means that every time that the server have a 
> problem we
> have to let the client to deal with it.
 
I agree with Brent too.  Its just that I agree with his first
paragraph and you (apparently) agree with his last paragraph.

V3 servers don't have (major) problems with retransmissions but
V4 fixes that (the lack of problems).  In V3, each request is one
operation which is either idempotent or non-idempotent and further,
all of the requests which have to return large amounts of data 
happen to be idempotent, which means the v3 server never has to
save large responses.

With V4, you can combine ops that return large amounts of data
with non-idempotent ops to create something with is both non-
idempotent and requires you to save a very large response.  I do 
not know how to deal with that in a practical way in a server.

COMPOUND creates some very large potential problems for the server.
In addition to the delights of combining idempotent and non-
idempotent operations so that you have the most troublesome
features of each, requests can be very long.  If we want 
servers of reasonable complexity, requiring them to handle
a request with a thousand ops and dealing properly with 
retransmission, does not make the protocol very easy to
implement.  

The problem is that long chains of ops may require a lot of
memory to process correctly.  Allocating enough memory to
handle the maximum is very memory-inefficient.  Allocating
incrementally opens you up to deadlock.  Once the server
does something non-idempotent, it cannot just forget about
the whole request.  If it is not allowed to say, "I did this
much, but can't do any more", how are you going to handle the 
situation.  If you feel that a large READ, followed by a 
non-idempotent op is something the server must handle, then
I'd like to know how your server would handle it.

If we want to change the current spec which allows the
server to return NFS4ERR_RESOURCE, then we have to limit
the size and composition of COMPOUND requests to those
that we can reasonably require a server to process without
having to resort to NFS4ERR_RESOURCE.  That isn't an easy
thing to formulate and get agreement on, unfortunately.

---

From: "Dan Trufasiu" <[email-masked]>
Date: Mon, 4 Oct 1999 15:17:54 -0400

> thing to formulate and get agreement on, unfortunately.

The Compound operation is the biggest challenge for NFS V4 implementation. I
am sure that this discussion is only the beginning. I would like that the
final version of the protocol to be more specific concerning the Compound
request (now it's a bit too much freedom for my taste).
I accept the idea that the server returns NFS4ERR_RESOURCE because it can
not manage the retransmission but I don't like the idea that the server that
initially answer NFS4_OK at the client request will answer error at the
retransmission in order to force the client to change the Xid and to be able
to deal with a completely new request.
In the worst scenario look at the sequence:

PUTFH  Dir1
RENAME A B
GETFH

At the retransmission the server answers NFS4ERR_RESOURCE. The client
retransmits and the server can now answer NFS4ERR_NOENT. How is suppose the
client to interpret it?

Dan