Subject: RE: COMPOUND and the nfs reply cache

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 4 Oct 1999 12:43:02 -0700

> > > I agree with Brent. I don't know any NFS V3 server that have
> > > a problem with
> > > retransmission. The NFS Version 4 remove some burdensome from
> > > the server,
> > > but this doesn't means that every time that the server have a
> > > problem we
> > > have to let the client to deal with it.
> >
> > I agree with Brent too.  Its just that I agree with his first
> > paragraph and you (apparently) agree with his last paragraph.
> >
> > V3 servers don't have (major) problems with retransmissions but
> > V4 fixes that (the lack of problems).  In V3, each request is one
> > operation which is either idempotent or non-idempotent and further,
> > all of the requests which have to return large amounts of data
> > happen to be idempotent, which means the v3 server never has to
> > save large responses.
> >
> > With V4, you can combine ops that return large amounts of data
> > with non-idempotent ops to create something with is both non-
> > idempotent and requires you to save a very large response.  I do
> > not know how to deal with that in a practical way in a server.
> >
> > COMPOUND creates some very large potential problems for the server.
> > In addition to the delights of combining idempotent and non-
> > idempotent operations so that you have the most troublesome
> > features of each, requests can be very long.  If we want
> > servers of reasonable complexity, requiring them to handle
> > a request with a thousand ops and dealing properly with
> > retransmission, does not make the protocol very easy to
> > implement.
> >
> > The problem is that long chains of ops may require a lot of
> > memory to process correctly.  Allocating enough memory to
> > handle the maximum is very memory-inefficient.  Allocating
> > incrementally opens you up to deadlock.  Once the server
> > does something non-idempotent, it cannot just forget about
> > the whole request.  If it is not allowed to say, "I did this
> > much, but can't do any more", how are you going to handle the
> > situation.  If you feel that a large READ, followed by a
> > non-idempotent op is something the server must handle, then
> > I'd like to know how your server would handle it.
> >
> > If we want to change the current spec which allows the
> > server to return NFS4ERR_RESOURCE, then we have to limit
> > the size and composition of COMPOUND requests to those
> > that we can reasonably require a server to process without
> > having to resort to NFS4ERR_RESOURCE.  That isn't an easy
> > thing to formulate and get agreement on, unfortunately.
> >
> >
> 
> The Compound operation is the biggest challenge for NFS V4 
> implementation. I
> am sure that this discussion is only the beginning. 

You're probably right :-(

> I would 
> like that the
> final version of the protocol to be more specific concerning 
> the Compound
> request (now it's a bit too much freedom for my taste).

Mine as well.  Of course, being a server person, I'm obviously
biased.

> I accept the idea that the server returns NFS4ERR_RESOURCE 
> because it can
> not manage the retransmission but I don't like the idea that 
> the server that
> initially answer NFS4_OK at the client request will answer 
> error at the
> retransmission in order to force the client to change the Xid 
> and to be able
> to deal with a completely new request.

I am not proposing returning different things at the initial 
request and at the the retransmission.  

Suppose someone issues: PUTFH, READ, WRITE.

I would return (on the initial request): NFS_OK, NFS_OK, NFS4ERR_RESOURCE
for each of the respective ops.

If there were a retransmission of that, the same form of response
would be returned each time.  Since the PUTFH-READ combination
is idempotent, that wouldn't be through caching the response
but though re-executing the same request.  You might get 
different data each time if somebody else was writing the
file, but that's OK, since the client gets accurate data
as of some time that the request is outstanding.

I'm assuming the client will recover from the NFS4ERR_RESOURCE
be issuing a new request PUTFH-WRITE which will have a 
separate xid and whose response will be cached.  The response
to that will normally be NFS_OK & NFS_OK.

> In the worst scenario look at the sequence:
> 
> PUTFH  Dir1
> RENAME A B
> GETFH
> 
> At the retransmission the server answers NFS4ERR_RESOURCE. The client
> retransmits and the server can now answer NFS4ERR_NOENT. How 
> is suppose the
> client to interpret it?
 
I don't see any problems of interpretation.  The first thing that
wasn't executed gets the resource error.  Everything that was
processed successfully shows up with NFS_OK.

If your sequence followed a READ, I would return RESOURCE on the
RENAME.  This does require the client to figure out the current
file handle and do its own PUTFH as part of reissuing the op 
in error, but that applies to all error recovery from compound.

One other COMPOUND recovery issue is that if I were to 
return RESOURCE on the GETFH, then the client might never be
able to find out the handle returned by RENAME.  I try hard not 
to return RESOURCE on GETFH for that reason.  When I limit the
number of ops in a COMPOUND (to about 256), I make a special
effort not to give you RESOURCE on the GETFH, even it is the
256th op.  If you make GETFH the 256th, 257th and 258th op
of a compound, I will lose patience and give you RESOURCE
at that point, but I figure that anyone that does that
deserves it.

---

From: "Dan Trufasiu" <[email-masked]>
Date: Mon, 4 Oct 1999 16:58:55 -0400

> to that will normally be NFS_OK & NFS_OK.

I don't like the idea to let to every server the freedom to decide what it
doesn't like. This means that a fully COMPLIANT V4 server can implement some
specific procedures and returns NFS4ERR_RESOURCE for everything that it
doesn't implement. This one will kill the idea of Compound operation.  Think
at the poor client. The client that implement a 100 ops in a COMPOUND,  has
to handle and to recover from 99 different NFS4ERR_RESOURCE response.
In my opinion we have to decide and to include in the protocol what
operation doesn't work together and to answer from a duplicate request cache
for non-idempotent operations.

dan

---

From: Mike Eisler <[email-masked]>
Date: Mon, 4 Oct 1999 14:53:09 -0700 (PDT)

"Dan Trufasiu" <[email-masked]> writes
> for non-idempotent operations.

NFS4ERR_RESOURCE  and NFS4ERR_EJUKEBOX are errors that
can occur at any time, not just the mix of idempotent and non-idempotent
operations that the server might want to handle reasonably.

Perhaps we should entertain a flag to COMPOUND to indicate what the
client is willing to tolerate as far as dup request cache handling,
or lack thereof.

In any case though, the client that implements 100 ops per COMPOOUND should not
have 100 if-then-else classes or case statements to deal with the 100
different positions that NFS4ERR_RESOURCE can appear in. Loops are your friend.

	-mre