Subject: [nfsv4] RE: more on server cache algorithm

From: "Khan, Saadia" <[email-masked]>
Date: Tue, 18 Nov 2003 11:45:21 -0800

Currently in such a sitation our server returns a RESOURCE error. I
believe the spec allows that. This is what the spec says about that:

NFS4ERR_RESOURCE         For the processing of the COMPOUND procedure,
                         the server may exhaust available resources and
                         can not continue processing operations within
                         the COMPOUND procedure.  This error will be
                         returned from the server in those instances of
                         resource exhaustion related to the processing
                         of the COMPOUND procedure.

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] 
> Sent: Tuesday, November 18, 2003 11:29 AM
> To: Khan, Saadia
> Cc: [email-masked]
> Subject: more on server cache algorithm
> 
> 
> > What do you do if you get a compound with only one 
> non-idempotent op 
> > and a large number of idempotent ops including operations such as 
> > read/readlink/readdir which will cause the reply to be very large. 
> > Saving the reply in this case might take up too much space.
> 
> Good point. As of now, the algorithm would save it.
> It might be possible to create a scale that indicates
> "how dangerous" redoing the various non-idempotent Ops are 
> and calculate a "total risk" (based on how many and which 
> non-idempotent Ops are in the RPC). Then use that vs "size of 
> reply" to decide whether or not to save it. (Trouble is, as 
> soon as you decide to not save some non-idempotent replies,  
> the cache almost doesn't seem worth the bother. In fact, the 
> V2,3 BSD  server that has been out there for several years 
> doing NFS over TCP, never  cached anything for TCP.)
> 
> One thing that might get added to the algorithm when I code 
> it is some sort of high water mark on "reply bytes saved" 
> that would allow the amount of storage used to be tuned, at 
> the expense of a less effective cache.
> 
> In general, I see a lot of places where an NFSv4 server can 
> get very memory hungry and I am anxious to see how mine works 
> under heavy load. (This summer, I am hoping to rig up one of 
> my teaching labs (40-60 machines) with a v4 client so I can 
> load test my server. My challenge at the moment is finding a 
> mature client I can use for this. If I succeed, others would 
> be welcome to come and do testing on it, as well, if they are 
> interested.)
> 
> Could be fun, rick
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 18 Nov 2003 12:41:08 -0800

At 02:28 PM 11/18/2003, [email-masked] wrote:
>"how dangerous" redoing the various non-idempotent Ops are and calculate
>a "total risk" (based on how many and which non-idempotent Ops are in the
>RPC). Then use that vs "size of reply" to decide whether or not to save it.

While this may be doable, I think it's important to ask if it's useful. The
global-LRU behavior of server caches really adds very little correctness
to the protocol (apologies to Chet Juszczak), because the "window of
correctness" is actually quite small on a busy server. Even smaller when
you consider that a disconnected client probably takes a while to get a
chance to reconnect. And it doesn't cover the case of server failure, where
the cache is usually lost altogether.

All this is addressed as the core of the sessions proposal that Spencer and
I published in May (draft-talpey-nfsv4-rdma-sess-00.txt). The session id is
assigned to a client persistently, until the client revokes it. And the number
of outstanding requests is bounded, so the server can accurately cache the
replies for the session. I envision the server caching them "forever", at least
until some cron job comes and cleans them out every week or so.

This kind of guarantee is really what's needed to really nail the fundamental
requirement here - correctness in the face of client failure, server failure, and
network partitioning. But the framework you describe is a good one as an
enabler for the cache implementation.

Tom.

>(Trouble is, as soon as you decide to not save some non-idempotent replies,
> the cache almost doesn't seem worth the bother. In fact, the V2,3 BSD
> server that has been out there for several years doing NFS over TCP, never
> cached anything for TCP.)
>
>One thing that might get added to the algorithm when I code it is some sort of
>high water mark on "reply bytes saved" that would allow the amount of storage
>used to be tuned, at the expense of a less effective cache.
>
>In general, I see a lot of places where an NFSv4 server can get very memory
>hungry and I am anxious to see how mine works under heavy load. (This summer,
>I am hoping to rig up one of my teaching labs (40-60 machines) with a v4 client
>so I can load test my server. My challenge at the moment is finding a mature
>client I can use for this. If I succeed, others would be welcome to come
>and do testing on it, as well, if they are interested.)
>
>Could be fun, rick
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 18 Nov 2003 13:20:55 -0800

>    > What do you do if you get a compound with only one non-idempotent op and
>    > a large number of idempotent ops including operations such as

> I think I thought of that once upon a time and then realized that COMPOUNDs
> that include READs probably wouldn't include anything else.  

Assuming reasonable clients, sure.  But sometimes, one finds unreasonable
ones :-(.

> Besides, the
> cached reply could leave out the idempotent parts.  Then the processing
> of the retransmitted request would form the reply from cached data and
> fill in the holes by reexecuting those parts.  It might not even take
> much code.

But that wouldn't necessarily work correctly.  If the idempotent request
precedes non-idempotent, then you are OK.  If it follows it, you aren't.
If you do READDIR followed by a CREATE, then re-executing the idempotent 
READDIR will show the created file, when it should by rights omit it,
since it didn't happen yet, in the context of the re-executed request.
Idempotent ops can be effectively non-idempotent in the context of non-
idempotent requests.  A non-idempotent request can even be made of all 
idempotent ops.  WRITE followed by READ for example.

You can argue that a client who does things like that does not deserve
strictly correct re-execution.  The problem is defining "things like that"
and the spec doesn't.  It does give the server an out and allows it to
return a RESOURCE error as the client's just deserts, when he does "things
like that".  Although a client might contest the justice of this, the
fact that servers are going to return this will mean that clients are
not going to depend on COMPOUNDs like this, no matter how creative the
client writer.  As far as I'm concerned, that's a good thing. 

> I could see COMPOUNDs that include CREATE, RENAME, etc. and READDIR....
>
>  Could be fun, rick
> 
> Could be fun to write a malicious client that makes subtle changes in
> the retransmitted requests too.

Everybody has their own conception of fun :-).  This would be a very useful
thing to bring to Connectathon.