Subject: RE: Caching issue (SHORT, really, I swear)

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 12 Aug 1999 08:43:05 -0700

> I am afraid that the write back on close is not a panacea because on
> Windows clients errors detected at this point cannot be 
> reported to the
> user application.  Quota/space errors are expected to be 
> reported in the
> context of write calls. 

Ouch.   

> Perhaps someone familiar with Windows NFS
> integration can comment on this in more detail.  In DFS, which does
> write back on close, we have found that various (dubious) tricks are
> necessary to avoid this problem.  To do a better job it be 
> nice to have
> some kind of reservation system so that the client can be reasonably
> sure that the server can accept all the dirty data it has.  This leads
> to a scheme like a) or c).  Because of the Windows problem I 
> don't think
> b) make a great fallback.  I think that the many advantages of write
> delegation make d) a bad choice.

So your order of decreasing preference is c), a), b), d)?

It seems also that the Windows issue strengthens the case for 
getting a better close-on-flush discrminator than file length
alone.  Mike Eisler used the word "critical".  I was thinking
that was an overstatement, but I'm coming around to the 
opinion that it isn't.

Are you OK with the resolution I proposed of making the space_limit
structure be just a file length for August but leaving it open
for a better implementation later?  This essentially a) with the
chance to easily substitute a c) when it becomes available.
I'm hoping that this will get people thinking in the space
reservation direction early on as we start building with the
option to change some of the details later

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 12 Aug 1999 11:07:54 -0700

Gee, where is that apathy that Mike Eisler and I were worrying 
about now that I really need it :-)

> I have always had problems with writing after the close() call!

But there is another group that would have problems with the absence 
of writing after close() and wants it in more cases than the
current proposal would allow.  I'm closer to agreeing with that
group but I'm stuck in the middle.  I would be willing to accommodate 
either position but I can't accommodate both.

> 
> What happens in the following circumstances:
> 
> 1) Writing to a disk which stores data in compressed form. At the
> time of the write it looks like there is enough room on disk given
> the current compression ratio, but when the data is actually written
> to the disk from the cache manager, the disk can become full because
> the compression ratio changes.

Although the language could be strengthened I'm sure, the proposal 
indicates that the server should set the file size limit (soon to be 
nfs4_space_limit) so it can do any necessary writes.  It should set 
it on a worst-case basis.

The conditions exported to the client for the flush-on-close test
are essentially a promise by the server.  A server shouldn't make
promises it can't keep.  Some servers will take their promises 
seriously and some less seriously.  Some will take such a promise 
so seriously that they won't ever make it.  This is a quality-of-
implementation issue.  The protocol needs to make clear
what the server is promising to the client.  Enforcement is up to
the marketplace.  If your server lies too much, use something else.
    
> 
> 2) Writing to a tape device and a media error occurs.
> 
> 3) Writing to a CD-RAM or DVD-RAM drive with media errors. This
> is exactly where you would want lazy-writing-after-close due to speed
> problems.
> 
> 4) Writing to a floppy disk with media-errors.

There are two classes of errors here.  Those in which it is a space
issue which can be dealt with probabilistically (space to skip over
some expected number of bad spots) and catastrophic errors (like
the tape breaking), but the tape can break after close() just as 
well.  The server has the ability to force the client to always
flush on close.  He can treat some set of media errors as an
event that he wants to take account of he can treat it something
like an asteroid hitting Earth (it cancels all promises).

> A competitor to ours added lazy-write-after-close to their NT NFS
> client and made hay about the speed of the drop-and-drag. This was
> using NFS V2/3 and if you ran out of disk space no error could be
> returned to the application. We responded by providing the same
> feature but the user must turn it on to make it work. (And they are
> warned about the possible loss of data).

This is different.  In your case the server made no promise and
the client has the choice of doing something risky or not.
In this case, the server will be making a promise (some of the
time), and the client has the choice of either accepting that
or always flushing on close, to be hyper-conservative.  If you
were to have an option to skip the flush even when the server 
didn't make the promise, that would be non-compliant, although
nobody is going to prevent you from actually doing it.  I'm
hoping that if servers have the ability to make reasonable
promises that clients will then accept (or not), then there will
be little demand to get so far out on a limb as skipping the
flush when the server doesn't promise that it is OK.

> 
> 
> Here is my 0.02 £ worth:
> 
> 	Lazy-Writes-After-Close are not a secure method of writing to
> any media and should be avoided. BUT the administrator should have the
> option, for speed over guaranteed writing considerations, to allow
> Lazy-Writes-After-Close. This means that no reservation is required to
> determine diskspace requirements, because the client knows it is
> taking a chance by using Lazy-Writes-After-Close.
 
Any server can force the client to do flush-on-close (and the client
should/must obey) and the client can flush-on-close if he 
chooses to do so.  If you prefer absolute security that's what
you'll do.  It makes sense to give people the ability to go faster
(yes they're taking some sort of risk in trusting any server)
without doing so blindly (with no reservation).  That would be
taking a risk of an entirely different order.

Remember, lots of local file systems do this fine and people are
(fairly) happy with them.  If they want to write to disk before
close they do an fsync() or set some mount option.  Some people
are more risk-averse than others and having NFS v4 accomodate 
that range of opinion makes sense to me.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 16 Aug 1999 09:07:36 -0700

> > > I've come to agree that the file size limit for write 
> caching is too
> > > restrictive and that a flush on close requirement would be better.
> > 
> > But a flush-on-close requirement is more restrictive??
> 
> I may not understand the caching proposal properly.
> 
> 	1) If a client does not have a write open delegation, is it
> 	   still allowed to cache writes until the file is closed
> 	   (ignoring file locking issues)?

Right.

> 	2) If a client has a write open delegation for files up to
> 	   a particular size, is it still allowed to cache writes
> 	   until the file is closed for files larger than the size
> 	   specified for the write open delegation?

Right also.

> 
> The answer to 1 appears to be "yes" from the text in section A.4.1
> of the proposal.  The answer to 2 appears to also be "yes" from the
> text in the last paragraph of section A.5.1.  I would find it helpful
> to have the caching constraints for write open delegations made
> more explicit.  For example, something like the following might
> work:
> 
> 	For modified files whose size is less than or equal to the
> 	size specified in the write open delegation, the client is
> 	not required to flush dirty data back to the server when the
> 	files are closed.  For those files larger than the specified
> 	size, all dirty data must be flushed to the server when the
> 	files are closed.
 
It would have to be modified a bit to account for the change that
Mike Eisler suggested and that I agreed to.  Something like:

For files which meet server-specified constraints, either by having
a size less than or equal to a server-specified file size or by
having the set of modified blocks being less than or equal to a server-
specified count, the client is not required to flush dirty data
back to the to the server when the files are closed.  For files
which do not meet these constraints, all dirty data must be flushed
to the server when the files are closed.  When there is no write
open delegation for a file, no server-specified flush-on-close
constraints apply and all modified data must be flushed to the 
server on close.

I may not have understood what you meant but when you said "the file
size limit for write caching is too restrictive and a flush on
close requirement would be better", but by "too restrictive"
I understood you to be saying that v4 should allow avoiding the
flush on close in more situations (which I think Mike Eisler's
suggestion will do).  The part I am confused about is what you
meant by a flush on close requirement being better.  I thought
you meant always flushing on close which I think of as much *more*
restrictive than what I had which I can't square with you finding
what I had to be "too restrictive".

I suppose if you are happy with what Mike and I agreed on,
the precise details of my confusion don't really metter.
Also, it allows for extensions, so if you have opionions on 
other forms such constraints might take, things can be extended
to include those.