Subject: RE: Caching proposal, take 2 (LONG)

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 11 Aug 1999 09:14:55 -0700

> There is one critical issue, which I will belabor one more time (at
> least until there've been attempts to implement it), is that 
> I consider
> the file size limit approach on write delegations to be an inadequate
> solution to the problem of running out of disk space on a server after
> the client flushes its dirty data. It is insufficient because 
> it limits
> the server to serving lots of clients with small files, or small
> numbers of clients with large files, but no lots of clients with lots
> of large files. In absence of a sufficient solution, I don't think
> write caching is worth doing.

One point here is that file size limit does not prohibit write
delegations for files above that size limit.  It only forces you
to flush modified data on close to the server when the size is 
above the limit (and you don't have to commit to stable storage).  
You can still keep the data for the file on the client and retain 
the delegation.  You have something that is certainly no worse than
v3 and in many respects, particularly for locking, is much better.
Maybe you recognize that and we are just using terms slightly 
differently.  If so, sorry. 

Let me try to summarize the situation.  There are three ways to go
here:

    a) What's in the proposal, which doesn't do as much as you or
       I would prefer that it do.

    b) Take out the file size limit and modify the proposal so
       that you always flush on close (just as if the file size
       limit were zero).

    c) Do something better.

As I understand things, you would prefer c), but want b) in
preference to a) since you feel that a) is so of so little use 
that it is not worth the effort.

My problem is that I don't know how to do c).  It may be that I
am overly space-reservation-complexity-phobic (I wonder what the
Greek word for that is?) but after doing the full-bore version 
for ADFS, I am reluctant to start down that slope.  

Given that, my preference would be to keep a) and allow the 
prototyping to decide.  If nobody implements a) substantively
(everybody sets the file size limit to zero), and people 
don't have real plans to change that, then we can simplify
things at that time.  Also, you or I or someone else may
get an idea about c), which would be great.

I guess I understand why you prefer b) to a), but I don't
understand why you think it is a critical issue.

Nevertheless, I don't want to stand in the way of consensus
here.  If you still prefer b), Spencer and I can jointly
implement that simplification if and when this goes into
the spec.

My only issue is that I want to make sure there would be 
consensus on this point, if I were to abstain.  I don't
want to go to b) and then have someone say, "But I *really*
want a)".

On the theory that some people may not find our debates as
fascinating as we do, and thus may not be giving them the
attention they deserve, I'll send a separate message laying
out the issue and asking if there is anyone out there who
really wants a).  Silence will imply that we go to b).

OK?
 
> 
> >   Changes addressing comments made about first version (I
> >   believe I did what commenter wanted):
> ...
> >        Giving the server more freedom to delay cancellation
> >        (now revocation) of a delegation if a client seems to be
> >        making progress in flushing state.  (Issue raised by Mike
> >        Eisler).
> 
> I looked for it, but couldn't find it.

Last two sentences in the seventh paragraph of section A.3.

> 
> > A.3.1.  Delegation recovery
> > 
> > There are three situations that delegation recovery must deal with:
> > 
> > o    Client reboot
> > 
> > o    Server reboot
> > 
> > o    Network partition (full or callback-only)
> > 
> > In the even of a client reboot, the failure to renew leases will
> > result in the revocation of record locks and share reservations. 
> > Delegations, however, may treated a bit differently.   
> > 
> > Because data associated with some delegations may be written to
> > stable storage
> ^^^^^^^^^^^^^^^
> 
> For clarity, the document should make it clear that "stable storage"
> refers to stable storage on the client.
> ...

OK.

> 
> > When a network partition occurs, delegations, like other locks
> > will subject to freeing when the lease renewal period expires,
> > although the server will normally extend the period in which
> > conflicting requests are held off in the case of delegations.
> > Eventually, however, the occurrence of a conflicting request
> > from another client will cause revocation of the delegation.  
> > A blockage of the callback (e.g. by later network configuration 
> > change) will have the same effect.  A recall request will fail 
> > and revocation of the delegation will result.  
> > 
> > A client normally finds out about revocation of a delegation 
> > when it uses a stateid associated with a delegation and 
> > receives the error NFS4ERR_EXPIRED.  It also may find out about
> > delegation revocation after a client reboot when it attempts
> > to reclaim a delegation and receives that same error.
> 
> The above doesn't explain what happens  if the delegation is revoked,
> and another client has written the file. If the revoked client had a
> read delegation this is one thing, but what if the revoked 
> client had a
> write delegation?  What does the client do with its dirty data, and
> what are some of the recommened actions with respect to the
> interfaction between the client and the application the client is
> operating on the nehalf of (in the common siutation when the
> application anbd client are layered entities).  This is covered in
> general terms in section A.6.1, and so a reference to A.6.1 is needed
> here.

I'll put the reference in.

> 
> Then there is the situation when the server revokes the write
> delegation during partition, and another client write the file, and
> then tserver reboots before the revoked client determines that it lost
> its delegation. This can be detected by looking at the change
> attribute. Note that I raised this issue in my review fo the first
> version of the his document. If it has been addressed elsewhere in the
> document, my mistake.

Sorry.  I must have missed that.

As I see it, this is exactly the problem that is addressed in the
last paragraph of section 7.5.  By making delegations a species of
the lock genus, I get all of the benefits of the work getting the 
locking model to work.  In the context of that paragraph, a 
delegation is a lock and the write you are woried about is 
a conflicting IO.  Maybe there needs to be a reference back to
this section?

Or am I missing something?

> 
> > A.4.3. Data caching and mandatory file locking
> > 
> > Client-side data caching needs to respect mandatory file
> > locking when this is in effect.  The presence of mandatory
> > file locking for a given file is indicated the result flags
> 					^^^^^^^
> 					typo: missing "in"
> > for an OPEN.

OK.

> 
> > A.5.  Open delegation
> ...
> > The following is a typical set of conditions 
> > that servers might use in deciding whether open should be delegated:
> ...
> > o    Availability of disk space on the server to guarantee that
> >      when modified files are written to the server, that fact 
> 							^^^^^^^
> 
> >      can be detected and reported to the user as part of the
> 
> which "fact" can be detected?

Ouch.  That got garbled.

How about:

o    Availability of disk space on the server to guarantee that
     when modified files are written to the server, any possible
     lack of disk space can be detected and reported to the user 
     as part of the close corresponding to the open under which 
     the writes were done.

>  
> >      close corresponding to the open under which the writes
> >      were done.
> 
> > A.5.  Open delegation
> ...
> > The use of delegation together with various other forms of 
> > caching creates the possibility that no server authentica-
> > tion will ever be performed on a given user since all of 
> > his requests might be satisfied locally.  Where the client is
> > depending of the server for authentication, it should make
> > sure that some authentication (via an ACCESS call) happens
> > for each user, even if an ACCESS call would not otherwise
> > be required.  The server may enforce frequent authentication
> > by returning an nfsacl4 denying all access with every open
> > delegation.  
> 
> In my review of the first version of this document, I noted:
> 
> 	"Another related issue, is what if there are no unexpired
> 	credentials available for the client after the file has been
> 	closed locally, user has logged off, and the server sends a
> 	revoke request? AFS and DFS, which use Kerberos authentication,
> 	must have had to deal with this problem, so perhaps WG members
> 	from MIT or TransARC could advise here."
> 
> Ted Anderson responded:
> 
> 	"No magic bullet here. Really, you need to track the expiration
> 	time of the credentials you are depending upon and flush dirty
> 	data well before it arrives. Further, if you can't get
> 			 ^^
> 
> 	notification of impending unlog (kpurge) operations which allow
> 	you to flush dirty data, you can make private copies of the
> 	creds you are using, and purge them after the dirty data is
> 	stored."
> 
> I assume that "it" refers the expiration time of the credentials.
> 
> The issue I raised and the solution Ted described to need to be
> mentioned somewhere in the document.

Fair enough.

> 
> > A.5.1  Open delegation and data caching
> ...
> > In order to maintain current semantics in which the non-availability
> > of storage to hold a file written by an NFS client is guaranteed
> > to be determined at or before the associated close operation,
> > the avoidance by the client of the requirement to flush data to
> > the server on close, is limited to files below a certain size, 
> > determined by the server, and communicated to the client as part
> > of the delegation.  It is up to the server to reserve sufficient 
> > space so that all files whose size is under that specified in a 
> > write open delegation can be successfully written, and to recall 
> > delegations appropriately to maintain that invariant.
> 
> See my comment at the start of this message.

Ditto.

> 
> > A.5.3.  Recall of open delegation
> ...
> > Special handling is needed for a GETATTR which occurs when a write
> > open delegation is in effect.  In this case, the client holding the 
> > delegation needs to be interrogated, using a BACK_GETATTR callback, 
> > if the GETATTR attribute bits include any of the attributes that a
> > write open delegate may modify (length, modify time, change).  
> 
> Is the above list of attributes intended to be an example, or 
> is it the
> complete set of attributes that a write delegation can modify?
> 
> It would be useful if the document prior the A.5.3 specified what
> exactly, beyond data, is delegated to the client in a delegation.

I'll add something to A.5 for this.

Basically it is specified in terms of actions rather than data
items.  You can open, close, lock, unlock, and do IO.  The attributes
in question are those that can be changed by doing IO.

> 
> > A.6.   Data caching and revocation
> > 
> > When locks (including delegations) are revoked, the assumptions
> > upon which successful caching depend, are no longer guaranteed.
> > Therefore the client, in addition to notifying the owner of a 
> 					^^^^^^^^^^^^^^		
> 		
> > record lock or share reservation, and processes holding opens 
> 				    ^^^^^^^^^^^^^^
> > for the delegation, needs to remove all data for the file from 
> > its cache.  In the case of modified data, it must be removed 
> > from the client's cache without being written to the server.
> 
> Notification to processes is going to be operating system specific.
> Nonetheless, suggestions of possibilities for notification would be
> useful to let the implementor understand the ramifications.

OK.

> 
> > A.6.1.  Revocation recovery for write open delegation
> > 
> > Revocation recovery for a write-open delegation poses
> > the issue in that there may be modified data in the client
> > cache while the file is not open.  In this situation, any
> > client which does not flush modified to the server on each 
> > close must make sure that the user receives appropriate
> > notification of the failure.
> > 
> > If there is modified data on the client, it must not be
> > flushed normally to the server.  A client may attempt to
> > provide a copy of the file data as modified during the
> > delegation under a different name, to ease recovery.  Use
> 
> The document should point out that this only works if the client has
> cached the entire file.

OK.

> 
> > of such a technique may be limited to files under a certain
> > size or may only be used when sufficient disk space is
> > guaranteed available within the target file system.
> 
> And if sufficient space (disk or otherwise) on the client's cache is
> guaranteed to be available.

You can do this a block at time so it shouldn't be too hard.
I guess there can deadlocks where all of your cache blocks are
dirty (and for this file) and doing a block-by-block copy is
the straw that breaks the camels back.  Hard to believe that
the client is going to survive if he avoids the copy once he
so close to the edge.  It needs to be made clear anyway that 
all of this on a best-effort basis.

> 
> > A.7.  Attribute caching
> 
> ...
> > Clients may cache file attributes obtained from the server and
> > use them to avoid subsequent GETATTR requests.  Such caching
> > is write-through in that modification to file attributes is
> > always done my means of requests to the server and should
> 	      ^^
> 		by

OK.

> 
> 
> > [[Add the following to page 129 in chapter 14, after the definition
> >   of fattr4_voltility:]]
> 
> > struct wcc4_info {
> > 	fattr4_change	before;
> > 	fattr4_change	after;
> > };
> 
> In this version of the document, the wcc4_info struct is now part of
> the response to several operations. The problems with this approach
> are:
> 
> 	- clients that don't want wcc are forced to process it, 
> even if to
> 		ignore it.

True but your proposal below would add to the data that must be 
processed by clients who do want wcc.  I think clients who want
it have to process (send or receive) 16 extra bytes they wouldn't
have had to process.  This is to save clients who don't want wcc
16 bytes.  This is exclusive of the discriminated union issue
which I'll treat separately.

So the question is which class is more numerous (and also how do
you count)?  I had assumed that we should count by client boxes
rather than client implementations (i.e. Joe's Term Project for
Protocols 302 doesn't count for that much).  As I understand it,
SUN's v3 client uses wcc_info and has found it to be beneficial.
I didn't do an exhaustive survey but that counts for a lot of
client boxes.

I put back wcc_info after asking whether people found it useful.
If it's not very useful, we shouldn't have it at all.  If the 
uses are few and far between then let's leave it out.  I just 
don't see why of all the v3 features that people have found
useful, this one needs to be singled out for special handling
to avoid burdening some clients with a relatively small number 
of extra bytes.  I think this adds needless complexity. 
   
> 
> 	- servers that can't or won't support wcc are forced to returns
> 		values for wcc4_info, which while accurate, aren't
> 		atomic with respect to the operation, and so the server
> 		is effectively lying.

Good point.

There are a few ways we can deal with this.

    Make wcc_info into a discriminated union which is part of
    the result of all of the operations

    Add a boolean atomicity field to wcc_info.

    Add a per-fsid attribute wcc_info_atomicity.

    Distinguish a special value of wcc_info as indicating no valid
    data.  For example, if before and after were both zero, then the
    client could treat this as no information since, given the fact
    that you are changing the directory, the value of change before
    and after the op had better be different.

I prefer the boolean atomicity field unless we are so concerned 
about four extra bytes, in which case I would go for the 
special value approach (Gee, that almost sounds like you get 
fries with that wcc_info :-)

> 
> 	- clients that are using wcc to determine directory 
> cache consistency
> 		have to have specialized code to in each 
> operation that has
> 		wcc results.

I don't see how your proposal helps this.

Presumably you can have a subroutine to process the wcc_info
so your specialized code for each of the ops, which you are 
going to have anyway, needs to make one call.

With your proposal, that specialized code is going to have
to call a routine (or a macro) to set up the wcc_start, to
set up the wcc_end and to process the wcc_end result.  There
would have to be a corresponding set of stuff in the 
with_target cases.  I can't see that this is simpler.
  
> 
> An alternative (short of supporting delegation on directories, which
> I've suggested previously) would be to introduce a pair of operations
> to "lock" and "unlock" the operation the client wishes to get wcc
> infomration for.
> 
> wcc_start would have a null args, (and would assume the current fh and
> its response would be an nfsstat4) wcc_end would have null args, and
> its response would be a discriminated union on nfsstat4, with a
> wcc4_info result.
> 
> wcc_start_with_target would have a single argument which is the file
> handle that operations like LINK and RENAME have (because those
> operations pick up the source directory from the current file handle,
> and the target directory is explicitly in the operations arguments).
> wcc_end_with_target's results have a discriminated union with two
> wcc4_info fields, or perhaps two discriminated unions, which I could
> imagine might be necessary if a server's API let it atomically do
> pre-op getattr, op, post-op getattr on one of the target or source
> directory, but not both. I favor two discriminated unions.
> 
> I place "lock" and "unlock" in quotations for two reasons:
> 
> 	- they are not operations that get a record lock, 
> mandatory lock,
> 		or share reservation on the file, but rather operations
> 		that cause the NFS server to acquire an operating
> 		system mutual exclusion primitive on the file before
> 		doing the pre-operation get change attribute request.
> 		The operations also cause the NFS server to release the
> 		mutual exclusion primitive after doign the
> 		post-operation get change attribute request.
> 
> 	- they do not necessarily immediately result in the 
> acquisition or
> 		release of said mutual exclusion primitive. I expect
> 		that in some NFS servers, such as the one I work on,
> 		that they are simply hints to engine processing
> 		COMPOUND to do the right thing.
> 
> wcc_start*/wcc_end wrap one and only one operation.

I suppose it can be any operation rather than the few for which
it is truly needed.  I think that adds complexity.

This is adding four new ops and a new form of interaction
between sets of ops (I think that complicates error recovery).
I think adding a few parameters to some existing ops is a
whole lot simpler.

> 
> >     struct OPEN4args {
> >             open4CLAIMtype   claim;
> > 	    open4CLAIM	     claim_specific;
> > 	    nfs_lockowner    owner;
> > 	    seqid4	     seqid;
> > 	    int32_t	     access;
> > 	    int32_t          deny;
> >     };
> > 
> ...
> >     union open4CLAIM switch (open4CLAIMtype claim) {
> 
> Given that "claim" appears in the open4CLAIM discriminated union,
> "claim"'s presence as a field in OPEN4args appears to be redundant.

OK.  I was confused by the conventions.  I could have sworn I
saw a case where somethng similar was done in the existing spec.
If I remember where it was, I'll point it out to Spencer.

> 
> >             case CLAIM_NULL:    /* No special rights to 
> file.  Ordinary
> >                                    OPEN of the specified file. */
> > 	            /* CURRENT_FH: directory */
> > 		    filename4        file;
> > 
> > 	    case CLAIM_PREVIOUS:/* Right to the file established by an
> >                                    open previous to server reboot.  
> >                                    File identified by 
> filehandle gotten
> > 		                   at that time rather than by name. */
> > 	            /* CURRENT_FH: file being reclaimed */
> >                     int32_t	      dlg_type;
> > 
> > 	    case CLAIM_DLG_CUR: /* Right to file based on a delegation 
> >                                    granted by the server.  File is 
> >                                    specified by name. */
> > 		    /* CURRENT_FH: directory */
> >                     open4CLAIMdlgCur dlg_cur_info;
> >  
> >             case CLAIM_DLG_PREV:/* Right to file based on a 
> delegation
> >                                    granted to a previous 
> boot instance of
> >                                    the client.  File is 
> specified by name. */
> >                     /* CURRENT_FH: directory */
> >                     filename4	     file;
> >     };
> 
> It would be helpful, to me at least, if the relevent text preceding
> chapter 14, would refer to CLAIM_NULL, CLAIM_PREVIOUS, CLAIM_DLG_CUR,
> and CLAIM_DLG_PREV. This way, as a client implementor, I'd 
> know when to
> send them and as a server implementor what to do with them. I realize
> that some of the forward references need to be made in the base
> document, so this is advice for Spencer as much as Dave.

Makes sense to me.

> 
> > [[In section 10.16, replace the result section with the following:]]
> > 
> >     /*
> >      * Delegation types
> >      */
> >     const OPEN4_DLG_NONE	= 0x0000;	/* No open delegation */
> >     const OPEN4_DLG_READ	= 0x0001;	/* Read open 
> delegation */
> >     const OPEN4_DLG_WRITE	= 0x0002;	/* Write open 
> delegation */
> 
> 
> Unless these bits can be ORed together, dlg_type should be an 
> enum, and
> the above values changed to decimal 0, 1, 2 in the enum.

They can't be Ored and an enum is fine.

> 
> > struct OPEN4writeDelegation {
> > 	stateid4	stateid;	/* Stateid for delegation */
> > 					   be flushed to the 
> server on close. */
> 					^^^^^^^^^^^^^^^^^^
> 
> something appears to be missing here.

Actually there is something extra.  The line you flagged
just plain shouldn't be there.  

> 
> >         bool            recall;         /* Pre-recalled 
> flag for delegations
> >                                            obtained by reclaim
> > (CLAIM_PREVIOUS) */        offset4		size_limit;	
> /* Maximum size of file which
> > will not
> >         offset4		size_limit;	/* Maximum size 
> of file which will not
> > 					   be flushed to the 
> server on close. */
> > 	nfsacl4		permissions;	/* Defines users who 
> don't need an
> > 					   ACCESS call as part 
> of a delegated
> > 					   open. */
> > };
> 
> 
> 	-mre
>

---

From: Mike Eisler <[email-masked]>
Date: Wed, 11 Aug 1999 11:40:18 -0700 (PDT)

> Date: Wed, 11 Aug 1999 09:14:55 -0700
> From: "Noveck, Dave" <[email-masked]>
> Subject: RE: Caching proposal, take 2 (LONG)
> To: "'Mike Eisler'" <[email-masked]>, [email-masked]
> 
>  

I recognize that there may be situations where there is a single writer,
and the proposed caching scheme allows the writer to lazily write
the data to the server after it unlocks the region, whereas with v3, the
application must flush the data immediately (before the unlock) to
ensure data consistency. (BTW, the document should note situations such as
the aforementioned, where the proposed caching scheme is beneficial).
What is not clear if such situations are compelling.
 
>        that you always flush on close (just as if the file size

(d) skip write delegations.

...
> understand why you think it is a critical issue.

It's critical because NFS V4's strongest appeal might be with Big Iron, and
Big Iron means lots of big files, and lots of clients to access them. 

> implement that simplification if and when this goes into

I don't want to stand in way of consensus either. I simply wanted to
make the point, one more time before implementation, to see if I was
alone on this.

In absence of opinions that agree with mine, I think the consensus is (a).

> really wants a).  Silence will imply that we go to b).

If as an implementor, I don't see benefit to (b), then I can simply implement
(a) and report a file size limit of 0 in the serve, and in the client, always
flush on close regardless of space reservation. So I propose a separate message
that lays out the issue, including the fact that (b) is merely an
implementation detail, and asks if there is anyone who really does not want
(a).

> Or am I missing something?

I think I was missing the discussion in 7.5. I don't understand how a lock
server (in absence of a call back protocol) can detect network partition:

"When there is a network partition and the lease expires, the server
   must record on stable storage the client information relating to
   those leases. "

The caching proposal might be able to detect partition via the call back
protocol, but since the locking protocol is supposed to work through
firewalls, it can't count on callbacks.  I'll complain about this in a
separate message.

So your proposal could refer to 7.5, noting that the callback protocol is used
to detect partition.

...
>      as part of the close corresponding to the open under which 

If there is disk space, then the server delegates the open. Then why wouldn't
there be disk space?

Also the above seems to be sayng that the open will be delegated if there
is sufficient disk space to report a lack of disk space.

...
> You can do this a block at time so it shouldn't be too hard.

I don't understand. I though the issue in A.6.1 is what happens when a write
delegation is revoked, and now the client is trying to write its modified
version to  a new file. Such a technique can't work if the client didn't have
enough room to cache the whole file before it dirtied it.

> which I'll treat separately.

An issue is not so much extra bytes on the wire, but coding complexity.

> I put back wcc_info after asking whether people found it useful.

At issue isn't whether wcc is useful, it is whether the V3 implementation of
it is appropriate.

...
> fries with that wcc_info :-)

I still prefer a separate operation for returning the information. However, in
the interest of moving forward with your caching proposal, I suggest that we
add wcc_info with the boolean atomicity (one boolean per wcc4_info). 
Meanwhile, time permitting, I'll prototype wcc_start/wcc_end and if I'm right,
show why it's better.

	-mre