Subject: mmap and locking (was "Caching proposal, take 2 (LONG)")

From: Mike Kupfer <[email-masked]>
Date: Mon, 09 Aug 1999 18:32:14 -0700

I've looked through your proposal, and I'm afraid I'm having problems
getting a handle on it.  So I'll just ask: what are the implications
of your proposal for applications that want to mmap a file,
particularly when the file is locked?

The reason I ask is that the Solaris v2/v3 client completely disallows
concurrent mmapping and locking of an NFS file.  This is to avoid
problems such as those that you discuss in Section A.4.2, where a
client needs to avoid writing the entire page if only part of the page
was locked.  This restriction will likely be relaxed some in Solaris
8, but we're getting customer requests to make mmap and locking play
together in all circumstances.

I don't have a problem with telling these customers that they're not
going to get what they want.  (In the general case they are asking for
complete cache consistency.)  But I want to understand what they
*would* get.  For example, would the client ever have to revoke a
previously granted mmap request?  And I'd like to make sure the
Working Group remembers mmap when it considers how much caching v4
should support.

cheers,
mike

---

From: Mike Eisler <[email-masked]>
Date: Tue, 10 Aug 1999 16:31:49 -0700 (PDT)

> Date: Tue, 3 Aug 1999 08:47:04 -0700 
> From: "Noveck, Dave" <[email-masked]>
> Subject: Caching proposal, take 2 (LONG)
> To: "[email-masked]'" <[email-masked]>

There is one critical issue, which I will belabor one more time (at
least until there've been attempts to implement it), is that I consider
the file size limit approach on write delegations to be an inadequate
solution to the problem of running out of disk space on a server after
the client flushes its dirty data. It is insufficient because it limits
the server to serving lots of clients with small files, or small
numbers of clients with large files, but no lots of clients with lots
of large files. In absence of a sufficient solution, I don't think
write caching is worth doing.

>   Changes addressing comments made about first version (I
>   believe I did what commenter wanted):
...
>        Giving the server more freedom to delay cancellation
>        (now revocation) of a delegation if a client seems to be
>        making progress in flushing state.  (Issue raised by Mike
>        Eisler).

I looked for it, but couldn't find it.

> A.3.1.  Delegation recovery
> 
> There are three situations that delegation recovery must deal with:
> 
> o    Client reboot
> 
> o    Server reboot
> 
> o    Network partition (full or callback-only)
> 
> In the even of a client reboot, the failure to renew leases will
> result in the revocation of record locks and share reservations. 
> Delegations, however, may treated a bit differently.   
> 
> Because data associated with some delegations may be written to
> stable storage
^^^^^^^^^^^^^^^

For clarity, the document should make it clear that "stable storage"
refers to stable storage on the client.
...

> When a network partition occurs, delegations, like other locks
> will subject to freeing when the lease renewal period expires,
> although the server will normally extend the period in which
> conflicting requests are held off in the case of delegations.
> Eventually, however, the occurrence of a conflicting request
> from another client will cause revocation of the delegation.  
> A blockage of the callback (e.g. by later network configuration 
> change) will have the same effect.  A recall request will fail 
> and revocation of the delegation will result.  
> 
> A client normally finds out about revocation of a delegation 
> when it uses a stateid associated with a delegation and 
> receives the error NFS4ERR_EXPIRED.  It also may find out about
> delegation revocation after a client reboot when it attempts
> to reclaim a delegation and receives that same error.

The above doesn't explain what happens  if the delegation is revoked,
and another client has written the file. If the revoked client had a
read delegation this is one thing, but what if the revoked client had a
write delegation?  What does the client do with its dirty data, and
what are some of the recommened actions with respect to the
interfaction between the client and the application the client is
operating on the nehalf of (in the common siutation when the
application anbd client are layered entities).  This is covered in
general terms in section A.6.1, and so a reference to A.6.1 is needed
here.

Then there is the situation when the server revokes the write
delegation during partition, and another client write the file, and
then tserver reboots before the revoked client determines that it lost
its delegation. This can be detected by looking at the change
attribute. Note that I raised this issue in my review fo the first
version of the his document. If it has been addressed elsewhere in the
document, my mistake.

> A.4.3. Data caching and mandatory file locking
> 
> Client-side data caching needs to respect mandatory file
> locking when this is in effect.  The presence of mandatory
> file locking for a given file is indicated the result flags
					^^^^^^^
					typo: missing "in"
> for an OPEN.

> A.5.  Open delegation
...
> The following is a typical set of conditions 
> that servers might use in deciding whether open should be delegated:
...
> o    Availability of disk space on the server to guarantee that
>      when modified files are written to the server, that fact 
							^^^^^^^

>      can be detected and reported to the user as part of the

which "fact" can be detected?
 
>      close corresponding to the open under which the writes
>      were done.

> A.5.  Open delegation
...
> The use of delegation together with various other forms of 
> caching creates the possibility that no server authentica-
> tion will ever be performed on a given user since all of 
> his requests might be satisfied locally.  Where the client is
> depending of the server for authentication, it should make
> sure that some authentication (via an ACCESS call) happens
> for each user, even if an ACCESS call would not otherwise
> be required.  The server may enforce frequent authentication
> by returning an nfsacl4 denying all access with every open
> delegation.  

In my review of the first version of this document, I noted:

	"Another related issue, is what if there are no unexpired
	credentials available for the client after the file has been
	closed locally, user has logged off, and the server sends a
	revoke request? AFS and DFS, which use Kerberos authentication,
	must have had to deal with this problem, so perhaps WG members
	from MIT or TransARC could advise here."

Ted Anderson responded:

	"No magic bullet here. Really, you need to track the expiration
	time of the credentials you are depending upon and flush dirty
	data well before it arrives. Further, if you can't get
			 ^^

	notification of impending unlog (kpurge) operations which allow
	you to flush dirty data, you can make private copies of the
	creds you are using, and purge them after the dirty data is
	stored."

I assume that "it" refers the expiration time of the credentials.

The issue I raised and the solution Ted described to need to be
mentioned somewhere in the document.

> A.5.1  Open delegation and data caching
...
> In order to maintain current semantics in which the non-availability
> of storage to hold a file written by an NFS client is guaranteed
> to be determined at or before the associated close operation,
> the avoidance by the client of the requirement to flush data to
> the server on close, is limited to files below a certain size, 
> determined by the server, and communicated to the client as part
> of the delegation.  It is up to the server to reserve sufficient 
> space so that all files whose size is under that specified in a 
> write open delegation can be successfully written, and to recall 
> delegations appropriately to maintain that invariant.

See my comment at the start of this message.

> A.5.3.  Recall of open delegation
...
> Special handling is needed for a GETATTR which occurs when a write
> open delegation is in effect.  In this case, the client holding the 
> delegation needs to be interrogated, using a BACK_GETATTR callback, 
> if the GETATTR attribute bits include any of the attributes that a
> write open delegate may modify (length, modify time, change).  

Is the above list of attributes intended to be an example, or is it the
complete set of attributes that a write delegation can modify?

It would be useful if the document prior the A.5.3 specified what
exactly, beyond data, is delegated to the client in a delegation.

> A.6.   Data caching and revocation
> 
> When locks (including delegations) are revoked, the assumptions
> upon which successful caching depend, are no longer guaranteed.
> Therefore the client, in addition to notifying the owner of a 
					^^^^^^^^^^^^^^				
> record lock or share reservation, and processes holding opens 
				    ^^^^^^^^^^^^^^
> for the delegation, needs to remove all data for the file from 
> its cache.  In the case of modified data, it must be removed 
> from the client's cache without being written to the server.

Notification to processes is going to be operating system specific.
Nonetheless, suggestions of possibilities for notification would be
useful to let the implementor understand the ramifications.

> A.6.1.  Revocation recovery for write open delegation
> 
> Revocation recovery for a write-open delegation poses
> the issue in that there may be modified data in the client
> cache while the file is not open.  In this situation, any
> client which does not flush modified to the server on each 
> close must make sure that the user receives appropriate
> notification of the failure.
> 
> If there is modified data on the client, it must not be
> flushed normally to the server.  A client may attempt to
> provide a copy of the file data as modified during the
> delegation under a different name, to ease recovery.  Use

The document should point out that this only works if the client has
cached the entire file.

> of such a technique may be limited to files under a certain
> size or may only be used when sufficient disk space is
> guaranteed available within the target file system.

And if sufficient space (disk or otherwise) on the client's cache is
guaranteed to be available.

> A.7.  Attribute caching

...
> Clients may cache file attributes obtained from the server and
> use them to avoid subsequent GETATTR requests.  Such caching
> is write-through in that modification to file attributes is
> always done my means of requests to the server and should
	      ^^
		by

> [[Add the following to page 129 in chapter 14, after the definition
>   of fattr4_voltility:]]

> struct wcc4_info {
> 	fattr4_change	before;
> 	fattr4_change	after;
> };

In this version of the document, the wcc4_info struct is now part of
the response to several operations. The problems with this approach
are:

	- clients that don't want wcc are forced to process it, even if to
		ignore it.

	- servers that can't or won't support wcc are forced to returns
		values for wcc4_info, which while accurate, aren't
		atomic with respect to the operation, and so the server
		is effectively lying.

	- clients that are using wcc to determine directory cache consistency
		have to have specialized code to in each operation that has
		wcc results. 

An alternative (short of supporting delegation on directories, which
I've suggested previously) would be to introduce a pair of operations
to "lock" and "unlock" the operation the client wishes to get wcc
infomration for.

wcc_start would have a null args, (and would assume the current fh and
its response would be an nfsstat4) wcc_end would have null args, and
its response would be a discriminated union on nfsstat4, with a
wcc4_info result.

wcc_start_with_target would have a single argument which is the file
handle that operations like LINK and RENAME have (because those
operations pick up the source directory from the current file handle,
and the target directory is explicitly in the operations arguments).
wcc_end_with_target's results have a discriminated union with two
wcc4_info fields, or perhaps two discriminated unions, which I could
imagine might be necessary if a server's API let it atomically do
pre-op getattr, op, post-op getattr on one of the target or source
directory, but not both. I favor two discriminated unions.

I place "lock" and "unlock" in quotations for two reasons:

	- they are not operations that get a record lock, mandatory lock,
		or share reservation on the file, but rather operations
		that cause the NFS server to acquire an operating
		system mutual exclusion primitive on the file before
		doing the pre-operation get change attribute request.
		The operations also cause the NFS server to release the
		mutual exclusion primitive after doign the
		post-operation get change attribute request.

	- they do not necessarily immediately result in the acquisition or
		release of said mutual exclusion primitive. I expect
		that in some NFS servers, such as the one I work on,
		that they are simply hints to engine processing
		COMPOUND to do the right thing.

wcc_start*/wcc_end wrap one and only one operation.

>     struct OPEN4args {
>             open4CLAIMtype   claim;
> 	    open4CLAIM	     claim_specific;
> 	    nfs_lockowner    owner;
> 	    seqid4	     seqid;
> 	    int32_t	     access;
> 	    int32_t          deny;
>     };
> 
...
>     union open4CLAIM switch (open4CLAIMtype claim) {

Given that "claim" appears in the open4CLAIM discriminated union,
"claim"'s presence as a field in OPEN4args appears to be redundant.

>             case CLAIM_NULL:    /* No special rights to file.  Ordinary
>                                    OPEN of the specified file. */
> 	            /* CURRENT_FH: directory */
> 		    filename4        file;
> 
> 	    case CLAIM_PREVIOUS:/* Right to the file established by an
>                                    open previous to server reboot.  
>                                    File identified by filehandle gotten
> 		                   at that time rather than by name. */
> 	            /* CURRENT_FH: file being reclaimed */
>                     int32_t	      dlg_type;
> 
> 	    case CLAIM_DLG_CUR: /* Right to file based on a delegation 
>                                    granted by the server.  File is 
>                                    specified by name. */
> 		    /* CURRENT_FH: directory */
>                     open4CLAIMdlgCur dlg_cur_info;
>  
>             case CLAIM_DLG_PREV:/* Right to file based on a delegation
>                                    granted to a previous boot instance of
>                                    the client.  File is specified by name. */
>                     /* CURRENT_FH: directory */
>                     filename4	     file;
>     };

It would be helpful, to me at least, if the relevent text preceding
chapter 14, would refer to CLAIM_NULL, CLAIM_PREVIOUS, CLAIM_DLG_CUR,
and CLAIM_DLG_PREV. This way, as a client implementor, I'd know when to
send them and as a server implementor what to do with them. I realize
that some of the forward references need to be made in the base
document, so this is advice for Spencer as much as Dave.

> [[In section 10.16, replace the result section with the following:]]
> 
>     /*
>      * Delegation types
>      */
>     const OPEN4_DLG_NONE	= 0x0000;	/* No open delegation */
>     const OPEN4_DLG_READ	= 0x0001;	/* Read open delegation */
>     const OPEN4_DLG_WRITE	= 0x0002;	/* Write open delegation */

Unless these bits can be ORed together, dlg_type should be an enum, and
the above values changed to decimal 0, 1, 2 in the enum.

> struct OPEN4writeDelegation {
> 	stateid4	stateid;	/* Stateid for delegation */
> 					   be flushed to the server on close. */
					^^^^^^^^^^^^^^^^^^

something appears to be missing here.

>         bool            recall;         /* Pre-recalled flag for delegations
>                                            obtained by reclaim
> (CLAIM_PREVIOUS) */        offset4		size_limit;	/* Maximum size of file which
> will not
>         offset4		size_limit;	/* Maximum size of file which will not
> 					   be flushed to the server on close. */
> 	nfsacl4		permissions;	/* Defines users who don't need an
> 					   ACCESS call as part of a delegated
> 					   open. */
> };

	-mre