Subject: RE: COMPOUND and the nfs reply cache

From: Brent Callaghan <[email-masked]>
Date: Mon, 4 Oct 1999 11:44:01 -0700 (PDT)

I don't have any problem with v4 servers returning NFS4ERR_RESOURCE
if the server encounters a resource problem that prevents it from
completing the compound request. The client needs to deal with that
by being conservative in its compound requests.

However, Dave's original posting mentioned that his server would
return NFS4ERR_RESOURCE for any non-idempotent compound request
that contained lots of data in the response.  The reason given,
is that large responses will overload his duplicate request cache.
Dave's server is able to complete the request successfully, but
instead of returning the response, it returns an error because
it doesn't want to stash it in the duplicate request cache.

There's no requirement in the v4 spec that servers use a
duplicate request cache.  Nor is there a requirement in
the v2 or v3 spec.  No doubt, they're a good idea to have
and server implementors should be encouraged to use them,
but nowhere does the protocol require any guarantees from
the server that it'll detect retransmitted requests (except
for special cases like the CREATE verifier).

On the other hand, the duplicate request cache is just a cache.
There's never any guarantee that a retransmission will be 
detected by the cache.  Eventually, entries will fall out
of the cache - especially if the server is under heavy load.
So the client will sometimes be exposed to the effects of
duplicate retransmissions.  

If the server hasn't got room in its duplicate retransmission
cache, then it's perfectly OK for the server not to cache the
response.  So my advice is: return the uncached response but
don't return a resource error.

	Brent

---

From: Carl Beame <[email-masked]>
Date: 04 Oct 1999 20:44:44 +0100

> for special cases like the CREATE verifier).

Just a note of commercial reality here...

There was an older version of a VMS NFS Server which did NOT have a
duplicate request cache for non-idempotent operations and our client
with Microsoft Word running over it kept failing. You see Microsoft
Word renames like mad and on a typical client network (ie. real
dirty), replies were lost on a regular basis and OUR client NFS was
crashing Microsoft Word. Imaging us trying to tell our customer that
our client NFS was not at fault it was the server ... which was fully
compliant to the NFS specification.

I propose that a non-idempotent response cache IS a requirement of NFS
V4. 

At the same time we should limit the size of any element in the
response cache which contains a non-idempotent COMPOUND request.
(maybe to one ethernet packet size). We should also give examples of
non-idempotent COMPOUND requests (ie. RENAME, READ/WRITE, DELETE, ...).

If we don't do this, each of us could create a NFS V4 compliant server
and find that some COMPLIANT client is causing us problems. And as
well all know, telling one's customer that the client NFS
implementation is at fault, not our server, DOES NOT WORK!

- Carl

---

From: Mike Eisler <[email-masked]>
Date: Mon, 4 Oct 1999 13:25:44 -0700 (PDT)

Several things, some random, have occured to me throughout this discussion:

(1) With nfs/tcp, dup requests are a rarity:

jurassic% nfsstat -s     

Server rpc:
Connection oriented:
calls         badcalls      nullrecv      badlen        xdrcall       
270034517     1             0             0             1             
dupchecks     dupreqs       
21106029      20            
Connectionless:
calls         badcalls      nullrecv      badlen        xdrcall       
768197        0             0             0             0             
dupchecks     dupreqs       
9422          94            

Server nfs:
calls         badcalls      
270762326     8606          

1% of non-idempotent UDP calls hit the dup request cache, and 
0.0001 % of TCP calls hit the cache. It is possible that my server's
cache is too small, but even if it were, I'd expect a higher hit rate.

The dup request cache is for caching responses to executed requests.
If we add the require that NFS V4 servers over TCP must not drop the
response to any executed request, except when the connection breaks,
then dup requests from the client requiring a hit in the dup
req cache are very rare.

(2) delegations further increase the rarity of duplicate requests.

(3) Server's don't have to return all N bytes requested in a READ or READDIR.
This would let a server return fewer bytes than asked for if the COMPOUND
contains a non-idempotent operation. For example,  the server could return the
full results in the initial request, and cache an abbreviated result in the
dup req cache.

(4) Not all non-state modifying ops result big results ... LOOKUP for example
has simple a 32 bit error code. True it is likely to have a file handle and
attributes tacked on in other operations, but that's often going to be about
the size of NFS V3's post op attributes for a WRITE responses.

(5) The problem is now reduced to the situation of state modifying ops (SMOs)
mixed with non-state-modifying op with big, non-truncatable results
(NSMOWBNTRs). An example of an NSMOWBNTR might be READLINK, or a GETATTR
with a big ACL.

While Dave proposed returning NFS4ERR_RESOURCE for each SMO that appeared in
the same COMPOUND as non-state modifying op (NSMO), it seems to me that
limiting this approach to SMOs mixed with NSMOWBNTRs is both sufficient and
rare enough to not be a significant issue.

	-mre

---

From: Carl Beame <[email-masked]>
Date: 21 Oct 1999 15:23:52 +0100

Section 5.5 Recommended Attributes

aclsupport 53	uint32	READ Indicates what ACLs are supported on the current filesystem
/*
Mask that indicates which Access Control Entries are supported.
*/
const ACL4_SUPPORT_ALLOW_ACL = 0x0001;
const ACL4_SUPPORT_DENY_ACL = 0x0002;
const ACL4_SUPPORT_AUDIT_ACL = 0x0004;
const ACL4_SUPPORT_ALARM_ACL = 0x0008;

The Server indicates which types of Access Control Lists of identifiers it supports:
 
ALLOW - Explicitly grants the access defined in acemask4 to the file or directory
DENY  - Explicitly denies the access defined in acemask4 to the file or directory 
AUDIT - LOG (system dependant) any access attempt to a file or directory which uses an
        access method which is a subset of acemask4.
ALARM - Generate a system ALARM (system dependant) when any access attempt is made to
        a file or directory which is a subset of acemask4

ACL	11	nfsace4<>	R/W	The Access Control List for the object.

***** Note that all references to nfsacl4 should be changed to nfsace4 in the draft *****
 
struct nfsace4 {
	acetype4 type;
	aceflag4 flag;
	acemask4 access_mask;
	utf8 who;
};

Each nfsace4 entry is assumed to be processed in order by the server. The first Access
Control Entry is used where both the "who" and the "access_mask" match the requester and the 
type of access desired. Any later additional Access Control Entries which also match are
ignored.

typedef uint32_t acetype4;
/*
acetype4 values, others can be added as needed.
*/
const ACE4_ACCESS_ALLOWED_ACE_TYPE = 0x0;
const ACE4_ACCESS_DENIED_ACE_TYPE = 0x1;
const ACE4_SYSTEM_AUDIT_ACE_TYPE = 0x2;
const ACE4_SYSTEM_ALARM_ACE_TYPE = 0x3;

typedef uint32_t aceflag4;
/*
ACE4_FILE_INHERIT_ACE can be placed on a directory and indicates that 
	this ACE should be added to each new non-directory file created.

ACE4_DIRECTORY_INHERIT_ACE can be placed on a directory and indicates 
	that this ACE should be added to each new directory created.

ACE4_INHERIT_ONLY_ACE can be placed on a directory but does not apply 
	to the directory, only to newly created files/directories as 
	specified by the above two flags.

ACE4_NO_PROPAGATE_INHERIT_ACE can be placed on a directory. Normally 
	when a new directory is created and an ACE exists on the parent 
	directory which is marked ACL4_DIRECTORY_INHERIT_ACE, two ACEs 
	are placed on the new directory. One for the directory itself and
	one which is an inheritable ACE for newly created directories.
	This flag tells the O/S to not place an ACE on the newly created
        directory which is inheritable by subdirectories of the created directory.

ACE4_SUCCESSFUL_ACCESS_ACE_FLAG and ACL4_FAILED_ACCESS_ACE_FLAG indicate for 
	AUDIT and ALARM which state to log the event.
	On every ACCESS or OPEN call which occurs on a file or directory
	which has an ACL that is of type ACE4_SYSTEM_AUDIT_ACE_TYPE or
	ACE4_SYSTEM_ALARM_ACE_TYPE, the attempted access is compared to the
	ace4mask of these ACLs. If the access is a subset of ace4mask 
	and the identifier match, an AUDIT trail or an ALARM is generated. 
	By default this happens regardless of the success or failure of the
	ACCESS or OPEN call.   

	The flag ACE4_SUCCESSFUL_ACCESS_ACE_FLAG only produces the AUDIT or
	ALARM if the ACCESS or OPEN call is successful. The
	ACE4_FAILED_ACCESS_ACE_FLAG causes the ALARM or AUDIT if the ACCESS or
	OPEN call fails.
 
ACE4_IDENTIFIER_GROUP indicates that the "who" refers to a GROUP as defined under Unix.

****** Area for Discussion *****************
	If a client or server does not have the concept of GROUP, it SHOULD (a) ignore
	this flag and treat to "who" as best it can, or (b) return an error ????

	It should be noted that global identifiers [email-masked] don't have a GROUP 
	attribute and thus it might not be programmatically possible to identify that
	the "who" in a SETATTR call is of type group or user. It should be noted that
	under Windows NT a group name and a user name cannot be the same, while under
	Unix they can. I personally would suggest that if the server can itself 
	determine if an identifier is a group it should ignore this flag on a SETATTR
	request and return it on a GETATTR request.
********************************************

*/
const ACE4_FILE_INHERIT_ACE = 0x0001;
const ACE4_DIRECTORY_INHERIT_ACE = 0x0002;
const ACE4_NO_PROPAGATE_INHERIT_ACE = 0x0004;
const ACE4_INHERIT_ONLY_ACE = 0x0008;
const ACE4_SUCCESSFUL_ACCESS_ACE_FLAG = 0x0010;
const ACE4_FAILED_ACCESS_ACE_FLAG = 0x0020;
const ACE4_IDENTIFIER_GROUP = 0x0040;

typedef uint32_t acemask4;

const ACE4_READ_DATA = 0x0001; // Permission to read the data of the file
const ACE4_LIST_DIRECTORY = 0x0001; // Permission to list the contents of a directory
const ACE4_WRITE_DATA = 0x0002; // Permission to modify the file's data
const ACE4_ADD_FILE = 0x0002; // Permission to add a new file to a directory
const ACE4_APPEND_DATA = 0x0004; // Permission to append data to a file
const ACE4_ADD_SUBDIRECTORY = 0x0004; // Permission to create a subdirectory to a directory
const ACE4_READ_STREAMS = 0x0008; // Permission to read the additional streams of a file
const ACE4_WRITE_STREAMS = 0x0010; // Permission to write the additional streams of a file
const ACE4_EXECUTE = 0x0020; // Permission to execute a file
const ACE4_DELETE_CHILD = 0x0040; // Permission to delete a file or directory within a directory
const ACE4_READ_ATTRIBUTES = 0x0080; // The ability to read basic attributes (non-acls) of a file 
const ACE4_WRITE_ATTRIBUTES = 0x0100; // Permission to change basic attributes (non-acls) of a file

const ACE4_DELETE = 0x00010000; // Permission to Delete the File, IF FILE BASED
const ACE4_READ_ACL = 0x00020000; // Permission to Read the ACL
const ACE4_WRITE_ACL = 0x00040000; // Permission to Write the ACL
const ACE4_WRITE_OWNER = 0x00080000; // Permission to change the owner
const ACE4_SYNCHRONIZE = 0x00100000; // Allow the forcing of mutual-exclusion to the file

const ACE4_GENERIC_READ = (ACE4_READ_CONTROL|ACE4_READ_DATA|ACE4_READ_ATTRIBUTES|ACE4_READ_EXTENDED_ATTRIBUTES|ACE4_SYNCHRONIZE);
const ACE4_GENERIC_WRITE = (ACE4_READ_CONTROL|ACE4_WRITE_DATA|ACE4_WRITE_ATTRIBUTES|ACE4_WRITE_EXTENDED_ATTRIBUTES|ACE4_APPEND_DATA|ACE4_SYNCHRONIZE);
const ACE4_GENERIC_EXECUTE = (ACE4_READ_CONTROL|ACE4_READ_ATTRIBUTES|ACE4_EXECUTE|ACE4_SYNCHRONIZE);

There are several special identifiers ("who") which need to be understood 
universally. Some of these identifiers cannot be understood when an NFS
client accesses the server, but have meaning when a local process accesses the
file. The ability to display and modify these permissions is permitted over NFS.

"OWNER" - The owner of the file.
"GROUP" - The group associated with the file.
"EVERYONE" - The world.
"INTERACTIVE" - Accessed from an interactive terminal.
"NETWORK"- Accessed via the network.
"DIALUP" - Accessed as a dialup user to the server.
"BATCH" - Accessed from a batch job.
"ANONYMOUS" - Accessed without any authentication.
"AUTHENTICATED" - Any authenticated user (opposite of ANONYMOUS)
"SERVICE" - Access from a system service.

To avoid conflict these special identifiers should be of the form "xxxx@".

- Carl Beame