Subject: Feedback on draft-ietf-nfsv4-01 - filesystems

From: Neil Brown <[email-masked]>
Date: Mon, 11 Oct 1999 14:13:25 +1000 (EST)

This is my second Email with feedback in the draft-ietf-nfsv4-01.

NeilBrown

fsid issues:

 Previous versions of NFS have had the fsid attribute, but never really
 used it at all (except NFSv3 said that LOOKUP must return a filehandle
 with the same fsid as the parent).  NFSv4 says a lot of specific
 things about a filesystem without every saying what a filesystem "is"
 (of course, we all know what a Unix filesystem is, but does an NFS
 filesystem have to be the same?).

 draft-01 says the following about filesystems:

   A filesystem imbues each file in it with an 'fsid' attribute which
   contains major and minor components (section 5.4).  Why does NFS
   care about the substructure of an fsid?

   A filesystem provides all the files in it with certain other
   attributes which are uniform across the
   filesystems. e.g. symlink_support. (Section 5.4).

   A filesystem may have a limit on the amount of data and the number
   of files stored therein, and may be able to report how much space
   is currently used, or is free. (Section 5.5)

   If a filehandle has a "fileid" attribute, then it is unique across
   a filesystem.

   A filesystem may be replicated or migrated as a whole (section 6)

   The directories in a filesystem must form a strict
   hierarchy. (implied by the existence of LOOKUPP section 12.2.14).

   A filesystem is closed under the "LINK" operation (Section 12.2.9)
 
   A filesystem is closed under the "RENAME" operation (Section 12.2.25)

 draft-01 does NOT say about filesystems:

   A filesystem has to be contiguous under lookup/lookupp operations.
   Presumably it should, but I think this should be clear stated if it
   is desired.  This relates to the discussion of pseudo filesystems
   in section 7.3  If filesystems do not need to be contiguous, then
   there only needs to be one pseudo filesystem.

 Having single concept, "The filesystem", that covers all of these
 different ideas seem un-necessary in general.  For example, the set
 of filehandles across which the fileid is unique may be much larger
 than the set of filehandles which will be migrated as a group.  And
 set of files that is closed under the "LINK" operation  may be much
 smaller (in AFS, LINK much be within a directory).

 An idea that occurred to me (triggered by something I read in the
 mail archive, can't remember what) seems to provide a useful
 interpretation of the fsid field.  The idea involves redefining the
 "homogeneous" attribute, making it something that isn't really an
 attribute... 

[PROPOSAL:]
 Rather than using the word "filesystem", let's use a different word,
 "fileset".  This simply refers to a set of files.  Sets may have
 subsets, they may intersect, the potentially exhibit all the
 generality of (finite) mathematical sets.

 The server knows all about all files and may decide to group some of
 them together into sets to help the client "understand" what is going
 on.
 Some sorts of sets might be:
    The set of all files that are stored in a specific device.
    The set of all files owned by a specific user.
    The set of all files in a particular fileid namespace (numberspace?)
    A set of all files which is closed under "LINK" (i.e. any file
        in the set can be linked into any directory in the set)
    A set of all files which is closed under "RENAME"
    A set of all files in a particular type of filesystem (e.g. all
        files in any FAT filesystem, all files in any UFS filesystem).

 The server encodes this file set information in the fsid field.  The
 implementor would choose a few sorts of sets that seem relevant, and
 would allocate bitfields within the fsid for each sort of set.

 Now when the client does a GETATTR (or similar) and asked for the
 "homogeneous" attribute as well as some others, the server returns, as
 the homogeneous attribute, a mask which can be used to determine
 which files are in the same set as the target file with respect to
 the attributes requested.

 So, if I use GETATTR to ask for a couple of attributes, say
 "symlink_support" and "link_support" as well as "homogeneous", then I
 know that any two filehandles which present fsids which are the same
 as the target file under the returned mask have the same vales of
 "symlink_support" and "link_support". 

 Homogeneity needs to mean something a little bit different to "has
 the same value".  Each attribute might have a subtly different
 meaning. e.g.: a set of "homogeneous" files under each individual
   attribute below holds the given properties:

 symlink_support:  all directories may have symlinks created within
    them.
 link_support:  any file in the set can be linked into any directory in
    the set.
 fs_locations: all files in the set can be found at any of the given
    locations.
 quota_used:  all files in the set contribute toward the same quota
    usage value as is reported by this attribute
 space_total: all files in the set are stored in a "Device" which is
    of the given size
 fileid:  all files in the set have unique fileids.

 I would then define a new attribute "contiguous" which is always
 "true", but ensures that the set described by the "homogeneous" mask
 is contiguous.

 Now, if I want to make sure that I notice "mountpoints" in the Unix
 sense of them, I would get the "homogeneous" mask for the root
 filehandle for attributes:
	    space_total,fileid,contiguous
 And use that when comparing fsid's.  Once I find a 'fsid' which is
 different under that mask, I retrieve the "homogeneous" mask for the
 same attributes of that file, create a new mountpoint, and keep
 going.
 This assures me that inode number uniqueness is guaranteed, and that
 "df" will do "the expected thing".

 When I get told NFS4ERR_MOVED, I get the "fs_locations" attribute and
 at the same time check the "homogeneous" attribute. I now know exactly
 the set of files that has moved.

 The "contiguous" attribute would need to have one or two "special"
 value which are not masks. This is because as it stands, there is no
 way to say "This attribute applies only to this file" (as applies to
 size, etc).
 I would suggest:
    1 - the set is exactly this file.
    2 - the set is all files in the same directory as this one (not
         sure if this is useful or not.  I am thinking of afs which
	 has ACL's per directory, not per file).

 Some common masked would be
    0 - all files are in the set
    1's - only files with exactly the same fsid as this one are in the
       set 

 The server implementer would have to ensure that no real mask has the
 value "1" or "2".  This should not be hard to achieve.

 The set implied by the mask would not be required to be the largest
 set which meets the criteria, but the server should endeavour to make
 as large as reasonably possible.

 A simple server implementation could use device-number fsids, and
 return "contiguous" as "all-1s" or 1 as appropriate.  A more sophisticated
 server might encode lots of information in the fsid (how many bits
 would we want?).  A client should make sure that it asks for what it
 needs without assuming anything about the fsid.

 It is an important point that the test is not
   "any pair of fsid's that are the same under this mask are in the
    same set with respect to these attributes"
 but is
   "any file with an fsid that is the same as the fsid of this file is
    in the same set as this file"
 This is because different filesystems in the server may use different
 encodings to create the fsid.

 Given this new interpretation on fsid, we need to ask "how big should
 it be?" I am inclined to make it a
		int fsid<>;
 so the server can use whatever seems best to it.

 A related thought which grew out of this idea is to provide an
 attribute "submounts" which returns a mask similar to the
 "contiguous" mask, but which defines a set of file containing all
 hierarchical descendants of the given file. (the server should return
 the smallest reasonable set).

 Then, if there are no submounts beneath a given directory, the
 "submounts" field would be "all-1's" (or maybe a mask that specifies
 the variability within the filesystem).

 Given this, the client could walk the filetree searching for
 mountpoints, efficiently trimming the walk when it found any
 directory which had no submounts.  The client could determine what it
 needed to think of as a "mount" as described above.  This would
 answer the "Finding all exports for a server" question that was
 discussed earlier this year.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 13 Oct 1999 11:15:44 -0700 (PDT)

>    A filesystem has to be contiguous under lookup/lookupp operations.

What does this mean? An example illustrating contiguuous and
non-contiguous behavior would be useful.

	-mre

---

From: Neil Brown <[email-masked]>
Date: Thu, 14 Oct 1999 13:04:21 +1000 (EST)

> non-contiguous behavior would be useful.

Ok, this is possibly a subtle point and may not be very important, but
explaining might help to illuminate the whole idea.

By contiguous, I mean internally connected, no separate parts.
e.g. the set

    /a /a/b /a/c /a/b/1

is a continguous set of files.  If you draw the tree, you will only
get one tree.
                         a
                        / \
                       b   c
                       \
                        1

However
     /a/b /a/c
is not contiguous, as /a is not in the set (remembering that by "set"
I mean the pure mathematical concept: any collection).

One way of describing a contiguous set of files (by which I really
mean filesystem objects including directories) is:
  Each non-directory in the set must be in at least one directory that
    is also in the set
  The directories in the set must form a single hiearchy with a single
    root. 

One could argue that all sets reported by the server should be
contiguous, and I probably wouldn't argue too hard about that, but it
is worth thinking about contiguity too see how important it is (or
isn't).

Most sets that "feel" like filesystems are, of course, continguous.
However there are some other sets that could meaningfully be specified
by the server which are not contiguous.

e.g.:
   The set of all files that contribute to a specific quota value. 
       Given that quota can be based on owner or groupowner or
       filetree or probably anything else, the client cannot determine
       if two files are in the same quotaclass without something like
       a file set indicating the homogeneity of "quota_used".
       Would a client care?  I'm not sure but I cannot definately say
      "no". (maybe a UI that helps you reduce your quota usage by finding
       files that are charged to you and suggesting deletion).

   The set of files that form the "Pseudo File-System" described in
       section 7.3. 
       That section mentions the need to discuss "multiple pseudo
       filesystems", but doesn't yet.
       Presumably a reasonable implementation would create a set of
       files (well, directories) which tie the various export sets
       into a namespace, and have these directories in a single fileid
       numberspace.   Some clients might not care that these pseudo
       directories aren't contiguous, others would.

That last one was what started me thinking about contiguity.  The only
reason that you might need "multiple pseudo filesystems" was if the
client couldn't deal with dis-contiguous filesets, and surely that is
the client's decision to make.

It is, admittedly, hard to come up with really convincing examples of
non-contiguous filesets that are really useful, and I would not be
strongly against requiring all filesets reported by the server to be
contiguous.  But either way, the protocol spec should make it clear
what the client can expect, and what the server may provide.

NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Oct 1999 03:31:25 -0700 (PDT)

> I mean the pure mathematical concept: any collection).

Given that today NFS servers can do:
	share /a/b
	share /a/c

where b and c are subdirectories of a common file system, a filesystem
is not contiguous under lookup,lookupp. An NFS V4 client that is the server's
/a/b that does a LOOKUPP gets into the server's pseudo file system that
represents /a.

	-mre

---

From: Neil Brown <[email-masked]>
Date: Fri, 15 Oct 1999 12:58:24 +1000 (EST)

> /a/b that does a LOOKUPP gets into the server's pseudo file system that

That's a very good point.  Should the client see these two "export
points" as one filesystem or two?  
If a (unix like) client created mount points at every fsid change, and
indexed these by fsid (+ server id) then I can see it getting confused.
And should the client be able to create hard links, or request renames
between these two places?

NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Fri, 15 Oct 1999 07:08:16 -0700 (PDT)

> points" as one filesystem or two?  

right now, nfs clients tend to see them as two different file systems.
i don't most UNIX client implementations will alter that for V4.

> indexed these by fsid (+ server id) then I can see it getting confused.

the indexing will need to include the server path mapping ot
the mount point, fsid, server id.

> between these two places?

that would be up to the client.