Subject: Feedback on draft-ietf-nfsv4-01 - volatile file handles

From: Neil Brown <[email-masked]>
Date: Mon, 11 Oct 1999 14:14:32 +1000 (EST)

This is my third Email with feedback on draft-ietf-nfsv4-01.

NeilBrown

Volatile File Handles in General.

I notice from the archives that there has been some discussion about
recovering from expired Volatile File handles in the list recently
(before I joined), so I hope I'm not duplicating too much with these
comments.

It occurs to me that Volatile handles were defined with some
particular situations in mind, and that it would be worth enumerating
these situations to see where there is enough commonality that a
single abstraction on a "Volatile handle" will suffice, or whether a
number of different flavours of Volatile handles (distinguished by
attributes similar to "unique_handles") are necessary.

Scenarios that I can see are:
  1/  Virtual file systems such as the "Pseudo" file systems mentioned
     in section 7, and a Linux  "/proc" file system (not that anyone
     in their right mind would export /proc ?-) 

     For these, the namespace is read-only so if the object has the
     same pathname, then it "is" the same object for all intents and
     purposes. 

     Thus some attribute like "path_safe" could be used to tell the
     client that re-processing the "lookup" sequence is the correct way
     to deal with NFS4ERR_FHEXPIRED.

  2/ Real filesystems that don't store a unique file identifier,
     such as the FAT file systems used by DOS. For such a filesystem
     there is no number that can reliably identify a file across
     rename, truncate, and reboot.

     Such a filesystem would chose some file handle that is a "best-effort"
     representation of the file, and would need to tell the client
     which operations might cause the filehandle to change.
     Some attributes like "rename_breaks_handle",
     "truncate_breaks_handle" might be needed, though we really need
     something more general than enumerating operations.

     Such a filesystem might disallow "rename" while a file is open to
     reduce problems (I gather than some filesystems do that).

     Maybe an attribute like "open_safe" which guarantees that the
     file handle will never change when the file is "open".

     Then, is a filehandle is not "persistent_fh" and not "path_safe"
     and not "open_safe", then every COMPOUND request that changes
     anything about the file should do a GETATTR(filehandle) incase
     the filehandle changed, and if the client ever gets
     NFS4ERR_FHEXPIRED, it should do a GETATTR(filehandle) incase the
     server has a cache mapping from the old to the new, and if this
     doesn't work, treat the error as NFS4ERR_STALE.

  3/ ?? What other scenarios are there?

Summary (changing the format of the attributes a bit):
   Instead of "persistent_fh" define an attributes "fh_type" which is
   a bit-or of
     1 - persistent
     2 - path_safe
     4 - open_safe

   As long as fh_type is non-zero, then you can be guaranteed of
   finding the file properly unless you really need to track the file
   when it is not-open, and I cannot see a real need for that.

   If fh_type is zero, then you can cannot guarantee anything, but the
   client can do a best-effort by calling GETATTR(filehandle)
   regularly, and the server can help by caching old->new mappings for
   as long as possible.

   If the server allows a filehandle to change due to some operation
   such as rename or truncate, then it should remember the mapping
   from old to new at least long enough to be able to respond usefully
   to a GETATTR(filehandle) in the same COMPOUND operation that caused
   the change, and preferable a little bit longer.  The server is NOT
   required to maintain the mapping across restart, and so it cannot
   be depended upon.

This doesn't provide much help for Allen Morris's problem of Linux
using filehandles that that are path name specific, but I think that
can be handled better by not using path name specific file handles
(which I am fairly sure is possible).

---

From: Mike Eisler <[email-masked]>
Date: Wed, 13 Oct 1999 11:32:46 -0700 (PDT)

> Scenarios that I can see are:
...
>      "truncate_breaks_handle" might be needed, though we really need

Right. The P/V table in the draft spec is a start, but until it is
actually mapped to a set of attributes, it doesn't solve the problem.

>      something more general than enumerating operations.

First enumerate the operations and potential attributes, then simplify.

>    when it is not-open, and I cannot see a real need for that.

However there are variations of path_safe and open_safe.

Variations of of path_safe are 

	- does the fh work if another client renames the file?
	- does the fh work if same client  does a rename? (The Linux
		problem)

Variations of open_safe are

	- does the fh work if another client deletes/renames an open file
	- does the fh work after server recover if another client
		deletes/renames an open file?

> (which I am fairly sure is possible).

Considering the installed base of Linux servers, any solution that doesn't
address Linux is inadequate.

What we need for Linux is a simple mandatory boolean attribute that
says: 
	will the fh expire to client A if client A does a rename?

When client does a "COMPOUND { open, getattr }" the client can request
this attribute in the getattr, and if the result is TRUE, then
the client will know that when renaming a file, it needs to do:

	COMPOUND { putfh d1 , rename "a" "b" , lookup "b"; getfh }

If file "a" were alreayd open, then it is unclear to me what the reovery
is, but if we had a set of attributes base don the P/V table, we
would know.

	-mre

---

From: Neil Brown <[email-masked]>
Date: Thu, 14 Oct 1999 16:00:10 +1000 (EST)

> 	- does the fh work if same client  does a rename? (The Linux

By path_safe, I meant that for this file, the path is ALWAYS a safe
way to get a file handle on the SAME file - guaranteed (possibly we
should say "a path from the nearest persistent file handle").

I am imagining the "server namespace" pseudo filesystem, or a CDROM
filesystem or a "/proc" like file system.  i.e. a filesystem that does
not support rename.

If another client can rename the file, then it is not a variation of
path_safe, is it simple NOT path_safe.

> 		deletes/renames an open file?

By "open_safe" I mean that the filehandle is guaranteed to be stable
while the file is open (barring sysadmin intervention and filesystem
corruption I guess).
A server filesystem might implement this by disallowing rename or
unlink while the file is open (I gather that is the DOS/Windows
semantic?).  This would include during the grace period after server
restart.

Alternately, a server could log the rename to stable storage before
letting it complete, and use that information to respond to any "old"
filehandle that is still open.

Basically, if a filehandle reports "open_safe", then the server will
either disallow or log any operation that would change the filehandle
so long as the file is open.

> is, but if we had a set of attributes base don the P/V table, we

There is nothing about Linux that makes is particularly different from
any other Unix in this respect.  I am currently preparing a patch to
the V2 server which can provide persistent file handles.

The only thorny issue, and this would apply to any Unix, is the
following:
   Unix access permissions are affected not only by the permission bits
   on the file, but also by the permission bits on all ancestor
   directories right up to the root.
   Specifically, if any ancestor does not give the accesser 'x'
   access, then they should not have any access to the file, not even
   "GETATTR" access.
   To test this properly, you need to know which directory(s) the file
   is in, and to check it and all it's parents (following "..").
   I don't know how other NFS servers deal with this situation.
   In the first instance, I plan to avoid it by (optionally) ignoring
   it, and just relying on the permission bits on the file.  I don't
   think that looses much.
   In the second instance, I am considering storing
   "who-has-ancestor-directory-access" in the filehandle, and
   cryptographically signing the handle.  Such a handle would not be
   persistently "presented", but would be persistently "honoured".
   [You could still get inappropriate access through the handle after
   a move but that is similar to the unix approach of
   "check-permission-on-open" which is expected.]

Having thought more about it, I don't think that "persistent" as it
stands is particularly necessary.
Consider the following:
    Any access with any filehandle may return NFS4ERR_FHEXPIRED
    except:
	    GETATTR(persistent_fh)

    If GETATTR(persistent_fh) returns "persistent", then
       GETATTR(filehandle) WILL return a new filehandle that can be used
       on the file.
    If GETATTR(persistent_fh) returns "path_safe", then an appropriate
       series of LOOKUPs should be used to find the new filehandle.
    If GETATTR(persistent_fh) returns "open_safe", then either:
         GETATTR(filehandle) will return a valid handle for the file
       or
         the server didn't think you have the file open (and hopefully
         the client didn't either), so this filehandle is simply a
         cached copy which is no-longer valid, and the client should
         act as though it didn't have a cached filehandle. (Actually,
         the server could probably have returned NFS4ERR_STALE in this
         case). 

    Providing GETATTR(persistent_fh) is not "none", the client has a
    guaranteed way to deal with NFS4ERR_FHEXPIRED.

    If GETATTR(persistent_fh) returns "none", then the client has NO
    guarantees about continued access to the file.  The server may in
    some instances honour GETATTR(filehandle) after an
    NFS4ERR_FHEXPIRED, but the client should probably call
    GETATTR(filehandle) on a regular basis, just in case, on the
    assumption that the server might cache changes, but for limited time.

This gives a clear proceedure for the client to follow, and should
provide "best-available" semantics for any filesystem.  It would also
allow the server implementer to "version" filehandles, and introduce
new layouts.  Any "old" filehandle received can cause an FHEXPIRED, and
the following GETATTR(filehandle) can be used to give the client a
new, up-to-date file handle.

NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Oct 1999 04:36:55 -0700 (PDT)

>        GETATTR(filehandle) WILL return a new filehandle that can be used

If a GETATTR on the old file handle works to return a new file handle,
then why do we need a new file handle?

FHEXPIRED ca't be returned on a filehandle with the persistent attribute,
otherwise it isn't persistent.

>     assumption that the server might cache changes, but for limited time.

Aside from the persistent_fh definition, your suggestion is a reasonable step
toward moving forward with the never ending debate around volatile file
handles.

If open_safe is true, and persistent_fh and path_safe are false, then the
server, even through server reboot/recovery, will ensure that the file handle
returned by open is valid as long as the file is open. The file handle is
indeed volatile, but there are no risks if there is an attempt to rename or
unlink the file by the client or another client. Either the rename or unlink
will be denied, or the server will record in stable storage, some state that
allows it to recover the file handle to file mapping after a server reboot.

This allows a server to return volastile file handles from writeable file
systems.

	-mre

---

From: Neil Brown <[email-masked]>
Date: Fri, 15 Oct 1999 14:16:08 +1000 (EST)

> then why do we need a new file handle?

We are assuming the possibility of filesystems for which the file
handle will change on operations like "rename".  It has been suggested
that with such filesystems, the client should use a sequence like:

That provides no guarantees as COMPOUND does not give atomicity, and
so some other client could have renamed "b" (unlikely I guess), and
also is completely in adequate in the face of other clients renaming
in general.
I proposed that for this case, the server might keep a cache of
renamed filehandles, and what the new filehandle was.  This cache
might be in stable storage, but the server wouldn't want to keep it
indefinately.
So, I suggested that the 
      COMPOUND { putfh f1, getattr(filehandle) , getfh }
sequence could be used to effectively do a cache lookup and find
the new filehandle.

I then wondered "If the client has to be ready to do this sequence
anyway, are there any other cases where it might be useful?"

It is hard to come up with a convincing example where you would need
to get all your clients to update their file handles.  The sort of
scenarios I image involve either upgrading filehandle to a new,
more reliable, format (Linux filehandles have had several formats), or
storing some sort of fast-path hint in the filehandle which may need
to be updated occasionally.

I'm trying to explore what really constitutes a filehandle, and was
wondering 
   "given that we need volatile file handles, and apparently different
   flavours of volatility, can we discard "persistant" in favour of
   some flavour of "volatile", and if so, what is the weakest
   semantic that would still work?"

> otherwise it isn't persistent.

I was experimenting with changing what we mean by "persistent".  I
probably shouldn't have used the word "persistent" at all.  Maybe
"reliable" would have been a better choice.

NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Fri, 15 Oct 1999 07:15:53 -0700 (PDT)

> sequence could be used to effectively do a cache lookup and find

Fine, but those type of file handles are clearly not persistent. You need
another bit (linux_fh perhaps).

	-mre