Subject: RE: recovery from NFS4ERR_FHEXPIRED

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 23 Sep 1999 06:31:34 -0700

> My $0.02.
> 
> The main reason for having NFS4ERR_FHEXPIRED is for server's which
> basically keep an in memory mapping from filehandle to local file
> access identifier. (Could be Path, or something). When the server
> reboots all in-core file handles are made invalid. When the server
> receives one of these old filehandles it recognizes it by possibly the
> filehandle containing the start time of the server. The server then
> returns FHEXPIRED and the client gets a new filehandle by using
> LOOKUP for the path to the file.
> 
> So what is the problem?
> 
> The problem seems to be the situation when Client A has the file open
> and Client B renames or deletes the file and then the server reboots.

Or it could be renamed or deleted by a local process on the server.

Or a directory in which the file resides or an ancestor directory is
renamed.

> I think the real question is:
> 
> Should we allow the delete or rename of a file open by another
> client?????

I don't think it's reasonable to prohibit that in general, but it
might be possible to prohibit it for those that use volatile
file handles.

The question is what to do about the other cases I mentioned:

     Local processes doing delete and rename can be considered
     outside the scope of the protocol.

     Implicit file renames because of directory renames have
     to be prohibited as well.  This may be difficult for 
     some servers, but I don't know if the servers that need
     volatile file handles are among them.

> If no, does this not solve the HFEXPIRED problem?
 
If we could do this (including the directory rename issue),
then almost.  Rename or delete during the grace period in 
which the server doesn't know what is really open yet
would have to be addressed.  

Another case in which the server doesn't know what is
really open is delegation.  I think it fits OK logically, 
but it does mean that an attempt to rename a directory 
would cause recall on all of the delegated files, or 
at least until you had some client that said he truly 
had the file open.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 23 Sep 1999 06:46:08 -0700

> >>>Carl Beame said:
> What you can do is expire any filehandle that points to a file
> that has been renamed.  Then only clients that do not use a
> filehandle between the time of the rename and the time of the
> server reboot would get a stale filehandle.
 
I don't see how this helps anything.  If I have a volatile handle
to A/B and someone renames it to A/C then if, instead of getting
STALE, I get FHEXPIRED which causes me to re-lookup A/B, then
I will either:

      Not find A/B in which case I am no better off than having
      gotten STALE in the first place.

      Find a different A/B that someone just created in which
      I now have the wrong file.

It seems that this gives me either essentially the same answer
as STALE, or else a wrong answer.

This is a case in which the the current procedure to refresh
expired handles does not do the right thing.  The server could
save the mapping between old and new handles at least until a
reboot but right now, there is no way for the client to avail
himself of the server's knowledge.

---

From: Carl Beame <[email-masked]>
Date: 23 Sep 1999 17:40:02 +0100

>      volatile file handles are among them.

As an example, under NT you cannot rename or delete ANY part of the
directory path to an open file. 

Maybe we should put restrictions on semantics of hosts which use
volatile file handles so that FHEXPIRED can be recovered from. 

- Carl

---

From: "G. Allen Morris III" <[email-masked]>
Date: Sat, 25 Sep 1999 13:12:25 -0700

>>>Carl Beame said:
 [snip]
 > directory path to an open file. 
    :-O
 > volatile file handles so that FHEXPIRED can be recovered from. 

There are two reasons that this is a bad idea:

First:  This is a horrible restriction to place on a filesystem.

Second: It is impossible.  This would require the server to have a
list of all open files, but the only reason for the server to
return FEXPIRE is that it has lost its cache of filehandles.

  Allen

---------------------------------
       G. Allen Morris III

---

From: Gordon Waidhofer <[email-masked]>
Date: Sat, 25 Sep 1999 14:11:56 -0700

I think maybe an earlier reply of mine got lost.
I've appended it to the end.

So, is a file handle a 128-bit inode number,
or is it a 128-bit file descriptor? For a lot
of reasons, I believe it is (or should be)
more the latter -- a "handle", a rendevouz
between the client and the server.

We all seem to be jumping through
countless hoops trying to preserve
file handles as file identities (128-bit
inode numbers). Give it up, and the
hoops all go away. Don't EVER worry
about file identity. Give up st_ino and
st_gen also, as they are problematic
or impossible for some server/file
models. OBTW, I believe a great
many cache issues get simpler
with the "file descriptor" model.
They sure do for SMB.

File semantics are the domain of the file
server, not the protocol. A file handle
is an opaque (means don't look at
it ever) rendevouz between a client
and it's server, and let no
(overzealous) working group come
between them ;-).

Cheers,
    -gww

Carl Beame wrote:
>
> On Thu Sep 23 05:10:27 1999, G. Allen Morris III wrote:
>
> >

BINGO! Should [the server] allow the delete or rename of a file
open by another client. This is a bona-fide question, and I
believe up to the implementation. At all times, the server must
be in control of the semantics.

> >
> > There is no way for the server to know who has a file `open' unless
> > there is a lock on the file.
> >
>
> Sorry. Here is section 8.8 from thr spec:
>
> 8.8.  OPEN/CLOSE Procedures
>
>    To provide correct share semantics, a client MUST use the OPEN
>    procedure to obtain the initial file handle and indicate the desired

>    access and what if any access to deny.  Even if the client intends
to
>    use a stateid of all 0's or all 1's, it must still obtain the
>    filehandle for the regular file with the OPEN procedure.  For
clients
>    that do not have a deny mode built into their open API, deny equal
to
>    NONE should be used.

Provided the server prevents **ITS** trans-reboot recovery story
for volatile file handles from breaking (by defending the file
name space), the contemplated FHEXPIRED recovery should work:
repeat the OPEN (pathname reduction) and continue. If a
server which returns volatile file handles does not defend the
file name space (or other volatile recovery method), the server
is broken.

All this really gets back to some basic philisophical questions.
Is a file handle a file identity, or is it a "handle" by which
the client requests fileops of the server. For Carl's server,
the name space is the definitive identity, and his server
(NT more generally) defends the name space to prevent such
issues. It's not a perfect recovery story -- it's littered
with client-side recovery race conditions. Nevertheless,
that's how it (NT) works. Hence, no fair making a heterogeneous
protocol (NFS) require the file handle be a file identity.

Carl, given the client-side races during recovery (e.g.
client B can rename() a file before client A can recover
the OPEN), what should clients do to protect file
content? I'm not a big believer in the change_time
or create_time heuristic.

--
==================================================================
  Gordon W Waidhofer                    Traakan Software, Inc.
  [email-masked]                       111 Main St., Suite #5
  650-947-1583  Direct                  Los Altos, CA  94042
  650-947-1580  Main                    http://www.traakan.com
  650-947-1588  Fax

---

From: Carl Beame <[email-masked]>
Date: 26 Sep 1999 10:43:34 +0100

> or create_time heuristic.

I guess I have been making an assumption which I find out is not true.
But it should be!!!

The OPEN procedure can return NFS4ERR_GRACE for new opens when the
server has just rebooted. This is to allow for recovery of shares and
other state information. With persistant file handles the recovery
takes place by using the file handle. With volatile file handles the
client MUST use the pathname / filename to re-open the file.

Given this it stands to reason that during the grace period the
filename space of the server should have any elements removed from it
if at all possible. This may not be possible for a local process, but
by returing NFS4ERR_GRACE for the REMOVE or RENAME routines we
eliminate the possiblility of a change to the namespace by NFS during
the re-opening of files.

- Carl

---

From: Gordon Waidhofer <[email-masked]>
Date: Sun, 26 Sep 1999 12:11:01 -0700

This is a good answer. It follows the client-side
implementations need to be prepared for NFS4ERR_GRACE
from all ops, not just OPEN or LOCK.

Is it safe to say the pathname is the only
possible recovery for volatile file handles?
Only repeated LOOKUPs and OPENs can recover
expired handles? I believe so. Comments?

The NFSv4 spec should at least *RECOMMEND*
that servers which return volatile file handles
defend the file name space and also use
the trans-reboot grace period as Carl
described. The spec should also *REQUIRE*
client implementations to handle NFS4ERR_GRACE
for all operations. A set of *RECOMMENDED*
heuristics for the client-side to defend
file content would also be a good topic
for the spec, but since file identity is
problematic, such heuristics should not be
required of either the client nor server.

The whole mechanism begs the question of a lost
CLOSE. There are similar issues for client-side
caching, and I would urge that the same
mechanism be used in this case.

OBTW, SMB ain't got a solution for the
client/client trans-reboot race condition.
Chalk one up for NFSv4. I love it when
that happens. Nice going, Carl.