Subject: RE: recovery from NFS4ERR_FHEXPIRED

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Sep 1999 08:22:03 -0700

> This discussion turns on what may the client use
> for file identity. There is a more important question:
> What may the client do with file identity?
> 
> This gets into the whole caching question. You can
> make client-side determination of file identity
> as elaborate and complete as you please.
> So what. Can the client-side then perform
> (shadow) server-side semantics?
> 
> Whether LOOKUP("X") done 100 times
> in a row produces 100 distinct file handles
> or 100 identical file handles does not matter.
> The server side ***MUST*** always
> be in control of file op semantics, with
> respect to the NFSv4 caching architecture.
> 
> So, the original question (I'm guessing based
> on the Subject:) was what do you do about
> NFS4ERR_FHEXPIRED. I think the
> precise question is:
>    "What does the client-side do with
>     cached data (possibly changed)
>     when it was authorized by the server
>     to cache the data, but subsequently
>     receives an NFS4ERR_FHEXPIRED."

This is an interesting question but it isn't
the question that we were discussing.  The 
problem with FHEXPIRED occurs independent of
caching.  I open a file.  I do a read and don't
do any client-side caching.  The server reboots
and I get FHEXPIRED and do my lookups get
a file that has the same name as the file I
opened.  If it isn't the same file, reclaiming
my opne and writing to it, whether I do any 
caching or not, is a Bad Thing (tm).  Various
people (including me) have mentioned various 
possible approaches to try to make sure that
you really have the same object as the one 
opened and none of these approaches are any
better than a band-aid.  The problem is not
that the server is not in control of the 
semantics of WRITE but that volatile file
handles seem not to give him any tools to 
enforce that a set of IO's go to the same
object once file handles expire.

As to caching and FHEXPIRED, suppose you have
modified data for a file.  You can then try to
recover and you may not be able to find the file
in question.  In this case, you have to throw
away the data.  If the file is still open you
would want to return an IO error to the user
on close or the next IO.  If the file is closed
and there is data that has not been flushed,
then you have to throw away the data and warn 
the user more directly (e.g. console messages,
pop-ups), just as you do when a lock or delegation
is revoked.

If you can find a file of the same name, the client
needs to do whatever it can to make sure it is the
same object (this is true whther you are caching 
or not), so it should make sure the fileid's 
match.  If they don't, this should be treated the
same as not being able to find the object in the
first place.  Given that fileid comparison is not
safe (a deleted file and one created subsequently
can have the same fileid) so I suppose a client
should also compare creation_time if it exists.

In the case in which the server does not support
the fileid attribute, you are skating on extremely
thin ice whether you are caching or not.  it seems
really bad to me to blindly write the cached data
to the server but it isn't any worse than blindly
writing new non-cached data to the server.

I'll try to update the caching sections to deal with
the issues posed by volatile file handles.  I think
flush-on-close should be strongly recommended for
volatile file handles since having to deal with
modified data when the file is close and you can't
recover from FHEXPIRED adds complexity.

---

From: Carl Beame <[email-masked]>
Date: 22 Sep 1999 23:30:56 +0100

My $0.02.

The main reason for having NFS4ERR_FHEXPIRED is for server's which
basically keep an in memory mapping from filehandle to local file
access identifier. (Could be Path, or something). When the server
reboots all in-core file handles are made invalid. When the server
receives one of these old filehandles it recognizes it by possibly the
filehandle containing the start time of the server. The server then
returns FHEXPIRED and the client gets a new filehandle by using
LOOKUP for the path to the file.

So what is the problem?

The problem seems to be the situation when Client A has the file open
and Client B renames or deletes the file and then the server reboots.

I think the real question is:

Should we allow the delete or rename of a file open by another
client?????

If no, does this not solve the HFEXPIRED problem?

- Carl

---

From: "G. Allen Morris III" <[email-masked]>
Date: Wed, 22 Sep 1999 21:10:27 -0700

>>>Carl Beame said:

What you can do is expire any filehandle that points to a file
that has been renamed.  Then only clients that do not use a
filehandle between the time of the rename and the time of the
server reboot would get a stale filehandle.

 > Should we allow the delete or rename of a file open by another

There is no way for the server to know who has a file `open' unless
there is a lock on the file.

 > If no, does this not solve the HFEXPIRED problem?

---------------------------------
       G. Allen Morris III

---

From: Carl Beame <[email-masked]>
Date: 23 Sep 1999 08:58:44 +0100

> there is a lock on the file.

Sorry. Here is section 8.8 from thr spec:

8.8.  OPEN/CLOSE Procedures

   To provide correct share semantics, a client MUST use the OPEN
   procedure to obtain the initial file handle and indicate the desired
   access and what if any access to deny.  Even if the client intends to
   use a stateid of all 0's or all 1's, it must still obtain the
   filehandle for the regular file with the OPEN procedure.  For clients
   that do not have a deny mode built into their open API, deny equal to
   NONE should be used.

---

From: Gordon Waidhofer <[email-masked]>
Date: Thu, 23 Sep 1999 04:05:36 -0700

> >  > Should we allow the delete or rename of a file open by another

BINGO! Should [the server] allow the delete or rename of a file
open by another client. This is a bona-fide question, and I
believe up to the implementation. At all times, the server must
be in control of the semantics.

>    that do not have a deny mode built into their open API, deny equal to

Provided the server prevents **ITS** trans-reboot recovery story
for volatile file handles from breaking (by defending the file
name space), the contemplated FHEXPIRED recovery should work:
repeat the OPEN (pathname reduction) and continue. If a
server which returns volatile file handles does not defend the
file name space (or other volatile recovery method), the server
is broken.

All this really gets back to some basic philisophical questions.
Is a file handle a file identity, or is it a "handle" by which
the client requests fileops of the server. For Carl's server,
the name space is the definitive identity, and his server
(NT more generally) defends the name space to prevent such
issues. It's not a perfect recovery story -- it's littered
with client-side recovery race conditions. Nevertheless,
that's how it (NT) works. Hence, no fair making a heterogeneous
protocol (NFS) require the file handle be a file identity.

Carl, given the client-side races during recovery (e.g.
client B can rename() a file before client A can recover
the OPEN), what should clients do to protect file
content? I'm not a big believer in the change_time
or create_time heuristic.

-- 
==================================================================
  Gordon W Waidhofer                    Traakan Software, Inc.
  [email-masked]                       111 Main St., Suite #5
  650-947-1583  Direct                  Los Altos, CA  94042
  650-947-1580  Main                    http://www.traakan.com
  650-947-1588  Fax