Subject: RE: recovery from NFS4ERR_FHEXPIRED

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 21 Sep 1999 09:46:43 -0700

Spencer Shepler wrote:
> "G. Allen Morris III" wrote:
> > 
> > >>>Mike Eisler said:
> >  > Correct, although if unqiue_handles attribute is TRUE, then
> >  > if the file hanbxles are diffetent they point ot to 
> different files.
> >  >
> >  > > Can you tell me how I can test if two filehandles 
> point to the same file?
> >  >
> >  > If unique_handles is FALSE you cannot. But what does 
> this have to do with
> >  > your proposed 'validate filehandle' operation? I still 
> don't understand
> >  > what the the proposed operation would do.
> > 
> > All it is is a compare filehandle operation.
> > 
> > It is VERY likely that this can be handled without adding a 
> new operation.
> > 
> 
> When we had the volatile file handle discussion, it was suggested
> that the client could check the 'change' and 'object_size' attributes
> upon recovery from receipt of NFS4ERR_FHEXPIRED.  Obviously 
> in the case
> that the client was able to find the filename upon recovery.
 
If you do this, you will fail to recover if the object in question
was written to by someone else.  It may be perfectly valid for the 
file to have been written to.  What you want to test for is whether 
it is the same file (perhaps with somewhat different contents).  I 
suppose you can compare the fileid attribute before and after 
recovery if it is present.  Since fileid is a recommended attribute, 
it is not clear what you can do if it isn't present.  Comparing
object_size and change attributes may spuriously reject some
recovery instances as noted above, but it also might spuriously
allow some.  In some environments, many files have the same
size.  Further, the change attribute, as I understand it, has 
no global uniqueness requirements.  You might very well 
implement 'change' as the number of changes since the object
was created, which would mean that you could have two 
different objects which had the same value of the change
attribute.

---

From: Brent Callaghan <[email-masked]>
Date: Tue, 21 Sep 1999 17:29:51 -0700 (PDT)

A server that's lucky enough to have access to both st_ino and st_gen
in the stat structure could do fairly well just to embed these
in a filehandle along with a pathname of some kind.

Ya, I know the pathname isn't immune to renames, but in the case
where the pathname does take you to something real and you want
to make sure it's the *right* file or directory, the server could
use the st_ino and st_gen to verify.

The st_gen attribute is supported by some BSD OS's.  It gets
bumped whenever an inode is re-used.

	Brent

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Sep 1999 09:00:02 -0700

> use the st_ino and st_gen to verify.

This makes sense.

When I read this (it appeared in a thread about FHEXPIRED), I
assumed we were talking about volatile file handles.  I'm not
sure if that was your intention or not.

Upon thinking about it, it appears that this would not be a
volatile file handle.  Suppose the file handle consisted
of name, inumber, and igen.  So if you had (a/b, ino=7,
igen=2), you would return STALE.  If you returned FHEXPIRED,
the client would just do lookups and not find a/b, unless
of course,  someone just created a new one at the wrong time
and then you would wind up with a new handle, say (a/b, ino=8,
igen=6), but at that point, since handles are opaque, the
client couldn't see that and the server would of course
accept the handle he just gave out as the proper handle
even though we now have a different file.

If a/b is renamed, you have the same situation.  The server
should return STALE because if he returns FHEXPIRED, the 
lookups are either going to fail or succeed giving the wrong
answer.

In this case, the server would never give FHEXPIRED so the
file handle would be persistent, even though STALE would
happen more often than it does with a classical nfs file
handle.  Or am I missing something here?

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Sep 1999 10:50:46 -0700

> >>>"Noveck, Dave" said:
>  > 
>  > If you have some unique-id (time of creation even) for the CDROM,
>  > you can put this in each volatile file handle together with
>  > whatever else you have in the handle (e.g. the name) Then when 
>  > these file handles are used in a situation in which a new CDROM 
>  > has been mounted, you return STALE if a comparison of this id 
>  > does not match.  You just have to make sure that you do this 
>  > before you check the rest of the handle for any of the conditions
>  > that will cause FHEXPIRED to be generated.
> 
> I'm sorry Dave, what am I comparing against.  The only reason that
> FHEXPIRED would be returned is if I can't find the file.  The server
> can only compare this unique-id after the client has performed a
> new lookup.

The context was that we have a CDROM and we want to return STALE when 
somebody uses a file handle that he got when one CDROM mounted when
that CDROM is no longer mounted (a different one was mounted).

So given some unique id that identifies the particular CDROM, the
server would incorporate that id in all handles it gave out which
were derived from that CDROM and further always have in memory the
id for the CDROM which is currently being used.  When a file handle
is received, the CDROM-id from handle is compared to the id for
the current CDROM and you return STALE if it doesn't match.  If 
you do this, you won't have any problem with FHEXPIRED, because
you will never return FHEXPIRED.

It is true that if you do return FHEXPIRED, all would be lost,
because LOOKUP would return the file handle (inlcuding CDROM-id)
for the current CDROM so all future comparisons would go through,
which you don't want them to do.

> 
>  > > 
>  > > But you can reduce the odds. And it seems to me any 
> thing is better
>  > > than nothing: in the current implimentaion there is a 
> 100% chance that
>  > > you won't catch the problem.
>  >
>  > It depends.  If you have decided that you are going to use
>  > volatile file handles, then a partial solution is better 
>  > than none.  On the other hand, a partial solution might cause
>  > people to use volatile file handles when that is expedient
>  > rather than absolutely necessary.  If we don't have a true
>  > solution to the problem, I'd rather that people weren't
>  > given a false sense of security, only to see some disaster
>  > occur later.
>  >
> 
> I think that this should be left up to the server implimentation,
> not the v4 spec.  I would point out that the spec does not say
> how many bits of generation number there should be in the
> filehandle, nor should it.

The spec doesn't mention generation number at all.  It says the 
handle should identify the object.  It is up to the server to
do that well or poorly.

As things stand, using volatile file handles will mean the client
will read or write the wrong file sometimes.  If the spec were to
define a procedure that the server could use to solve the problem
then it would make sense to leave it up to the server to decide
how much uniqueness is enough.

I just don't want the spec to define a procedure that is not a
true solution to the problem, no matter what the server does
to implement it.

---

From: "Uresh Vahalia" <[email-masked]>
Date: Thu, 23 Sep 1999 04:25:10 -0400

-----Original Message-----
From: Carl Beame <[email-masked]>
To: [email-masked] <[email-masked]>
Date: Wednesday, September 22, 1999 6:36 PM
Subject: RE: recovery from NFS4ERR_FHEXPIRED

>
>My $0.02.
>
>The main reason for having NFS4ERR_FHEXPIRED is for server's which
>basically keep an in memory mapping from filehandle to local file
>access identifier. (Could be Path, or something). When the server
>reboots all in-core file handles are made invalid. When the server
>receives one of these old filehandles it recognizes it by possibly the
>filehandle containing the start time of the server. The server then
>returns FHEXPIRED and the client gets a new filehandle by using
>LOOKUP for the path to the file.
>
>So what is the problem?
>
>The problem seems to be the situation when Client A has the file open
>and Client B renames or deletes the file and then the server reboots.
>
>I think the real question is:
>
>Should we allow the delete or rename of a file open by another
>client?????

If a server reboots, it loses the knowledge about the client having the file
open.

>
>If no, does this not solve the HFEXPIRED problem?
>
>
>- Carl
>
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 27 Sep 1999 11:33:16 -0700

> "Noveck, Dave" wrote:
> > If a server is going to take an open file and
> > all of a sudden connect it to a different object
> > then I'm not sure what heuristics a client could
> > use to defend it's files other than to not use
> > such a server.
> 
> It shouldn't be sudden. The risk exists for all
> file models which lack a file identity (like
> ino+gen) other than pathname.

You go through the FHEXPIRED stuff and you're 
connected to a different file, whether you call
that sudden or not.  I still don't understand what 
heuristics you are going to use, if there is no
such thing as file identity.

> 
> > >
> > > The whole mechanism begs the question of a lost
> > > CLOSE. There are similar issues for client-side
> > > caching, and I would urge that the same
> > > mechanism be used in this case.
> > 
> > I'm confused.  What mechanism begs what question
> > regarding a lost CLOSE? What are the similar
> > issues for client-side caching and what mechanism
> > needs to be used in that case?
> 
> For caching, a lost CLOSE means that the current
> copy may still be on the client. How long and hard
> does the server try to recover file content from
> the client before unilateraly revoking the
> caching "delegation"?

The server times out the delegation if he recalls it
and the client does not return it in a resasonable 
time.  The actual time is up to the server.

The server may also allow a client that appears to be
deligently flushing state to the server an extension 
on the normal time to return the delegation, but the
spec says that there has got to be some ultimate
limit beyond which the server revokes the delegation,
the consent of the client notwithstanding.

> 
> For volatile file handles, a whole lot of names
> in a path are essentially locked pending
> conclusion of the volatile file handle.
> If no CLOSE, then how is the volatile file
> handle concluded? How are the locks on the
> pathname released?

This is an issue for any server whose semantics do
not allow renames of open files or parent directories,
whether they have volatile handles or not.

Even apart from renaming semantics, if you have a
file open then some other operations are going to 
be prevented.  Even if you specify DENY=NONE, other
opens specifying DENY=something will get failed.

The server cannot limit the amount of time that 
a file can be open.  If you have a rogue client
that spuriously OPEN's file and forgets about them,
servers are going to have to be able (in self-defense)
to identify the bad guys and revoke the opens.

Clients are going to have to be very careful to 
avoid doing this.  For example, if are doing an
OPEN, and you time out the RPC or the fs is mounted
with the intr option and someone ^c-s it, then you
have the possiblity that the OPEN was done on the
server and the reply was lost.  That also will look
like a lost CLOSE because the client never knew 
about the open file and so can't issue a close.
Meanwhile, the associated lock on the server is
being implcitly renewed as long as the client is
functioning.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 27 Sep 1999 11:47:21 -0700

> "Noveck, Dave" wrote:
> > 
> > >
> > > So, is a file handle a 128-bit inode number,
> > > or is it a 128-bit file descriptor? For a lot
> > > of reasons, I believe it is (or should be)
> > > more the latter -- a "handle", a rendevouz
> > > between the client and the server.
> > >
> > > ... Don't EVER worry about file identity. ...
> > 
> > It seems to me that if you can't tell if whether
> > two handles refer to the same object, cache issues
> > become very simple.  You can't cache on the client
> > because any write to any file might be a write to
> > any other file you have cached.  So you might as
> > well not cache on the client.  This simplifies things
> > but I think this is case where "evrerything should
> > be as simple as possbile, but not more so."
> 
> The *CLIENT* can't tell if two handles refer to the
> same object, ever. The only rule that makes sense
> is two returned file handles which are the same
> MUST refer to the same file. The server can tell
> when two file handles refer to the same object.
> Further, the server should be aware of potential
> caching conflicts, and deny client-side caching
> in such circumstances. There was a lot of discussion
> a few months ago about what identity to use for
> cache tags. My proposal then -- and now --
> is to make a cache identity a deliberate attribute
> under the control of the server, rather than
> overloading file handles and attribute fields
> with the cache-identity duty.

Right now, the spec says he can compare filehandles
to determine whether two handles designate the
same object, if the server allows it.  The server can
set unique_handles to true for the fs, in which
case two different handles denote two different
objects.  If the server returns false for 
unquie_handles, then the client can compare
the fileid attributes.  If they are the same,
then you have two handles for one object and
if they are different, then you have one object.

We could have a separate cache_identity attribute
if we needed it.  If two objects could have the 
same fileid yet should be treated as separate
for caching purposes, then this would make
sense.  

> 
> > 
> > This basically creates the protocol equivalent of
> > the C pointer-aliasing problems.  Handles are like
> > pointers and you have a choice between a very simple
> > but poorly-performing naive implementation and great
> > deal of complexity trying to be clever enough to
> > recover information about how things are used,
> > information that the model denies you.
> 
> If you want cache identity, define it independently
> of all other values.

If there is a need to do that, fine.

> 
> > 
> > > File semantics are the domain of the file
> > > server, not the protocol. A file handle
> > > is an opaque (means don't look at
> > > it ever) rendevouz between a client
> > > and it's server, and let no
> > > (overzealous) working group come
> > > between them ;-).
> > 
> > If you sent every file operation to the server,
> > the server-as-absolute-monarch model might make
> > sense.  As soon as the client can do things
> > locally, the server can no longer do whatever
> > it chooses.  There have to be shared expectations
> > between the server and the client.  This doesn't
> > mean gratuitously restricting the possible server
> > semantics but it does mean that the protocol
> > definition cannot avoid some involvement with file
> > semantics.
> 
> Agreed. When the server or file model is a mismatch
> for the NFS caching architecture, the server must
> be able to advise the client "no caching, you'll
> get it wrong."
 
Right now, if the server returns unique_handles as
false (or doesn't support the attribute) and doesn't
support the fileid attribute, then the client is not
supposed to cache because he has no way to reliably
get it right.