Subject: RE: Persistent Filehandle: contradiction in spec and possible imp lementation

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 25 Jun 2001 13:33:46 -0700

Som wrote: 
> I have some more doubts:
> 
> 1) Can a NFS client assume that a persistent
> filehandle is never allocated ( after its deleted ) as
> a volatile filehandle? 

As the spec is written right now, I don't think this
possiblity is properly excluded.  It says that if an 
object is deleted, then if the persistent handle is 
presented, STALE should be returned.  The intention 
is that that requirement apply to all subsequent 
presentations of that handle, even those that might
be in the guise of a volatile filehandle.  Because 
persistent and volatile filehandles are discussed
separately, I don't think that comes through.

> [ In section 4.2.4 of the spec it tries to define the
> format of volatile filehandle...but, was it part of
> spec or just an example? so looks to me, a persitent
> filehandle can never be equal to a volatile filehandle
> ..is that true? ]

What is in section 4.2.4 is an example.  I think the
implication of the initial one bit indicating volatile
is that it if the server followed that approach it would
be a good idea for it to have an initial zero-bit in its 
persistent handles so that it can tell the difference, but 
there is no corresponding example of a persistent filehandle.

Right now I don't think the spec says this anywhere,
but I think it should say that if a server supports
both persistent and volatile file handles, it should
be able to tell the difference (i.e that the sets of
values that peristent and volatile handles may take
for a given server be disjoint).

> 2) Can a NFS server steal few bits of persistent
> filehandle to denote as "reincarnation bits"?...the
> purpose of this is to inform client that a "deleted
> persistent filehandle" [ DPF ] being re-used ( so
> don't trust this handle anymore...a new error status
> is needed in that case ).

I don't understand how these re-incarnation bits would
differ from the generation number that is present in
Spencer's example, or how the error would differ from
NFS4ERR_STALE.

> If we don't do 2), then a NFS client may misuse a DPF
> handle thus corrupting a file.
> [ however, reincarnation bits don't fix the problem
> 100%, but its better than no information at all ]

It is up to the server to make the generation number big
enough so that this cannot happen.  If a 32-bit generation
number does not provide sufficient security (and as servers
get faster maybe this is something that we will no longer
be able to take for granted), a 64-bit generation number 
will do the job.  If a server can do 1000 create-remove pairs
per second in the worst case environement (the filesystem
has only a single free inode).  In that case a filehandle
could not be re-used in less than about 100 million years,
so that I don't see this as a problem.  The 1000 per second 
is good bound for normal NFS access but local file system 
access (or something like DAFS) may be somewhat faster but
anything that requires a client to stay up for millions of
years seems safe enough to me.

With 32-bit generation numbers, then you have something that
could in the absolute worst case turn over in some number of
months.  At least in the nfs-only case 1000 crate-removes
per second is probably unrealistic.  With 500/s, the generation
number could turn over in three months.  So, given the number
of things that would have to happen for this to corrupt a file:

      Client gets a filehandle

      Avoids using it any way (and thus detecting its staleness)
      for some number of months.

      Another client has to assiduously do nothing but create and 
      remove a file for months (and if he does anything real with
      the files, then the filehandle re-use time is going to be
      years).

      There is nothing in the server filesystem code to cause 
      a different inode index to be selected during these months
      (or any such code is defeated by a circumstance such as 
       having only a single inode free, with that condition 
       maintained for months).

      Just at the wrong time, the client wakes up (a one-in-four-
      billion co-incidence) and uses the filehandle.

You're not 100% safe, but I think that it is far more likely that
your file is going to get corrupted by undetected multi-bit 
memory errors, than by persistent filehandle re-use.

So I don't worry about this corrupting files.  On the other hand,
If you forsee servers and file systems and file system protocols
getting 1000 times faster, then 32-bit generation numbers start to
seem kind of wimpy.  If I were starting from scratch today, I'd
definitely go for a 64-bit generation number (or maybe 48-bit).

> 3) NFS clients need not know anything about those
> special bits in persistent filehandle, but the new
> error status will be needed.

The client doesn't know anything about the generation number
but he does recognize NFS4ERR_STALE.

> thanks, som.

--- Spencer Shepler <[email-masked]> wrote:
> On Thu, Som wrote:
> > 1.
> > In section 4.2.2 Persistent filehandle is defined
> as:
> > 
> > ".... having a fixed value for the lifetime of the
> > file system object to which it refers"
> > 
> > Is this also true when the file system object is
> > removed? As per the specification probably that's
> not
> > true. But in the next chapter it says server must
> > return an error when a removed persistent file
> handle
> > is presented. Only way server can implement this
> is to
> > store even the deleted file system handle for its
> > lifetime....I mean, as per the implementation goes
> > server can not relocate a persistent filehandle
> even
> > after its deleted, since it has to return error in
> > future.
> > 
> > But as per the definition of persistent filehandle
> > goes, its only valid within the lifetime of
> filesystem
> > object!
> 
> Right.  Later in that section it explicitly states:
> 
>   "The persistent filehandle will be become stale or
> invalid when the
>    file system object is removed.  When the server
> is presented with a
>    persistent filehandle that refers to a deleted
> object, it MUST return
>    an error of NFS4ERR_STALE."
> 
> > I see a contradiction here...
> > To get rid of the contradiction one may modify
> section
> >  4.2.2 like...
> > 
> > ".... having a fixed value for the lifetime of the
> > file system (note: not object ) to which the
> object
> > belongs"
> > But that's going to be very difficult to
> > implement...so probably a practical timeout value
> has
> > to be decided on persistent filehandles after
> > deletion!
> 
> I can't seem to find the reference in the RFC at the
> moment but there
> are comments somewhere that talk about the eventual
> reuse of a
> persistent filehandle and that the server should
> wait as long as
> possible before such reuse.  If these statements are
> not in the
> current RFC, I will make sure they are in the
> update.
> 
> In any event, this description takes into account
> the common Unix
> implementation of NFS filehandles.  Historically,
> the filehandle
> contains the inode number and generation number of
> that particular
> inode.  It also contains other identifiers but these
> are the most
> important.  If the inode is reallocated after the
> filesystem object is
> removed, the generation number is incremented to
> signify a distinct
> filesystem object.  That generation number will
> eventually wrap but it
> has typically been a 32bit quantity so the time it
> would take to wrap
> is "long".  With this mechanism, the server does
> store the necessary
> persistent state to avoid duplicate filehandles for
> different objects
> and that state is in the filesystem itself.
> 
> > 2. Nothing is told about migration/replication of
> > deleted persistent filehandles..should that
> > information be also migrated/replicated? Probably
> yes,
> > but the didn't find that in spec..
> 
> Right.  It is implied in that if the filesystem is
> migrated and the
> filehandles for that filesystem are presented as
> persistent, then the
> new server will employ similar filehandle allocation
> techniques.
> Maybe the RFC is a sparse in this area but I believe
> the current
> intent is sufficient but maybe I am a little too
> close to the RFC as
> it stands. :-)
> 
> -- 
> 
> - Spencer -
> 
> 

=====
"Chess holds its master in its own bonds,
shackling the mind and brain so that the 
inner freedom of the very strongest must 
suffer"-Albert Einstein

__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

---

From: Som <[email-masked]>
Date: Tue, 26 Jun 2001 22:22:44 -0700 (PDT)

> Som wrote: 
<< stuff deleted >>

> Spencer's example, or how the error would differ

Currently, a persistent filehandle has only two states

a) its a good filehandle which represents a valid
filesystem object

b) its a filehandle which is deleted and won't be
reallocated till we run out of filehandles

But, I was trying to bring the concept of another
state which divides state a) in two states:

a.1) its good filehandle which represents a filesystem
object when it was opened

a.2) its a good( from server's perspective) filehandle
which no longer represents a filesystem object what a
client x has ( rollover happend for that client ), but
for client y it may be perfectly allright.

But as per Dave, state a.2) normally won't happen with
32-bit generation numbers...but, if it happens, the
new state "NFS4ERR_HANDLE_REUSED" is definitely
different from "NFS4ERR_STALE". 

Hope I clarified the doubt. Spencer also sent me an
email on this which I am going to forward to the group
since it contains some good information.

thanks, som.

=====

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 27 Jun 2001 18:38:15 -0500

> different from "NFS4ERR_STALE". 

It is unclear to me what problem you are proposing to
solve and exactly how a server would use this.
Generally, the server generates filehandles in a way
that they are the same for all clients.  The protocol
allows the server to create filehandles in a way that 
they would be client specific since clients do not
share filehandles but it would unnecessarily add to
the burden of the server implementation.

Without a clear definition of what the perceived problem,
I am not sure this discussion can proceed much further.

Spencer