Subject: [nfsv4] Comments on draft-noveck-nfsv4-referrals-00.txt

From: Ted Anderson <[email-masked]>
Date: Fri, 22 Apr 2005 10:29:29 -0400

Comments on "Implementation Guide for Referrals in NFSv4", by David
Noveck, Rodney C. Burnett, Feb 2005,
http://www.ietf.org/internet-drafts/draft-noveck-nfsv4-referrals-00.txt

In section 2.1, PUTFH is described as receiving "fh for absent volume",
but it should be something like "fh within absent filesystem".  This is
the only use of the term "volume" in the document.  I think "filesystem"
would be better and more consistent.

In section 3.1, the assuptions for READDIR include:
      o    When fs_locations is among the attributes requested, the
           server may provide a subset of the other requested attributes
           together with fs_locations for roots of absent fs's, without
           causing any error for the READDIR as a whole.  If rdattr_error
           is also requested and there are attributes which are not
           available, then rdattr_error will receive the value
           NFS4ERR_MOVED.

This seems inconsistent with the recommendation for GETATTR, which says
that if fs_locations is requested, will not return NFS4ERR_MOVED.  Since
rdattr_error is described as "Error returned from getattr during
readdir", what is the motivation for this behavior?  Is the thinking
that this is a handy way of indicating an absent filesystem?

In section 3.2.1, this example is confusing:
 >    A key reason for always providing an fsid can be seen by
 >    considering client behavior.  When a client gets the error
 >    NFS4ERR_MOVED, it has a number of key steps to get through.  It
 >    must determine if it has accessed the filesystem on the server that
 >    the moved error involves. If so, then it represents a data
 >    migration event in the strict sense, rather than a referral.  The
 >    client must recover state and properly use rootpath data from the
 >    old and new server to graft the new server's namespace into the
 >    client's local space. If a client gets an NFS4ERR_MOVED error and
 >    it cannot get an fsid, then it must use fs_locations, fs_root, and
 >    rootpath data to locally determine if it has actually accessed the
 >    data or if this is a referral for this client.  If it uses fs_root
 >    and there have been renames along the path to the fsid, then the
 >    client will not be able to distinguish a migration from a referral.
 >    The best solution here is for the client to always get an fsid. All
 >    a server has to do is generate and fsid value that cannot represent
 >    actual exported data at the server. We want to have a model where
 >    referrals can still function even if renames or other events change
 >    the path to the referral point.  Always providing fsid's improves
 >    the usefulness of the NFS namespace since referral and migration
 >    events can continue to be handled in the face of namespace
 >    management. This also helps to be reduce client complexity.

I guess with "if it has accessed" you mean "if it has previously
accessed".  Otherwise it sounds like you are talking about the operation
that received the NFS4ERR_MOVED error.  Similarly, "to locally determine
if it has actually accessed the data or if this is a referral for this
client" might be clearer as "to locally determine if it has been
accessing the filesystem or if the error applies to a fh in an different
(absent) filesystem".  The key here is recognizing the transition
between filesystems so the scope of the affected fhs can be determined.
The client's question is: "Did the filesystem holding the current fh
suddenly move, or has a filesytem transition occurred (e.g. as a result
of lookup) and the new filesystem is absent?"  Clearly, discovering the
fsid of the fh would resolve this simply.

In "renames along the path to the fsid", I think "fsid" should be "fh"
or "filesystem".

In "generate and fsid value": s/and/an/.

In a few places the prose implies that NFS4ERR_MOVED is an error that is
specific to certain clients, e.g. the phrase "this is a referral for
this client" in this paragraph.  The text in section 1.2 "Pure
Referrals" makes a similar implication.  Is this intentional and what
circumstances would make this appropriate?

In section 3.2.2, "Note first that the fileid attribute, since it is
within a new fs and thus a new fileid space, will not be unique within
the directory."  Should be "may not be unique".

 >    Neither of these problems arise in the case of mounted_on_fileid
 >    since that fileid is in the context of the mounted-on fs and unique
 >    within it.  Servers need to support and always return valid data
 >    for mounted-on-fileid, even for the case of an absent filesystem.
 >    The returned data represents a fileid for the entry in the
 >    directory that represents the absent fs. It's not the fileid of the
 >    root of the absent fs itself. ...

As mounted_on_fileid is an optional attribute, perhaps "Servers need to
support" is too strong a statement.

This is an interesting suggestion, though.  Is it really normal for the
client readdir to return to the application the inode numbers of the
mounted on fileid in the case where the fsid changes?  I presume that
this would be different from the inode reported by stat(), which would
return the fsid/fileid of the new filesystem.  This suggest that the
traditional implementation of pwd, would need to actually stat the
children and not rely on the inode number returned by readdir.

What about servers that do not actually have a mounted on object in the
parent filesystem, but fabricate the mountpoint using some other
mechanism?  If a fileid is to be invented for mounted_on_fileid, does it
have to be unique only within the directory or within the whole
filesystem?  What use do applications actually make of the value
returned by readdir?  Since mounted_on_fileid is an optional attribute,
servers can just decline to support it, but then clients are put in the
same position of inventing a value for readdir.

In section 3.3.2:
 >    When a server implements pure referrals for a large number of
 >    filesystems, it may be convenient for clients to cache the
 >    destination fs_locations and interrogate the change attribute to
 >    see if there has been any change in the locations attribute to
 >    require a refetch.  Therefore, the server must update the change
 >    attribute when the target of the referral change.

This seems like an unnecessary requirement.  The client could just
request the fs_locations attribute instead of the change attribute.  Is
the assumption that the latter is much cheaper than the former?  Also
the change attribute is described as a per filesystem object attribute,
even though fs_locations is per filesystem.  If the filesystem is
absent, to what filesystem object does the change attribute refer?

 >    If the server does provide this information, the client must
 >    understand that there is no necessary continuity between the change
 >    values on referring server and on the target, as there may not be
 >    in the general migration case.  Clients are best advised to
 >    periodically refetch such values on the target server and assume
 >    that it is possible that a change to the object may have occurred
 >    before the fetch of the change attribute on the target, and that
 >    this fact means that data and attribute caches on he client are
 >    best flushed.

Why "periodically refetch"?

Reading the descripton of "change" in RFC3530, it does describe the
attribute as "created" and not of property of the object, per se.  This
is consistent with your statement that the sequence of values cannot be
assumed to survive a migration event.  However, this means that data
cannot be cached over a migration, which is rather unfortunate.  Though,
I guess, data protected by an appropriate session lock can be relied
upon to survive a migration.

In section 4.1:

In "the implementation herein", perhaps "recommendations" would be
better.

This draft contains lots of good information and many helpful
clarifications.  It is a valuable addition to the RFC.

Thanks,
Ted Anderson
IBM Almaden Research Center

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 28 Apr 2005 17:39:56 -0400

> > Maybe we need something in a minor version that will either allow
the
> > change value continuity or discontinuity for a migration event to be
> > queried by the client.  Or maybe change value continuity should be
> > required.  It is hard to generate interest in addressing these
issues
> > as long as general migration is always a bit over the horizon.  It
> > seems that nobody wants to sign up for writing a inter-server
> > migration protocol.

> This highlights a deficiency in the change attribute.  It is defined
to
> change "if file data, directory contents or attributes of the object
> have been modified".  Since fs_locations is an attribute, it makes
sense
> for its change attr to be different when the filesystem moves.
However,
> this makes using the change attribute to protect a cache impossible
> across a move or even a replica fail-over.  What we really want is
> something that describes changes to file data and directory contents
> alone (maybe including the length), like an unsettable, fine-grained
> mtime.

If you take that statement literally and then go through the attribute 
list, I think you can find plenty of cases in which saying that the
change time has to change when that attribute changes is more than a
little bizarre.  For example, if I create a file in an fs then
files_free
and files_avail will probably change for every object in the fs.  Even
though the spec literally does say that the change attribute should
change in this case no sane implementation will do so.

With respect to the change attribute and migration (rather than pure
referrals), the issue with the change attribute is not simply that it
is an attribute (and must therefore change), but the other text you
noted that makes the particular form of change attribute up to the 
server.  This makes the issue of continuity of monotonic increase
problematic across migration between servers when those servers are
of different types.  It is problematic, but not insolubly so.  If the
spec mandated such continuity, then if you used some sort of fine-
grained function of time of last change (data or "real" file attributes)
and I used an update count, then if you migrated an fs to me, I could
just do an increment with our change value , while you could use a
delta between migrate time and time of last change and add my base
change number.  But the spec doesn't mandate those contortions when
migration happens.  Maybe it should in a minor version at least
give the client a way to determine whether there is continuity
or not. 

Note that in this case, the issue is not that we need (because we
think the spec mandates it) or want (because we think it helps) to
increment the change value, but that if there is a lack
of change monotonic increment continuity, the client can assume
nothing about the relationship between change values coming from
two different servers.  They are not comparable.  Despite the fact
that the client can compare two 64-bit integers, the results of 
that comparison have no significance when they derive from 
different servers.

The idea that the change value has to change when an fs_locations
in the case of change in referral target is motivated by the need
to give the client notice of a change in what is the major piece 
of information associated with the referral, where it goes.  In 
that case, fs_locations is not a random piece of associated fs 
state but the real contents of the referral point.  Note that
if we had the assured continuity of change value in the general
the effect on client caching of bumping the change value for
the referral point would be minimal.  It would only affect caching
of the root directory of the migrated fs.

>  >>Though, I guess, data protected by an appropriate session lock can
be
>  >>relied upon to survive a migration.
>  >
> > Not sure exactly what you mean here.

> By "session lock" I meant "share reservation".  In other words, if the
> server state transferral accompanying a migration, as is alluded to in
> the RFC, could preserve lock state, then a client with a file opened
> with OPEN4_SHARE_ACCESS_READ could assume that the file contents was
the
> same in the new location even if the change attribute was different.

Only if you specified DENY-WRITE.  In that case I believe it is
reasonable
to suppose that if the new location allowed you to reclaim that lock,
the
possibility of a conflicting access (i.e. a write) would be excluded.  I
need to check the exact text.  The spec is very explicit on this issue
in
the case of reclaim after a reboot but may not be so clear in the case
in
which you are establishing state on the target after migration.

-----Original Message-----
From: Ted Anderson [mailto:[email-masked]] 
Sent: Tuesday, April 26, 2005 11:35 AM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] RE: Comments on draft-noveck-nfsv4-referrals-00.txt

On 4/25/2005 13:49, Noveck, Dave wrote:
 > Ted Anderson wrote:
 
 >>>   since that fileid is in the context of the mounted-on fs and
 >>>   unique within it.  Servers need to support and always return
valid
 >>>   data for mounted-on-fileid, even for the case of an absent
 >>>   filesystem.  The returned data represents a fileid for the entry
 >>>   in the directory that represents the absent fs. It's not the
 >>>   fileid of the root of the absent fs itself. ...
 >>
 >
 >>As mounted_on_fileid is an optional attribute, perhaps "Servers need
 >>to support" is too strong a statement.
 >
 > My co-author and I went round and round on this issue and as I expect
 > that this document will eventually turn into a working group document
 > at some point, the working group will have to go round and round on
 > the issue as well. ...

Agreed.  Since this document is providing recommendations or guidelines
for interoperability all statements like this should be taken in that
context.  It is appropriately deviod of "MUST"s, "SHALL NOT"s etc.

 >>This is an interesting suggestion, though.  Is it really normal for
 >>the client readdir to return to the application the inode numbers of
 >>the mounted on fileid in the case where the fsid changes?
 >
 > I believe that it is and this is why the mounted_on_fileid attribute
 > was created, i.e. to allow that value to be returned.  As you can see
 > from the fact that its attribute number is not where it would be
based
 > on alphabetical order of name, it was a late addition, as we found
out
 > servers needed to return that data for cleint to function properly in
 > many cases.

I tracked down what I belive to be the thread[1] where this was
discussed.  I remain a bit confused about the value of using readdir
inode numbers to implement pwd/getcwd functionality.  Perhaps it is used
when looking for mount points, rather than when looking for the name of
the child.  It seems to me that comparing st_dev's would be the more
obvious approach.  At any rate, the point is that some applications do
this and so it would be nice not to break them.  The details matter if
you are trying to *invent* a value to use, but in this case we just want
to make sure that readdir can work as well as possible assuming the
server is already providing mounted_on_fileid for resident filesystems.

 >>is consistent with your statement that the sequence of values cannot
 >>be assumed to survive a migration event.
 >
 > I wasn't specifically relying on that, although, as you say it does
 > seem to give support to what I was saying.  To me, referrals,
 > particularly pure referrals are a special case in which it is
 > unreasonable to expect that a server implementing referrals, can
 > co-ordinate it's change value with that of the root of the fs that it
 > refers to (or even the different roots on different servers when the
 > fs moves).
 >
 >
 >>However, this means that data
 >>cannot be cached over a migration, which is rather unfortunate.
 >
 > Very unfortunate.  I think there are a number of significant problems
 > here.  One is there can be significant data cached particularly when
 > we are talking about high-latency far-distant clients, including data
 > cached on disk.  Having the client have to verify the up-to-date-ness
 > of such data is one thing, but making verification useless and
forcing
 > a data refetch is another.
 >
 > The other issue is that for now, we don't have an inter-server
 > migration protocol so migration events will for a while be
 > intra-vendor using compatible filesystem implementations (not
 > necessarily so in the pure referral case).  In that environment,
 > continuity of change values would normally be provided by servers as
 > matter of course and clients will come to depend on that, any "value
 > created by the server" language in the spec notwithstanding.
 >
 > Maybe we need something in a minor version that will either allow the
 > change value continuity or discontinuity for a migration event to be
 > queried by the client.  Or maybe change value continuity should be
 > required.  It is hard to generate interest in addressing these issues
 > as long as general migration is always a bit over the horizon.  It
 > seems that nobody wants to sign up for writing a inter-server
 > migration protocol.

This highlights a deficiency in the change attribute.  It is defined to
change "if file data, directory contents or attributes of the object
have been modified".  Since fs_locations is an attribute, it makes sense
for its change attr to be different when the filesystem moves.  However,
this makes using the change attribute to protect a cache impossible
across a move or even a replica fail-over.  What we really want is
something that describes changes to file data and directory contents
alone (maybe including the length), like an unsettable, fine-grained
mtime.

 > So one thought about how to approach this is to look at how much of
 > the work of migration might be implementated by reading and writing
 > the data from the source and to the target servers using v4 and ask
 > ourselves precisely else we need to complete the task.  Are we
talking
 > about a relatively small set of things that could be ops added to a
 > minor version?  Would the task be less daunting conceived in that
way?
 > Might we get started and make incremental progress in that fashion?
 > Is that something that we are more likely to get volunteers for than
 > a new inter-server migration protocol?

I agree that a few enhancements to the client/server protocol might be
the way to go.  It would certainly seem to be easier from a
standardization perspective.  In practice, third-party or multi-vendor
approaches are going to have to support this mode as a fall back anyway.
If performance is the concern that a server-to-server protocol is
intended to address, then maybe extensions (such as pNFS?) to tackle
that head on might be the winning strategy.

 >>Though, I guess, data protected by an appropriate session lock can be
 >>relied upon to survive a migration.
 >
 > Not sure exactly what you mean here.

By "session lock" I meant "share reservation".  In other words, if the
server state transferral accompanying a migration, as is alluded to in
the RFC, could preserve lock state, then a client with a file opened
with OPEN4_SHARE_ACCESS_READ could assume that the file contents was the
same in the new location even if the change attribute was different.

Ted Anderson

[1] 
http://playground.sun.com/pub/nfsv4/webpage/nfsv4-wg-archive-dec-96-jan-
03/2058.html