Subject: [nfsv4] RE: Comments on draft-noveck-nfsv4-referrals-00.txt

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 25 Apr 2005 13:49:39 -0400

Ted Anderson wrote:
> Comments on "Implementation Guide for Referrals in NFSv4", by David
> Noveck, Rodney C. Burnett, Feb 2005,
>
http://www.ietf.org/internet-drafts/draft-noveck-nfsv4-referrals-00.txt

> In section 2.1, PUTFH is described as receiving "fh for absent
volume",
> but it should be something like "fh within absent filesystem".  This
is
> the only use of the term "volume" in the document.  I think
"filesystem"
> would be better and more consistent.

Agreed. 

> In section 3.1, the assuptions for READDIR include:
>       o    When fs_locations is among the attributes requested, the
>            server may provide a subset of the other requested
attributes
>            together with fs_locations for roots of absent fs's,
without
>            causing any error for the READDIR as a whole.  If
rdattr_error
>            is also requested and there are attributes which are not
>            available, then rdattr_error will receive the value
>            NFS4ERR_MOVED.

> This seems inconsistent with the recommendation for GETATTR, which
says
> that if fs_locations is requested, will not return NFS4ERR_MOVED.
Since
> rdattr_error is described as "Error returned from getattr during
> readdir", what is the motivation for this behavior?  Is the thinking
> that this is a handy way of indicating an absent filesystem?

If there were an "absent" (or "present") attribute, then there would be
no value in this, so yes, this is the motivation.

As far as the seeming inconsistency, I would argue that "GETATTR" has to
denote the specific op and that the reason that GETATTR must not return
an error in this case is simply that it would be impossible to get the 
needed information if it is requested.  So the logic of the exception 
that GETATTR should not return an error when fs_locations together with
additional attributes are requested, is not that there is no error when 
one such as fh is asked, but that returning an error in this case would 
force the client either not to get the desired information or play a 
guessing game that doesn't make sense.  I don't think it implies that 
the act of getting attributes (i.e. "getattr") has no error.  And I
think
does make sense to say that the act of getting an attribute for an
object
withn and absent filesystem that the server does not allow in that
context,
is gounds to return a MOVED error.

> In section 3.2.1, this example is confusing:
> >    A key reason for always providing an fsid can be seen by
> >    considering client behavior.  When a client gets the error
> >    NFS4ERR_MOVED, it has a number of key steps to get through.  It
> >    must determine if it has accessed the filesystem on the server
that
> >    the moved error involves. If so, then it represents a data
> >    migration event in the strict sense, rather than a referral.  The
> >    client must recover state and properly use rootpath data from the
> >    old and new server to graft the new server's namespace into the
> >    client's local space. If a client gets an NFS4ERR_MOVED error and
> >    it cannot get an fsid, then it must use fs_locations, fs_root,
and
> >    rootpath data to locally determine if it has actually accessed
the
> >    data or if this is a referral for this client.  If it uses
fs_root
> >    and there have been renames along the path to the fsid, then the
> >    client will not be able to distinguish a migration from a
referral.
> >    The best solution here is for the client to always get an fsid.
All
> >    a server has to do is generate and fsid value that cannot
represent
> >    actual exported data at the server. We want to have a model where
> >    referrals can still function even if renames or other events
change
> >    the path to the referral point.  Always providing fsid's improves
> >    the usefulness of the NFS namespace since referral and migration
> >    events can continue to be handled in the face of namespace
> >    management. This also helps to be reduce client complexity.

> I guess with "if it has accessed" you mean "if it has previously
> accessed".  Otherwise it sounds like you are talking about the
operation
> that received the NFS4ERR_MOVED error.  

I can see that this should be clarified.

> Similarly, "to locally determine
> if it has actually accessed the data or if this is a referral for this
> client" might be clearer as "to locally determine if it has been
> accessing the filesystem or if the error applies to a fh in an
different
> (absent) filesystem".  

OK.

> The key here is recognizing the transition
> between filesystems so the scope of the affected fhs can be
determined.
> The client's question is: "Did the filesystem holding the current fh
> suddenly move, or has a filesytem transition occurred (e.g. as a
result
> of lookup) and the new filesystem is absent?"  Clearly, discovering
the
> fsid of the fh would resolve this simply.

Yes.

> In "renames along the path to the fsid", I think "fsid" should be "fh"
> or "filesystem".

Will fix.

In "generate and fsid value": s/and/an/.

> In a few places the prose implies that NFS4ERR_MOVED is an error that
is
> specific to certain clients, e.g. the phrase "this is a referral for
> this client" in this paragraph.  The text in section 1.2 "Pure
> Referrals" makes a similar implication.  Is this intentional and what
> circumstances would make this appropriate?

That isn't the intention.  The idea is that client A and client B (and
every other client) will both/all get MOVED, but that  you may have 
different situations depending on the past history of A and B.  If 
A has not previously accessed the absent fs, he will see and process 
the event as a referral.  On the other hand, if B gets the error after 
having accessed the file system (when it wasn't absent), then B will 
see a migration event and not a referral.

> In section 3.2.2, "Note first that the fileid attribute, since it is
> within a new fs and thus a new fileid space, will not be unique within
> the directory."  Should be "may not be unique".

How about "need not be unique"?

> >    Neither of these problems arise in the case of mounted_on_fileid
> >    since that fileid is in the context of the mounted-on fs and
unique
> >    within it.  Servers need to support and always return valid data
> >    for mounted-on-fileid, even for the case of an absent filesystem.
> >    The returned data represents a fileid for the entry in the
> >    directory that represents the absent fs. It's not the fileid of
the
> >    root of the absent fs itself. ...
>

> As mounted_on_fileid is an optional attribute, perhaps "Servers need
to
> support" is too strong a statement.

My co-author and I went round and round on this issue and as I expect 
that this document will eventually turn into a working group document
at some point, the working group will have to go round and round on the 
issue as well.

One thing to note is that fileid is also an optional attribute.  If
you asked me whether your server needs to support fileid, my answer
would be "yes".  I recognize that RFC3530 says that if you don't support
it, you have a legal implementation, but it may be an implementation 
that nobody will be able to cleanly interoperate with.  So perhaps
my answer to the previous question should not be "yes", but rather
"Do you feel lucky?".

The document reflects some implementation experience and that has all
been with servers that support filed and mounted_on_fileid and clients
that use those values.  My inclination is to stick with what we know 
works until somebody, for example, does referrals without
mounted_on_fileid
supported and makes it work.  

> This is an interesting suggestion, though.  Is it really normal for
the
> client readdir to return to the application the inode numbers of the
> mounted on fileid in the case where the fsid changes? 

I believe that it is and this is why the mounted_on_fileid attribute was
created, i.e. to allow that value to be returned.  As you can see from
the
fact that its attribute number is not where it would be based on
alphabetical
order of name, it was a late addition, as we found out servers needed to
return that data for cleint to function properly in many cases.

> I presume that
> this would be different from the inode reported by stat(), which would
> return the fsid/fileid of the new filesystem.  This suggest that the
> traditional implementation of pwd, would need to actually stat the
> children and not rely on the inode number returned by readdir.

I believe that mounted_on_fileid was needed in part to make pwd work
in the case where fsid changed.

> What about servers that do not actually have a mounted on object in
the
> parent filesystem, but fabricate the mountpoint using some other
> mechanism?  If a fileid is to be invented for mounted_on_fileid, does
it
> have to be unique only within the directory or within the whole
> filesystem?  

I believe it has to be unique within the filesystem which would be easy 
enough given that fileid is 64 bits, but we know that there are clients 
that do not like fileid's that are not in the 32-bit range and they can 
cause applications to get errors that they would not get when the
fileids 
are within the 32-bit range.  This points up again the fact that
reasonable
implementation advice may be more restrictive than what the spec allows:
the spec is perfectly OK with 0x0000000100000000 as a fileid but my 
advice to a server would be to not return it. 

> What use do applications actually make of the value
> returned by readdir?  Since mounted_on_fileid is an optional
attribute,
> servers can just decline to support it, but then clients are put in
the
> same position of inventing a value for readdir.

Inventing a value or returning an error to the caller.

If a server doesn't support fileid, you have the same issue for stat.

> In section 3.3.2:
>  >    When a server implements pure referrals for a large number of
>  >    filesystems, it may be convenient for clients to cache the
>  >    destination fs_locations and interrogate the change attribute to
>  >    see if there has been any change in the locations attribute to
>  >    require a refetch.  Therefore, the server must update the change
>  >    attribute when the target of the referral change.

> This seems like an unnecessary requirement.  The client could just
> request the fs_locations attribute instead of the change attribute.
Is
> the assumption that the latter is much cheaper than the former?  

It is cheaper.  I don't know about "much cheaper", but I want to 
clarify that the intention is that only if a serer supports the change
attribute is it required to update it at this point.  I think lots of
servers will not return the change attribute for fh's in absent fs's
and just return fs_locations.  If a server does support this, it should
provide a useful value for someone using this as referral information.

> Also
> the change attribute is described as a per filesystem object
attribute,
> even though fs_locations is per filesystem.  If the filesystem is
> absent, to what filesystem object does the change attribute refer?

The one on which you do a fs_locations (and whose fh you can never see
:-).
We have the concept that fs's can be absent and still have the
fs_locations
attribute (and possibly other attributes).

>  >    If the server does provide this information, the client must
>  >    understand that there is no necessary continuity between the
change
>  >    values on referring server and on the target, as there may not
be
>  >    in the general migration case.  Clients are best advised to
>  >    periodically refetch such values on the target server and assume
>  >    that it is possible that a change to the object may have
occurred
>  >    before the fetch of the change attribute on the target, and that
>  >    this fact means that data and attribute caches on he client are
>  >    best flushed.

> Why "periodically refetch"?

Because I was confused when I did some editing.  Will fix.

> Reading the descripton of "change" in RFC3530, it does describe the
> attribute as "created" and not of property of the object, per se.
This
> is consistent with your statement that the sequence of values cannot
be
> assumed to survive a migration event. 

I wasn't specifically relying on that, although, as you say it does seem
to give support to what I was saying.  To me, referrals, particularly
pure referrals are a special case in which it is unreasonable to expect
that a server implementing referrals, can co-ordinate it's change
value with that of the root of the fs that it refers to (or even the 
different roots on different servers when the fs moves). 

> However, this means that data
> cannot be cached over a migration, which is rather unfortunate. 

Very unfortunate.  I think there are a number of significant problems
here.  One is there can be significant data cached particularly when
we are talking about high-latency far-distant clients, including data
cached on disk.  Having the client have to verify the up-to-date-ness
of such data is one thing, but making verification useless and forcing
a data refetch is another.

The other issue is that for now, we don't have an inter-server migration
protocol so migration events will for a while be intra-vendor using
compatible filesystem implementations (not necessarily so in the pure
referral case).  In that environment, continuity of change values 
would normally be provided by servers as matter of course and clients
will come to depend on that, any "value created by the server" language 
in the spec notwithstanding.

Maybe we need something in a minor version that will either allow the
change value continuity or discontinuity for a migration event to be
queried by the client.  Or maybe change value continuity should be
required.  It is hard to generate interest in addressing these issues
as long as general migration is always a bit over the horizon.  It 
seems that nobody wants to sign up for writing a inter-server migration 
protocol.

So one thought about how to approach this is to look at how much of
the work of migration might be implementated by reading and writing 
the data from the source and to the target servers using v4 and ask 
ourselves precisely else we need to complete the task.  Are we talking
about a relatively small set of things that could be ops added to a
minor version?  Would the task be less daunting conceived in that way?
Might we get started and make incremental progress in that fashion?
Is that something that we are more likely to get volunteers for than
a new inter-server migration protocol?      

> Though,
> I guess, data protected by an appropriate session lock can be relied
> upon to survive a migration.

Not sure exactly what you mean here.

In section 4.1:

> In "the implementation herein", perhaps "recommendations" would be
> better.

OK.

> This draft contains lots of good information and many helpful
> clarifications.  It is a valuable addition to the RFC.

It will be more valuable due to your comments.  Thanks.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Ted Anderson <[email-masked]>
Date: Tue, 26 Apr 2005 11:34:52 -0400

On 4/25/2005 13:49, Noveck, Dave wrote:
 >>>   unique within it.  Servers need to support and always return valid
 >>>   data for mounted-on-fileid, even for the case of an absent
 >>>   filesystem.  The returned data represents a fileid for the entry
 >>>   in the directory that represents the absent fs. It's not the
 >>>   fileid of the root of the absent fs itself. ...
 >>
 >
 >>to support" is too strong a statement.
 >
 > at some point, the working group will have to go round and round on
 > the issue as well. ...

Agreed.  Since this document is providing recommendations or guidelines
for interoperability all statements like this should be taken in that
context.  It is appropriately deviod of "MUST"s, "SHALL NOT"s etc.

 >>the client readdir to return to the application the inode numbers of
 >>the mounted on fileid in the case where the fsid changes?
 >
 > I believe that it is and this is why the mounted_on_fileid attribute
 > was created, i.e. to allow that value to be returned.  As you can see
 > from the fact that its attribute number is not where it would be based
 > on alphabetical order of name, it was a late addition, as we found out
 > servers needed to return that data for cleint to function properly in
 > many cases.

I tracked down what I belive to be the thread[1] where this was
discussed.  I remain a bit confused about the value of using readdir
inode numbers to implement pwd/getcwd functionality.  Perhaps it is used
when looking for mount points, rather than when looking for the name of
the child.  It seems to me that comparing st_dev's would be the more
obvious approach.  At any rate, the point is that some applications do
this and so it would be nice not to break them.  The details matter if
you are trying to *invent* a value to use, but in this case we just want
to make sure that readdir can work as well as possible assuming the
server is already providing mounted_on_fileid for resident filesystems.

 >>attribute as "created" and not of property of the object, per se. This
 >>be assumed to survive a migration event.
 >
 > I wasn't specifically relying on that, although, as you say it does
 > seem to give support to what I was saying.  To me, referrals,
 > particularly pure referrals are a special case in which it is
 > unreasonable to expect that a server implementing referrals, can
 > co-ordinate it's change value with that of the root of the fs that it
 > refers to (or even the different roots on different servers when the
 > fs moves).
 >
 >
 > The other issue is that for now, we don't have an inter-server
 > migration protocol so migration events will for a while be
 > intra-vendor using compatible filesystem implementations (not
 > necessarily so in the pure referral case).  In that environment,
 > continuity of change values would normally be provided by servers as
 > matter of course and clients will come to depend on that, any "value
 > created by the server" language in the spec notwithstanding.
 >
 > seems that nobody wants to sign up for writing a inter-server
 > migration protocol.

This highlights a deficiency in the change attribute.  It is defined to
change "if file data, directory contents or attributes of the object
have been modified".  Since fs_locations is an attribute, it makes sense
for its change attr to be different when the filesystem moves.  However,
this makes using the change attribute to protect a cache impossible
across a move or even a replica fail-over.  What we really want is
something that describes changes to file data and directory contents
alone (maybe including the length), like an unsettable, fine-grained
mtime.

 > a new inter-server migration protocol?

I agree that a few enhancements to the client/server protocol might be
the way to go.  It would certainly seem to be easier from a
standardization perspective.  In practice, third-party or multi-vendor
approaches are going to have to support this mode as a fall back anyway.
If performance is the concern that a server-to-server protocol is
intended to address, then maybe extensions (such as pNFS?) to tackle
that head on might be the winning strategy.

 >>Though, I guess, data protected by an appropriate session lock can be
 >>relied upon to survive a migration.
 >

By "session lock" I meant "share reservation".  In other words, if the
server state transferral accompanying a migration, as is alluded to in
the RFC, could preserve lock state, then a client with a file opened
with OPEN4_SHARE_ACCESS_READ could assume that the file contents was the
same in the new location even if the change attribute was different.

Ted Anderson
IBM Almaden Research Center

[1] 
http://playground.sun.com/pub/nfsv4/webpage/nfsv4-wg-archive-dec-96-jan-03/2058.html

---

From: Carl Burnett <[email-masked]>
Date: Tue, 26 Apr 2005 11:10:27 -0500

So I just have a couple of remarks below. Dave has addressed many of Ted's 
items. I see there have been new recent exchanges so I will catch up and 
may comment further.

Carl

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

> at some point, the working group will have to go round and round on the 

So while the RFC lists mounted_on_fileid as a recommended attribute, its 
really the case that a POSIX/UNIX platform cannot reasonably do an NFSv4 
implementation without mounted on fileid. Perhaps the adjustement to this 
documenet should be to say that for servers that support 
mounted_on_fileid, its essential that it be also be supported in the case 
of a directory entry that represents a junction to an absent fs.

> my answer to the previous question should not be "yes", but rather

Just as a note that, to my knowledge, every implementation that has 
successfully interoperated supports fileid. I think several of the 
implementations will operate without. I know for the AIX client, we won't 
work if the server can't provide a fileid (modulo the referral and 
migration cases).

---

From: Ted Anderson <[email-masked]>
Date: Thu, 28 Apr 2005 11:26:26 -0400

From 
 > 3.2.1.  fsid
 > ...
 >      To avoid misunderstanding, it should be noted that the fsid
 >      provided [along with fs_locations] is solely so that the fs
 >      boundaries can be properly noted and that the fsid returned will
 >      not necessarily be valid after resolution of the referral or
 >      migration event.  The logic of fsid handling for NFSv4 is that
 >      fsid's are only unique within a per-server context.  This would
 >      seem to be a strong indication that they need not be persistent
 >      when file systems are moved from server to server, although RFC
 >      3530 does not specifically address the matter.

and

On 4/26/2005 17:49, Noveck, Dave wrote:
 > One issue that I think we may be confronted with is that of creating
 > filesets out of existing directory trees.  What does not make sense as
 > a fileset this year may grow and need to be converted to that status
 > next year.  I think we will wind up having to tackle this issue.  The
 > solution may be as simple as establishing rules about fsid validity
 > (and requirements to refetch) when volatile filehandles expire, or we
 > may also need some corresponding support where volatile filehandles
 > are not suitable.  Anyway I don't think this is requirement for a
 > global namespace but I think that the reverse is quite likely.  The
 > existence of a global namespace may wind up making this a requirement
 > for smooth operation and reconfiguration.

These two comments bring to mind a question.  As the referrals draft
says, the fsid is a server-local concept.  When a filesystem moves to
another server, therefore, the fsid of all the objects in the old
server's filesystem may change.  Is it required that all objects on the
new server share the same new fsid?

If the moved objects do not all have to be in the same filesystem on the
new server, then the client must call GETATTR on each object it retains
(e.g. open or cached) in order to determine its new fsid.  In the case
of expiring volatile file handles it needs to do this anyway, so perhaps
there is no extra traffic required to accomplish this.  With persistent
file handles, the need is less clear, though at least this attribute
would need to be marked as requiring validation.

This concern applies to the fileid attribute as well.  It may also
change on a migration event, especially if the fsid is changing.
Generally, is is assumed that "normal" attributes do not change when an
object is migrated, but are faithfully copied to the new server.
Clearly, this should be true for user-visible attributes such as size
and time_modify, but less clear for attributes whose implementation may
be different on the new server, such as ACL, space_used and many of the
per filesystem attributes.  The uncertain status of the change attribute
was dicussed in an earlier exchange.

Clearly, some attributes are controlled by the operations defined by the
protocol or are otherwise well-defined.  Others describe properties of
the filesystem or the server.  An attribute like space_avail can be
expected to change frequently and often in a way unrelated to operations
of the client.  Others, like symlink_support, are probably assumed to be
immutable.  The fs_locations attribute is one of these, except that the
NFS4ERR_MOVED error can be returned as a sort of signal that
fs_locations has changed.  It might be worth considering the idea that
the same error could be used to signal the change of these other
quasi-static attributes.  Normally a change in these attributes would be
a result of a real migration to a new server, but this need not
necessarily be so.  In any case, receiving this special error code would
provide a reasonably clear-cut barrier allowing the client to cleanly
manage changes in the fundamental operating conditions of the server.

While it is probably out of scope of the referrals draft to clarify
these issues, migration may be difficult to implement in an
interoperable fashion without resolving the question of attribute
stability.  A simliar draft providing guidance for implementing
migration may be necessary.

Ted Anderson