Subject: RE: Replication/Migration issues list

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 4 Oct 2002 10:27:15 -0700

> On 10/3/2002 10:26, Noveck, Dave wrote:
> > Ted Anderson wrote:
> >>On 9/10/2002 13:25:51 MDT, Robert Thurlow wrote:
> >>>I don't really see a way for this not to be disruptive,
> >>>unfortunately.  Ideas?
> >
> >>My suggestion is to move away from persistent file handles and the use
> >>of file ids (i.e. inode numbers) as well as functions that depend upon
> >>them.  Pathnames become the preeminent file identifiers.  This means
> >>doing away with hardlinks, in favor of symlinks, and deprecating
> >>anonymous (zero link-count / ZLC) files.  Windows systems do fine
> >>without these features (though I've never understood why symlinks got
> >>left out).
> >
> > I don't see that we have the option of legislating all of the
> > troublesome features of the UNIX file system interface out of
> > existence.  I'd outlaw hard links in a second if I had that power, but
> > I think we have to be able to support the link system call and
> > applications that depend on removing open files, and doing stat calls
> > (which returns the fileid).  If I'm wrong and we could reconstitute
> > ourselves as a benevolent API dictatorship, I'm sure we could make the
> > world better, but I don't think anyone is going to let us.
> >
> > In the Windows case, they do have an API dictatorship, and opinions on
> > how well that works vary.
>
> I was not suggesting that we legislate troublesome features away, but
> only that we encourage people not to use them.  We don't need a
> dictatorship to accomplish this.  Multi-platform environments such as
> Java and Perl motivate developers to eschew these kind of features
> because they cannot be relied upon to work well on all systems.  More
> generally applications written with Unix and Windows portability in mind
> face some of the same motivations.  Distributed file systems are similar
> in that both clients and servers are multi-platform.  So even
> Unix-specific applications that expect to work with distributed file
> systems will want to avoid these features too.  Another example is the
> Data Grid community, which is constructing a wide-area distributed file
> system based on an appallingly low common denominator, ftp, which also
> doesn't support these semantics.  So these forces are already at work.
> I would expect a declining fraction users which depend upon hardlinks
> and ZLC files, and that NFSv4's wide acceptance would accelerate this
> trend.

I would also expect a declining fraction users to use those features,
but I expect a very long tail.  Old applications can stay around for
a very long time and if only a single application in your environment
requires a feature your environment requires that feature, at least if
you don't want to be bothered changing it, and many users most certainly
do not.

I'm doubtful about the claim that NFS-v4 will, of itself, accelerate
this trend.  It is true that the V4 spec allows the server a lot of
freedom (e.g. to reject remove of an open file), but the issue is
how users will react to this.  It's possible that they will avoid
the features, but if the other trends that you mentioned haven't
caused that (because the user can't change the application or for
other reasons), I think it is more likely that they will select V4
servers that have the desired characteristics.  After all, the range
of possible behaviors from v4-spec-compliant server is so large 
(e.g. almost all attributes are optional), that I don't think
people will be able to assume that any compliant server will do for
their application. 

> I am only suggesting a change in bias which anticipates this trend.
> This change could take the form of an official statement discouraging
> these features or even just describing them as non-portable.  Is it an
> issue that this would erode the differentiation between NFS and CIFS?

The spec says that the server may deny remove of open files and may 
deny rename of open files (and also rename of directories including
open files).  That constitutes an official statement (at least I expect
it to become official in a few weeks) that those features are non-
portable, but some of them are things that I would hate to do without
and you would as well.

I think the official statement you are looking for would distinguish
between "good" non-portable features and "bad" non-portable features,
and I really don't relish trying to get a consensus on that.  (and if
that's not legislation, it's awfully close.  Sort of like a joint
congressional resolution) 

> <vague-handwaving-argument>

> This new bias would support consideration of features that improve the
> NFSv4 protocol primarily for applications that avoid these features.
> While ZLC files and hardlinks would continue to work when applied to
> Unix file servers, applications that avoid them might, for example, be
> able to enjoy better transparency in the face of replication and
> migration.  Generally, it would allow development and design efforts to
> be better allocated towards features that increasingly distinguish
> large scale distributed file systems.  Such a change in perspective
> might allow adoption features which benefit users of volatile file
> handles but which don't help (or even hinder) users of persistency file
> handles, while, in the present climate, such a feature would be
> unacceptable.  The protocol seems too heavily weighted in favor of
> persistent file handles and that legacy.

> </vague-handwaving-argument>

If features are very rarely used, or we have very good reason to believe
that they will be very rarely used  when the extensions we are designing
will be available, then we certainly should consider that as a relevant
fact.  When the extensions in question are made significantly more
difficult, I expect people to ask whether supporting a particular
little-used feature makes sense.  The answers that people give are
going to depend on the specific trade-offs involved.

I think the features you are talking about don't even begin to be
in that category yet.  There are lots of people that use them and
while they may avoid these features at some point in the future,
we really don't know when.  They have these features on their
local file systems and there is no reason they should not have them
on their file servers.

I know these features make life more difficult for us, but I don't
think the increment of difficulty is so great that we should simply
not support them.

As for volatile file handles, I remain to be convinced of their value
in a general setting (not readonly, in the presense at least of renames
of open files).  Do we have any implementations that have gone this way
yet?  I think we have to see them in real use before we start considering
persistent file handles a legacy format that is on its way out.

> > I think that if we want to be able to re-partition in a fashion which
> > is minimally disruptive, and have to support the expectations of
> > existing operating systems, what an implementation needs to do is to
> > maintain the image of the existing partition structure for current
> > mounts while new ones see the split.  This involves some sort of
> > remapping on the server which is considerable implementation work for
> > the servers but leaves the protocol fairly unencumbered.  The idea is
> > that the remapping is temporary once all the existing mounts go away.
> > What's required on the V4 protocol side is the re-introduction of
> > mount, not a separate protocol, but a new V4.1 operation which might
> > be better called open_fs (and also a close_fs).  It allows us to
> > delimit the period for which we have to maintain certain invariants
> > (in this case that the boundaries of an fs don't change).  In fact,
> > this would be useful even for those that did not want to tackle the
> > compelxities of the remapping implementation.  A simple server could
> > support repartitioning of a non-mounted file system, with the server
> > knowing whether the fs in question was mounted or not.

> This certainly seems like a reasonable suggestion.  It would allow
> well-defined behavior for attributes that apply to whole filesystems,
> though it might raise again the question of their granularity.  Even
> better, though perhaps too ambition, would be to allow the server to
> force expiration of such a mount.  This would allow prompt changes on
> the server while placing some of the remapping burden on the client.

I think it would be possible to put the remapping on the client rather
than the server, or there might be a reason to split the responsibility
in some particular way.  What I don't think will work is for the protocol
not to clearly assign the responsibility.  If the server and the client
can both do it, then the likely result will be the client will wind up doing
a poor job because the server is going to do it and he will only have to 
do it in extremis, and the server will be very free about finding exigent
circumstances that force him to punt the job to the client, since the
client should have that capability.