Subject: RE: Replication/Migration issues list

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 3 Oct 2002 07:26:01 -0700

Ted Anderson wrote: 
> On 9/10/2002 13:25:51 MDT, Robert Thurlow wrote:
> > I think repartitioning [replication subsets] is a disruptive one-time
> > event that is just as painful as it is today, but the hope is that
> > once the filesystems are "right-sized", this doesn't need to happen
> > again.  This may not hold true over the life of the data, I can see
> > now that you mention it.

> One of the benefits of the AFS/DFS approach which provides uniform names
> and fileset mobility is that pathnames become very long lived.  One can
> really count on file names surviving all manner of managerial and
> technological disruptions.  A consequence of this is that filesets do
> not stay right-sized over the long run, and both AFS and DFS suffer
> because they have no easy way to rearrange volume/fileset boundaries.
> To the extent that NFSv4, with its improving support for replication and
> migration, is moving in the direction of long-lived namespaces, this
> problem is likely to arise here as well.

I agree.

> > I don't really see a way for this not to be disruptive, unfortunately.
> > Ideas?

> My suggestion is to move away from persistent file handles and the use
> of file ids (i.e. inode numbers) as well as functions that depend upon
> them.  Pathnames become the preeminent file identifiers.  This means
> doing away with hardlinks, in favor of symlinks, and deprecating
> anonymous (zero link-count / ZLC) files.  Windows systems do fine
> without these features (though I've never understood why symlinks got
> left out).

I don't see that we have the option of legislating all of the troublesome
features of the UNIX file system interface out of existence.  I'd 
outlaw hard links in a second if I had that power, but I think we have
to be able to support the link system call and applications that depend
on removing open files, and doing stat calls (which returns the fileid).
If I'm wrong and we could reconstitute ourselves as a benevolent API 
dictatorship, I'm sure we could make the world better, but I don't think
anyone is going to let us.

In the Windows case, they do have an API dictatorship, and opinions on how
well that works vary.  

I think that if we want to be able to re-partition in a fashion which is
minimally disruptive, and have to support the expectations of existing
operating systems, what an implementation needs to do is to maintain
the image of the existing partition structure for current mounts while 
new ones see the split.  This involves some sort of remapping on the 
server which is considerable implementation work for the servers but 
leaves the protocol fairly unencumbered.  The idea is that the 
remapping is temporary once all the existing mounts go away.  What's
required on the V4 protocol side is the re-introduction of mount,
not a separate protocol, but a new V4.1 operation which might be better
called open_fs (and also a close_fs).  It allows us to delimit the
period for which we have to maintain certain invariants (in this case
that the boundaries of an fs don't change).  In fact, this would be
useful even for those that did not want to tackle the compelxities 
of the remapping implementation.  A simple server could support 
repartitioning of a non-mounted file system, with the server knowing
whether the fs in question was mounted or not.  

> Rename remains problematic for open files and persistent caches.  The
> Windows approach of outlawing all renames that affect open files seems
> unreasonable.  One idea might be to allow a client to register to
> receive renames affecting certain directories: either an explicit list
> or perhaps for an entire filesystem.  Directory renames are uncommon so
> this wouldn't represent a large amount of traffic, but would allow
> clients to track changes affecting their volatile file handles.

---

From: Ted Anderson <[email-masked]>
Date: Fri, 04 Oct 2002 11:18:30 -0400

On 10/3/2002 10:26, Noveck, Dave wrote:
 >>>I don't really see a way for this not to be disruptive,
 >>>unfortunately.  Ideas?
 >
 > I don't see that we have the option of legislating all of the
 > troublesome features of the UNIX file system interface out of
 > existence.  I'd outlaw hard links in a second if I had that power, but
 > I think we have to be able to support the link system call and
 > applications that depend on removing open files, and doing stat calls
 > (which returns the fileid).  If I'm wrong and we could reconstitute
 > ourselves as a benevolent API dictatorship, I'm sure we could make the
 > world better, but I don't think anyone is going to let us.
 >
 > In the Windows case, they do have an API dictatorship, and opinions on
 > how well that works vary.

I was not suggesting that we legislate troublesome features away, but
only that we encourage people not to use them.  We don't need a
dictatorship to accomplish this.  Multi-platform environments such as
Java and Perl motivate developers to eschew these kind of features
because they cannot be relied upon to work well on all systems.  More
generally applications written with Unix and Windows portability in mind
face some of the same motivations.  Distributed file systems are similar
in that both clients and servers are multi-platform.  So even
Unix-specific applications that expect to work with distributed file
systems will want to avoid these features too.  Another example is the
Data Grid community, which is constructing a wide-area distributed file
system based on an appallingly low common denominator, ftp, which also
doesn't support these semantics.  So these forces are already at work.
I would expect a declining fraction users which depend upon hardlinks
and ZLC files, and that NFSv4's wide acceptance would accelerate this
trend.

I am only suggesting a change in bias which anticipates this trend.
This change could take the form of an official statement discouraging
these features or even just describing them as non-portable.  Is it an
issue that this would erode the differentiation between NFS and CIFS?

<vague-handwaving-argument>

This new bias would support consideration of features that improve the
NFSv4 protocol primarily for applications that avoid these features.
While ZLC files and hardlinks would continue to work when applied to
Unix file servers, applications that avoid them might, for example, be
able to enjoy better transparency in the face of replication and
migration.  Generally, it would allow development and design efforts to
be better allocated towards features that increasingly distinguish
large scale distributed file systems.  Such a change in perspective
might allow adoption features which benefit users of volatile file
handles but which don't help (or even hinder) users of persistency file
handles, while, in the present climate, such a feature would be
unacceptable.  The protocol seems too heavily weighted in favor of
persistent file handles and that legacy.

</vague-handwaving-argument>

 > I think that if we want to be able to re-partition in a fashion which
 > is minimally disruptive, and have to support the expectations of
 > existing operating systems, what an implementation needs to do is to
 > maintain the image of the existing partition structure for current
 > mounts while new ones see the split.  This involves some sort of
 > remapping on the server which is considerable implementation work for
 > the servers but leaves the protocol fairly unencumbered.  The idea is
 > that the remapping is temporary once all the existing mounts go away.
 > What's required on the V4 protocol side is the re-introduction of
 > mount, not a separate protocol, but a new V4.1 operation which might
 > be better called open_fs (and also a close_fs).  It allows us to
 > delimit the period for which we have to maintain certain invariants
 > (in this case that the boundaries of an fs don't change).  In fact,
 > this would be useful even for those that did not want to tackle the
 > compelxities of the remapping implementation.  A simple server could
 > support repartitioning of a non-mounted file system, with the server
 > knowing whether the fs in question was mounted or not.

This certainly seems like a reasonable suggestion.  It would allow
well-defined behavior for attributes that apply to whole filesystems,
though it might raise again the question of their granularity.  Even
better, though perhaps too ambition, would be to allow the server to
force expiration of such a mount.  This would allow prompt changes on
the server while placing some of the remapping burden on the client.

Ted Anderson