Subject: Re: Replication/Migration issues list

From: Ted Anderson <[email-masked]>
Date: Thu, 03 Oct 2002 08:20:42 -0400

This note addresses some of the harder questions about scope and goals.

Regarding file system state, it seems to me that there are three cases:
read-only replicas, writable replicas (e.g. a fully populated cache) and
migration.  All these require addressing the question of file handles
(and file ids), since this involves issues of file system implementation
which may differ from one replica/host to another.  The first is
distinguished because it has very limited (or absent) requirements for
managing opens, shares and delegations.  For the latter two, however,
this management is important because of the need to coordinate writers
with other users.  Consistency of writable replicas depends upon
communicating information about multiple readers and writers with a
central coordinating authority.  On the other hand, migration involves
transferring this coordination from one authority to another.  These
latter two activities seem to require different mechanisms.  Thus, I
conclude there is a need for three mechanisms to support the full range
of file sharing functionality.

As mentioned in the previous message in connection with right-sizing
filesystems, I think we should move towards depending upon pathnames and
deprecating file ids, hardlinks, and ZLC files, though directory renames
remain an issue.  This shift, already begun by the introduction of
volatile file handles, eases the job of transferring file handles and
file ids between heterogeneous servers.

For writable replicas or caches, the NFSv4 client/server protocol
contains a suite of mechanisms that are arguably adequate to consistency
management.  I don't see any moves to add this capability to the
repl-mig protocol.  Is support for writable replicas (or even strongly
consistent read-only replicas which would benefit from something like
delegation) out of scope for the repl-mig protocol?

To support the migration of filesystems, there seem to be two minds on
the subject of this meta file system information.  Does the repl-mig
protocol rely on clients to reestablish state with the new server, like
after a crash, or does the protocol need to help.  I've heard opinions
both ways on the WG mailing list.  Has a consensus developed?

The approach of relying on the server crash recovery logic seems
workable.  I am not aware of any show stoppers with this approach,
except the grace period limit on high-availability[1].  Probably a few
corner cases that present some difficulties do exist.  We should collect
problematic scenarios so they can be evaluated based on the application
transparency goal.

The grace period problem, might admit to a third option, namely to
enhance the client-server protocol to ease the transfer of client state.
Specifically, a callback could be issued by the source server which
tells clients to start transferring their state to the target server.
Then clients send a finish message to advise the target that they have
completed their state transfer, which would allow the target to exit
this state promptly.  It does require that clients in an environment
requiring high-availability support callbacks, but this does not sound
onerous.

On 9/10/2002 13:25:51 MDT, Robert Thurlow wrote:
 >>Updates can be "pulled" as well as "pushed".
 >
 > The current proposal deals only with making information flow
 > from the master copy out to the replicas, and this flow is
 > unidirectional (modulo rsync-like examination of files).  The
 > way a transfer is initiated is unspecified - earlier thoughts
 > of also defining an third-party admin protocol to control this
 > were abandoned since we have things like SNMP and WBEM/CIM.
 > That means that the impetus to perform an update can come from
 > a human being, from the destination server, or from a "cron"
 > job, and happen out-of-band to cause the source to contact
 > the destination server(s) via this protocol.  We do need to
 > also define the SNMP MIB or WBEM/CIM schema to do this.

Okay, clearly at the lowest level all updates have to be "pushed" from
where writes occur to other sites.  The rest is "control".

So if this out-of-band mechanism is where all this control is located,
then this repl-mig protocol just ships the data when directed.  I
suppose with a sufficiently sophisticated control protocol to determine
what and where to send data, one could implement any kind of
replication.  For example, such a repl-mig protocol doesn't need support
for describing updates, such as incremental dumps.  Instead, the
contents of an update is negotiated out-of-band and the protocol just
ships the selected data.

Replication specification and maintenance has several components:
    * Where are the replicas positioned - location - list of sites.
    * When are updates received at the replicas - consistency -
      guarantees made about data to readers regarding currency and
      writers regarding safety.
    * Which objects does the replica house - subset - lots of choices are
      possible here, including "recently used" and "all".
    * What name does the replica have - read-only - writable secondaries
      usually have the same name as the primary so they are interposed,
      while read-only secondaries have different names from the writable
      primary.
These are all control-like functions which I guess would be out of
scope.  While I can see how SNMP could be used to manage the list of
replica sites, it seems less suitable for maintaining the consistency of
a replica.

If this protocol is targeted at shipping the filesystem data, then what
distinguishes its goals from those of the client-server protocol?  My
tentative answer is bulk transfers are optimized relative to requests
for information about individual files[2].  A protocol limited to file
transfer is likely to be significantly simpler than the client-server
protocol burdened with operational semantics (e.g. opens, reads, etc).
But is this the whole list of benefits?  Is there any data to support
the assumption that repl-mig can perform better than "big honking
compounds ops"[3]?

Ted Anderson

[1] http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2001/0463.html
[2] http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2001/0476.html
[3] http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2002/0246.html

---

From: Ted Anderson <[email-masked]>
Date: Thu, 03 Oct 2002 08:19:33 -0400

To simplify following the discussion, I'll collect and rearrange my
responses on all three Replication/Migration messages under this topic.

On 9/10/2002 13:25:51 MDT, Robert Thurlow wrote:
 > This appears to be related to work discussed by Juan Gomez in
 > http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2002/0256.html
 > and the following thread.  I'm skeptical - is there really that
 > much duplication in segments to justify this work?  Juan was
 > going to do some analysis of how this would work with real-world
 > data.

Yes, this is the same material.  The benefits gained by using hashes to
recognize redundant segments depend a lot the data in the file system.
Both the LBFS and Venti hash all the segments in the whole file system,
not just those contained in a single filesystem transfer/update.  I
think the utility of such a system improves at its scale increases.  I
don't have any specific numbers on how efficacious this approach is in
"typical" usage.  However, it seems that supporting this capability
could fall out of a suitably general approach to compression plug-ins.

On 9/10/2002 13:27:38 MDT, Robert Thurlow wrote:
 > If you're doing a replica update, you need a way to note that
 > a chunk of the file has been deallocated without sending over
 > zero-filled blocks.

I see, good point.  So, like sending creates, removes and renames,
sending IS_HOLE helps transmit updates efficiently.

 > [Filesystem version numbers have] been discussed on the alias before,
 > and is not out of scope, but it isn't reflected here since it seems
 > more of an application-level thing.  I'd expect that when versions are
 > really needed, there's an application which can put that info into a
 > file in the root of the filesystem so it knows the state of the data.
 > This doesn't support "I want version 2.23".  How would the NFS client
 > (as opposed to a particular application) make use of this?

A filesystem version number helps the NFS client provide better
semantics in several ways.  The basic idea is that each replica has a
single number that identifies which version of the filesystem it is
providing and clients track this number.  Whenever the number changes
the client knows that the replica has been updated and can immediately
mark all its cached content from that replica for revalidation.  This
insures that an application never sees data that goes backward in time,
in the sense that it might reference one file from replica N and
subsequently reference another file from replica N-1.  Similarly, when a
client fails over from one replica to another it can use the advertised
replica version number to avoid sites that are less current than the one
it has been using.

On 9/10/2002 13:27:38 MDT, Robert Thurlow wrote:
 >> > OPEN_SESSION_NEW - create new transfer session
 >> > - Security token
 >> > - Session ID
 >> > - Source FS path
 >> > - Destination FS path
 >>
 >>How does the source know the destination path?
 >
 > This would come in through the SNMP or WBEM/CIM operations
 > controlling the transfer.

Maybe a session should include a description of the data being sent.
The form of this description is probably not specified by this protocol,
but depends on the out-of-band control protocol (SNMP or whatever).  One
approach is for the Session ID itself to provide a link with a
conversation in the related control protocol that established what data
should be shipped.  For this to work the selection of the Session ID
shouldn't be encumbered with other constraints.  In any event, including
a some provision annotations or inter-protocol linkage would probably be
prudent.

On 9/10/2002 13:25:51 MDT, Robert Thurlow wrote:
 >>In section 1.1.1 I'd quibble a bit.  Replication is basically used for
 >>two reasons: making a copy "here" for locality/performance and making
 >>a copy "there" for redundancy/availability.
 >
 > True - using multiple copies nearby to increase performance
 > could be interesting, though this might be more relevant to
 > services other than NFS due to cache consistency issues.

Many NFS clients already support multiple replicas and even use some
kind of ping metric to help select a close one, so performance and
locality are already factors.  As long as the replicas are read-only and
(the source is) slowly changing the consistency issues shouldn't be a
big deal.  Another way to view the here/there dichotomy is that "here"
is a more reader-centric perspective while "there" is writer-centric.

On 9/10/2002 13:25:51 MDT, Robert Thurlow wrote:
 > ... But on the other side, if you want to implement filesystem
 > consistency, where a complete database transaction is propagated
 > intact, the current filesystem-wide update is better.  I'm not sure
 > which compels more.  I'd like to see NFS be able to use proxy caches,
 > but I wonder if the NFSv4 core protocol isn't a better place to work
 > on it?

There is something to be said for AFS's (manual) release consistency
model and I think there will always be a place for it.  Any kind of
time-based consistency which is released automatically is bound to cause
trouble once in a while if the replica is propagated in the midst of
some sequence of updates.  Still it is an easy level of consistency to
provide and has value in some circumstances.

On 9/10/2002 13:25:51 MDT, Robert Thurlow wrote:
 >>In section 1.1.2, is it useful to explicitly describe migration as a
 >>form of replication where the "primary" designation moves from one
 >>site to another?
 >
 > The working group has considered having multiple replicas with
 > one designated the master, to which writes would be directed.
 > I don't think this was ever declared to be out of scope, but
 > the cache consistency issues in NFS client implementations and
 > the current lack of a standard way to tell the client about it
 > cast long shadows over it.  How important does this seem to you?

What I meant by my comment here is that replication and migration should
work smoothly together.  For example, imagine that I have a primary at A
and a secondary at B and I wish to move the filesystem from A to B.
Then I would expect the final result to be a primary at B and a
secondary left intact at B.  Clearly, this involves no extra data
movement but just entails some bookkeeping.  This bookkeeping is what I
was referring to as moving the "primary-ness", though it might be more a
job for the control protocol.

On 9/10/2002 13:25:51 MDT, Robert Thurlow wrote:
 > I think repartitioning [replication subsets] is a disruptive one-time
 > event that is just as painful as it is today, but the hope is that
 > once the filesystems are "right-sized", this doesn't need to happen
 > again.  This may not hold true over the life of the data, I can see
 > now that you mention it.

One of the benefits of the AFS/DFS approach which provides uniform names
and fileset mobility is that pathnames become very long lived.  One can
really count on file names surviving all manner of managerial and
technological disruptions.  A consequence of this is that filesets do
not stay right-sized over the long run, and both AFS and DFS suffer
because they have no easy way to rearrange volume/fileset boundaries.
To the extent that NFSv4, with its improving support for replication and
migration, is moving in the direction of long-lived namespaces, this
problem is likely to arise here as well.

 > I don't really see a way for this not to be disruptive, unfortunately.
 > Ideas?

My suggestion is to move away from persistent file handles and the use
of file ids (i.e. inode numbers) as well as functions that depend upon
them.  Pathnames become the preeminent file identifiers.  This means
doing away with hardlinks, in favor of symlinks, and deprecating
anonymous (zero link-count / ZLC) files.  Windows systems do fine
without these features (though I've never understood why symlinks got
left out).

Rename remains problematic for open files and persistent caches.  The
Windows approach of outlawing all renames that affect open files seems
unreasonable.  One idea might be to allow a client to register to
receive renames affecting certain directories: either an explicit list
or perhaps for an entire filesystem.  Directory renames are uncommon so
this wouldn't represent a large amount of traffic, but would allow
clients to track changes affecting their volatile file handles.

Ted Anderson

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 4 Oct 2002 16:59:35 -0600 (MDT)

Ted wrote:

> subsequently reference another file from replica N-1.

I see the value, but this is pretty different from current
NFS consistency semantics, so I wonder how much interest
there will be in this direction.

> Maybe a session should include a description of the data being sent.

It could; what kind of metadata do you have in mind here?
I had a hard time thinking of an example to nail down what
you wrote.

> movement but just entails some bookkeeping.

Ah, I understand.  Yes, this should be easy to do when required,
and seems to all be in the scope of that control protocol and
the namespace the clients use to find the master location.  We
probably need to have another discussion about what we might do
to standardize the namespace.

> problem is likely to arise here as well.

Interesting; can one not move a volume/fileset and get the
data right-sized again?  Or are you thinking of a large
directory which has only a couple of files that are now
interesting at a later point in the life cycle?

Rob T

---

From: Ted Anderson <[email-masked]>
Date: Tue, 08 Oct 2002 08:54:15 -0400

On 10/4/2002 18:59, Robert Thurlow wrote:
 > It could; what kind of metadata do you have in mind here?  I had a
 > hard time thinking of an example to nail down what you wrote.

The sort of information I had in mind would be something like this:
   - incremental dump since time X
   - directory information only
   - subtree rooted at x/y/z
   - transaction id X
Without a design for the control protocol it is hard to know what will
be needed to coordinate information sent from one server to another.
But one could imagine multiple operations occurring concurrently or the
data getting staged somewhere, such that the "Destination FS path" is
insufficient to sort it out conveniently.  This information might also
be useful in an advisory capacity (i.e. auditing or debugging).

At the protocol level I think some kind of opaque field would be
sufficient, since the control protocol running on each peer would
presumably be able to agree on its interpretation.  As I mentioned
earlier, it might also be possible for the session ID to be used to help
out with this.  However, that depends upon the session ID values being
unencumbered with semantics at the repl-mig protocol level.  In other
words, the repl-mig spec would have to state that the session ID is
arbitrary and can for example be an input from the control protocol for
its own purposes.  Adding an opaque metadata field for use by the
control protocol might be simpler, though.

 >>One of the benefits of the AFS/DFS approach which provides uniform
 >>names and fileset mobility is that pathnames become very long lived.
 >>One can really count on file names surviving all manner of managerial
 >>and technological disruptions.  A consequence of this is that filesets
 >>do not stay right-sized over the long run, and both AFS and DFS suffer
 >>To the extent that NFSv4, with its improving support for replication
 >>and migration, is moving in the direction of long-lived namespaces,
 >>this problem is likely to arise here as well.
 >
 > Interesting; can one not move a volume/fileset and get the data
 > right-sized again?  Or are you thinking of a large directory which has
 > only a couple of files that are now interesting at a later point in
 > the life cycle?

The context of this discussion was the means of specifying replication
subsets, namely units of replication (or migration) smaller than a whole
filesystem.  Section 6.3 of the repl-mig draft[1] suggests using the
fsid for this.  In the current NFSv4 specification, however, the fsid is
an immutable property of a file.  I was concerned that using fsid to
label subsets was impractical, at least in part because of the need to
occasionally repartition (or rebalance of right-size) the subsets.  This
is where the discussion turned to the open_fs / close_fs idea.

Ted Anderson

[1] 
http://www.ietf.org/internet-drafts/draft-thurlow-nfsv4-repl-mig-design-00.txt

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 10 Oct 2002 11:54:32 -0600 (MDT)

> migration.

We could imagine more cases, as well.  This group has already
discussed read-write replication, where a set of replicas
exist for read-only use, but where a single server is named
as the location where changes should be sent.  This seems to
me to be far simpler than replicas which can accept changes.
And regarding caches, we could also talk about a cache which
is more akin to a conventional HTTP proxy cache, sparsely
populated and coordinating with the master as needed.

> delegation) out of scope for the repl-mig protocol?

I believe so.  I'm aiming for something we don't have now, a
bulk-data protocol which can be used to send entire filesystems
and known updates to replicas.  The kind of interactive traffic
I would imagine in maintaining a cache would seem more suited
to an extension of the NFSv4 protocol itself.  We could attempt
to define how a server could behave when handed a filehandle
which it recognizes not as one it issued, but which belongs to
the master server, and we could specify behaviours such as
write-around or write-through.  The problem space is interesting,
but using the protocol I'm tinkering with to solve these problems
seems unwieldy.

> both ways on the WG mailing list.  Has a consensus developed?

I'm not sure I'm following you here.  The client currently
gets NFS4ERR_MOVED after a migration, and may consult the
old server only to get new location information.  It is then
up to the client to react to that change without further
interaction with the old server.  Changes to the information
available from the old server have been discussed, but only
to be able to let the client know more of what to expect
when it does contact the new server (e.g. are filehandles
unchanged?)  The client is still responsible for reacting
to the migration.  If I'm not remembering a proposal outside
of these limits, would you remind me of it?

> problematic scenarios so they can be evaluated based on the application

Agreed that uptime guarantees are an issue, and that we should
collect scenarios.

> tells clients to start transferring their state to the target server.

Yes, that could be useful to explore, and could work wherever
we can use delegation.

> where writes occur to other sites.  The rest is "control".

Yes, this is exactly how I split the work.

> replica sites, it seems less suitable for maintaining the consistency of

Some of this could be part of the namespace, which we may want
to have some parallel discussions about.  Certainly locations
of replicas are properly revealed there.  With some of the other
bits, I wonder how widely they need to be visible; a WBEM/CIM
abstraction of a replicated filesystem could store how updates
should be pushed and would provide visibility to the admin who
cares and to the mechanism that propagates updates, but not to
users, the client, or replica servers.  That doesn't seem like
a problem to me, as the elements that care will know.

> the assumption that repl-mig can perform better than "big honking

That's the biggest motivation.  I think some others exist,
such as, read-only attributes in NFSv4 couldn't be "pushed",
and lock and share state can't be transferred with the
current NFSv4 protocol.  I also hope that this can be more
simple and perhaps more useful in other situations than
extensions to NFSv4 would be.

Rob T