Subject: Replication/Migration conference call minutes, 5/29

From: Robert Thurlow <[email-masked]>
Date: Thu, 30 May 2002 10:16:41 -0600 (MDT)

Conference call minutes, May 29 2002, as reported by Rob Thurlow

The following people met today in a conference call to discuss
NFSv4 replication and migration:

	"Dave Noveck" <[email-masked]>
	"Spencer Shepler" <[email-masked]>
	"William A.(Andy) Adamson" <[email-masked]> and the CITI team
	"Brent Callaghan" <[email-masked]>
	"Brad Cain" <[email-masked]">
	"Rob Thurlow" <[email-masked]>

Rob started by saying that he had an Internet Draft in the works
which covered design principles for repl-mig; this tries to set
the scope of the initial work.  Rob said the draft will encourage
a focus on using repl-mig with data shared by NFSv4 servers, while
trying not to design out use with other types of data.  A general
discussion of scope followed.  Points of discussion:

- security: Rob has assumed that alignment with NFSv4 security
  mechanisms would be important, but it was suggested that things
  could be simplified greatly by relying on strong host-based
  security, such as IPSec or SSH, since we're really dealing
  with a simpler case.  Rob was concerned about admins having
  to administer yet another security infrastructure, but agreed
  that using null authentication with IPSec or SSH was needed.
  We did not decide whether or not it would be sufficient to
  point to IPSec or SSH and not require a more generic GSS-API
  or RPCSEC_GSS security capability.

- admin client to server: it was asked whether it was appropriate
  for this protocol to specify admin-client-to-server messages
  when other management options existed.  Rob said that this was
  motivated by a desire for the admin to be authenticated as
  strongly as the servers in the transaction; this may indeed
  not be a compelling argument.  This should be discussed with
  someone more knowledgeable about the state of current management
  frameworks.

- NDMP: Rob admitted a motivation to include admin client to
  server traffic in the protocol was influenced by NDMP, which
  uses this arrangement.  We discussed an alternate way of
  delivering replication/migration protocol functionality: to
  design a canonical format for NDMP servers to exchange, rather
  than the opaque, vendor-proprietary blobs of data NDMP servers
  exchange today.  Then, the NDMP infrastructure could be used
  to manage repl-mig for NFSv4, reducing the new work required
  a great deal.  This led to discussion of the status of NDMP
  within the IETF and discussion of the second NDMP BOF held at
  IETF 53.  They were not able to be chartered as a working
  group because of an overly vague solve-world-hunger charter,
  but it was thought that with a tighter charter, they could
  be an IETF WG and benefit from IETF guidance on issues such
  as security.  Several people, including our WG chair Brian
  Pawlowski, were engaged in helping them with their charter.
  Several people expressed interest in discussing this with
  the NDMP folks, but Rob said we have to wait until NDMP
  becomes an IETF working group; there would also be some
  time before it would be clear what architectural changes
  the IETF would mandate.

- filesystem subsets: in offline discussions with Brent Callaghan,
  Brent expressed again a need for migrating sub-directories of
  exported filesystems, which was discussed in the previous call.
  RFC 3010 specifies that an NFS4ERR_MOVED would apply to all
  files with the same fsid, and most participants thought that
  the server implementation was free to deal with subsets of
  filesystems it wanted to manage separately by sharing them
  separately and assigning different fsids to them and different
  filehandles for their files.  It was discussed that there was
  a thorny problem of general server reconfiguration that some
  would want to solve, but that a solution was not necessarily
  practical.

- state preservation: should state such as extant record locks
  and delegation state be transferred as metadata to the new
  server in a migration?  Brent thought that this was a rathole
  to be avoided upfront; others thought that the problem should
  be investigated to some degree and specified if acheivable.
  Rob thought delegation state could be factored out by recalling
  delegations, and that record locks and share reservation info
  should not be too hard to transfer.  Consensus was to try to
  understand this better before ruling it out.

After the above discussions, we opened up discussion to new
topics:

- Brad Cain asked if there would be anything in the protocol
  to control or support filesystem snapshots.  The answer was
  that so far, snapshots were an optimization that did not
  appear to need protocol support.  If it was really needed,
  we speculated that something generic like the NFS Version 4
  "change" attribute could be included without burdening the
  protocol too much.  [Rob adds out-of-band that the latest
  NDMP adds verbs to manipulate snapshots, but thinks that
  this is probably not a good thing.]

- Brad asked about how metadata would be expressed in the
  replication/migration protocol.  Rob said that he hoped to
  define attributes as parts of "profiles" which could be
  defined to suit the data being transferred, and that the
  first profile would include the attributes defined by the
  NFS Version 4 spec.  Dave Noveck asked about what multi-
  protocol filers should do, and we agreed that it should
  be possible for both a CIFS attribute profile and an
  NFSv4 profile be sent per file when useful.  We should
  be able to make this extensible, such that future RFCs
  for e.g. WebDAV could define a metadata profile.  We also
  discussed how named attributes should be handled more as
  files with peculiar characteristics than as attributes
  in one of these profiles.

  Brad raised another interesting issue: how should file
  data and metadata be arranged on the wire?  It seemed
  that for some devices, having data and metadata sent
  over separate connections could be an advantage, and that
  at least, there were probably more situations where it
  was better to send metadata ahead of file data.

- Andy asked about whether compression of data on the wire
  had been considered.  Rob said this had not yet come up,
  and that it would be worth searching for prior art within
  the IETF about data compression.

- Some discussion about encoding of attributes led to a
  consensus that we should try to re-use as much from NFSv4
  as we could; for example, there was no reason to invent a
  new way to express directories when the XDR encoding for
  READDIR was available.  Some asked how much different the
  repl-mig protocol had to be than "one long compound op";
  we agreed that there were differences, and that in some
  cases e.g. read-only attributes, we would have to do some
  things that could not be done with NFSv4 ops.

- Finally, Rob asked for some feedback from the design team
  on the draft he's working on.  To get some discussion going
  before IETF 54 in Tokyo, he said he hoped to submit the
  "design principles" draft by early next week as a working
  group document if possible.  Rob will send the draft out
  after the meeting.

Rob T

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 31 May 2002 12:25:11 -0400

>   things that could not be done with NFSv4 ops.

If you're moving something from a source to a target, then wouldn't it
make more sense for the target to do reads and gettattrs and such, instead
of the source doing writes and setattrs?

What's wrong with the following as a migration protocol: Given a source
machine with a directory tree that we want to move to a target machine:

  1. The target machine mounts the directory tree from
     the source.
  2. The target machine then exports this same tree.  At
     this point, the target is just another client to
     the source machine, but can also serve the same
     tree to other clients.
  3. The target machine tells the source that it wants
     to take over.
  4. The source machine then starts sending
     NFS4ERR_MOVED in response to any further requests,
     except for requests coming from the target, which
     the source continues to process normally.

So now the clients of the source have all become clients of the target,
and the target is a client of the source.  The target could just do the
equivalent of a "cp -a" at this point to move the data from the source,
probably dropping requests from the clients in the meantime, or it could
choose to only make the requests to the source that it needs to make in
order to service incoming client requests.  If it was smart, it would
probably do some combination of the two.

No additional protocol is required here, except maybe a single extra RPC
at step 3.  This doesn't solve the problem of transferring state, but
no-one knows how to solve that problem anyway right now.  As a first
attempt, we've said we're content with migrations that look to clients
like server reboots.

Maybe there are obvious problems with this.  But if someone could point
out those problems, maybe that would help me understand better why we
need an entirely separate protocol to do migration, and what features
such a protocol must have if we do need one.

--Bruce Fields (@citi.umich.edu)

---

From: Robert Thurlow <[email-masked]>
Date: Fri, 31 May 2002 10:59:51 -0600 (MDT)

> of the source doing writes and setattrs?

I think this is worse because the source has much better awareness
of what is to be transferred.  If the target has to walk the file
tree to discover what needs to be pulled, that seems to introduce
unnecessary round-trips and traffic.  Also remember that we need a
way to send only changes for a replica update; we'd like to send
only parts of file data and metadata that have changed, and only
the source can know that.  I'm convinced we need a "push" protocol
for this reason.

> probably do some combination of the two.

I'd summarize this as "arrange the mappings and then let page
faults take care of moving the data."  You absolutely need to
take action to move all the data, or else you can never know
when the source can go offline?  Also, your clients could pay
quite a penalty in performance when two network hops need to
happen; what's the advantage of not having the source field
their requests until the transfer is done?

> This doesn't solve the problem of transferring state, but

Opinions vary, but state transfer is still of interest to some
as a way to get the best transparency.

> such a protocol must have if we do need one.

I hope the above helps.  Another motivation is the hope that we
can do this more efficiently than big honking compound ops; that
will obviously need to be proven.

Rob T