Subject: [nfsv4] [FedFS] Junctions in file system snapshots

From: <[email-masked]>
Date: Mon, 21 Dec 2009 15:38:35 -0500

Hi,

Somebody just asked me an interesting question...

What is the expectation for the client experience when they encounter a
junction in a snapshot of a fileset snapshot?  That is, the client is
traversing the point in time view of what the fileset looked like at
some point in the past.   That fileset happened to contain a junction
when it was created.  The user decides to traverse through that junction
- what do we expect will happen?

I can think of three possible options:

a) Nothing - the junction appears and behaves as an empty directory.

b) They end up in the current, production, copy of the target fileset -
assumes the target fileset still exists.

c) They end up in a snapshot of the target fileset that represents the
same point in time as the one they've just come from - assumption that
such a snapshot exists. 

Both (a) and (b) would easy to implement, but potentially confusing for
users (in different ways).   The third (c) seems like a nightmare to
implement and may only work a small portion of the time.

Help?

Thanks,

Chris

-- 
Chris Stacey - Senior Technologist, Storage Architecture and Virtual
Storage Group, Unified Storage Division, EMC Corporation (New Zealand).
Phone: +64 21 225 3747

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 21 Dec 2009 14:58:11 -0600

> Somebody just asked me an interesting question...

Indeed.

> implement and may only work a small portion of the time.

Let's add a few more options:

d) An option to specify which junctions' filesets should be snapshot
when a filesystem is snapshot.  

e) Junctions in snapshots are unchanged by the snapshot operation, but
can be manually updated to point to snapshots of their filesets.  This

f) Add support for recording the FSNs of snapshot fileset names in FSNs
for live filesystems.  That way when a junction in a snapshot is resoved
the server can try to resolve it to equally named snapshots of the
target filesets.

IMO (c) and (d) cannot be adopted without having a standardized protocol
for taking and accessing snapshots, as well as a minimum of standardized
semantics for snapshots.  That seems like a very difficult path to take.
And even then, to require that snapshots be taken recursively -- that
is, recursively following all junctions and snapshotting their filesets
-- strikes me as a bad idea.  At least (c) is out, and (d) should be
too.

(e) is inappropriate because snapshots should be read-only.  For the
same reason (a) seems inappropriate.

(f) requires that we have a standard method for naming snapshots, which
does seem tractable.

Only (a) and (b) can be done without any further ado.

My preference: (f), then (b).

Nico
--

---

From: Philip Love <[email-masked]>
Date: Mon, 21 Dec 2009 13:38:40 -0800

> (f) requires that we have a standard method for naming snapshots, which

What should we present to the client if we cannot resolve this snapshot junction in the db?

Phil

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 21 Dec 2009 15:44:07 -0600

> What should we present to the client if we cannot resolve this snapshot junction in the db?

IMO:

 - The snapshot name does not appear in the NSDB? -> the current FSLs.

 - The snapshot name does appear but it's FSLs cannot be resolved? -> error
   (which error?).

(b) is the best and simplest option in that we don't even have to
acknowledge snapshots in the protocol standards.

I'm not sure that (f) is more desirable from a user perspective than
(b).  But if we can agree on snapshot naming then we could certainly
provide (f).

Nico
--

---

From: <[email-masked]>
Date: Mon, 21 Dec 2009 17:08:02 -0500

Why do we have to agree on snapshot naming for (f) to work?   Surely,
the naming convention is likely to be vendor/implementation specific.  I
would have thought that it would be up to the thing managing the
creation of the snapshots to insert/update the entry in the NSDB rather
than requiring a standardized snapshot naming scheme.   

Thanks,
Chris 

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Tuesday, 22 December 2009 10:44 a.m.
To: Love, Phil
Cc: [email-masked]; Stacey, Chris
Subject: Re: [nfsv4] [FedFS] Junctions in file system snapshots

IMO:

 - The snapshot name does appear but it's FSLs cannot be resolved? ->
error
   (which error?).

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 21 Dec 2009 17:33:48 -0500

Option (b) violates the least-surprise principle.  It lets the user (the
NFSv4 user) think that they're traversing the same filesystem, but they
are not.  Suddenly, instead of traversing a snapshot, they are in a live
filesystem.

Unless we want to bite off the larger problem of providing synchronous
snapshots everywhere in a vendor-independent fashion, I recommend
solution (a): let snapshots of junctions lead to nothing, returning an
error rather than traversing into something surprising.

		Craig

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Monday, December 21, 2009 4:44 PM
To: Philip Love
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] [FedFS] Junctions in file system snapshots

IMO:

Nico
-- 
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 21 Dec 2009 16:19:26 -0600

> than requiring a standardized snapshot naming scheme.   

Well, we'd have to agree that snapshots have names and, less obviously,
what form those names take (opaque? UTF-8 string? and if so, with what
stringprep?).  The rest can all be implementation-specific.  Snapshot
names would appear only in FSN objects, or better, below them.

So I suppose (f) is not hard...  (f) it is for me then.

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 21 Dec 2009 17:46:09 -0500

I'm not sure I understand what's under discussion.  Quoting Nico's
earlier message:
-------
f) Add support for recording the FSNs of snapshot fileset names in FSNs
for live filesystems.  That way when a junction in a snapshot is resoved
the server can try to resolve it to equally named snapshots of the
target filesets.
-------

So what's the information flow?  When I create a snapshot of a fileset
("X"), am I doing something with respect to the NSDB, or the data in the
junctions of the parent fileset, or the junctions in fileset X, or the
children of fileset X, or the data in the junctions in snapshots of the
parent fileset?  If I create a new FSN in the NSDB for the snapshot of
fileset X, what is going to point to it?

I'm really lost.

		Craig

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Monday, December 21, 2009 5:19 PM
To: [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] [FedFS] Junctions in file system snapshots

Nico

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 21 Dec 2009 16:41:29 -0600

> error rather than traversing into something surprising.

I don't see how (a) doesn't also violate that principle: (a) loses
information when you take a snapshot.  Surely that makes no sense.
Which surprise is less surprising?

Nor can I see why users should expect that snapshots are atomically
recursive.  Oh, they might, but we really can't implement that.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 21 Dec 2009 16:54:34 -0600

> fileset X, what is going to point to it?

In (f) you'd do this:

 - snapshot some filesystem/fileset FSa.

 - later you decide that you want to have some junction in FSa that
   referse to FSb to now refer to a snapshot of FSb with the same
   snapshot name as in FSa snapshot that you just took, so you'd:

    - snapshot the target fileset with the same snapshot name as in the
      fileset that has the junctions

    - update the fedfsFsn object for the target of that junction to list
      the new snapshot and the FSLs of that snapshot.

Alternatively you'd:

 - first snapshot the targets of junctions in FSa; use the desired
   snapshot name;
 - update the FSN objects for those targets as above;
 - snapshot FSa and ensure that its junctions refer to snapshots of the
   original targets, all with the same desired snapshot name.

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Mon, 21 Dec 2009 18:21:02 -0500

(a) is surprising in that the user sees that the junction no longer
works, but it's not surprising in that it doesn't pretend to be what it
cannot be.  No information is lost when taking the snapshot; the
junction is still there, presumably exactly the same bits as were in the
live file system.

In fact, (a) is essentially impossible to mandate.  The concept of
"snapshot" (a NetApp TM, AFAIK) has no meaning to NFS; referrals is
referrals.  What we're discussing is how a file server might represent a
junction if it is in a snapshot.  The easiest thing is (b), that a file
server represent a snapshot ("image"?, for non-marked names) of a
referral as something other than a referral.  AFS (IBM mark?) would
represent such a referral as a symbolic link that pointed to the name of
a fileset; perhaps in this case symlink text might render the FSN.  So
the information would be there, but wouldn't "work" transparently.

		Craig

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]] 
Sent: Monday, December 21, 2009 5:41 PM
To: Everhart, Craig
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] [FedFS] Junctions in file system snapshots

Nico

---

From: James Lentini <[email-masked]>
Date: Mon, 21 Dec 2009 18:24:02 -0500 (EST)

On Mon, 21 Dec 2009, [email-masked] wrote:

> What is the expectation for the client experience when they 
> encounter a junction in a snapshot of a fileset snapshot?

The behavior is dependent on the snapshot implementation. The sensible 
thing would be to match the snapshot implementation's symlink 
behavior.

With symlinks, the common behavior is to keep them intact. For example 
if a symlink called "/u/jlentini/foo" points to 
"/usr/bin/some_program" and a snapshot of /u/jlentini is created, the 
snapshot's copy of the foo symlink will point to /usr/bin/some_program 
for the life of the snapshot, regardless to changes in the active 
filesystem.

I think implementations would be insane to do anything different with 
a FedFS junction, but that is a decision implementers will have to 
make.

-james

---

From: Daniel Ellard <[email-masked]>
Date: Mon, 21 Dec 2009 18:28:31 -0500

The "sane" use of f) may be easy, but the corner cases
are gruesome.

Imagine that we have two filesets, T1 and T2, that both have
junctions referring to fileset C.   T1 and T2 are managed separately,
by different entities, who might not even be aware of each other
(and/or belong to different security domains, making them invisible
to each other).

Now imagine that T1 and T2 both take snapshots of themselves,
but at different times.  Included in those snapshots are junctions
referring to snapshots of C at different moments.

Now we have a very confusing state of affairs.  In the "live" copy,
the target of the junctions in T1 and T2 are the same fileset, but
in the snapshotted copies of T1 and T2, they are not.  Since the
defining characteristic of a snapshot is that it is a self-consistent
view of the state of the namespace, the thing we've ended up with
isn't really a snapshot at all.

I don't think there's a way to shoehorn "global" snapshot semantics
on top of NSDB without enormous amounts of support from the
underlying file systems, and this support does not exist in any
portable mechanisms right now.  Even if it did, I suspect that it
would violate some of the notions of federation.

-Dan

---

From: <[email-masked]>
Date: Mon, 21 Dec 2009 18:37:28 -0500

James,

> for the life of the snapshot, regardless to changes in the active

The problem is that the symlink doesn't point to a *snapshot* of
/usr/bin/some_program.  If that some_program binary is replaced after
the
snapshot is taken, following the symlink from the snapshot may lead to
something inconsistent with the source of the symlink (e.g., if a script
is run from /u/jlentini).

That said, I think this yields an useful framing of the question - wrt
snapshots, should FedFS junctions behave like symlinks, hard links or
some approximation of one or the other?

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of James Lentini
> Sent: Monday, December 21, 2009 6:24 PM
> To: Stacey, Chris
> Cc: [email-masked]

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 21 Dec 2009 17:27:32 -0600

> junction is still there, presumably exactly the same bits as were in the

Information is lost if the client can't see it.  The only way a client
can see a junction is by stepping into it and getting a referral.  (Or
by being an ADMIN client instead of/in addition to an NFSv4 client.)

> the information would be there, but wouldn't "work" transparently.

I'd not want to standardize such behavior.

Consider a referral+ that says "this referral may not match what was
intended at some point in the past when this read-only fileset was
created".  What would a client do with that?

Why are snapshots any more special than read-only filesets?  Should
read-only filesets not be allowed to have junctions leading to
modifiable filesets?  That question convinces me.  We must use (b).  (f)
would be nice too, for bonus points.

Nico
--

---

From: James Lentini <[email-masked]>
Date: Mon, 21 Dec 2009 18:59:10 -0500 (EST)

On Mon, 21 Dec 2009, [email-masked] wrote:

> /usr/bin/some_program.  If that some_program binary is replaced 
> after the snapshot is taken, following the symlink from the snapshot 
> may lead to something inconsistent with the source of the symlink 
> (e.g., if a script is run from /u/jlentini).

Yes, this is the behavior I was describing. I would not characterize 
this as a "problem", but that is a different topic.

> some approximation of one or the other?

I would strongly recommend following the symlink behavior, but, as 
with other aspects of snapshots, implementations may choose a 
different behavior.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 21 Dec 2009 18:34:40 -0600

> with other aspects of snapshots, implementations may choose a 

+1.

What is a snapshot?  It's a read-only fileset.

Do we need/want special handling of junctions when a fileset is made or
created read-only (possibly by copying another fileset)?  No.

Why would we want special handling of junctions in filesets when they
are snapshot?  I think we don't.  At most we should want (f) (add an
NSDB notion of named snapshots of filesets, referenced by FSNs).

I think (f) is potentially useful, but I'd like to hear that from
customers.  (a) is not appropriate (because snapshots should not be
destructive).  That leaves us with (b) (treat snapshots as the moral
equivalent of copying a fileset and making the copy read-only).

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Tue, 22 Dec 2009 10:57:34 -0500

inline

> Sent: Monday, December 21, 2009 7:35 PM
> To: Lentini, James
> Cc: [email-masked]; [email-masked]; [email-masked]

No, it's a point-in-time image of some other fileset.  It is *not* an
arbitrary read-only fileset.  People often use snapshots because they
are aware of the point-in-time nature of what they're looking at, and
the relationship to some other data.
 
> created read-only (possibly by copying another fileset)?  No.

That's not the question here.  The question is how to handle junctions
in snapshot filesets.

> NSDB notion of named snapshots of filesets, referenced by FSNs).

Here I would err on the side of caution.  I don't know how to provide
the functionality implicit in (f), in which I believe users will expect
that they are traversing the same point in time on both parent and child
sides of a junction, which we generally strive to make invisible to
them.  That's why I continue to recommend not doing close-but-no-cigar
substitution.
 
> equivalent of copying a fileset and making the copy read-only).

You can treat a snapshot as any other read-only fileset, but that
doesn't make it one.  I have no problem with making junctions from
general read-only filesets work, and in general I think they must work.
But that's not what I believe customers think of when they traverse
snapshots.

I'm not interested in going too many more rounds without declaring
unbridgeable semantic gaps, but I think the non-equivalence in the
customer's eyes of "snapshot" and "read-only data" is under-appreciated.

		Craig

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 22 Dec 2009 10:56:24 -0600

> the relationship to some other data.

I agree that the point-in-time aspect of snapshots is essential, not
incidental to snapshot.  That a snapshot is a read-only copy of a
fileset seems even more essential however.  And I don't agree that
recursion is essential, however.  For example, in ZFS the snapshot
operation is not recursive by default.

Are there servers where snapshot operations are always, or by
default, recursive?

I suspect that user expectations already are that snapshots are not
recursive by default.

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Tue, 22 Dec 2009 15:18:17 -0500

Nico,

I agree that snapshots are not "recursive by default", and that making
them so would be incredibly difficult.  I disagree, though, that its
being a "read-only copy" of a fileset is more important than its
relationship with the original.  It is a read-only copy with a
particular property; I'd say that its being read-only is incidental.  If
people could easily use it to revert to a given point-in-time, so that
it was writable, I bet some customers would be takers.

Given that I see the point-in-time nature as more important to users, I
see our inability to provide an accurate recursive, inter-vendor view of
a point-in-time for a collection of filesets as a problem.

There may be other uses for point-in-time technologies as a mechanism
for choosing which version of a fileset to publish.  I'm not talking
about those uses--what AFS used to call ".readonly" versions of a
fileset, as opposed to its ".backup" versions.  I'm talking about users
choosing which point-in-time image of a directory tree to look at,
usually to find a historical view from a particular time.

		Craig

> Sent: Tuesday, December 22, 2009 11:56 AM
> To: Everhart, Craig
> Cc: Lentini, James; [email-masked]; [email-masked];
> [email-masked]

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 22 Dec 2009 14:26:43 -0600

> it was writable, I bet some customers would be takers.

Reverting a fileset to a snapshot is akin to replacing the fileset with
a read-write copy of the snapshot.  The snapshot must, obviously,
preserve all sorts of internal detail that relates to file handles.

But now I'm picking nits.  It suffices that we agree that the snapshot
operation is not recursive.

> a point-in-time for a collection of filesets as a problem.

The key in your statement is "recursive".  I don't think there's any
need to make recursive snapshot taking a requirement for anyone, much
less to count junctions in (as opposed to local "child" filesets, as in
ZFS).

Nico
--