Subject: [nfsv4] Fed-fs: Root fileset discussion continued

From: "Renu Tewari" <[email-masked]>
Date: Thu, 6 Nov 2008 10:25:24 -0800

Two topics for discussion on the root fileset.

1. Revisit the single master root NSDB and other slave root NSDBs in light
of  LDAP replication.

I verified that OpenLDAP supports master-slave replication. Mario raised an
important concern that Active Directory and iPlanet do not support
mater-slave replication instead every LDAP server is a master and the last
update wins.
Here are some documents on OpenLDAP replication (syncrepl) that describes
the master-slave and multi-master modes.
http://www.openldap.org/doc/admin24/replication.html
http://www.openldap.org/doc/admin22/syncrepl.html

2. Define the root fileset schema.
There are two options here:

a.  Each (virtual) directory node is represented as  a record in LDAP
For example,
<fsnUUID=ROOT_FSN
 dirName= b,
 parent = fed-fs,
 last_update_time = update_seq#,
 type= junction,
 Target_FSN= fsn_uuid1>
This approach requires the NFSv4 server query each level of the root
fileset hierarchy on a client request.

b. The root fileset is represented as a record of all the paths from the
root to each leaf
<Root_filest
<fsnUUID=ROOT_FSN
 Target_FSN= fsn_uuid1,
 path = /fed-fs/b,
<fsnUUID=ROOT_FSN
 Target_FSN= fsn_uuid2,
 path = /fed-fs/a/d,
<fsnUUID=ROOT_FSN
 Target_FSN= fsn_uuid3,
 path = /fed-fs/a/e,
<fsnUUID=ROOT_FSN
 Target_FSN= fsn_uuid4,
 path = /fed-fs/c/f,
<fsnUUID=ROOT_FSN
 Target_FSN= fsn_uuid4,
 path = /fed-fs/c/g,
last_update_time = update_seq#

The NFSv4 server can download the entire root fileset representation. Any
changes made to the schema can be pulled periodically.
Any updates (add/delete) to the internal nodes will require changing all
the paths. Useful for small scale setups.

=================================================================
Definitions

1.  Root Fileset: A fileset that implements the "top" of the namespace is
called a root fileset.  The root fileset consists entirely of directory
nodes and junctions -- no other type of file system object is permitted.
The leaf nodes of the root fileset are junctions or empty directories.

2.  Root FSN: A root fileset is uniquely identified by an FSN called the
root FSN. The FSLs of instances of the root fileset are called root FSLs.
Each root FSL corresponds to a root fileserver exporting the root fileset.

3.  Root fileservers: Fileservers that host instances of the root fileset
are called root fileservers.  The NFS client can locate the root fileserver
using an external protocol such as DNS (using SRV records).

4. Root NSDBs: A set of NSDBs that store the representation of the root
fileset(s) and the root FSN and FSL information.
    A root NSDB should respond to following queries:
    i)  search for the root FSN of the federation  [Discuss:]
    ii) search for the root FSL's given a root FSN
    iii)  search for the layout of the root fileset tree.  [Discuss]

5.  Master Root NSDB: An NSDB that stores the authoritative representation
of the root fileset in a federation.  All updates to the root fileset are
made at the master. All other root NSDBs replicate the root fileset
information from the master NSDB. The replication protocol across root
NSDBs is external to fed-fs. For example, we can rely on the LDAP server's
replication protocol.
[Discuss] Can it work cross vendor.

6. Root Fileset Schema:  A schema the defines the content/structure/layout
of the root fileset tree. This schema is used to store the records for the
root fileset representation at the root NSDBs.

Root fileserver and Root NSDB Interaction

PULL mode: Each root fileset server queries a root NSDB for information
about the contents of the root fileset. The fileserver can rely on a TTL
value to reduce the frequency of requests. It can also use a query to find
out if anything has changed since the last request using a last modified
time equivalent.  If contents of the root fileset have changed it should be
possible to request only the new changes (deltas) instead of the entire
data.
[Discuss] details of the deltas and last modified times.
[Discuss] How does the root fileserver find which root NSDB to query? Does
the admin when creating the  root FSL create a junction for the root that
stores the root FSN information that includes the root NSDB?

Root Fileset Schema
[Discuss] Are there any constraints on the layout of the root fileset?
Is the root fileset one-level, like AFS. Or can the root fileset have
multiple levels?

[Discuss] Do we want to be able to limit updates to parts of the root
fileset by ACL or some other mechanism, so that updates to the
part of the namespace related to one federation member can be
controlled by that federation member?

[Discuss]
Consider an example (in the picture which may be garbled) where the root
fileset consists of the following paths from the root to each leaf
/fed-fs/a/d
/fed-fs/a/e
/fed-fs/b
/fed-fs/c/f
/fed/-fs/c/g

The root fileset consists of records for each node in tree.
Each record in the above example is  represented as the tuple:

<fsnUUID=ROOT_FSN
 dirName= fed-fs,
 parent = /,
 type= internal>

<fsnUUID=ROOT_FSN
 dirName= a,
 parent = fed-fs,
 type= internal>

<fsnUUID=ROOT_FSN
 dirName= d,
 parent = a,
 type= junction>

Root Fileserver State
[Discuss:] How does the root fileserver represent the root fileset
structure for the NFS clients.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 06 Nov 2008 16:13:51 -0700

> using an external protocol such as DNS (using SRV records).

Should this be "a root fileserver", to acknowledge that there
could be several of them?

>     iii)  search for the layout of the root fileset tree.  [Discuss]

I don't think I know what iii) means.

> of the root fileset in a federation.

Last week, weren't we closing on having a single (logical)
NSDB for a domain, which could be replicated?  In that case,
the NSDB should always contain root fileset records, and we
don't need this definition/distinction.

> Is the root fileset one-level, like AFS. Or can the root fileset have

I have been expecting multiple levels, with the norm being
a small number.  I don't know that I would be concerned
with cutting back to a single level.

On a somewhat related note, I wondered if we should define
other filesets that are strictly virtual - i.e. their
authoritative contents could be in the NSDB instead of
on spinning rust.  The example that made me think of it
was the "home" fileset.  (We could indeed be getting too
automounter-ish here, of course.)  It might be good to
think of the motivation for creating separate root
filesets, and see if they a) hold water and b) apply
to other collections of junctions we can imagine.

Rob T

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 6 Nov 2008 21:30:56 -0500

> From: Robert Thurlow [mailto:[email-masked]] 
> Renu Tewari wrote:
> 
> ...
> 
> > 5.  Master Root NSDB: An NSDB that stores the authoritative 
> > representation of the root fileset in a federation.
> 

I wildly disagree with the one-NSDB-per-domain story.  To me, the
fileserver-NSDB association comes because they both come from the same
vendor (more or less), and I want to allow a single organization to
build a single (logical) site out of stuff from multiple vendors (and
multiple NSDBs).  Even if the top-level fileset is replicated somehow
(whether through LDAP or something else), I believe that's done so that
there's interchangeability of vendors in the organization's view.

> I have been expecting multiple levels, with the norm being a 
> small number.  I don't know that I would be concerned with 
> cutting back to a single level.

I agree.

> authoritative contents could be in the NSDB instead of on 
> spinning rust.  The example that made me think of it was the 
> "home" fileset.  (We could indeed be getting too 
> automounter-ish here, of course.)  It might be good to think 
> of the motivation for creating separate root filesets, and 
> see if they a) hold water and b) apply to other collections 
> of junctions we can imagine.
> 
> Rob T

My guess is that you're looking for something else as "home", not the
same kind of mechanism as has been posited for a root fileset.  I don't
have a deep understanding of what it is you're looking for, clearly.
I've seen other schemes used for something like "home" semantics, ones
that don't rely on automounter-like functionality.  But I don't want to
suggest something that's not even close to what you're thinking about.

So that's to say: what are the requirements?

		Craig

---

From: Rob Thurlow <[email-masked]>
Date: Wed, 12 Nov 2008 12:25:01 -0700

>> NSDB for a domain, which could be replicated?

> multiple NSDBs).

It sounds like you think of a set of gear coming in on a
forklift load, which is interesting.  But I don't think
the specs we have now cope with multiple non-identical
NSDBs in a single domain, so you may want to scan them
to look for other assumptions.  I personally don't think
I understand how we do things like enumerate the namespace
if we have a bunch of NSDBs per domain; we only know how
to find the root NSDB now (via DNS).

> So that's to say: what are the requirements?

I think the root fileset case is a lot stronger, but I see
a similarity with some types of directories in the namespace.
/home on a Solaris system looks like that, too - it is a
nest of referrals to other places, without any regular files,
and there could be value in the whole thing being hosted in
the NSDB rather than being on spinning disks, so that it would
be easier to widely replicate.  My main point is that we might
want to consider generalizing the behaviour we're talking about
for the root fileset to other use cases.

Rob T

---

From: "Everhart, Craig" <[email-masked]>
Date: Wed, 12 Nov 2008 14:35:46 -0500

> From: Rob Thurlow [mailto:[email-masked]] 
> Sent: Wednesday, November 12, 2008 2:25 PM
> To: Everhart, Craig
> Cc: Renu Tewari; [email-masked]
> Subject: Re: [nfsv4] Fed-fs: Root fileset discussion continued
> 
> Everhart, Craig wrote:
> forklift load, which is interesting.  But I don't think the 
> specs we have now cope with multiple non-identical NSDBs in a 
> single domain, so you may want to scan them to look for other 
> assumptions.  I personally don't think I understand how we do 
> things like enumerate the namespace if we have a bunch of 
> NSDBs per domain; we only know how to find the root NSDB now 
> (via DNS).

Sure, the root fileset thought is that whatever describes the root
fileset should be replicated.  Does that mean that all NSDBs that host
the root fileset have to be identical all the way down?

I agree that we've got a way to go from DNS for a domain to a set of
file servers and that those file servers could/should be configured to
present this root fileset that's replicated in multiple NSDBs.  I admit
that I've been assuming that the only way to enumerate the namespace is
to use the fileservers to do it, encountering junctions.  We've
discussed the meme that NSDBs could also capture something of the
fileset-to-FSN mappings too, and we've bantered around the idea that the
NSDB's memory of those mappings might be inaccurate or stale.  I've
heard the argument that the NSDB's memory should be treated as
authoritative, but I don't yet see how that works.

Even relying on that meme (NSDBs capture fileset-to-FSN mappings,
representing junctions from a given fileset to its namespace children),
it would be straightforward for such a FSN to refer to some NSDB besides
itself, right?

		Craig

---

From: Rob Thurlow <[email-masked]>
Date: Wed, 12 Nov 2008 13:40:15 -0700

> the root fileset have to be identical all the way down?

No, but that wasn't my argument.  I'm probably paging in
multiple non-identical NSDBs, and how that works.

> it would be straightforward for such a FSN to refer to some NSDB besides

Certainly - that part is fine.  I guess the missing link
I'm noticing is this: if we have multiple NSDBs, and we
think it is a good idea to be able to enumerate parts of
the namespace by talking to NSDBs (and I know we're not
done with that discussion), how do we find any NSDB other
than the root?  A related question: how can an admin or
an admin tool know what NSDB to manipulate?

Rob T

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 13 Nov 2008 08:29:44 -0500

Part of the FSN is a pointer to an NSDB.  So if NSDBs contain
fileset-to-child-FSN pairings, doesn't that achieve the transitive
closure you'd want?  You could go from the root NSDB to pointers to
other NSDBs.

		Craig 

> -----Original Message-----
> Sent: Wednesday, November 12, 2008 3:40 PM
> To: Everhart, Craig
> Cc: [email-masked]
> > representing junctions from a given fileset to its namespace 
> > children), it would be straightforward for such a FSN to 
> refer to some 
> > NSDB besides itself, right?
> 
> Certainly - that part is fine.  I guess the missing link I'm 
> noticing is this: if we have multiple NSDBs, and we think it 
> is a good idea to be able to enumerate parts of the namespace 
> by talking to NSDBs (and I know we're not done with that 
> discussion), how do we find any NSDB other than the root?  A 
> related question: how can an admin or an admin tool know what 
> NSDB to manipulate?
> 
> Rob T
>

---

From: James Lentini <[email-masked]>
Date: Thu, 13 Nov 2008 10:50:47 -0500 (EST)

On Wed, 12 Nov 2008, Rob Thurlow wrote:

> > > Last week, weren't we closing on having a single (logical) NSDB for a
> > > domain, which could be replicated?
> 

The current DNS SRV proposal allows the client to find a root 
fileserver via DNS, not the root NSDB. Under the current proposal, the 
root fileserver is configured to use the root NSDB to obtain the root 
fileset, but the NFS client is not aware of this.

With regards to the multiple NSDB's per domain, there is nothing I 
know of in the current specifications that prevent this. As you've 
mentioned, we should consider if a mechanism to discover NSDBs would 
be useful. For administrative purposes, I think such a mechanism would 
be valuable.

> > > On a somewhat related note, I wondered if we should define other filesets
> > > that are strictly virtual - i.e. their authoritative contents could be in
> > > the NSDB instead of on spinning rust.  The example that made me think of
> > > it was the "home" fileset.  (We could indeed be getting too
> > > automounter-ish here, of course.)  It might be good to think of the
> > > motivation for creating separate root filesets, and see if they a) hold
> > > water and b) apply to other collections of junctions we can imagine.
>

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 13 Nov 2008 10:08:44 -0700

> closure you'd want?  You could go from the root NSDB to pointers to

I suppose that if I enumerate all junctions in the root NSDB,
I can examine them to find other NSDBs.  That's more work
than I like, but it may be enough.

Thanks,
Rob T

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 13 Nov 2008 10:11:01 -0700

> fileset, but the NFS client is not aware of this.

Yeah, I realized I'd got this wrong awhile after I sent it.

> be useful. For administrative purposes, I think such a mechanism would 

Agreed.  So the relavent questions seem to be:

How do we get an admin tool bootstrapped into the namespace?
How does a fileserver find the root NSDB?

Rob T