Subject: RE: [nfsv4] Global Namespace - LDAP Proposal for Location Resolution

From: Manuel Pereira <[email-masked]>
Date: Tue, 26 Apr 2005 16:38:02 -0700

Hello Dave,

Following are a few terse remarks:

> So a "fileset" is basically what NFSv4 calls a "filesystem".  So why
> the different term?

Right, the first footnote of the first post [1] states:
"fileset - a unit of namespace management.  A fileset is like an NFSv4
filesystem, with a single root and leaves that are either files,
directories or other filesets."

Our reasoning for using "fileset" instead of "filesystem" was primarily for
granularity differentiation, which you mention may not be necessary--if in
fact filesystems can be represented in this way.  As stated, our
intentional focus is representing a "unit of namespace"; are there any
non-namespace constraints on NFSv4 filesystems?

> One issue that I think we may be confronted with is that of creating
> filesets out
> of existing directory trees.  What does not make sense as a fileset this
year
> may grow and need to be converted to that status next year.  I think we
will
> wind up having to tackle this issue.

We too have considered this issue and would like to discuss in a later
post.

> So the issue here is that for most purposes, you don't want the junction
to be
> visible to users and applications.  Most users and application would
> see not the
> junction but the directory/fileset that the junction refers to.  The
> client would
> presumably see a referral, i.e. a MOVED error and an fs_locations
> value derived
> from the junction using your mapping scheme.

Right, what I was attempting to communicate there was that the "mountpoint"
is a visible directory entry that resides along side standard file and
directory entries.  As you mention, there may be some auxiliary purpose for
allowing management applications, for instance, access the content of the
junction, but this aspect was not addressed in the original post.  So
therefore your comment about not being exactly parallel is correct, since
the intention is that the server would return a MOVED error when a request
targets a junction.

> Has "nfs" been registered?

Yes:  http://www.ietf.org/rfc/rfc2224.txt

Furthermore, section 4 states the following:
"The client must first attempt to use NFS version 3.  If the server returns
an RPC PROG_MISMATCH error then the client must assume that NFS version 3
is not supported, and retry the operation with an NFS version 2 public
filehandle."

> Maybe mapping once again is ok in this case but
> the point is you have to stop mapping some time and access the data and
this
> requires mount points of real filesystems which are not junctions.

Agreed, we were simply attempting to leverage the principal of the pseudo
filesystem in hopes that some would be familiar with this concept.

Regards,
Manuel Pereira
IBM Almaden Research Center
408-927-1935 [T/L 457]

[1] http://www1.ietf.org/mail-archive/web/nfsv4/current/msg01653.html

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Ted Anderson <[email-masked]>
Date: Thu, 28 Apr 2005 09:53:32 -0400

[This is the first of two message responding to Dave and Manuel on the 
namespace topic.  I split my response because I wanted to clear up a 
confusion Dave mentioned at the end of his note, before addressing the 
other topics raised in the discussion.]

On 4/26/2005 19:38, Manuel Pereira wrote:
 > On 4/26/2005 17:49, Noveck, Dave wrote:
 >>> In NFSv4 the server's pseudo filesystem contains only (virtual)
 >>> directories and junctions (to real filesystems).  [from [1]]
 >>
 >> Some server pseudo-fs's have to have mountpoints of real filesystems
 >> and not be the style of junctions that you are talking about.
 >>
 >> Suppose netapp_eng_root is a server and has
 >> /usr/dnoveck/ontap/referral-prototype which is a junction that
 >> points, through your mapping scheme, to the server nane-17 and the
 >> volume/fs /vol/dnoveck-tree-xyz within it.  So, in that case /vol is
 >> virtual directory in the pseudo-fs.  The dnoveck-tree-xyz has better
 >> not be a junction or else we have to map once again.  Maybe mapping
 >> once again is ok in this case but the point is you have to stop
 >> mapping some time and access the data and this requires mount points
 >> of real filesystems which are not junctions.
 >
 > Agreed, we were simply attempting to leverage the principal of the
 > pseudo filesystem in hopes that some would be familiar with this
 > concept.

The reference to the NFSv4 pseudo-fs was an *analogy*, the connections
the pseudo-fs contains to real filesystems are *like* junctions.  But
the association leads to confusion, so we have to be careful.  While
abstractly the pseudo-fs contains virtual directories and junctions, it
is really used by the server for publishing exports to clients.  The
connections between the pseudo-fs's virtual directories and the real
filesystems are not subject to an additional external location lookup
step.

The topic of this thread is a global namespace that clients can assemble
for users from individual filesystems (aka filesets) exported by
servers.  The magic of NFSv4, relative to previous versions on NFS, is
the referral, which allows this namespace to cross server boundaries and
even the boundaries of individual exports on the same server.  This
capability allows filesystems to be linked together to create a larger
namespace for files.  A junction is the entity that creates these
linkages.

   S1:/x/fs7   S1:/x/fs4   S2:/x/lv3   S1:/x/fs2
  |---------| |---------| |---------| |----------...
  / a / b / C / d / e / F / g / h / I / j / k / l...
        |           |       |       |
   S1:/x/fs7/a/b    |       |       +-> S2:/x/lv3/g/h/I is a junction
             S1:/x/fs4/d/e  |           S1:/x/fs2/ is what users see
                     S2:/x/lv3/g

The global pathnames that represent junctions are /a/b/C, /a/b/C/d/e/F
and /a/b/C/d/e/F/g/h/I.  They may or may not exist in the various real
filesystems, at S1:/x/fs7/a/b/C, S1:/x/fs4/d/e/F, or S2:/x/lv3/g/h/I,
respectively.  The junctions can be real objects, such as the
symlink-like objects that AFS calls mountpoints, or they may be virtual
objects that the server somehow inserts into their parent directories
based on configuration files, an external repository or some other
source of namespace information.  What the client sees when it accesses
these server paths is NFS4ERR_MOVED and a fs_locations attribute that
indicates their actual locations, namely C => S1:/x/fs4, F => S2:/x/lv3,
I => S1:/x/fs2.  I won't belabor this example, but maybe this review was
useful for some.

Another perspective is that referral, as a *verb*, is what the server
does when a client accesses a junction, as a *noun*.  The client doesn't
see the junction itself, but is referred and as a consequence can infer
its existence.  Since clients don't see junctions the RFC does not talk
about them, but concentrates on the client / server interaction when
they are encountered.

Because the fs_locations attribute contains filesystem addresses, the
server must perform the junction target lookup that yields the
address(es).  Therefore this thread is mostly concerned with what goes
on behind the scenes on the server: how it places junctions in the
namespace and how junction targets are resolved to servers and
filesystems.

Another source of confusion relates to the top-levels of the global
namespace that provides the initial connections to the real filesystems
exported by servers.  This part of the names is often envisioned as
stored outside a traditional filesystem such as UFS.  Possibilities
include DNS, LDAP, configuration files or similar repositories.  To
expose this part of the namespace to clients, however, it is convenient
to imagine an NFSv4 server acting as a *proxy* for these sources and
exposing them as filesystem to clients.  Naturally, this special
filesystem contains only directories and junctions with no actual files
or ordinary data.  While the similarities with the pseudo-fs are even
closer in this case, it is better to think of this proxy service as just
another filesystem (like procfs or cachefs) exported by the server.

Hope this Helps,
Ted Anderson

---

From: Ted Anderson <[email-masked]>
Date: Thu, 28 Apr 2005 10:14:58 -0400

> Following are a few terse remarks:

And some less terse ones from me.

 > Our reasoning for using "fileset" instead of "filesystem" was
 > primarily for granularity differentiation, which you mention may not
 > be necessary--if in fact filesystems can be represented in this way.
 > As stated, our intentional focus is representing a "unit of
 > namespace"; are there any non-namespace constraints on NFSv4
 > filesystems?

Dave is basically right about the reasons for our use of the term
fileset.  Part of the purpose is to jar readers out of their familiar
mindset about (big) filesystems and consider the possibilities of the
protocol with a new perspective.  Pretty clearly NFSv4 supports
exporting lots of smallish filesets.  But in the end maybe introducing a
new term is more confusing than helpful?

 >> filesets out of existing directory trees.  What does not make sense
 >> as a fileset this year may grow and need to be converted to that
 >> status next year.  I think we will wind up having to tackle this
 >> issue.
 >

 >> The solution may be as simple as establishing rules about fsid
 >> validity (and requirements to refetch) when volatile filehandles
 >> expire, or we may also need some corresponding support where volatile
 >> filehandles are not suitable.  Anyway I don't think this is
 >> requirement for a global namespace but I think that the reverse is
 >> quite likely.  The existence of a global namespace may wind up making
 >> this a requirement for smooth operation and reconfiguration.

Indeed.  The specification is quiet on the question of what happens when
a fsid and/or fileid change.  The implication is that this is
inconcievable.  Certainly, there are many assumptions implicit and
explicit in the RFC (e.g., the file identity rules in 9.3.4) that depend
upon fsid being immutable, though the spec doesn't say it not change.
Perhaps a type of virtual file handle that has properties like
"unique_handles" could be used temporarily to bridge a change of
fsid/fileid for a set of file handles while preserving file identities.

 >> So the issue here is that for most purposes, you don't want the
 >> junction to be visible to users and applications.  Most users and
 >> application would see not the junction but the directory/fileset that
 >> the junction refers to.  The client would presumably see a referral,
 >> i.e. a MOVED error and an fs_locations value derived from the
 >> junction using your mapping scheme.
 >
 > Right, what I was attempting to communicate there was that the
 > "mountpoint" is a visible directory entry that resides along side
 > standard file and directory entries.  As you mention [below], there
 > may be some auxiliary purpose for allowing management applications,
 > for instance, access the content of the junction, but this aspect was
 > not addressed in the original post.  So therefore your comment about
 > not being exactly parallel is correct, since the intention is that the
 > server would return a MOVED error when a request targets a junction.

I would say that the client doesn't actually see the junction.  The
client learns of the junctions's existence if the target is absent
(generating MOVED errors) or, for resident filesets, if it notices
changes in fsid and/or fs_locations attributes.

 >> On the other hand, management applications and administrators would
 >> like to see what is in the junction and perhaps change it or
 >> establish new junctions.  In the case of symbolic links, the nfs
 >> client always sees them and may (by following them) hide them from
 >> applications/users.  In this case, we have something that is not
 >> exactly parallel, since, if I understand you, it is the server that
 >> is going to either show these (READDIR/GETATTR says the type is
 >> "junction" and allows you, via READJUNCTION(?) to get the contents)
 >> or doesn't (READDIR/GETATTR may give you MOVED and the corresponding
 >> fs_locations value).  I think we need a way to specify whether the
 >> client wants these to be visible objects or not but beyond that I'm
 >> nor clear on the exact requirements, as far as to granularity, etc.

To amplify a bit further, from the NFSv4 perspective, we aren't
proposing a new object type.  On the server, or a namespace respository
the server uses, the junction might look like a special symlink, but the
NFSv4 client would see only the fileset root directory or MOVED errors.

There are situations were we might like the client to understand
junctions more explicitly.  One strategy would be for the junction
contents to be encoded somehow in the fs_locations information.  Thus,
using GETATTR(fs_locations) could serve as a sort of READJUNCTION
equivalent.  Clients could ignore this if they were only interested in
physical addresses, but management applications could use the
information to assemble a namespace view at the fileset level.

Whether we could devise an interpretation of the fs_locations attribute
to include junction information that would not actually require a change
to the NFSv4.0 specification is unclear.  But it does seem plausible as
a small tweak for a minor revision.  Creating junctions within the NFSv4
protocol is a bit more difficult, but perhaps making the fs_locations
attribute writable would be one way to approach this.  We aren't
proposing to do this now.  Indeed, even making junctions visible via an
enhancement to the interpretation of fs_locations is not absolutely
necessary.

 >>> Finally, we propose a fully qualified URL string serve as the
 >>> representation of a physical address. The URL syntax must comply
 >>> with RFC 1738, being in the form:
 >>> <file system protocol>://<hostname>[:<port>]/<path to data>...
 >>
 >> <file system protocol> is worrisome in that nfsv2, nfsv3, nfsv4.0,
 >> nfsv4.1 are all different protocols, even though I would imagine that
 >> we would not want a fileset to have different locations for different
 >> nfs protocols.

One point this raises, is that the "same" replica may actually have
multiple entries in the location information for a fileset.  Each entry
might reflect a different access path to the same data, such as multiple
protocols or alternate IP addresses (in a multi-homed or cluster
server).  A precise definition of a replica's identity needs to be
worked out, but this two-level structure for location information may be
useful.  This is the sort of thing a fileset location schema could
define.

Presumably, an NFSv4 server would normally flatten and filter this
information for presentation to clients in fs_locations.  However,
advanced clients interested in doing more sophisticated replica
selection or clients that were really management applications might be
interested in the details.  For them, fs_locations could be enhanced, as
I suggested above, to include either the logical name (i.e. the key into
the location repository so the client can query the location information
themselves), or the contents of the location repository itself in some
format.

Ted Anderson

---

From: Carl Burnett <[email-masked]>
Date: Thu, 28 Apr 2005 11:08:15 -0500

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

> for users from individual filesystems (aka filesets) exported by

The real mapping to filesets above would really be the NFSv4 fsid concept 
and its related wire attribute. How that maps to a filesystem, some 
portion of it, or some underlying storage unit is impelementation 
dependent. So the NFSv4 fsid is really the identifier for a container of 
objects (dirs, files, etc) that share a set of common properties (supports 
ACLs, quota info, etc.). Its unfortunate that RFC3530 uses the term 
filesystems a bit too loosly.