Subject: RE: [nfsv4] Global namespace for v4.1?

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 12 Dec 2003 10:13:56 -0800

> > > More below.
> > > 
> > > On Tue, Dec 09, 2003 at 08:22:24AM -0800, Noveck, Dave wrote:
> > > >                                  The existing text about MOVED does 
> > > > not really properly address the situation of descending into a new 
> > > > fs which isn't there and so the suggestion that the server will only 
> > > > return fs_locations needs to be amended to suggest that fsid as well 
> > > > will normally be returned.  No big deal.
> > >
> > > Fsid?  Why?
> > 
> > Here's the case that caused me to want this.
> > 
> > Let's same I'm looking for /nfs/netapp/src/ontap/main on the global namespace
> > server.  Suppose that /nfs/netapp/src is in one file system (probably the 
> > pseudo-fs) and /nfs/netapp/src/ontap is the root of a second fs.  So suppose
> > I have the fh for /nfs/netapp/src and I now do LOOKUP of ontap/main.  I'm
> > going to get MOVED when I try to lookup main, because that directory is 
> > isn't there and in fact is part of an location somewhere else.  The question
> > is what filesystem?  Normally, you track the filesystem boundaries by looking
> > for fsid changes.  So, we are going to be able to get an fh coresponding
> > to /nfs/netapp/src/ontap but we can't do anythng with except GETATTR.  If
> > GETATTR only returns FS_LOCATIONS, we know that something moved but we
> > don't have a clear indication (via fsid changing from the parent) that this
> > is the root of new fs.

> Note that the MOVED error should be returned for any attempt to do
> anything with the "ontap" directory component of that example path,
> including crossing it in a LOOKUP.
> 
> And the answer to the "what filesystem" question is this: that which you
> find by looking up the "rootpath" of an fs_location4 against an
> fs_location4 server (and you can find the fs_id then too).

Huh?  rootpath is the path on the destination server but that doesn't
tell you the fs on the original server that is replaced.  And fsid is 
of no significance.  I don't care what the fsid is at that point
(and I assume it might be different from the one on the source).  The
only reason I can see for needing the fsid is to see the boundaries of
fs's, and so for that I have to know what the boundaries fo the fs are
on the original server, not the destination server.

> > It may be that context (e.g. /nfs/netapp/src doesn't give you MOVED while
> > /nfs/netapp/src/ontap) gives you enough to figure out the boundaries and 
> > there is also the possiblity that fs_root in fs_locations may help here.

> I believe fs_locations4 tells the client what it needs to know: the
> names of other servers to go to and the server-side paths ('rootpath')
> to look up at each to get the FH for the directory/file covering the
> referral trigger path.

I'm not sure what you mean by the referral/trigger path.  From my point
of view, the critical thing is that it is an fs that moves and the 
important thing for the client is to know the set of objects that have
been relocated (i.e. which ones belong to the fs).

> I'm not sure what fs_root is all about, actually -- but rootpath sure
> looks useful.

Useful?  It's essential.  You have to be able to find the fs on the
destination server.  

I guess fs_root can be used as in the third paragraph of
section 6.3, but it seems to me that the client's architecture will
determine whether fs_root is at all useful.

> IMO fs_root is redundant - the client should already be able to
> reconstruct the full server-side path that elicits the MOVED error
> anyways 

If I do a READ using a fh I have and I get MOVED, then I may know the
path of the file I opened, but it isn't necessarily the name of the
file now.  So I would think of the basic model of migration being 
fs-based in that I know all of the fh's associated with a given fs
and look for them on a different server.  In some cases I may have 
the name, or because of volatile file handles be forced to use a 
best-guess, but, the first and most desirable thing is that you 
be able to things transparently without oh-by-the-way-that-file-you-
had-open-might-be-a-different-file-or-may-not-exist-live-with-it
kind of stuff.

> - and if it isn't then won't need to use fs_root as in that
> example in section 6.3.

OK.  I now see how this could be used.  The example gives it in 
terms of a name-based implementation but I see how it could be used
if your implementation made fs boundaries, rather than paths,
the focus.

> > However, it just seems better if we have available fsid so that normal
> > method of noting fs boundaries continues to work.  Since the server is
> > assumed to have enough information within the fh to figure out what fs 
> > you are talking about (otherwise he wouldn't know what fs_locations value
> > to give you), I assume that he can give you fsid as well.
> 
> I don't.  Fsid in practice is often derived from information that is
> really private to the server that's serving the associated files.  In
> fact, the very design of fsid parallels the *nix major/minor device
> number idea (why? why was opaque not good enough?), and I think I won't
> be the only one here not wanting fsid's to be cached elsewhere.

The v4 spec makes looking for fsid changes within a given server the
way to see that you have crossed into a new fs.  That's the only
use of fsid that makes sense.  There is no need to use it outside
the boundaries of a given server.  It would have probably been better
if we simply had a this-is-the-root-of-an-fs attribute, and not had
fsid explicit in the protocol.

> > > > The only other thing that I think is needed is a little more work 
> > > > to explain how a client should deal with various fs unavailability 
> > > > situations.  These are all things that you could argue a client might
> > > > derive from the existing spec plus a store of good sense, but 
> > > > since definitions of "good sense" tend to differ, I think the spec 
> > > > should lay these out.
> > > > 
> > > > The things I'm thinking about are:
> > > > 
> > > >      If you are directed via migration to a server which becomes 
> > > >      unresponsive, you should go back to the source that directed 
> > > >      you there to see where the server for this fs is, and not assume 
> > > >      it will never move.
> > > > 
> > > >      Clients should interrogate fs_locations when crossing into new
> > > >      fs's and periodically thereafter to make sure that the best path 
> > > >      to a resource is used.
> > 
> > > "Best path?"  I think we need something better than just "the server
> > > sorts the list properly given information it may have about the fs
> > > locations and/or client, and the client prefers the first entry in the
> > > list" - which is what I'm assuming you mean.
> > 
> > That was what I meant.  
> > 
> > > Or maybe not - if there's multiple fs_locations then either they are all
> > > equivalent (module replication latency, say) or one is read-write and
> > > the others are read-only.
> > 
> > I don't think you can make that assumption.  There are architectures in
> > which servers can co-operate so that the effect from the clients' point of
> > view is that all can be written with the effects visible on all of them.
> > The multiple server fs's may be instantiated on the same physical disks.

> Right.  I don't want to make that assumption.

> > In that kind of a context, directing clients to the best server appearance
> > (from an efficiency point of view) makes sense.
>
> Sure.  But we're talking about changing the semantics of fs_locations4.

OK.

> > > >      In connection with such periodic interrogation, if fs_locations
> > > >      is valid but does not include the current server, this is a very
> > > >      strong suggestion that the client should use another path/replica, 
> > > >      as the way to access that fs, if at all possible.
> > > 
> > > Referral revocation, eh?
> > 
> > We can effect revocation by the server simply returning MOVED.  Given that
> > clients who support migration right now, are kind of, how shall I put it,
> > non-existent, I'm looking for something a bit weaker.  I'd like to tell
> > anyone who is with it (and I assume such clients will someday exist) to 
> > move without blowing less competent clients out of the water.  The assumption
> > is that we can accommodate such clients by forwarding their requests but
> > would prefer not to do so in the case of clients that can support the 
> > migration.

> Proxy forwarding of client's requests to other servers won't work unless
> the proxy has full rights to impersonate any users to the real server,
> because the client won't be using the right GSS-API acceptor principal
> name for the real server when talking to the proxy.  I think this means
> that proxy forwarding for dumb clients will not be generally useful.

The client is thinks he is talking to a given server and to him it is the
real server.  That server's job is to effect the client's wishes.  How he
does that is up to him.  I gues he would be able to terminate GSS stuff 
and make, for example, auth-sys requests of the real/ultimate server or
have some sort of private protocol to support this function.  What am I 
missing?

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 12 Dec 2003 10:23:31 -0800

> on the original server, not the destination server.

The MOVED error is certainly an indication of an fs boundary on the
server that returns it.

> Useful?  It's essential.  You have to be able to find the fs on the

Ok, I know what's going on: I'm thinking of MOVED/fs_locations as a
mechanism for implementing referrals, which we need in order to
implement a global a namespace, but it is also intended for indicating
that a filesystem really moved while the client was using it.

So yes, fs_root may be essential for the case where a filesystem
actually moved, but it's redundant where MOVED/fs_locations is used
purely as a referral mechanism.

> had-open-might-be-a-different-file-or-may-not-exist-live-with-it

I assume FHs are to be preserved across migration, yes?

> have some sort of private protocol to support this function.  What am I 

Nothing.  That's pretty much what I said -- the proxy would have to be
able to use AUTH_SYS to talk to the real server.  Of course, there is
the problem here in that AUTH_SYS is limited to 16 (17) groups, a
restriction that does not apply with RPCSEC_GSS.

Cheers,

Nico
--

---

From: Brent Callaghan <[email-masked]>
Date: Fri, 12 Dec 2003 11:42:21 -0800

> that a filesystem really moved while the client was using it.

This discussion keeps bringing up the notion of a
"Global Namespace" - a concept for which we lack a either
a problem description or requirements. It would be good
to have some idea of what we're talking about, before
talking about referrals as a component of its implementation.

Anyone care to take a stab at either a problem statement
or some requirements - even as a simple bullet list ?

	Brent

---

From: Julian Satran <[email-masked]>
Date: Mon, 15 Dec 2003 00:59:10 +0200

Great point. I was looking (lurking) on this thread and trying to 
understand how one can build in a guaranteed unique way a tree (or DAG) 
using only subtrees.

Julo

Brent Callaghan <[email-masked]> 
Sent by: [email-masked]
12/12/2003 21:42
Please respond to
[email-masked]

To
Nicolas Williams <[email-masked]>
cc
[email-masked]
Subject
Re: [nfsv4] Global namespace for v4.1?

                 Brent

---

From: Ted Anderson <[email-masked]>
Date: Mon, 15 Dec 2003 10:05:53 -0500

> fsid explicit in the protocol.

It is all well and good for you to say this here in a message buried in 
a long thread on namespaces, but this interpretation seems at best 
implicit in the spec.  I would expect that many clients will build the 
server fsid deep and hard into their implementations, which will prevent 
them from working with servers that support migration and perhaps 
referrals.  So servers will have to construct fsid mapping mechanisms to 
work-around these clients which will lead to mysterious interoperability 
problems.

This is where the connectathon can highlight where these assumptions 
have crept into the spec and various implementations.  Test cases at 
connectathon then depend upon the problem description and requirements 
mentioned Friday by Brent.

Ted Anderson

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 15 Dec 2003 09:11:17 -0800

IMO we need:

 - a referral mechanism (got that)
 - a "root" finding mechanism (we can build that today as a set of
   conventions, really)

Given which a "global namespace" can be constructed that satisfies most
sysadmins' requirements with minimal client configuration (in some cases
even none).

Of course, one can build namespaces on the client side without servers'
help (think automounter), and one can do it with minimal client
configuration (think directories), but "crossing domains" is difficult
or impossible.

You're correct about DAGs.  NFSv4 referrals don't provide a guarantee
that there won't be cycles in the namespace, and while within a single
domain it won't be hard to prevent them, when crossing administrative
domains loop prevention becomes rather difficult.

So clients will have to watch out for loops (which otherwise could make
for a DoS).  Note that this is true today for any client that correctly
follows MOVED/fs_locations -- this isn't a new problem introduced by
some strange "global namespace" proposal, it's there now.

As I see it the WG should consider "global namespace" "root" finding
conventions (or extensions to v4, if not conventions).  And the WG
should consider extensions to fs_locations to better support protocols
other than NFSv4, as well as any related clarifications that might be
needed (e.g., fsid).

One more thing: it would be nice if there were a standard way to admin
NFS namespaces.  IMO, whether the client, through an automounter, say,
or the server, through referrals, drives the namespace is not quite
relevant to how the namespace is administered in a directory.

Cheers,

Nico

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 15 Dec 2003 10:17:38 -0800

On Mon, Dec 15, 2003 at 10:12:13AM -0800, Brent Callaghan wrote:
> Yes, a couple of important components, though, I should point out,
> these elements are just a few in a long list of the list of features
> that we expect of any general-purpose name service, e.g. LDAP, NIS, DNS etc.

I acknowledged as much, and should have put that on my list.

> I think what Julian is trying to point out is that a namespace is more
> than just a collection of exported filesystems.  There's some expectation
> that these filesystems will be organized in some way, most likely one or
> more hierarchical trees, and their shape will be determined by the customer.
> 
> I predict that as the requirements of the namespace become more clearly
> defined, the advantages of using an established name service will become
> more apparent.

Surely.  But how the information is distributed to either smart clients
or smart servers is a problem that I see as separate from the question
of whether or not we should support dumb clients (for which we need
referrals).

Nico
--

---

From: Ted Anderson <[email-masked]>
Date: Mon, 15 Dec 2003 16:12:18 -0500

> or some requirements - even as a simple bullet list ?

Rob Thurlow had a good problem statement in section 2 of his May 2003
personal Internet Draft[2].

Here are some requirements and some thoughts about their necessity and
application to NFSv4.

Heterogeneous file system protocols -- In addition to NFS and CIFS, FTP
and many other file repositories may be included through mechanisms such
as LUFS[3].  Heterogeneity is important if the namespace portion of the
hierarchy is to have wide appeal.  This suggests that the namespace is
something apart from the particular protocol used to access it.

Global scope for namespace -- In other words, literally global with
inter-enterprise scope.  AFS experience suggests that this is valuable,
though I can't really disagree with Dave's 99.44% figure[1].  However, I
think the extra 0.56% is an important and probably growing constituency.
In addition to AFS users, Grid Virtual Organizations, for instance,
would find this capability invaluable.  A key argument in favor of true
global naming is that it decouples the administrative domains from the
usage domains.  This is a good thing for transparency.  We do need to
completely avoid getting bogged down in defining the ownership of the
top level of the global namespace.  A simple way to do this is to
delegate responsibility to another authority, such as a local
configuration file and/or DNS.  In other words, in the interest of
expediency, we should pass the buck on the "root finding" problem.

Referrals -- client support is crucial to support multiple file system
protocols.  It would seem that the NFSv4 Unix client could provide
integrated support for autofs functionality, by letting the OS handle
mounts when crossing fsid boundaries.  This would allow straightforward
support of a namespace that includes all sorts of file systems.  It
might be useful to extend fs_locations with type information which could
allow clients to select the appropriate handler from a wider set than
just NFSv4 servers.

Location independence -- the NFSv4 assumption is that the server will
provide the physical server name, or list, to the client.  However,
there are reasons why the client should lookup and evaluate the choices
to determine the "best" source for the data.  Often the client is in a
better position to consider its network connectivity than the server is,
and this is especially true if the server is far away or supports very
many and disparate clients.  Dave alluded to this issue with his comment
about a "weaker" form of MOVED error.  There is also an argument for
separation between the namespace, which imbues pathnames with meaning,
and the provisioning layer, which may move filesystems among servers and
establishes replicas to meet demand throughout the network.
Fundamentally these tasks have very different time-scales and
responsible authorities, so keeping them separate will have some
architectural advantages.

Consistency semantics -- since the namespace consists entirely of
directories strong consistency at this level must wait for directory
delegation.  On the other hand, time-based consistency is increasingly
appropriate higher in the hierarchy.

Access Control -- NFSv4 supports ACLs, which is probably sufficient.

A paper prepared in June for the Grid community[4] discusses some of
these issues from that perspective.

Ted Anderson

[1] http://www.ivory.com/history.htm#pure
[2] http://www.ietf.org/internet-drafts/draft-thurlow-nfsv4-namespace-00.txt
[3] http://sourceforge.net/projects/lufs
[4] 
http://forge.gridforum.org/docman2/ViewProperties.php?group_id=90&document_content_id=299
     http://tinyurl.com/m3pb

---

From: Julian Satran <[email-masked]>
Date: Tue, 16 Dec 2003 16:44:18 +0200

Brent,

You are right - a namespace without any structure is pretty useless to a 
client. On the other hand it would be hard to have a client organize the 
name space and present it in a coherent way - beyond the trivial preset 
model. 
I would find also hard to consider the naming issue completely separate 
from the access control.
Structure has to be imposed from outside. The only neat model that 
everybody liked in the past was AFS (and its clones).
It handles in a neat fashion names and administrative domains. I don't 
know how good it scales however (although I've seen it deployed 
successfully in pretty large enterprises :-)).

Regards,
Julo

15/12/2003 20:12
Please respond to
[email-masked]

To
[email-masked]
cc

Subject

Yes, a couple of important components, though, I should point out,
these elements are just a few in a long list of the list of features
that we expect of any general-purpose name service, e.g. LDAP, NIS, DNS 
etc.

I think what Julian is trying to point out is that a namespace is more
than just a collection of exported filesystems.  There's some expectation
that these filesystems will be organized in some way, most likely one or
more hierarchical trees, and their shape will be determined by the 
customer.

                 Brent