Subject: [nfsv4] Global namespace for v4.1?

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 9 Dec 2003 08:22:24 -0800

There has been some discussion to the effect that v4 extensions to 
support a global namespace should be done in v4.1.  I'm not sure 
if this was on the working group list or not, but I have had some 
discussions with people about this at interoperability events.

Given the desirability of drawing a line under v4.1 and getting going
on actually producing the thing, I'd like to understand where this 
stuff is, if anywhere.  Is anybody currently working on this with 
the intention of producing a document soon?

On the assumption that the answer to the question above will be 
negative, let me offer my suggestions on the minimal set of namespace-
related changes that should be done in v4.1.  It is a very small set, 
since the basics of migration give you most of what you need, already.

The existing migration support would allow you to have a single server 
represent within a single namespace all of the fs's within a large set 
of servers.  All you would have to do is to return MOVED for anything 
not actually hosted on this server and have fs_locations direct the 
client to the correct location.  The existing text about MOVED does 
not really properly address the situation of descending into a new 
fs which isn't there and so the suggestion that the server will only 
return fs_locations needs to be amended to suggest that fsid as well 
will normally be returned.  No big deal.

The only other thing that I think is needed is a little more work 
to explain how a client should deal with various fs unavailability 
situations.  These are all things that you could argue a client might
derive from the existing spec plus a store of good sense, but 
since definitions of "good sense" tend to differ, I think the spec 
should lay these out.

The things I'm thinking about are:

     If you are directed via migration to a server which becomes 
     unresponsive, you should go back to the source that directed 
     you there to see where the server for this fs is, and not assume 
     it will never move.

     Clients should interrogate fs_locations when crossing into new
     fs's and periodically thereafter to make sure that the best path 
     to a resource is used.

     In connection with such periodic interrogation, if fs_locations
     is valid but does not include the current server, this is a very
     strong suggestion that the client should use another path/replica, 
     as the way to access that fs, if at all possible.

     If fs_locations does include the current server but it is not the 
     first location, this is a suggestion that the client consider using
     another path/replica for that fs.

I think this would allow a fairly reasonable namespace facility without 
any big protocol changes.  The problem is that it is not "global" in that 
there is no definition of server-server protocol features to allow the
global namespace server to find all of the relevant fs's and their status.
The situation is similar to the basic migration situation in that we have
tackled the client-server interactions with the server-server interactions
left TBD.  The difference is that the server-server interactions for
fs location discovery are considerably smaller and might reasonably fit
within a v4.x extension, as opposed to actual server-server data migration 
where we are talking about a significant separate protocol.

So is anybody out there of a mind to tackle the server-server fs discovery,
issues soon, so that they might get into v4.1?  I don't have time to do it,
even though it is very desirable.  I'm going to go ahead and try to write 
up a draft for the stuff laid out above, the client-server stuff, but if 
someone wants to quickly do the server-server part we could combine our 
two drafts into a single draft that addresses global namespace issues for 
v4.1.  Any takers? 

Note that if we don't do server-server discovery extensions for v4.1, we
could still do this feature as a separate protocol (and therefore without
having to wait for v4.2), but it generally winds up that the overhead of 
even a simple separate protocol is substantial, both at the spec and the 
implementation level.  That's why tackling this now, as a part of v4.1, 
would be nice.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Brent Callaghan <[email-masked]>
Date: Tue, 09 Dec 2003 14:48:36 -0800

Seems that in building this kind of support for fs_locations
and server-server fs discovery, you're building a private
name service for NFS.  This would be fine if there weren't
already lots of general purpose alternatives, e.g. Active Directory,
LDAP, DNS etc.  These alternative name services have already
implemented such features as master-slave replication,
security, update protocols etc.  Wouldn't it be better to
leverage these existing standards, implementations, and
deployments ?

Rather than bounce clients around from server to server
via MOVED/fs_locations, why not have the client query
the name service once and go directly to the right place ?

Then, if a MOVED error does occur (rare we hope), the client could check
fs_locations (per the spec) and if it's null, just re-query the
name service - from whence it got the original reference ?

Rather than add to the list of NFS v4.x extensions, or write a
new server-server fs discovery protocol, why not work on an
LDAP schema and a description of its use by NFS ?

	Brent

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 11 Dec 2003 14:48:43 -0800

The fs_locations/MOVED facility is mostly enough to implement a global
namespace - the missing ingredient is about finding the "root" of a
global namespace.

Some possible methods for finding a global namespace root:

 - configure it in on clients (ick)

 - use the automounter and the directory service, or SLP, or ... (ok,
   but surely some will not like this, no?)

 - use a well-known server-side path which, by convention and along with
   fs_locations/MOVED, names the global root

    - and use DNS SRV RRs, PORTMAP indirect RPC broadcasts, or whatever
      else to find a server, any server

 - add a v4.x operation that returns the fs_locations for the global
   root - combine this with DNS SRV RRs / PORTMAP indirect RPC
   broadcasts (see above)

More below.

On Tue, Dec 09, 2003 at 08:22:24AM -0800, Noveck, Dave wrote:
>                                  The existing text about MOVED does 

Fsid?  Why?

>      fs's and periodically thereafter to make sure that the best path 

"Best path?"  I think we need something better than just "the server
sorts the list properly given information it may have about the fs
locations and/or client, and the client prefers the first entry in the
list" - which is what I'm assuming you mean.

Or maybe not - if there's multiple fs_locations then either they are all
equivalent (module replication latency, say) or one is read-write and
the others are read-only.

>      as the way to access that fs, if at all possible.

Referral revocation, eh?

>      another path/replica for that fs.

See above.

Cheers,

Nico
--

---

From: David Robinson <[email-masked]>
Date: Mon, 15 Dec 2003 12:00:18 -0600

The issue of a global namespace is very broad and
we tend to get stuck in rat holes when we deal with
parts of the problem. But, on the positive side
it can be a partitionable problem.

At the core of any global namespace is the need for
a global repository of the naming information, a
mechanism to publish data into the repository and
a mechanism to retrieve data from the repository.
There is a secondary issue of the syntax to be used
and the policy that determines where something should
exist in the namespace, while it is a necessary thing
to decide, it is more political than technical so
I will ignore it for now.

 From a filesystems protocol perspective the primary
work is in defining when the repository should be
consulted and who should do the retrieval. At this
level the form of the repository and to a lesser
extent the retrieval protocol can be separated out.
I think LDAP should be a top contender as it already
exists, but in the absence of such a standard
interim adhoc mechanisms can be used to provide a
set of good "first steps".

Breaking things down further there are three different
models that can be constructed to provide global naming:

	1) Intelligent clients: The clients are primarily
	   responsible for performing all naming operations
	   from the repository.  This is much like the
	   current automounter based products. The client
	   has a full naming protocol support (LDAP/NIS/DNS)
	   and build the tree from existing servers. A weakness
	   is that for open files it is hard to handle
	   migration and not all automounters are as dynamic
	   as they could be. In this world servers are dumb.

	2) Modestly smart clients, modestly smart servers:
	   The first part of this is to recognize that if
	   the server has knowledge about naming issues it
	   should speak up instead of having the client do
	   all the work.  This is what we have with fs_locations
	   where the server tells the client, I am no longer
	   the droid you are looking for, go ask that one
	   over there. This makes the server a client of
	   the repository as it now needs to have some
	   mechanism to determine where "there" is.  It also
	   allows the client to have a simplier implementation
	   as it can defer much of that heavy lifting to the
	   server.

	3) Dumb clients, intelligent servers:  In this world the
	   client can simply latch on to one server and access
	   all of the namespace through it.  The server is now
	   effectively a proxy for all accesses.  This will work
	   great for things like cell phone based clients.

I think from an industry perspective we will have to do all
three.

Of the three, #3 is the easiest to do without any
standards work in a relatively homogenous server world. But
as you try to implement it in a broader world you discover
that your proxy server is now really just a client operating
in either mode #1 or #2. So we can mostly ignore this case
but think about it as one weird client using #1 or #2.
I fear that Dave's server-server discussions are pushing
towards this type of server, when it is really just
a case of the server acting as a client (#1).

Mode #1 is what Brent is advocating, and I agree that it
will be very useful to have.  To some extent we can separate
out the problem from the others as it is a primarily client
only function. As a simple example of need, virtually all
vendor's automounters have some mount option that is unique
to their platform. Storing existing flat automounter strings
in a network database is inadequate because clients have no
rules to decide what to do with options they don't understand.
Is the option manditory or not? Developing a standard schema
using a standard protocol such as LDAP is a very good thing
to solve a problem we have today.  The WG should take it on
as an action item, but it is not a gating item.  Although it
is clear that in cases #2 and #3 it is highly desireable to
use the same repository mechanisms.

 From an incremental change in a minor version perspective,
case #2 is where the most fruit can be harvested.  We already
have a good first step but it is clearly has some problems
that we should address.  A first observation is that the existing
NFS protocols are already a naming protocol.  LOOKUP and READDIR
are just very limited methods of translating names into
objects, albeit one that returns very limited data back.
A GETATTR of fs_locations is just a bastardized form
of LOOKUP.

So I take Dave's original question to be, do we have the
right mechanisms for a server to assist a client in participating
in a global namespace.  To answer this I think it is important
to keep in mind how this fits within the bigger picture, but
also realize that there is considerable things we can do
without building the total solution.

My first take on what we need to fix is to make the fs_locations
mechanism a bit more flexible and extensible.  For example,
in the case of a tightly coupled clustered server it is
possible that the referring server knows the filehandle of
the destination server, so why not allow that sort of
information to be returned.

More generally fs_locations should communicate the following
information back to the client:
	1) Who is the new server
	2) How to access the new server
	3) What you should pass the new server in the request

In today's fs_locations #2 is implicitly NFSv4 LOOKUP and
#3 is the mount path of the root.  But if we do this sort
of generalization we can have the response range from
passing back (server, NFSv4:PUTFH, file_handle) for
tightly coupled servers to (server, ldap:, key_value)
for loosely couple servers that happen to use a directory
server as a repository. An advantage to having such a
mechanism is that as we make progress on generalized
global naming, we can add new access methods for the client
to use by simply extending the union.

To take things one step further, a bigger change, but one
that fits within the model is to extend LOOKUP so that it
can return more than just a filehandle, but also return
back the same redirection information without causing
an extra over the wire operation.

	-David

---

From: Rob Thurlow <[email-masked]>
Date: Fri, 19 Dec 2003 17:32:24 -0700

I'm very late in this thread, for which I apologize.
I hope I have some content worth the wait :-)

First off, I should remind folks about my I-D:
http://www.ietf.org/internet-drafts/draft-thurlow-nfsv4-namespace-00.txt

This will expire soon, and I want to update it in the
New Year to get some more stuff into it.  To that end,
please either send me comments on changes or let me
know if you want to own a chunk of it and become a
co-author.  Or, if it bugs you enough, go ahead and
write another draft.  In the next spin, I'd like to
focus on clearly stating options (without making any
decisions yet) for finding the root node, and for
Internet-wide vs. enterprise-wide scope.  The
requirements could be beefed up, too.

Next, what should we be doing?  I agree that we need
to Just Get On With It, and based on that feeling, I
showed up at the last Bakeoff with client code to
try to deal with NFS4ERR_MOVED, and was able to take
the client a lot farther and also get a swag at the
server side.  I was able to test with Dave, but
despite trading bugs for awhile, we didn't get so
far as having a successful migration event.  My code
is dumb as a rock - it is smart enough to parse a
dotted-quad format IPv4 string, and doesn't try to
do an upcall yet - that needs to be done.  I know
I intend to show up with this code again, hopefully
in better shape, and would certainly like to see
other people in the migrate-a-thon as well (sounds
like Trond has something more that at the Bakeoff? :-)

However, while we mess with these prototypes, I do
think we need to keep half a lobe on what we should
try to get to in the future.  fs_locations is a start,
but it's not enough, or even specified clearly enough
yet.

In the short term, we can leave OTW encoding alone
and permit some spec reinterpretations.  Dave has
talked about his fsid issue, and I guess there's not
closure on whether that is strictly necessary, but if
it's not harmful, it'd be nice for the server to return
these if asked.  I have another issue - as David had
mentioned, the spec is actually only set up for migration
of something you've already been using - you can't
get a filehandle for a filesystem that's moved before
you got there, because the only ops you're supposed
to be able to use are PUTFH and GETATTR.  So we should
permit LOOKUP as well in practice.

Beyond that, we need more work.  We should not have to
guess what type of thing is encoded in fs_locations,
as we now do - we should have a tag to tell hostnames
from IPv4 addresses from IPv6 addresses, and whatever
we come up with should be extensible to other formats.
Also, I feel a need for metadata about the locations -
is this location read-only or will it accept writes?
It's been argued here that the client can find that
out if it needs to know, but that's ugly.  I think
David pointed to a desire to know that some locations
were alternate paths to the same storage, and that's
another thing we could do if we defined some kind of
attribute (say, an opaque global ID tied to the
underlying storage).

I think prototyping will permit us to do a good job
at making a wish list for the future, and that that
wish list should become an RFC as soon as it can be
done.  If that's before V4.1, cool - we can consider
it.  It's not wasted effort if not, though.  (And
if you wonder about the V4.1 timeline, please do
ask your friendly WG chairs :-)

Some other random notes:

- David's taxonomy of client and server capabilities
   was useful.  I tend to not see why folks like Brent
   and David want the client to consult the namespace
   repository directly - can someone tell me why that's
   a better architecture at all?  I know it's another
   step to have the server involved, but it's only
   another cache, and I think we can do that.  I am
   obviously sympathetic to the idea that if we have
   to implement migration anyway, we should use that
   mechanism to do namespace referrals as well, and
   I don't recognize a trigger that says that that's
   not good enough for a smarter client.

- We haven't talked about management enough, and I
   don't think that's healthy.  Should we have one
   master repository of all this stuff, or should it
   be decentralized as a hierarchy of namespaces?
   Who should get to make changes and how should
   they do that?  What standard protocols make sense
   here?  It's a big area.

I hope folks have a great Christmas break - mine
starts tomorrow, so I won't see further traffic
for awhile.

Rob T

---

From: David Robinson <[email-masked]>
Date: Fri, 19 Dec 2003 18:48:24 -0600

>   not good enough for a smarter client.

Here is an alternative viewpoint. If we are going to
provide server based referals within the namespace, servers
will need to have some place to store and retrieve what
referals exist, where they are in the global namespace,
and where they point to.  This repository needs a
standard set of semantics and a standard access method,
otherwise you will only be able to handle a homogenous
set of servers. Another way to put this is that the
NFS servers are clients of the namespace.

If we have standardized the namespace information, then
it is relatively simple for a smart client to directly
query it and skip the need to use the server for a
referral.  This may help environments where there are
many referrals or the NFS latency is high.

But I think if we adequately design the semantics of
the repository and its access protocol, that is 99.44% of
what a smart client will need.

So I don't think we need to dwell on the smart client case
(#1 in my taxonomy), if we do the rest well it should
fall out almost for free.

	-David

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 22 Dec 2003 02:06:22 -0800

> we come up with should be extensible to other formats.

We can't have IP addresses in the protocol.  They're bad, bad news.

To wit:

 - NAT, NAT, NAT (no need to explain :(  )

 - Security, security, security

   Specifically, how does one format a GSS-API hostbased service
   principal name given an IP address?

   By resolving it to a hostname, which makes the use, on the wire, of
   IP addresses instead of hostnames pointless as a way to avoid upcalls
   *and*, most importantly, forces the use of a name service, most
   likely DNS, and most likely _not_ DNSSEC, which is a problem.

   Whereas there's no need to do any name service canonicalization of
   otw hostnames to create GSS-API hostbased service principal names.

   Canonicalization of principal names through insecure name services,
   such as DNS (ignoring DNSSEC), invites redirection attacks (e.g., by
   DNS spoofing).

   Arguing that IP addresses otw are ok when using AUTH_SYS/NONE is lame
   -- it's too easy to predict that in the end they'd also be used otw
   when using RPCSEC_GSS, causing the security problem described above.

> attribute (say, an opaque global ID tied to the

I'm not sure that the server issuing a referral can be considered
authoritative for such info.  Certainly not always, which argues that
what you propose is an optimization that won't always be applicable --
clients will just have to be able to discover such info by asking the
server(s) they are referred to.

Looked upon from this angle I don't think there's any ugliness here, but
elegance instead.

That said, referring servers could tell clients to treat the referred-to
filesystem as read-only even if it accepts writes -- but I'm not sure
that that would be of much use.

>   a better architecture at all?

It's needed for v2/v3 clients, that's it.  Also, I contend that the same
namespace repository, if properly designed, can be used by v2/v3 clients
and by v4 servers, the latter to issue referrals.  Which, if I'm
correct, means that the argument is somewhat moot because we'll want to
specify the repository side of things and it will be useful for both,
the smart client and smart server models.

Besides, the fact is we have referrals in the protocol and they can be
used for cool things, like building namspaces w/o client smarts, and if
we have referrals we'll want clients to implement them, and if we have
client support folks will want to implement the namespace w/o client
smarts.

So, either we get rid of referrals or the dumb-client model will come to
pass naturally, though the smart-client model will remain feasible.

End result: we let the admins choose what model is best for them.

The biggest problem with a referrals-based namespace, particularly from
a security point of view, lies in how the entry points into the
namespace are to be found.  But this does not prevent deployment since
the simplest answer is not inpractical: configure such info where it's
needed.

>                                  I know it's another

Agreed.

>   be decentralized as a hierarchy of namespaces?

Both!  :)

Referrals allow for both anyways, with referrals that cross domains
acting as the glue for decentralized hierarchies (like DNS zone
delegation, if you wish).

>   they do that?

The owners of the relevant servers.  For smart clients that would be the
directory's owners; for dumb clients that would be the NFSv4 servers'
owners, possibly the directory's owners if the NFSv4 servers are so
configured as to use the directory as the namespace repository.

>   here?

Directory service protocols; LDAP, basically.  The abtract model for the
namespace data is almost^W more important than the protocol.

>          It's a big area.

Basically, whatever we do for representing automount information in the
directory should work for smart clients and smart servers alike, at
least if we do it right.

> starts tomorrow, so I won't see further traffic

Happy holidays,

Nico
--