Subject: RE: [nfsv4] Global namespace for v4.1?

From: Mike Eisler <[email-masked]>
Date: Mon, 15 Dec 2003 11:03:38 -0800

I like this taxononomy:

 > Breaking things down further there are three different
 > models that can be constructed to provide global naming:
 >
 > 	1) Intelligent clients: The clients are primarily
 > 	   responsible for performing all naming operations
 > 	   from the repository.  This is much like the
 > 	   current automounter based products. The client
 > 	   has a full naming protocol support (LDAP/NIS/DNS)
 > 	   and build the tree from existing servers. A weakness
 > 	   is that for open files it is hard to handle
 > 	   migration and not all automounters are as dynamic
 > 	   as they could be. In this world servers are dumb.
 >
 > 	2) Modestly smart clients, modestly smart servers:
 > 	   The first part of this is to recognize that if
 > 	   the server has knowledge about naming issues it
 > 	   should speak up instead of having the client do
 > 	   all the work.  This is what we have with fs_locations
 > 	   where the server tells the client, I am no longer
 > 	   the droid you are looking for, go ask that one
 > 	   over there. This makes the server a client of
 > 	   the repository as it now needs to have some
 > 	   mechanism to determine where "there" is.  It also
 > 	   allows the client to have a simplier implementation
 > 	   as it can defer much of that heavy lifting to the
 > 	   server.
 >
 > 	3) Dumb clients, intelligent servers:  In this world the
 > 	   client can simply latch on to one server and access
 > 	   all of the namespace through it.  The server is now
 > 	   effectively a proxy for all accesses.  This will work
 > 	   great for things like cell phone based clients.
 >
 > I think from an industry perspective we will have to do all
 > three.

I think this WG has to do #2. As far as what the industry should
with respect to #3, that's a choice each vendor has to make, and
may the best competitor win.

 > but think about it as one weird client using #1 or #2.
 > I fear that Dave's server-server discussions are pushing
 > towards this type of server, when it is really just
 > a case of the server acting as a client (#1).

I don't see Dave's ideas pushing us toward #3, but if they are
that's (1) out of scope, and (2) wrong. :-) In Dave's
email that kicks off this thread, he is clearly talking
about clients in that are smarter than #3.

 > Mode #1 is what Brent is advocating, and I agree that it
 > will be very useful to have.  To some extent we can separate

#1 is an evolutionary dead end.

 > So I take Dave's original question to be, do we have the
 > right mechanisms for a server to assist a client in participating
 > in a global namespace.  To answer this I think it is important
 > to keep in mind how this fits within the bigger picture, but
 > also realize that there is considerable things we can do
 > without building the total solution.
 >
 > My first take on what we need to fix is to make the fs_locations
 > mechanism a bit more flexible and extensible.  For example,
 > in the case of a tightly coupled clustered server it is
 > possible that the referring server knows the filehandle of
 > the destination server, so why not allow that sort of
 > information to be returned.

Fine, but until NFSv4.next is defined and implemented to
support this new flexibility, why not see what NFSv4.0 can do
in this area? Cleaner specs are nice, but running code is
more important. Running code will tell us what we could change in
NFSv4.next to make fs_locations et. al. better.

What I'd like to see from client implementors is enough fs_locations
support to do the following, in priority order:

(1) when the _MOVED error comes back, do a GETATTR for fs_locations.
	Support enough of fs_locations to interpret an ascii
	IP address within the kernel and modify the in-kernel
	mount information accordingly (but save the original
	server somewhere else for supporting #2).

(2) If a server times out, then go the original server the mount
	was done with.

No upcalls, to automounter complexity, etc. Do this, and I predict NFSv4
servers will evolve quickly to take advantage. Customers will get scalable,
global namespace without having to wait 3 years for this WG to give
it to them. Even NFSv4 servers built on UNIX and Linux could take advantage.
Using tools like the LOFI driver in Solaris 8 (I'm told there's
similar concept in Linux), servers can migrate and replicate data in
ways that preserve metadata.

 > To take things one step further, a bigger change, but one
 > that fits within the model is to extend LOOKUP so that it
 > can return more than just a filehandle, but also return

LOOKUP doesn't return a file handle. :-) LOOKUP; GETFH
does.

 > back the same redirection information without causing
 > an extra over the wire operation.

fs_locations is an attribute.
COMPOUND { ... LOOKUP ; GETFH; GETATTR fs_locations }
does it one round trip.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 15 Dec 2003 11:19:51 -0800

> 	server somewhere else for supporting #2).

Not an IP address, please.  That would mean having to do a DNS lookup in
order to determine what the other server's GSS-API principal name should
be.

Yes, that means an upcall to resolve the hostname.  But using an IP
address means an upcall to resolve the IP to a name, so it all evens
out.  And no, we don't want IP-address based principal names.

[...]
> COMPOUND { ... LOOKUP ; GETFH; GETATTR fs_locations }

But this is wasteful to do on every LOOKUP!

Cheers,

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Mon, 15 Dec 2003 11:40:20 -0800

> out.  And no, we don't want IP-address based principal names.

Point, but:

Which authentication flavor will have more NFS traffic in the next
5 years: AUTH_SYS or RPCSEC_GSS?

Which is a higher barrier to implementation: modifying existing NFS clients to do an
upcall to resolve a hostname, or modiying existing NFS/RPCSEC_GSS clients to add
an upcall to an existing gssd (Solaris and Linux have them, others must
have something similar if they support RPCSEC_GSS) that converts <ip-address>@server
to <hostname>@server?

> But this is wasteful to do on every LOOKUP!

Given a choice between:

- NFSv4.0 has imperfect migrastion/replication support; let's wait for
the NFSv4.1 protocol specification death march to conclude before do anything?

- Let's do a wasteful GETATTR for fs_locations on our 1Gbit and 10Gbit
network between our 4Ghz client and server processors

I'll take the latter.

Some of us NFS old timers might not live till you youngsters
perfect the NFS spec :-)

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 15 Dec 2003 11:44:07 -0800

> 5 years: AUTH_SYS or RPCSEC_GSS?

Dunno.  I want this to be secure though, and forming a GSS-API hostbased
principal name from an IP address presents security issues.

> have something similar if they support RPCSEC_GSS) that converts 

We already have to do upcalls.  Adding an upcall, particularly one
per-mount or one per-referral, has not nearly as significant an impact
as having to add an upcall facility in the first place would have/has
had.  In other words, don't worry about this and don't optimize here, as
there be dragons.

> network between our 4Ghz client and server processors

Well, ok.  Besides, this will not affect LOOKUP perf/latency/bandiwdth
utilization all that much, and if it does the client can be configured
to go back to the two round-trip method.

> Some of us NFS old timers might not live till you youngsters

Heh.

Nico
--

---

From: Trond Myklebust <[email-masked]>
Date: 15 Dec 2003 15:18:40 -0500

>>>>> " " == Mike Eisler <[email-masked]> writes:

    >> address means an upcall to resolve the IP to a name, so it all
    >> evens out.  And no, we don't want IP-address based principal
    >> names.

     > Which authentication flavor will have more NFS traffic in the
     > next 5 years: AUTH_SYS or RPCSEC_GSS?

     > Which is a higher barrier to implementation: modifying existing
     > NFS clients to do an upcall to resolve a hostname, or modiying
     > existing NFS/RPCSEC_GSS clients to add an upcall to an existing
     > gssd (Solaris and Linux have them, others must have something
     > similar if they support RPCSEC_GSS) that converts
     > <ip-address>@server to <hostname>@server?

I'm not sure that argument is really valid. Any client with full
support for fs_locations is supposed to be able to deal with DNS
lookups since it is already allowed for in RFC3530.

The other advantage of using the DNS domain rather than the IP address
is that it doesn't involve the root server in the routing decisions
for how the client should be contacting the new server.

For instance, a lot of existing WAN setups involve the use of NAT in
order to translate private IP addresses to public ones. A client on
the internal network will not want to use the same IP address as one
on the outside (I can think of at least one AFS setup that I've
personally used, and where this is the case).

Ditto problem if/when IPv6 finally gets implemented, and we get a
choice of 2 possible transports to use.

Cheers,
  Trond

---

From: David Robinson <[email-masked]>
Date: Mon, 15 Dec 2003 14:24:59 -0600

> may the best competitor win.

I agree that #3 is primarily a best vendor win, but in that
case the server is ultimately a client of the namespace
and if we don't consider this model we may do things that
will make it hard to integrate such servers into networks
that also have type #1 and type #2. May not be much work
but we shouldn't ignore them.

> about clients in that are smarter than #3.

I guess I was reading into the comments about having a server
fronting others as a case of #3, maybe a 2.5.  The server to
server protocol discussion seems to be the wrong model, it
should be client of namespace to server of namespace
repository. Those themselves may be filers, but when someone
brought up the NxM update matrix problem it identified a
problem.  If a filer want to act as the
repository for naming model great, but we still should have a
client-server model for naming.

> #1 is an evolutionary dead end.

I disagree for a lot of reasons, but importantly we will
need to define some standard mechanism to determine
for a particular file where it is. Even in a server
driven model we need a way to lookup the repository.
So having a smart client that can query the repository
directly instead of first having to bounce off of
a NFS server can be a useful implementation. That said,
having just a smart client without the ability for a
server to redirect is an incomplete solution.

> NFSv4.next to make fs_locations et. al. better.

I have no problem with this. We can both refine what we have
to make it more useful as well as to design what the next
version should be.  The only issue here is to make sure
that we are very clear when we are doing one versus another
and not allow the latter to hijack the former.

> LOOKUP doesn't return a file handle. :-) LOOKUP; GETFH

According to the spec it returns cfh, the current file handle
which can be retrived with GETFH. :-) Depends on your
definition of "return". :-)

> COMPOUND { ... LOOKUP ; GETFH; GETATTR fs_locations }

Nope. LOOKUP returns NFS4ERR_MOVED, according to the RFC
that indicates that the COMPOUND must terminate.  You can't
do either the GETFH or GETATTR. As I said before, LOOKUP
should be fixed.

	-David

---

From: Mike Eisler <[email-masked]>
Date: Mon, 15 Dec 2003 13:20:46 -0800

> lookups since it is already allowed for in RFC3530.

Just as the spec says clients have to support in-place mounting
when it encounters fsid boundary; LIPKEY is mandatory, NFSv4/UDP
is verboten, etc.

The fs_locations is a big bite for David Robinson's class 2 clients.
I'm suggesting that clients bite off in smaller pieces if that will
get stuff out the door.

Partially implementing any feature always has drawbacks. That's not
an argument for implementing nothing.

> for how the client should be contacting the new server.

Yes of course full implementation of fs_locations is most useful.
There's lots of talk about how good this would be but where's the
code; which clients fully implement fs_locations? If you are saying
that Linux 2.6 has it, then great; that works for me, and I'll
shut up. :-)

> personally used, and where this is the case).

We have that same problem with delegation callbacks. Delegations are
still considered useful enough to implement. I've had more AFSers complain
to me about the aforementioned callback problem than the fact that
fs_locations support isn't present.

	-mre

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 15 Dec 2003 17:56:01 -0500

> that Linux 2.6 has it, then great; that works for me, and I'll

It doesn't have it yet (I'm doing the delegation support right now) but
I'm certainly not intending to implement fs_locations more than once.
The plan is to resolve those fs_locations problems by issuing an upcall
to a userland automounter that is to do all the hairy DNS-style work for
us.

Upcalls should by now be a solved problem for most NFSv4 clients. At
least in the Linux case, the cost in terms of code of adding a new one
ought not to be much greater than that of doing the mount in the kernel
itself.

> fs_locations support isn't present.

Sure, but lack of support for delegations should not be a showstopper
feature that will cause breakages at the user level (unless the hardware
melts down due to the resulting increase in network and server load). If
the return path fails to work, then the server is not supposed to issue
the delegation, and so we all go on living happily ever after as NFSv3
clients.

A supported, but broken fs_locations will OTOH have immediate and
measurable consequences for user applications. Were you planning on
manning the support helplines to handle all those "my client hangs
whenever I type 'ls /nfs/broken.com/'" queries personally? 8-)

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Mon, 15 Dec 2003 16:28:03 -0800

> whenever I type 'ls /nfs/broken.com/'" queries personally? 8-)

NetApp employees have been staffing NFS client support lines for years. :-)
Somw clients generate more customer calls than others. :-)

Implementing fs_locations completely is a massive piece of work.
Not only do you have to do the upcall, but you have to be prepared to
get new file handles and/or re-establish state. Not in 2.6 I gather.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 15 Dec 2003 22:59:08 -0500

> get new file handles and/or re-establish state. Not in 2.6 I gather.

Most of the above is already done and available for testing. We'll have
to see whether or not we can still get it into 2.6. It won't be in 2.6.0
though...

Cheers,
  Trond