Subject: RE: [nfsv4] Global namespace for v4.1?

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 12 Dec 2003 07:59:34 -0800

> The fs_locations/MOVED facility is mostly enough to implement a global
> namespace - the missing ingredient is about finding the "root" of a
> global namespace.
>
> Some possible methods for finding a global namespace root:
>
> - configure it in on clients (ick)

Ick, yes.  But, if we compare the ickiness of configuring the location
of one thing versus the ickiness of configuring the locations of lots
of fs's, the ickiness ain't so bad.  My thought/hope is that this is 
something that can be done in v4.0.  Let that be the moderately-icky 
version and then, when something better is done for v4.1, only one thing, 
the way in which the global namespace root is found, needs to change. 

One issue about this area might be characterized as "global" name space
vs. "more global" name space.  For most people, an enterprise-wide
NFS name space would cover at least 99.44% of their NFS access.  Having
them simply specify their mounts as being on the enterprise-wide root
namespace machine with the path being the enterprise-namespace-relative
name of the fs, gives them most of the value of a global namespace, in
the enterprise context.  Not truly global but global enough to be of
considerable value.

>  - use the automounter and the directory service, or SLP, or ... (ok,
>    but surely some will not like this, no?)

I think Rob Thurlow proposes SLP in his document.  As far as people who
don't like any particular thing, we should listen to the alternatives 
that they propose.  "I don't like it", with no alternative, isn't very
helpful.

> - use a well-known server-side path which, by convention and along with
>   fs_locations/MOVED, names the global root
>
>    - and use DNS SRV RRs, PORTMAP indirect RPC broadcasts, or whatever
>      else to find a server, any server

That sounds pretty reasonable to me.

> - add a v4.x operation that returns the fs_locations for the global
>   root - combine this with DNS SRV RRs / PORTMAP indirect RPC
>   broadcasts (see above)

I think the one above is fine.

> More below.
> 
> On Tue, Dec 09, 2003 at 08:22:24AM -0800, Noveck, Dave wrote:
> >                                  The existing text about MOVED does 
> > not really properly address the situation of descending into a new 
> > fs which isn't there and so the suggestion that the server will only 
> > return fs_locations needs to be amended to suggest that fsid as well 
> > will normally be returned.  No big deal.
>
> Fsid?  Why?

Here's the case that caused me to want this.

Let's same I'm looking for /nfs/netapp/src/ontap/main on the global namespace
server.  Suppose that /nfs/netapp/src is in one file system (probably the 
pseudo-fs) and /nfs/netapp/src/ontap is the root of a second fs.  So suppose
I have the fh for /nfs/netapp/src and I now do LOOKUP of ontap/main.  I'm
going to get MOVED when I try to lookup main, because that directory is 
isn't there and in fact is part of an location somewhere else.  The question
is what filesystem?  Normally, you track the filesystem boundaries by looking
for fsid changes.  So, we are going to be able to get an fh coresponding
to /nfs/netapp/src/ontap but we can't do anythng with except GETATTR.  If
GETATTR only returns FS_LOCATIONS, we know that something moved but we
don't have a clear indication (via fsid changing from the parent) that this
is the root of new fs.

It may be that context (e.g. /nfs/netapp/src doesn't give you MOVED while
/nfs/netapp/src/ontap) gives you enough to figure out the boundaries and 
there is also the possiblity that fs_root in fs_locations may help here.
However, it just seems better if we have available fsid so that normal
method of noting fs boundaries continues to work.  Since the server is
assumed to have enough information within the fh to figure out what fs 
you are talking about (otherwise he wouldn't know what fs_locations value
to give you), I assume that he can give you fsid as well.

> > The only other thing that I think is needed is a little more work 
> > to explain how a client should deal with various fs unavailability 
> > situations.  These are all things that you could argue a client might
> > derive from the existing spec plus a store of good sense, but 
> > since definitions of "good sense" tend to differ, I think the spec 
> > should lay these out.
> > 
> > The things I'm thinking about are:
> > 
> >      If you are directed via migration to a server which becomes 
> >      unresponsive, you should go back to the source that directed 
> >      you there to see where the server for this fs is, and not assume 
> >      it will never move.
> > 
> >      Clients should interrogate fs_locations when crossing into new
> >      fs's and periodically thereafter to make sure that the best path 
> >      to a resource is used.

> "Best path?"  I think we need something better than just "the server
> sorts the list properly given information it may have about the fs
> locations and/or client, and the client prefers the first entry in the
> list" - which is what I'm assuming you mean.

That was what I meant.  

> Or maybe not - if there's multiple fs_locations then either they are all
> equivalent (module replication latency, say) or one is read-write and
> the others are read-only.

I don't think you can make that assumption.  There are architectures in
which servers can co-operate so that the effect from the clients' point of
view is that all can be written with the effects visible on all of them.
The multiple server fs's may be instantiated on the same physical disks.
In that kind of a context, directing clients to the best server appearance
(from an efficiency point of view) makes sense.

> >      In connection with such periodic interrogation, if fs_locations
> >      is valid but does not include the current server, this is a very
> >      strong suggestion that the client should use another path/replica, 
> >      as the way to access that fs, if at all possible.
> 
> Referral revocation, eh?

We can effect revocation by the server simply returning MOVED.  Given that
clients who support migration right now, are kind of, how shall I put it,
non-existent, I'm looking for something a bit weaker.  I'd like to tell
anyone who is with it (and I assume such clients will someday exist) to 
move without blowing less competent clients out of the water.  The assumption
is that we can accommodate such clients by forwarding their requests but
would prefer not to do so in the case of clients that can support the 
migration.

> >      If fs_locations does include the current server but it is not the 
> >      first location, this is a suggestion that the client consider using
> >      another path/replica for that fs.
> 
> See above.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 12 Dec 2003 09:26:20 -0800

> That sounds pretty reasonable to me.

And note that the indirect RPC bcasts could be done as just a bcast
indirect ping (call to NULL) for the NFSv4 program.

> I think the one above is fine.

It does have the potential for creating conflicts, but the good news is
that administrators, rather that WG members, get to set the convention.

> don't have a clear indication (via fsid changing from the parent) that this

Note that the MOVED error should be returned for any attempt to do
anything with the "ontap" directory component of that example path,
including crossing it in a LOOKUP.

And the answer to the "what filesystem" question is this: that which you
find by looking up the "rootpath" of an fs_location4 against an
fs_location4 server (and you can find the fs_id then too).

> there is also the possiblity that fs_root in fs_locations may help here.

I believe fs_locations4 tells the client what it needs to know: the
names of other servers to go to and the server-side paths ('rootpath')
to look up at each to get the FH for the directory/file covering the
referral trigger path.

I'm not sure what fs_root is all about, actually -- but rootpath sure
looks useful.  I guess fs_root can be used as in the third paragraph of
section 6.3, but it seems to me that the client's architecture will
determine whether fs_root is at all useful.

IMO fs_root is redundant - the client should already be able to
reconstruct the full server-side path that elicits the MOVED error
anyways - and if it isn't then won't need to use fs_root as in that
example in section 6.3.

> to give you), I assume that he can give you fsid as well.

I don't.  Fsid in practice is often derived from information that is
really private to the server that's serving the associated files.  In
fact, the very design of fsid parallels the *nix major/minor device
number idea (why? why was opaque not good enough?), and I think I won't
be the only one here not wanting fsid's to be cached elsewhere.

> The multiple server fs's may be instantiated on the same physical disks.

Right.  I don't want to make that assumption.

> (from an efficiency point of view) makes sense.

Sure.  But we're talking about changing the semantics of fs_locations4.

> would prefer not to do so in the case of clients that can support the 

Proxy forwarding of client's requests to other servers won't work unless
the proxy has full rights to impersonate any users to the real server,
because the client won't be using the right GSS-API acceptor principal
name for the real server when talking to the proxy.  I think this means
that proxy forwarding for dumb clients will not be generally useful.

Cheers,

Nico
--