Subject: [nfsv4] Global namespace for v4.1?

From: Jon Haswell <[email-masked]>
Date: Wed, 24 Dec 2003 09:58:26 -0800

As a followup to my previous thoughts on why a client should not be allowed
to independantly query the namespace, i.e. not through the NFS server.

When one is building a clustered set of NFS servers that rely on a
distributed filing system a single effective namespace is being exported by
all servers and all data may be accessed from any server (although to the
clients different FSID's may be exported to represent different file sets
or containers of data within the actual physical single namespace in the
true filing system).

It is to be expected that the ERR_MOVED return may be used by the servers
in the cluster to balance workload by moving clients to other servers when
an individual machine becomes overloaded. In this case if clients are
allowed to query data locations independant of the server then it becomes
much harder to co-ordinate this movement of clients, where not all clients
will be presented with the same locations information, and we introduce a
third logical machine in the movement of a client, i.e. the old NFS server
that has been servicing the client, the new NFS server that will service
the client and now the namespace server.

I really would like to avoid having to try and represent this type of
dynamic and client specific location information from an LDAP schema or the
like and be able to force the clients, at least in this case, to get their
namespace information from the NFS server directly, even when this NFS
server cluster is also part of a larger global namespace, i.e. I want to be
able to use the NFS servers as a filter on the view of the global namespace
stored in LDAP or wherever.

Jon Haswell
IBM Research
Tel 408 927 2074 Tie 457 2074
Fax 408 927 3030 Tie 457 3030

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Juan Gomez <[email-masked]>
Date: Mon, 29 Dec 2003 10:30:14 -0800

Seems as we have two issues that must be addressed differently from the
naming point of view:

1.-Perhaps Inter-cluster, real data migration, where clients do need to go
to the new server to actually get to the data: In this case I think it
makes sense to use LDAP to reflect data location. These movements must be
reflected equally to all clients, and they do not occur that frequently.

2.-Intra-cluster data movements, which may be virtual, done on the basis of
load balancing algorithms: LDAP may not be the best solution for this due
to latencies, and Jon's points, etc. Having v4 servers address this by
referrals does not seem best either. Perhaps we need to formalize what has
been done in the past for this....These movements may occur frequently and
each client may have a different picture of them.

Juan

                      Jon Haswell/San                                                                                               
                      Jose/IBM@IBMUS           To:       [email-masked]                                                             
                      Sent by:                 cc:                                                                                  
                      nfsv4-admin@ietf.        Subject:  Re: [nfsv4] Global namespace for v4.1?                                     
                      org                                                                                                           

                      12/24/03 12:17 PM                                                                                             

Brent Callaghan wrote :

>Keep in mind that name services like LDAP, Active Directory etc are
already
>widely deployed, have published APIs and libraries, trained admins and
nice
>administrative tools.  With an NFSv4 based referral service, you'd have to
>start from scratch.

I agree that LDAP et al are available, tested and deployed and I expect us
to use them as the underlying infrastructure for the global namespace. What
I am arguing against is allowing the NFS clients (and by extension other
filesystem clients) direct access to them. I want to see the NFS servers
accessing them and acting as proxies for passing this information to the
NFS clients. The reason for doing this is so the NFS servers can act as
smart proxies and manipulate this data, if they so desire, prior to it
being received by the NFS clients.

When trying to optimize a clustered NFS server as part of a larger global
namespace the NFS servers may choose to move clients between nodes in the
cluster on the fly, present different location information to different
clients etc. I do not want to have to push this type of dynamic, on the
fly, changes to far up the global namespace - in particular I don't see how
we could easily have different views of the namespace presented to
different clients simulateneously via LDAP or the like.

I absolutely agree with you in that we should avoid at all costs trying to
recreate LDAP or the like within V4, I am only arguing for the V4 clients
to only access this data via the V4 server using the already established V4
protocol (with minor corrections to the bits of it that may be broken).

>I don't understand your reasoning.  Data movement between NFS servers
>may not use the NFS protocol at all.  A data movement tool can just
>as easily make namespace changes with a back-end namespace server
>as it can with a network of NFSv4 referral servers.  As long as the
>data movement and namespace update steps are done in the right order,
>there needn't be a sync problem.

I agree that it can work as you describe, in this case I am saying it is
simpler to coordinate, in all error situations etc, if the clients are
getting their information from one level in the hierarchy, as opposed to
some getting NFS V4 referrals and some going direct to the namespace
servers.

>
>Keep in mind that the market doesn't sell "NFS Servers" - they're
>muli-protocol NAS servers that handle a variety of file access
>protocols: most notably NFS v2/v3, CIFS, AFP, Netware, FTP, HTTP etc
>Any cross server movement of data must take into account all these
>other access methods - not just NFSv4 clients.

I'm not sure how manadating NFS clients obtain their namespace information
via the NSF V4 protocol breaks any of the other protocols. Yes if data
movement is going to happen then all supported protocols are going to need
to handle this movement either by redirection or by proxy methods for
support of legacy or little used protocols.

We typically find that any one machine/cluster/data set is predominently
accessed by a single protocol and we will frequently only load balance for
that protocol and leave all other protocols on a single node etc, i.e. for
the non-domin ant protocol we provide access but not performance.

>
>My guess is that NAS cluster vendors will continue to design for multiple
>protocols and assume the dumbest clients.  If you already have a
transparent
>solution for switching v2/v3 and CIFS clients from node to node, then it's
>easier to extend the same solution to v4 clients, rather than build v4
specific
>solution - even if it's "better".
>

Firstly I agree with the dumbest clients statement, hence unless there is a
really good reason why complicate a spec by defining or allowing a more
sophisticated client that will not be implemented. With regard to
transparent switching solutions for V2/V3 I do not like the current
transparent solutions, and I'm thinking virtualization boxes here, and want
to avoid them for V4

>I'd expect an NFS cluster to advertise itself, and be administered as a
single,
>virtual NFS server, i.e. the cluster will likely load balance without
making
>any changes to the name service or the perceived location of the data.

To do this requires a front end node that can switch the traffic between
backend nodes. This is inherently more expensive than a cluster that
exports the IP addresses of all it's nodes and directs the clients between
them. While this may not seem as clean it is far more scalable and
virtually as easily managed once the initial IP addresses are assigned.

Jon Haswell
IBM Research

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 29 Dec 2003 13:52:40 -0800

Juan Gomez wrote:
> Having v4 servers address this by referrals does not seem best either.

Could you expand on this a bit?

-----Original Message-----
From: Juan Gomez [mailto:[email-masked]]
Sent: Monday, December 29, 2003 1:30 PM
To: Jon Haswell
Cc: [email-masked]
Subject: Re: [nfsv4] Global namespace for v4.1?

Juan

                      Jon Haswell/San                                                                                               

                      12/24/03 12:17 PM                                                                                             

already
nice

transparent
specific

single,
making

Jon Haswell
IBM Research

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 30 Dec 2003 07:23:27 -0800

Same color (in italics ) for me. 

"Noveck, Dave" <[email-masked]> wrote on 22/12/2003 17:07:34:

> 
> 
> 
> > It is not the granularity that I am after. I could live with directories
> > or even subtrees.
> 
> OK, glad to hear it.
> 
> I was going by your statements:
> 
> > IMHO a good global namespace should be imposed without artificially 
> restricting 
> > the way components are placed. 
> 
> > It should also not have (or not have it standardized) a mapping to 
> local namespaces. 
> > I.E., in theory a file in the global namespace may be located anywhere.
>  
> > Probably you may want to restrict this to directories but you should not go 
> > beyond this. 
> 
> >From which I concluded that the current v4 model, which made fs's (significant
> trees) the unit of migration and referral, was not acceptable to you.
> 
> > The thing that I am concerned about is that building a global map based on an 
> > "arbitrary" composition of local maps will not get you the transparency that 
> > every enterprise is looking for - location transparency, migration 
> transparency, 
> > replication transparency. Transparency is what will keep applications and 
> > administration separate. 
> 
> Maybe I'm not understanding the problem you are worried about.  As far as what
> I have proposed, each fs can be at an arbitrary position within the namespace,
> and that typically will not depend on the server or servers on which the fs is 
> currently hosted.  Does that address your concern?
> 
Arbitrary in reference to what? If it is in a fixed position in the global namespace that addresses my concern. 
Arbitrary with respect to its position in the local namespace of the server hosting it.  My understand of your concern was what you were worried that there was some sort binding between the local name and the position in the global namespace and I don't think anyone is proposing that there is. 
 
> > As for teleport - I never though about AFS in those terms (should I have?).
> 
> I don't know AFS well enough to understand how it deals with the granularity
> issue.  I know there is a global namespace but is it the case that directories
> within it can be arbitrarily located and still manipulated as if they were
> within a single fs?  If so, then it does seem to me kind of science-fictional 
> and something that is rather distant, in the context of a protocol like NFS 
> where multiple separately-derived implementations need to interoperate.
> 
Not necessarily - in AFS the "local names" in the local filespace have no relevance to the global names 
However the same effect could be achieved with "bigger granules".  Interoperability - I am not aware of any interopearbility issues that become simpler by that or another choice of building the global tree. 
Let me  try again.  If your granules are visible to clients as file systems, then interoperability becomes a matter of the server providing some version of acceptable v4 semantics and the client being able to do that.  If you have granules that are smaller than that, in that a client sees two files or directories on different servers as part of the same fs, then the ante on interoperability goes up quite a bit.  For better or worse, v4 allows different servers to implement their own fs semantics (within limits) and takles the position that it is not the jobof the protocol to impose a semantic uniformity, but merely to provide remote access to the fs semantics exported by the server.   Further, an action like renaming a directory on one server between two parent directories on two different servers becomes very interesting if we have to test with the possible range of implementations of client, target directory server, source parent directory server, destination parent directory server. 

As any decent implementation will use some form of name caching any local naming action is bound to have global effects. As for semantic uniformity I am not sure that I understand or agree. The naming structures and rules have to be uniform over the global space for it to be really useful (you are not suggesting changing character casing rules or name lengths when crossing implementation boundaries). 

I am not suggesting that such variability would be a good thing or something that I would implement in a global namespace.  The problem is precisely how to prevent it.  V4 servers may implement case-sensitive or case-insensitive naming (and it allows servers to refuse to rename open files, or to allow that, etc.)  That kind of variability wouldn't be my choice but it was the choice of the working group.  If you have a global namespace which is fine-grained (i.e. the distribution units are smaller than a file-system), then this is an issue that you have to address, somehow and it is one of many in which the definition of v4, by allowing servers different semantic choices, complicates the implementation of a fine-grained global namespace.  It is one of many issues that one needs to address. if you want to go down this route.
 
 I think that a clear set of export rules a global namespace constructors will simplify interoperability not complicate it and an attempt to build something that is no more that the chain of mounts in previous versions is not very useful. 

If you can define such rules, and the group agress to them within a reasonable time, then that would be great, but I fear that it may be quite difficult.  As far as the usefulness of something more limited and coarse-grained, I guess we just disagree.  I'm willing to settle for something not as good, that I know can be built, without confronting some basic issues in v4.  I think the result is still quite useful. 
------------------- 

If you limit yourself to referrals servers or the referal-servers will have to guarantee the "unique" structure of the global name-space. 
Don't know what you mean here. 
My understanding is that you will use referrals wherever you reach a "mount point". Those could be different on each sever (or you have to replicate on all or use a separate global name/referral server.

There are two classes of referrals here to deal with.  The first is those that are referrals from mount points within a real filesystem (i.e. a filesystem that has real files within it and not only directories that lead to referrals).  For this class, you only have a consistency issue when the filesystem in question is replicated and I would presume that the identity of the referral will be assimilated to the general issue of the identity of the data within the individual replicas of the replicated fs.  There has been in the past some discussion of a v4 server-to-server migration and replication protocol to allow such replicas to be co-ordinated, even among different types of servers, but there has been no progress on that recently.  Without that, replication is going to be implemented individually and it is the responsibility of the servers implementing the repllicas to keep them the same including the same referrals.  As a practical matter, the server implementations must be the same.
 
The second class of referrals are those in what we might call global pseudo-fs's, which implement fs's devoted to providing a global or enterprise-wide or site-wide namespace, by means of a set f referrals within the fs.  Today, we are in a similar situation to the first case in that only implementaton-specific replication is available.  My assumption is that work will proceed on a naming service definition (LDAP or whatever) and then referral servers or referral fs implementatins will fetch data from a common name service with multiple server sproviding the same data sicne they are obtaining them from the same source (issues of dealing with stale data from the name service as configutations are changing is left as an exercise for the I-D writer).

> The v4 approach relies, for its simplicity, on the fact that the separately
> located subtrees are separate fs's and thus you cannot link files between them,
> move directories between them, etc.  That's why I characterized that 
> as "walking",
> i.e. something we know how to do, can be done within the bounds of RFC3530,
> albeit with a number of rough edges, and only requires that we travel the
> road laid out for us, putting one foot in front of the other.
> 
I am not sure that I would characterize moving directories as a bad thing

I wouldn't characterize it as a bad thing, either.  I would however, characterize it as a difficult thing when the participating directories are in different servers.  If people want this functionality badly enough, then the people in this group will work to provide it but it would not be easy.  So maybe this indicates that perhaps "teleportation" is the wrong metaphor.  Maybe a better analogy would be a controlled fusion power reactor.  The basic physics is understood and it is just a matter of figuring out the engineering of confining a plasma at the appropriate temperatures and pressures.  It's just work and money and time. 

You seem to suggest that it will never work (or it will take as long as fusion power). But AFS was (is) here for a long time and people love it 
 
It works in the context of AFS, and that's fine.  Making it work in the context of NFS presents a whole different set of challenges.  To stretch my analogy a bit more (probably to or beyond the breaking point), the sun has been here for a long time and people love it and it produces power from thermonuclear fusion with no problem. The problem is transferring that form of power production to an environment in which a strong gravitational field is not available to neatly solve the confinement problem.  There are lots of things that AFS can do easily that NFS has difficulty doing.  We want to retain NFS's basic structure ot modify it incrementally.  This imposes constraints on our ability to get some of the desirable features of AFS into NFS, as much as we'd like them.
 
Perhaps I'm being unduly pessimistic.  My perception of the difficulty is basically a reflection of the fact that I don't know how to do this.  If you (or someone else) knows how to do this within the constraints of NFS (making only moderate changes to NFS, or making radical changes and convincing the working group as a whole that these are justified) , then I would love to see an I-D explaining how to do this.   I think doing this on fine-grained basis is very difficult and that we are going to have enough work handling a global namespace on an fs granularity basis.  However, I would be quite happy to be proved wrong.