Subject: [nfsv4] Referrals questions from nfsv4-referrals and 3530bis

From: Tom Haynes <[email-masked]>
Date: Wed, 20 Jan 2010 13:29:52 -0600

Okay, so I'm editing draft-ietf-nfsv4-referrals-00.txt into 3530bis and 
I have some
questions:

1)

4.1.  Referral Example (LOOKUP)

     o    PUTROOTFH
     o    LOOKUP "src"
     o    LOOKUP "linux"
     o    LOOKUP "2.7"
     o    LOOKUP "latest"
     o    GETFH
     o    GETATTR fsid,fileid,size,ctime

Why do we need to wait for the GETFH to determine NFS4ERR_MOVED?

Let's say that "2.7" was the first component absent from the server,
wouldn't the reply be:

     o    PUTROOTFH  --> NFS_OK
     o    LOOKUP "src" --> NFS_OK
     o    LOOKUP "linux" --> NFS_OK
     o    LOOKUP "2.7" -->  NFS4ERR_MOVED

I.e., even if the current fh is never returned to the client, the
fact that we can't determine it internally means that we have to
mark it as NFS4ERR_MOVED at that point.

That in turn would let the client know exactly which op in the
compound crossed a fsid boundary. And we could avoid the GETATTR(fsid)
dance.

2)

Assume the server didn't return the NFS4ERR_MOVED until it was
asked for a fh, doesn't that in turn imply that it will accept any
LOOKUP until the GETFH?

I.e., if "2.7" is the referral/migration location and the client
continues with

LOOKUP "foo"
LOOKUP "bar"
LOOKUP "nfs4.h"
GETFH

and "foo" does not exist in the namespace, how does the server
recognize that?

I think the answer is the same as in 1).

3) 

Assume the sever worked as described in the document. Then the
client sends:

     o    PUTROOTFH
          GETATTR(fsid)
     o    LOOKUP "src"
          GETATTR(fsid)
     o    LOOKUP "linux"
          GETATTR(fsid)
     o    LOOKUP "2.7"
          GETATTR(fsid)
     o    LOOKUP "latest"
          GETATTR(fsid, fs_locations)

I don't understand how the client knows to only send the request for
fs_locations to "latest".

  A) Is this an edited shortcut?
  B) Should the client be issuing fs_locations in each GETATTR? I.e.,
     the first to match will end the chain.
     1) And if the fsid is local, a NULL fs_locations will be returned.

4)

Where I think this dance might really apply is when the client talks to
the referred to server.

If the client did not have existing state (a referral), it is pretty clear
that it will have to traverse the name space of the new server.

If the client does have state (i.e., an open file, assume a persistent fh),
it could just present that fh to the new server. But I think this is naive,
doesn't it have to establish the path to the new filesystem? Besides
not being able to assume filsystem attributes are the same, doesn't
the client need to move the reference to the filesystem/file to a valid
position in the namespace?

I.e., the new location may not exist for the client.

3530bis talks about it this way:

6.3.  Interpretation of the fs_locations Attribute

   As an example, there is a replicated filesystem located at two
   servers (servA and servB).  At servA the filesystem is located at
   path "/a/b/c".  At servB the filesystem is located at path "/x/y/z".
   In this example the client accesses the filesystem first at servA
   with a multi-component lookup path of "/a/b/c/d".  Since the client
   used a multi-component lookup to obtain the filehandle at "/a/b/c/d",
   it is unaware that the filesystem's root is located in servA's name
   space at "/a/b/c".  When the client switches to servB, it will need
   to determine that the directory it first referenced at servA is now
   represented by the path "/x/y/z/d" on servB.  To facilitate this, the
   fs_locations attribute provided by servA would have a fs_root value
   of "/a/b/c" and two entries in fs_location.  One entry in fs_location
   will be for itself (servA) and the other will be for servB with a
   path of "/x/y/z".  With this information, the client is able to
   substitute "/x/y/z" for the "/a/b/c" at the beginning of its access
   path and construct "/x/y/z/d" to use for the new server.

On a first blush, I could take this as the client being able to do a
string modification of the path. But this can't hold true, can it?

I'm pretty sure the referrals draft mentions this in Section 5:

     When clients make use of servers that implement referrals and
     migration, care should be taken so that a user who mounts a given
     filesystem that includes a referral or a relocated filesystem
     continue to see a coherent picture of that user-side filesystem
     despite the fact that it contains a number of server-side
     filesystems which may be on different servers.

     One important issue is upward navigation from the root of a server-
     side filesystem to its parent (specified as ".." in UNIX).  The
     client needs to determine when it hits an fsid root going up the
     filetree.  When at such a point, and needs to ascend to the parent,
     it must do so locally instead of sending a LOOKUPP call to the
     server.  The LOOKUPP would normally return the ancestor of the
     target filesystem on the target server, which may not be part of
     the space that the client mounted.

So the client has to stitch up its internal namespace before it can
resume the compound which got the NFS4ERR_MOVED.

As an example, the client is reading file "/a/b/c/d/nfs4.h" and it
gets back NFS4ERR_MOVED. Before it can reissue the read compound, it
has to issue a GETTATTR(fs_locations) to the origin server. The client
gets back "/x/y/z" as the root of the filesystem. It
has to basically do:

     o    PUTROOTFH
     o    LOOKUP "x"
     o    LOOKUP "y"
     o    LOOKUP "z"
     o    LOOKUP "d"

before it can reissue the read compound. It may be the case that
any of the path components are in fact referrals in their own right.

I.e., servA moved "/a/b/c/d" to servB at "/x/y/z/d", but servB in turn
moved "/x/y" (and "child" filesystems) to servC at "/m/n/o/p".

So each GETATTR(fs_locations) yields a list of (IP, path) pairs. The
client has to evaluate the path to make sure it is still on that server
(and valid). As a result of this evaluation, the client may end up
sending GETATTR(fs_locations) to the new server. Only when the client
has found either

 o a server which has that filesystem
 OR
 o a server which has no referral information for that filesystem

can it allow that blocked READ request to progress. I.e., if the
client determines fs_locations via get_fs_location() and it issues
an over-the-wire GETATTR(fs_locations), then that
function is inherently recursive.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 20 Jan 2010 15:28:12 -0500

1-3) are basically related issues and focus around the fact that MOVED
is only returned when the current fh at the *start* of the op is not
there.  Many people find this weird/wrong and I probably did at first,
but I don't now, and if you go through it enough, you find out that the
seemingly natural alternative doesn't work.  I went round and round on
this and when I switched to the at-the-start alternative, things were
weird but the refractory problems went away.

Let's look at 1).  The basic reason you don't give MOVED on the LOOKUP
of "latest" is that if you did, there would be no way you could do a
GETATTR of fs_locations on the fh for "latest", except every attempt to
do that winds up tying you into knots.  The
simplest thing is to allow the server to have a current fh in an absent
file system but not allow the client to ever see it.

> mark it as NFS4ERR_MOVED at that point.

You may mark it internally as absent, but idea is that you can do a
(very) few things with an absent fs (get attributes like fs_locations)
and nothing else.

> compound crossed a fsidAnd we could avoid the GETATTR(fsid)
> dance.

If the client is going through multiple LOOKUPs he can cross multiple
fsid boundaries.  If that is important to him, he needs to
GETATTR(fsid), and if not he doesn't.  As long as determining when you
crossed the boundary to the absent fs, the client just as clear no
matter which rule you use (MOVED if at absent fs at start of op vs.
MOVED if absent fs as end of op) as long you are consistent.

2)

Yes the answer is the same as 1) :-)

If you have a current fh at an absent fs at the start of an op and the
op is not a GETATTR of FS_LOCATIONS (or a very few other things), then
you return MOVED.

So in your example, the LOOKUP of "foo" would return MOVED since it
isn't a GETATTR.

3) If you got MOVED error with an fh that resulted from a particular
LOOKUP that is the LOOKUP you use to generate the fh to use as the
current fs to do a GETATTR of fs_locations.

A) Not sure I understand the question.
B) No.  It is possible that he gets something from the fs_locations, but
if the has not got moved associated with that fs, all he is getting the
alternate locations (for replication).

4) 

>  o a server which has no referral information for that filesystem

I think it is just a server which has the file system in the sense that
you can use the fs_locations to find the fs in the namespace and when
you do so, you don't find an absent fs.  An absent fs is not one the
server truly "has".

-----Original Message-----
From: Tom Haynes [mailto:[email-masked]] 
Sent: Wednesday, January 20, 2010 2:30 PM
To: [email-masked]
Subject: [nfsv4] Referrals questions from nfsv4-referrals and 3530bis

1)

     o    PUTROOTFH
     o    LOOKUP "src"
     o    LOOKUP "linux"
     o    LOOKUP "2.7"
     o    LOOKUP "latest"
     o    GETFH

     o    PUTROOTFH  --> NFS_OK

2)

LOOKUP "foo"
LOOKUP "bar"
LOOKUP "nfs4.h"
GETFH

3) 

     o    PUTROOTFH
          GETATTR(fsid)
     o    LOOKUP "src"
          GETATTR(fsid)
     o    LOOKUP "linux"
          GETATTR(fsid)
     o    LOOKUP "2.7"
          GETATTR(fsid)
     o    LOOKUP "latest"

4)

If the client did not have existing state (a referral), it is pretty
clear
that it will have to traverse the name space of the new server.

If the client does have state (i.e., an open file, assume a persistent
fh),
it could just present that fh to the new server. But I think this is
naive,
doesn't it have to establish the path to the new filesystem? Besides
not being able to assume filsystem attributes are the same, doesn't
the client need to move the reference to the filesystem/file to a valid
position in the namespace?

     o    PUTROOTFH
     o    LOOKUP "x"
     o    LOOKUP "y"
     o    LOOKUP "z"
     o    LOOKUP "d"

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Tom Haynes <[email-masked]>
Date: Wed, 20 Jan 2010 14:49:45 -0600

Okay, lets see if I understand this:

     o    PUTROOTFH
     o    LOOKUP "src"
     o    LOOKUP "linux"
     o    LOOKUP "2.7"
     o    LOOKUP "latest"
     o    GETFH

We return NFS4ERR_MOVED on the GETFH. We actually now know that
the "latest" has been migrated and none of the preceding path components.

I.e., if the migration point was "2.7", we would have gotten the 
NFS4ERR_MOVED
from the LOOKUP "latest". (this was items 1 and 2 in my list)

And the reason we send GETATTR(fsid) to all of the LOOKUPs except for
the one sent to "latest", which gets GETATTR(fsid, fs_locations) (item 3 
in my list)
is because we do know that we are going to treat it differently.

Another reason we don't return NFS4ERR_MOVED is that it is a stub in the
server's name space. I.e., the server has some internal representation 
to indicate
that "latest" refers to another machine. So we allow the LOOKUP to proceed,
since it operates in the name space. But other operations, such as GETFH and
LOOKUP of a child should return NFS4ERR_MOVED since they refer
to content in the migrated name space.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 20 Jan 2010 16:11:56 -0500

> We return NFS4ERR_MOVED on the GETFH. We actually now 
> know that the "latest" has been migrated and none of 
> the preceding path components.

Just to drive you crazy, some of those others could move after that and
the one that moved could move back, but you are right assuming no
subsequent change.

> And the reason we send GETATTR(fsid) to all of the LOOKUPs 
> except for the one sent to "latest", which gets 
> GETATTR(fsid, fs_locations) (item 3 in my list) is 
> because we do know that we are going to treat it differently.

The reason we are treating it differently is that it is different.

Note that issuing GETATTR(fsid) for the others is not a recommendation
but a common example which allows us to talk about what they might
receive.

> Another reason we don't return NFS4ERR_MOVED is that it 
> is a stub in the server's name space. I.e., the server 
> has some internal representation to indicate that
>  "latest" refers to another machine. So we allow the 
> LOOKUP to proceed, since it operates in the name space. 
> But other operations, such as GETFH and LOOKUP of a 
> child should return NFS4ERR_MOVED since they refer

Yes.  "Content" is stretching it a bit for fh but your point is
basically right.  Everything about the fs is on the other server and the
only thing that we have is the information to get him there
(fs_location).  If we let the client see the fh absent fs, it would only
be a piece of data that could add confusion.

-----Original Message-----
From: Tom Haynes [mailto:[email-masked]] 
Sent: Wednesday, January 20, 2010 3:50 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Referrals questions from nfsv4-referrals and
3530bis

     o    PUTROOTFH
     o    LOOKUP "src"
     o    LOOKUP "linux"
     o    LOOKUP "2.7"
     o    LOOKUP "latest"
     o    GETFH

We return NFS4ERR_MOVED on the GETFH. We actually now know that
the "latest" has been migrated and none of the preceding path
components.

in my list)

Another reason we don't return NFS4ERR_MOVED is that it is a stub in the
server's name space. I.e., the server has some internal representation 
to indicate
that "latest" refers to another machine. So we allow the LOOKUP to
proceed,
since it operates in the name space. But other operations, such as GETFH
and
LOOKUP of a child should return NFS4ERR_MOVED since they refer
to content in the migrated name space.