Subject: [nfsv4] NFS4ERR_MOVED and PUTPUBFH, PUTROOTFH

From: Tom Haynes <[email-masked]>
Date: Fri, 26 Mar 2010 03:00:03 -0500

> The NFS4ERR_MOVED error is returned for all
 > operations except PUTFH and GETATTR.

This is page 60 in RFC 3530bis.

The text claims it is valid for PUTFH and GETATTR.

Text fails to mention it for PUTPUBFH, PUTROOTFH,
RENEW, SETCLIENTID, SETCLIENTID_CONFIRM, and
RELEASE_LOCKOWNER.

I agree with the text and argue that the line on page
60 needs to change.

But looking in RFC 5661, I see it also contends that
both PUTPUBFH and PUTROOTFH can not return
NFS4ERR_MOVED. Why not?

Consider a server in which we migrate everything to
another server - isn't it better to return NFS4ERR_MOVED
on the root rather than have the client determine
that for each subdir of the root?

What if the only filesystem being exported is the one
being used as the root? The implication currently is
that the root has to move to the pseudo-fs. I.e., if
it does not, then we have the root not being allowed
to return NFS4ERR_MOVED, but all other file objects
in the filesystem will return NFS4ERR_MOVED.

And if the server originally had shares for NFSv3 and
NFSv4, but the admin decided to migrate the NFSv4
load and not the NFSv3 load, then we might also get
into this state.

The argument for PUTPUBFH is a bit stronger. The
admin might migrate the filesystem which is the
public FH and not remember the correlation. I.e.,
if fsA is the ROOT and /fsA/fsB is the PUB, then a
migration of fsB to another machine will cause
undetermined behavior until the time that the
admin changes the PUBFH on the server.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 26 Mar 2010 08:22:43 -0400

> both PUTPUBFH and PUTROOTFH can not return

How would the client query fs_locations if PUTROOTFH/PUTPUBFH return an
error? You need at least to be able to set up a compound of the form

     PUTROOTFH; GETATTR fs_locations

Cheers
  Trond

---

From: Tom Haynes <[email-masked]>
Date: Fri, 26 Mar 2010 09:36:54 -0500

>      PUTROOTFH; GETATTR fs_locations

In other words, the first use of the FH is what is problematic and not the
actual operation.

---

From: <[email-masked]>
Date: Fri, 26 Mar 2010 11:25:27 -0400

> operations except PUTFH and GETATTR.

I contend that this is wrong and it should be valid for GETATTR, if the
GETATTR is asking for ops other than things like locations.

There has been a lot of confusion about PUTFH and it appears that some
of it has migrated to RFC5661, despite lots of efforts to clean things
up.

I think PUTFH, PUTPUBFH, and PUTROOTFH should all be the same in this
regard, but I think this means that they should not be allowed to return
NFS4ERR_MOVED. 

Remember that NFS4ERR_MOVED does not mean that the current fh at the end
of the operation has migrated (or is a referral).  It means that the
current fh at the start of the operation is used by the operation and
the operation cannot be executed because of the fact that it requires
the fs containing that fh to be present (and it has migrated or is a
referral).

So there is nothing special about the root.  PUTFH, PUTPUBFH, and
PUTROOTFH do not return NFS4ERR_MOVED because they do not do anything
with the current fh at the start of the operation (but rather obliterate
it) and just set the current fh to something.  A subsequent operation
that uses that fh will normally return NFS4ERR_MOVED if the fs has
migrated, although there are special exceptions for using GETATTR to get
FS_LOCATIONS and similar attributes.

So I think that RFC5661 saying that PUTFH can return NFS4ERR_MOVED is
best dealt with via the errata mechanism.

I think RESTOREFH is similar but I need to think about this some more.

I'll send a proposal to the working group for comment before reporting
this/these as an erratum. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Tom Haynes
Sent: Friday, March 26, 2010 4:00 AM
To: [email-masked]
Subject: [nfsv4] NFS4ERR_MOVED and PUTPUBFH, PUTROOTFH

The argument for PUTPUBFH is a bit stronger. The
admin might migrate the filesystem which is the
public FH and not remember the correlation. I.e.,
if fsA is the ROOT and /fsA/fsB is the PUB, then a
migration of fsB to another machine will cause
undetermined behavior until the time that the
admin changes the PUBFH on the server.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Tom Haynes <[email-masked]>
Date: Fri, 26 Mar 2010 10:55:43 -0500

> FS_LOCATIONS and similar attributes.

Okay, I like that explanation.

While LOOKUP also can return NFS4ERR_MOVED, it needs
an exception as well, right?

Say that /a/b/c are each a filesystem and we have a FH for 'a'
but not 'b' and we migrate 'b':

PUTFH{a}, LOOKUP{b}, GETATTR{fs_locations}

Also works as:

PUTROOTFH, LOOKUP{a}, LOOKUP{b}, GETATTR{fs_locations}

Further, if we also have a FH for 'c', the following should
work:

PUTFH{c}, LOOKUPP, GETATTR{fs_locations}

If LOOKUPP returns NFS4ERR_MOVED, then it must be the
case that one of the above has to work.

BTW - you could argue that 'c' needs to be moved in the
namespace of the server. There is no way to get to it
any more via the namespace. But those pesky FHs
have a long half life.

---

From: <[email-masked]>
Date: Fri, 26 Mar 2010 12:52:46 -0400

> an exception as well, right?

Depends what you mean by an exception.

> PUTFH{a}, LOOKUP{b}, GETATTR{fs_locations}

LOOKUP(b) does not return MOVED since a is the starting fh and it is
there.

> PUTROOTFH, LOOKUP{a}, LOOKUP{b}, GETATTR{fs_locations}

LOOKUP(a) does not return MOVED since / is the starting fh and it is
there.
LOOKUP(b) does not return MOVED since a is the starting fh and it is
there.

> PUTFH{c}, LOOKUPP, GETATTR{fs_locations}

If c has not moved, then LOOKUPP will not return MOVED since c is the
starting fh and it is there.

> case that one of the above has to work.

LOOKUPP can return MOVED but will not in this case.

It seems to me that all three work.

-----Original Message-----
From: Tom Haynes [mailto:[email-masked]] 
Sent: Friday, March 26, 2010 11:56 AM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] NFS4ERR_MOVED and PUTPUBFH, PUTROOTFH

Also works as:

---

From: Tom Haynes <[email-masked]>
Date: Fri, 26 Mar 2010 12:05:33 -0500

So

PUTFH{b}, LOOKUP{c}

is an example of when LOOKUP can return NFS4ERR_MOVED?

I.e., we have the FH for b and when we go to use it in LOOKUP,
that is when we get the error.

The corresponding case for LOOKUPP is simply:

PUTFH{b}, LOOKUPP

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 26 Mar 2010 13:26:17 -0400

> GETATTR is asking for ops other than things like locations.

Isn't there also an exception for the fsid?

Trond

---

From: <[email-masked]>
Date: Fri, 26 Mar 2010 13:59:11 -0400

Not in the same way.  If you are just asking for fsid and the fs is
moved, then you get MOVED.

However, if you ask for fs_locations (or fs_locations_info or fs_status
for v4.1) then you can also ask for, and should get, fsid,
mounted_on_fileid (and change_policy for v4.1).

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Friday, March 26, 2010 1:26 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] NFS4ERR_MOVED and PUTPUBFH, PUTROOTFH

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 26 Mar 2010 15:08:30 -0400

> mounted_on_fileid (and change_policy for v4.1).

Right. My point was just to ensure we don't forget to put this into
3530bis too...

Cheers
  Trond

---

From: <[email-masked]>
Date: Mon, 29 Mar 2010 14:38:25 -0400

Yes. 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Tom Haynes
Sent: Friday, March 26, 2010 1:06 PM
Cc: [email-masked]
Subject: Re: [nfsv4] NFS4ERR_MOVED and PUTPUBFH, PUTROOTFH

So

PUTFH{b}, LOOKUP{c}

PUTFH{b}, LOOKUPP