Subject: [nfsv4] NFSv4 Referral handling by client

From: Rahul Shah <[email-masked]>
Date: Fri, 14 Nov 2014 19:14:16 -0500

I am working NFSv4 referral implementation on the server side and noticed
one of the occasion client never requested GETATTR with fs_locations even
though server returns NFS4ERR_MOVED.

I am running two set of test.

*TEST_1: works as expected and client is redirected to remote server. *

1) Client mounts to server_a
2) cd /a/b
3) From the server set referrals on directory /a/b/c and point it to some
remote server
4) cd "c"
5) What I observed is, client sends PUTFH (of "b"), LOOKUP (of "c"), GETFH,
GETATTR
6) Server returns NFS4ERR_MOVED for GETFH
7) Clients sends PUTFH (of "b"), LOOKUP (of "c"), GETATTR (requesting
fs_locations)
8) Server returns all referral information and client happily connects to
server_b

*TEST_2 : Client never requests fs_locations*

1) Client mounts to some path on server_a
2) cd /a/b
*3) READDIR "ls" on /a/b which returns filehandle of "c"*
4) From the server set referrals on directory /a/b/c and point it to some
remote server
5) cd to "c"
5) While catching the tcpdumps, I observed client sending requests
    PUTFH (of "c" it got fh from previous READDIR), ACCESS (of "c"),
GETATTR (without fs_locations)
As per RFC 3530, NFS4ERR_MOVED is a valid return code from ACCESS.

>From the server I tried returning NFS4ERR_MOVED for ACCESS. Client just
gives up  and NEVER sends  PUTFH (of "c"), LOOKUP (of "c"), GETATTR
(requesting fs_locations)
I also tried returning NFS4ERR_MOVED for PUTFH. Client again gives up  and
NEVER sends  PUTFH (of "c"), LOOKUP (of "c"), GETATTR (requesting
fs_locations)

Thanks,
Rahul

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 14 Nov 2014 20:26:19 -0500

On Fri, Nov 14, 2014 at 7:14 PM, Rahul Shah <[email-masked]> wrote:
> TEST_1: works as expected and client is redirected to remote server.
>
> TEST_2 : Client never requests fs_locations
>
> 3) READDIR "ls" on /a/b which returns filehandle of "c"
>     PUTFH (of "c" it got fh from previous READDIR), ACCESS (of "c"), GETATTR
> (without fs_locations)

A full description of how NFSv4 referrals are expected to work can be
found in the section on "absent filesystems" in the latest draft of
RFC3530bis:
    http://tools.ietf.org/html/draft-ietf-nfsv4-rfc3530bis-34#section-8.2

If you read through that, you will see that the most likely cause of
the client's confusion is that your server returned a filehandle in
the READDIR operation. Most existing clients expect that a server that
has an absent filesystem will instead implement the behaviour
described in
    http://tools.ietf.org/html/draft-ietf-nfsv4-rfc3530bis-34#section-8.3.2

Cheers
  Trond

---

From: Rahul Shah <[email-masked]>
Date: Fri, 14 Nov 2014 21:40:23 -0500

Hi Trond,

Thanks for your reply. I understand the behavior of 8.3.2 with respect to
READDIR.
However If you look closely in my case TEST_2, Client has performed READDIR
 in "step 3" while the filehandle was still PRESENT at the server. Once
server returns the present filehandle to client, then after few seconds I
am setting a referral on the directory "c" at server side.
So basically client has stale filehandle. In step 6 when the client does
"cd c", it is sending
PUTFH (of "c" it got fh from previous READDIR), ACCESS (of "c"), GETATTR
(without fs_locations)

Even though I return NFS4ERR_MOVED for ACCESS, client never attempted to
send new GETFH or GETATTR (with fs_locations) to the server.

If you want, I can send you the pcap for both the above cases.

Thanks,
Rahul

---

From: Trond Myklebust <[email-masked]>
Date: Sat, 15 Nov 2014 13:13:33 -0500

On Fri, Nov 14, 2014 at 9:40 PM, Rahul Shah <[email-masked]> wrote:
> server returns the present filehandle to client, then after few seconds I am
> setting a referral on the directory "c" at server side.
> So basically client has stale filehandle. In step 6 when the client does "cd
> c", it is sending

The IETF NFSv4 mailing list is not the correct venue for filing bug reports.

However, in clarification: a referral is an object that does not ever
have a filehandle, but is just a placeholder for a missing filesystem.
What you are describing above is considered to be a migration event.
You've handed out a filehandle for an object, and are now saying that
the object pointed to by that filehandle has been migrated to a
different server. Most clients today do not yet have full support for
the NFSv4 migration protocol since dealing with objects that move adds
whole new levels of complexity to state management and recovery.

Cheers
  Trond