Subject: Re: [nfsv4] What is NFSv4 READDIR doesn't return a filehandle....

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 20 Mar 2013 23:53:47 +0000

Please note that I have cc'ed the NFSv4 list over at the IETF.

On Mar 20, 2013, at 3:40 PM, Christopher T Vogan <[email-masked]> wrote:

> All,
> 
> Sorry for this late posting, a coworker was made aware of this thread 
> recently and
> has these replies to make.  Our server implementation is the one being
> complained about in this thread.
> 
> 
> Quoted text is from various entries in this forum.
> 
> 
> Neil Brown:
> ===========
>> Just a thought: while coping with broken servers would not be a good 
> path to
>> follow, detecting protocol violations and reporting an error might be...
>> should the NFS client treat a missing filehandle and a malformed reply?
> 
> The server is allowed to remove an attribute bit from an entry in the 
> readdir
> reply, this is not "broken" nor is the reply "malformed".  The lack of a 
> filehandle in the reply is deterministic.
> 
> 
> 
> Trond Myklebust:
> ================
>>> A customer has come across a server which does not return the 
> filehandle
>>> information (is that allowed?).
>> 
>> The filehandle attribute is a mandatory attribute according to RFC3530, 
> so I believe that the answer is "no".
> 
> Mandatory is described in RFS 3530 as that the server must return the 
> attribute
> on a GETATTR.  (Section 5, page 36).  There is nothing saying that it is
> mandatory to return on a READDIR.  Our server will return the filehandle
> on a LOOKUP/GETATTR every time.

GETATTR and READDIR both return attributes. To be precise, they return
a fattr4.

Looking at Section 15.26.4 (roughly pages 277-279 of the most recent copy)
of 3530bis (3530 is on the way to being replaced), READDIR states:

   The READDIR operation retrieves a variable number of entries from a
   filesystem directory and returns client requested attributes for each
   entry along with information to allow the client to request
   additional directory entries in a subsequent READDIR.
...
   On successful return, the server's response will provide a list of
   directory entries.  Each of these entries contains the name of the
   directory entry, a cookie value for that entry, and the associated
   attributes as requested.  The "eof" flag has a value of TRUE if there
   are no more entries in the directory.

Any client implementation has no way to request that any server implementation
return the filehandle because the filehandle is not an attribute which
can be requested. I.e., it is mandatory.

If the intent was to allow the server to not return a filehandle for READDIR but to
allow it to return one for GETATTR, then it would have been made
OPTIONAL.

Whether or not the client used to work with such a server implementation
or not is immaterial as far as standard compliance is concerned.

If you like, we can clean up the corresponding language in 3530bis to
explicitly state that REQUIRED attributes are indeed required whether
in response to GETATTR or READDIR.

> 
> 
> Trond Myklebust:
> ================
>> My concern is that the client can't objectively judge what constitutes a
>> valid filehandle and what does not until it tries to use it in an RPC
>> call.
> 
> Sure it can.  In the context of this discussion, if the readdir entry
> has the filehandle attribute bit off in the reply, there is no filehandle.
> 
> I would note in the scenario we sent a trace for, the Linux client had 
> a filehandle for the node in question, and "misplaced" it after a 
> readdir to the directory containing that node did not return the 
> filehandle
> for that entry.  So calling the server "broken" is objectively inaccurate.
> 
> I would also note that this problem occurred after we upgraded to RHEL 
> 6.3;
> the prior version did not have this issue, and our server did not change 
> its
> behavior.  My conclusion is that the means to obtain the filehandle was 
> either broken by the client adding logic to assume a filehandle and 
> obliterate 
> the prior value after a readdir reply chose not to return it, or 
> previously 
> had a code path to lookup the node and obtain the filehandle, and removed 
> that code path for some reason.  So again, pointing to server "breakage" 
> is objectively inaccurate.
> 
> There are many references to "brokenness" or "server bugs" in this 
> thread, and I take exception to it from a design and protocol conformance 
> point of view.
> 
> This condition is easily recoverable, as ANY missing attribute on a 
> readdir
> is with a lookup/readdir.  Our decision to not return the FH on a readdir 
> reply under certain narrow conditions is not one of convenience but of 
> limitations of some underlying filesystem types, and if "convenience" is
> to be used as an accusative, it can easily be returned to the client's 
> refusal to deal with the legal withholding of an attribute on a readdir
> reply.
> 
> 
> Signed,
> Steve Huntington
> IBM z/OS NFS
> 
> 
> Christopher Vogan
> Dept. W98 NFS Development & Test
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email-masked]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 21 Mar 2013 00:33:44 +0000

> in response to GETATTR or READDIR.

It might rather be useful to add language to point out that the
"filehandle" attribute SHOULD NOT be used for the GETATTR case. We have
GETFH, and AFAICR, all the language around referrals etc is written with
the assumption that clients _won't_ use GETATTR to retrieve the
filehandle.

Ditto for the case of "rdattr_error", which makes absolutely no sense
whatsoever in a GETATTR request.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Rick Macklem <[email-masked]>
Date: Wed, 20 Mar 2013 21:41:40 -0400 (EDT)

> > allow it to return one for GETATTR, then it would have been made
Well, I don't see any difference between a mandatory attribute and a
recommended attribute that the server claims to support via the
supported_attributes attribute.

I do believe that the server can choose not to return all of the
mandatory and supported recommended attributes in the readdir reply.
(If not, why have a bitmap of returned attributes?)

One example here is the mounted_on_fileid, which some servers choose
to only "support" for server mount points. (The FreeBSD server returns
this attribute for all file handles, setting it to the same value as
fileid for non-mount-points, but I am pretty sure some other servers
do not return mounted_on_fileid for non-mount-points.)

> > in response to GETATTR or READDIR.
I assume that by REQUIRED you are referring to "mandatory attributes".

> whatsoever in a GETATTR request.
And as I mentioned above, can a server choose to return mounted_on_fileid
for only some FHs when it claims to support the attribute? (I think that
is allowed and since I don't see a difference between mandatory and supported
recommended attributes I think the same applies to FH.)

Btw, when a server chooses to not return an FH in the readdir reply although
it was requested, the FreeBSD client "readdirplus" essentially falls back to
a basic "readdir" for the file name (ie. it doesn't prime the name and attribute
caches for it).

rick

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 21 Mar 2013 02:04:59 +0000

> (If not, why have a bitmap of returned attributes?)

That's for dealing with OPTIONAL attributes. Section 5.2 of RFC3530
states:

"A client may ask for any of these attributes to be returned by setting
a bit in the GETATTR request but must handle the case where the server
does not return them."

Section 5.1 says:

"These MUST be supported by every NFS version 4 client and server in
order to ensure a minimum level of interoperability." There are no
exceptions made for READDIR.

> I assume that by REQUIRED you are referring to "mandatory attributes".

I'm using the language from RFC5661 and RFC3530-bis, since that is the
most up-to-date.

> recommended attributes I think the same applies to FH.)

The fileid and mounted_on_fileid are both OPTIONAL attributes, and as
section 5.2 says, the client is required to be able to deal with the
case where the server does not return them. As stated earlier, there is
no such exception in 5.1.

> a basic "readdir" for the file name (ie. it doesn't prime the name and attribute

Implementations are only relevant here in as far as they limit the
protocol changes that RFC3530bis can make. Given that there are clients
out there that assume a strict interpretation of RFC3530, then it is too
late to have RFC3530bis relax that interpretation.

NetApp

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 21 Mar 2013 02:37:35 +0000

> late to have RFC3530bis relax that interpretation.

To put it differently: if you disagree, then please point to the text in
RFC3530, or RFC5661 (or RFC3530bis at this point) that supports your
position that Mandatory/REQUIRED attributes such as "filehandle" may be
ignored by a server in READDIR requests.
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Wed, 20 Mar 2013 23:38:54 -0400 (EDT)

Trond Myklebust wrote:
> > > > On Mar 20, 2013, at 3:40 PM, Christopher T Vogan
> > > > <[email-masked]>
> > > > wrote:
> > > >
> > > > > All,
> > > > >
> > > > > has these replies to make. Our server implementation is the
> > > > > one
> > > > > being
> > > > >> Just a thought: while coping with broken servers would not be
> > > > >> a
> > > > >> good
> > > > > path to
> > > > >> should the NFS client treat a missing filehandle and a
> > > > >> malformed
> > > > >> reply?
> > > > >
> > > > > The server is allowed to remove an attribute bit from an entry
> > > > > in
> > > > > the
> > > > > readdir
> > > > > reply, this is not "broken" nor is the reply "malformed". The
> > > > > lack
> > > > > of a
> > > > >> The filehandle attribute is a mandatory attribute according
> > > > >> to
> > > > >> RFC3530,
> > > > > Mandatory is described in RFS 3530 as that the server must
> > > > > return
> > > > > the
> > > > > attribute
> > > > > on a GETATTR. (Section 5, page 36). There is nothing saying
> > > > > that
> > > > > it is
> > > > Looking at Section 15.26.4 (roughly pages 277-279 of the most
> > > > recent
> > > > copy)
> > > >    The READDIR operation retrieves a variable number of entries
> > > >    from
> > > >    a
> > > >    filesystem directory and returns client requested attributes
> > > >    for
> > > >    each
> > > >    On successful return, the server's response will provide a
> > > >    list
> > > >    of
> > > >    attributes as requested. The "eof" flag has a value of TRUE
> > > >    if
> > > >    there
> > > > If the intent was to allow the server to not return a filehandle
> > > > for
> > > > READDIR but to
Yes, I just read them. I had forgotten (never realized) that for GETATTR
there is the rule that mandatory/required attributes must be returned.

You are also correct that there is no exception for READDIR. In fact,
I don't see any mention of READDIR in Sec. 5.1, 5.2. Both Sec. 5.1 and 5.2
refer specifically to GETATTR when stating whether they must be returned.

Then, when I read the description for READDIR, it seems to say that
the server is to set the rderror attribute if it cannot return all the
requested attributes (no exception for recommended/optional ones) or
fail the entire READDIR if the client hasn't asked for rderror.

Since this isn't what actually happens for mounted_on_fileid, I still
think the same should apply to other attributes requested by the
READDIR request and I don't see that the 5.1, 5.2 rule for GETATTR
applies to READDIR (ie. FH must be returned, but mounted_on_fileid
doesn't need to be).

> > > > If you like, we can clean up the corresponding language in
> > > > 3530bis
> > > > to
I could nitpick and note that rfc3530bis is simply a draft at this point
in time and that RFC3530 is at this time the specification for NFSv4.0.
(The truth is I just haven't bothered reading rfc3530bis. Once it becomes
 an RFC and I get bit by changes, like the "uid # in the owner string"
 change, then I guess I'll have to.;-)

Btw, Sec. 5.1 also states that a client should be able to handle a
server that only supports the mandatory/required attributes.
Have you tested against a server that doesn't support the "fileid"
attribute yet? (I can guarantee that the FreeBSD client will
never work against such a server, so I don't have to worry about
trying to be fully conformant.;-)

rick

> > > GETFH, and AFAICR, all the language around referrals etc is
> > > written
> > > with

---

From: Rick Macklem <[email-masked]>
Date: Thu, 21 Mar 2013 00:04:28 -0400 (EDT)

> can be requested. I.e., it is mandatory.
When I read this, all I think "requested" refers to is whether or not
the client sets the bit "requesting" that attribute.

> allow it to return one for GETATTR, then it would have been made
I don't know what the author's intent was, but since 5.1 only refers
to GETATTR, nothing in RFC3530 indicates that the same rule applies
to READDIR (ie. MANDATORY-->must always return it for READDIR).

> in response to GETATTR or READDIR.
Personally, I think it should be "clarified" that for READDIR, the server
can choose to not reply any of the attributes. That allows the server implementor
more flexibility and since the old spec. didn't clearly require the REQUIRED
ones, I don't see why the new spec. should, unless it in impractical for clients
to allow this flexibility (and that is where the implementation evidence is
relevent, imho).

rick

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 21 Mar 2013 04:04:45 +0000

> refer specifically to GETATTR when stating whether they must be returned.

No, but section 5.5 states very specifically that the "filehandle"
attribute is "primarily for readdir requests". What is the meaning of a
"Mandatory/Required" attribute if it doesn't apply to the primary (read
"only useful") case?

Ditto question for rdattr_error, which (as indicated earlier) _only_
applies to READDIR.

> fail the entire READDIR if the client hasn't asked for rderror.

I accept that.

> applies to READDIR (ie. FH must be returned, but mounted_on_fileid

Where do you see support for this in the spec?

> trying to be fully conformant.;-)

I'm not certain that we've tested against a server that only supports
the mandatory attributes, but I do see some protocol problems there. Not
least, there is a known problem with determining the current value for
the change attribute; hence my proposal of the "change_attr_type"
OPTIONAL attribute for NFSv4.2. Unless someone has a valid objection,
I'm planning on proposing that attribute for REQUIRED status in
NFSv4.3...

NetApp

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 21 Mar 2013 09:37:13 -0400

> do not return mounted_on_fileid for non-mount-points.)

That doesn't sound like traditional unix readdir behavior, and isn't the
behavior described by the spec; looking at 5661 5.8.2.23 (haven't
compared other rfcs):

	"If the server detects that there is no mounted point at the
	target file object, then the value for mounted_on_fileid that it
	returns is the same as that of the fileid attribute."

Also, the supported_attrs attribute is a filesystem-wide attribute.  I
don't think we generally allow attributes to vary between "supported"
and "unsupported" on a single filesystem.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 21 Mar 2013 10:15:47 -0400 (EDT)

> and "unsupported" on a single filesystem.
Well, I'm pretty sure some server did (maybe it has been changed more
recently) would only return mounted_on_fileid in the Readdir reply
if it was at a mount point.

I remember because it broke the FreeBSD client and I had to tweak
the code.

I'll try and remember to stick something in the FreeBSD client to
spot this for the next Bakeathon.

rick