Subject: [nfsv4] Can NFSv4.2 operations be optional on a per server file system basis?

From: Rick Macklem <[email-masked]>
Date: Sun, 10 Oct 2021 14:07:48 +0000

Hi,

I'm guessing the answer is "no", but I figured I'd ask just to make sure,
since RFC7862 states that operations are optional, but does not clarify
beyond that.

So, if an NFSv4.2 server exports two file systems, where one of them can
perform the operation and the other cannot, is the server allowed to
reply NFS4ERR_NOTSUPP for the operation when attempted on files
in the file system that cannot perform the operation?

Thanks, rick
ps: In FreeBSD, UFS can do Allocate, but ZFS cannot.

---

From: Thomas Haynes <[email-masked]>
Date: Sun, 10 Oct 2021 11:16:12 -0700

> On Oct 10, 2021, at 7:07 AM, Rick Macklem <[email-masked]> wrote:
> 
> Hi,
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

Hi Rick,

The answer is “no”.

But the question is very interesting. NFS4ERR_INVAL might be the most appropriate of the allowed errors, but pretty sure you do not want to be returning that.

Of the not allowed errors, I think that NFS4ERR_OFFLOAD_DENIED is closest to what you want. You actually want an NFS4ERR_ALLOCATE_DENIED. It would mean that ALLOCATE was supported, but not allowed for the given file.

And I have no clue how easy it would be to drive that into the standard’s body any more. :-)

Tom

---

From: Rick Macklem <[email-masked]>
Date: Sun, 10 Oct 2021 19:32:56 +0000

>> https://www.ietf.org/mailman/listinfo/nfsv4
That's what I thought.

>But the question is very interesting. NFS4ERR_INVAL might be the most appropriate of the allowed errors, but >pretty sure you do not want to be returning that.
Yea, EINVAL is the correct response for posix_fallocate(), but the Linux fallocate() uses ENOTSUPP and uses
EINVAL for lots of other things. Since returning EINVAL for a ZFS export was flagged as a bug by testers
during last week's Bakeathon, I'd agree that EINVAL wouldn't work.

>Of the not allowed errors, I think that NFS4ERR_OFFLOAD_DENIED is closest to what you want. You actually want an >NFS4ERR_ALLOCATE_DENIED. It would mean that ALLOCATE was supported, but not allowed for the given file.
>
I think I'll just have to NFS4ERR_NOTSUPP for Allocate unless the server knows that all file systems
(and DSs for the pNFS case) can do it.

Thanks, rick
ps: Tom, is hs-anvil going to stick around on the Bat VPN for a while? I have a client side race/deadlock
      when testing against it and, if I come up with a fix, it would be nice to be able to test against it?

Tom

---

From: David Noveck <[email-masked]>
Date: Fri, 15 Oct 2021 07:03:12 -0400

I think the right error to return in the case in which allocate is
implemented by the server but not supported on a particular fs is
NFS4ERR_SERVERFAULT. The case we are talking about is an error that is not
otherwise provided for, just as the definition of that error states.

On Sun, Oct 10, 2021, 3:33 PM Rick Macklem <[email-masked]> wrote:

> >But the question is very interesting. NFS4ERR_INVAL might be the most
> appropriate of the allowed errors, but >pretty sure you do not want to be
> returning that.
> >Of the not allowed errors, I think that NFS4ERR_OFFLOAD_DENIED is closest
> to what you want. You actually want an >NFS4ERR_ALLOCATE_DENIED. It would
> mean that ALLOCATE was supported, but not allowed for the given file.
> >

---

From: Rick Macklem <[email-masked]>
Date: Fri, 15 Oct 2021 23:43:10 +0000

David Noveck wrote:
>I think the right error to return in the case in which allocate is implemented by the >server but not supported on a particular fs is NFS4ERR_SERVERFAULT. The case we >are talking about is an error that is not otherwise provided for, just as the >definition of that error states.
Just to clarify. Are you thinking that a return of NFS4ERR_SERVERFAULT to
an Allocate operation should be "normal expected behaviour" for a server
with a mix of exported file systems, where some can do Allocate and others
not?

rick
ps: It should be easy for the FreeBSD client to map NFS4ERR_SERVERFAULT
      to EINVAL, which is the error return expected from posix_fallocate()
      for this case.

On Sun, Oct 10, 2021, 3:33 PM Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:
Thomas Haynes wrote:
>> On Oct 10, 2021, at 7:07 AM, Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> Hi,
>>
>> [email-masked]<mailto:[email-masked]>
That's what I thought.

Thanks, rick

Tom

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: [email-masked] (J. Bruce Fields)
Date: Tue, 19 Oct 2021 14:05:11 -0400

For what it's worth, I looked at the Linux knfsd code, and it looks like
it will in fact return NFS4ERR_NOTSUPP in this case.

That's probably not the only case.  E.g. (again from just looking at the
code) I suspect that's what we do you request an xattr from a filesystem
that lacks xattr support (even though GETXATTR is supported and might
succeed given a filehandle on another export).

Without testing right now, I *think* the Linux client just passes that
back to the application (probably as EOPNOTSUPP).  It doesn't give up on
ever using FALLOCATE against that server again.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 21 Oct 2021 00:10:08 +0000

Well, what do others think of this?

> The answer is “no”.
Well Tom, when I first red RFC7862 I assumed that the "optional features" were
"per server", but when I look in the RFC I cannot see that stated.
In Sec. 1.2:
   NFSv4.2 is a superset of NFSv4.1, with all of the new features being
   optional.  As such, NFSv4.2 maintains the same compatibility that
   NFSv4.1 had with NFSv4.0.  Any interactions of a new feature with
   NFSv4.1 semantics is described in the relevant text.

It clearly states that all new features are optional. It does not state if the
option is "per server" or  "per server file system" (or "per server file" for that
matter).

Then in Sec. 1.3:
   A major goal of the enhancements provided in NFSv4.2 is to take
   common local file system features that have not been available
   through earlier versions of NFS and to offer them remotely.  These
   features might
This kinda hints that the new features are based on local file systems
in the server, so supporting the new feature "per server file system"
doesn't seem too much of a stretch?

I could not see any statement that optional features were "per server"
in the RFC (but I might have just missed it?).

> ever using FALLOCATE against that server again.
So, it appears that the Linux knfsd server is shipping with optional features
on a "per server file system" basis and that the Linux NFSv4.2 client handles
that.

The FreeBSD NFSv4.2 client is slightly broken, in that it has an "optimization"
where, if it sees a NFS4ERR_NOTSUPP reply, it marks the operation not supported
and does not attempt it again. This optimization could easily be removed for the
next release and appears to be broken w.r.t. extant Linux knfsd servers, so I should
remove it anyhow.

So, could we just clarify that NFSv4.2 optional features are "per server file system"?

rick

--b.

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 20 Oct 2021 18:50:29 -0700

> On Oct 20, 2021, at 5:10 PM, Rick Macklem <[email-masked]> wrote:
> 

Honestly I went with what I *thought* I intended. The best supporting evidence is here:

15.1.1.5.  NFS4ERR_NOTSUPP (Error Code 10004)

   Operation not supported, either because the operation is an OPTIONAL
   one and is not supported by this server or because the operation MUST
   NOT be implemented in the current minor version.

Which implies that the if OPTIONAL the operation is either entirely supported by the server or not.

What is a client supposed to do if it tries ALLOCATE and the server accepts it -- it tries again and the server replies NFS4ERR_NOTSUPP.  Is the support based on time of day? Did something change on the server?

You and I may know it is by filesystem, by how is that exposed to the client?

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 20 Oct 2021 22:18:34 -0400

On Wed, Oct 20, 2021 at 06:50:29PM -0700, Thomas Haynes wrote:
>    Operation not supported, either because the operation is an
>    OPTIONAL one and is not supported by this server or because the
>    operation MUST NOT be implemented in the current minor version.
> 
> What is a client supposed to do if it tries ALLOCATE and the server
> accepts it -- it tries again and the server replies NFS4ERR_NOTSUPP.

Return 0 to the application in the first case, and ENOTSUP in the
second?  Is that a problem?

> Is the support based on time of day? Did something change on the
> server?
> 

Unless the client's trying to do some optimization, as in Rick's case,
it doesn't sound likely to cause a problem.

I could see how EINVAL could cause practical problems, though.  E.g. an
application probably knows how to handle NOTSUPP--it can fall back on
writing zeroes, or something.  EINVAL, though, sounds like something's
just broken, and it's not clear how to recover.

Ditto a tar application getting NOTSUPP on getxattr.

You're probably right on the letter of the spec.

But we definitely need a way to indicate these features aren't supported
on a per-filesystem basis, and it sounds like NFS4ERR_NOTSUPP is likely
to work in practice, so maybe just going with that is the least bad
option....

--b.

---

From: David Noveck <[email-masked]>
Date: Thu, 21 Oct 2021 01:02:12 -0400

Section 15.1.1.5 is a problem, although, as Bruce points out, it may not be
problem returning NOTSUPP in practice.

I'm thinking about updating this definition in rfc5661bis-03 together with
dealing with a bunch of the remaining erratta reports and a few updates to
the security discussion to respond to the changes in security-03.
I would submit this right after security-03, even though the big split up
of the bis is still a ways in the future.

 I think we can spend a few minutes discussing this at the meeting next week

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 21 Oct 2021 11:20:51 -0700

> On Oct 20, 2021, at 10:02 PM, David Noveck <[email-masked]> wrote:
> 
> Section 15.1.1.5 is a problem, although, as Bruce points out, it may not be problem returning NOTSUPP in practice.
> 

Perhaps the solution is to add a per-file system attribute to state whether or not it supports the optional feature.

> I'm thinking about updating this definition in rfc5661bis-03 together with dealing with a bunch of the remaining erratta reports and a few updates to the security discussion to respond to the changes in security-03.
> I would submit this right after security-03, even though the big split up of the bis is still a ways in the future.
> 
> On Wed, Oct 20, 2021, 10:18 PM J. Bruce Fields <[email-masked] <mailto:[email-masked]>> wrote:
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>

---

From: Rick Macklem <[email-masked]>
Date: Thu, 21 Oct 2021 21:01:09 +0000

>Perhaps the solution is to add a per-file system attribute to state whether or not it supports the optional feature.
That would be a lot more bother to implement in FreeBSD, since the nfsd only knows if an operation
(VOP_xxx() call) is supported by calling it, and there is no convenient file VOP_ALLOCATE() could be
called using it to test for support.
--> Yes, some new FreeBSD specific pathconf values could be added to do this, but it would be a lot more
       work than just translating the error return for not supported (typically EINVAL in FreeBSD's case) to
       NFS4ERR_xxx.

It also doesn't deal with the extant Linux knfsd servers that are already shipping to users.

rick

On Wed, Oct 20, 2021, 10:18 PM J. Bruce Fields <[email-masked]<mailto:[email-masked]>> wrote:

--b.