Subject: [nfsv4] correct error reply when RPC-over-TLS is required and is not being used?

From: Rick Macklem <[email-masked]>
Date: Tue, 25 May 2021 23:02:45 +0000

Hi,

When I implemented RPC-over-TLS for the NFSv4 server, I included
exports options that allowed the server sysadmin to require use of
RPC-over-TLS on a per-filesystem basis.

At this time, if a client attempts to perform an RPC without using
RPC-over-TLS for such a case, the NFSv4 server replies NFS4ERR_WRONGSEC.
I now think returning NFS4ERR_WRONGSEC may not be a good choice.
1 - NFS4ERR_WRONGSEC has a fairly specific meaning that does not really
     include this case.
2 - A client might attempt a Secinfo/Secinfo_no_name operation after
     receiving NFS4ERR_WRONGSEC and that will fail as well.
3 - It will be replied for operations which do not list it as a reply error
     in RFC5661.

So, what should the server do?
When testing yesterday, Chuck mentioned he expected the connection
was going to be dropped. If the server does this, I am concerned that a
non-RPC-over-TLS aware client will just do TCP reconnects over and over...
(I think Chuck agrees that an error reply is preferred, but I'm sure he will
 clarify this if I am incorrect.)

I had thought that, since the NFS layer decides if RPC-over-TLS is required,
that an NFS layer error was appropriate. I am now thinking that, since
RPC-over-TLS is in the RPC layer, that an RPC layer error reply would be
more appropriate (plus no current NFS4ERR_xxx seems to fit well).
--> How about MSG_REJECTED with AUTH_ERR plus some value for
       "enum auth_stat"?
      - AUTH_TOOWEAK might be considered a reasonable fit.
      Alternately, adding new entries to "enum auth_stat" might be better.
      (I hesitate to suggest that the draft be revised at this point, but it does
       seem that having a well defined way for a server to tell a client that
       "RPC-over-TLS is required" could be useful.)
     Multiple new values might be useful.
     For example:
     AUTH_NEEDS_TLS - client must use RPC-over-TLS
     AUTH_NEEDS_TLSWITHX509 - client must use RPC-over-TLS plus it
                                                       must provide a X.509 certificate during handshake

So, what do others think?

rick

---

From: Chuck Lever III <[email-masked]>
Date: Wed, 26 May 2021 14:51:02 +0000

> On May 25, 2021, at 7:02 PM, Rick Macklem <[email-masked]> wrote:
> 
> Hi,
> 

I was thinking of the case where the TLS handshake fails. I thought we had
some language that required the server to disconnect in that case, but I
could be misremembering. The client would be aware of the handshake failure,
and could report that to the consumer.

The client in this test isn't even doing a handshake. I hadn't considered
that case.

The SECINFO result is:

 typedef secinfo4 SECINFO4resok<>;

Seems like it could include RPC_AUTH_TLS in that list?

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

--
Chuck Lever

---

From: Rick Macklem <[email-masked]>
Date: Wed, 26 May 2021 23:26:42 +0000

Chuck Lever III wrote:
>could be misremembering.
Ok, I see this in the draft...
   If spurious
   traffic appears on a TCP connection between the initial clear-text
   AUTH_TLS probe and the TLS session handshake, receivers MUST discard
   that data without response and then SHOULD drop the connection.

Ok, so for your test, the unencrypted ExchangeID would be considered
"spurious traffic". As such, it looks like dropping the connection is the correct
server response for this case.

To be honest, for my implementation, all this is buried inside OpenSSL's
SSL_accept() call, which is called once the STARTTLS reply is sent to the client.
--> I can implement "close the TCP connection" when SSL_accept() returns
      an error in the server, although that applies to both "spurious traffic"
      and other handshake failures.

> The client would be aware of the handshake failure,
Yes, and then it presumably should drop the TCP connection, so it shouldn't
be a problem for the server to do so.

>The client in this test isn't even doing a handshake. I hadn't considered
Actually, the draft addresses what your test did, since it sent "spurious traffic"
after the STARTTLS probe. I just didn't realize that.

The cases I was thinking of are related to Sec. 4.2 of the draft:
   Server-only Host Authentication
      In this type of deployment, the client can authenticate the server
      host using the presented server peer TLS identity, but the server
      cannot authenticate the client.  In this situation, RPC-over-TLS
      clients are anonymous.  They present no globally unique identifier
      to the server peer.
Suppose the server wants Mutual Host Authentication, but the client
provides Server-only Host Authentication. (Or, the client does not
provide a X.509 certificate and the server wants on, if you prefer.)

   Mutual Host Authentication
      In this type of deployment, the client possesses an identity that
      is backed by a trusted entity (e.g. a pre-shared key or a
      certificate validated with a certification path).  As part of the
      TLS handshake, both peers authenticate using the presented TLS
      identities.  If authentication of either peer fails, or if
      authorization based on those identities blocks access to the
      server, the peers MUST reject the association.
Here it states "reject the association".
My question could be rephrased as "how does the server do the reject?".

There is also the case where the client does not do any STARTTLS probe,
but the server configuration requires that RPC-over-TLS be used.

>Seems like it could include RPC_AUTH_TLS in that list?
Well, for a case where the server requires TLS, it would
have to allow at least ExchangeID, CreateSession, some PutFH variant and
Secinfo or Secinfo_no_name to all be done before the STARTTLS probe.
For the case where it requires Mutual Host Authentication, the above would
have to be allowed for at least Server-only Host Authentication via TLS.

This does not seem to be a good plan to me. For most cases, I think the server
will not want to allow any non-NULL RPCs to succeed for cases where the
server configuration requires the client to use RPC-over-TLS.

>>                                                       must provide a X.509 certificate during handshake
I suppose this might be named something like
         AUTH_NEEDS_TLS_MUTUAL_HOST - to use the terminology in the draft

Note that RFC5531 allows additional "enum auth_stat" values be added through
the IANA registry.

rick

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 27 May 2021 12:09:53 -0400

It seems to me that two separate server policies are being discussed.

The first policy is the one that Rick initially introduced, that of a
server that requires use of RPC-over-TLS for accessing particular server
file systems.  Despite Rick's reservations about returning
NFS4ERR_WRONGSEC, I believe that is a good choice for that case and should
be used.  This is not currently documented but it should be in the new
nfsv4 security document being done in connection with rfc5661bis.

The second policy concerns a server that requires use of RPC-over-TLS or
use of RPC-over-TLS with client authentication for all nfsv4 access.  That
is dealt with by dropping the connection connection, with appropriate error
codes.  Those codes are not mentioned in the current RFC candidate but we
do not need to do a last-minute change to that document.  The needed error
codes could be added in the new nfsv4 security document or in a document
defining use of RPC-over-TLS by NFS.   Having those updating the rpc-tls
document is a possibility but appears not to be necessary.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 28 May 2021 02:00:27 +0000

David Noveck wrote:
>The first policy is the one that Rick initially introduced, that of a server that requires use of RPC-over-TLS for accessing >particular server file systems.  Despite Rick's reservations about returning NFS4ERR_WRONGSEC, I believe that is a good >choice for that case and should be used.  This is not currently documented but it should be in the new nfsv4 security >document being done in connection with rfc5661bis.
>
>The second policy concerns a server that requires use of RPC-over-TLS or use of RPC-over-TLS with client authentication >for all nfsv4 access.  That is dealt with by dropping the connection connection, with appropriate error codes.  Those codes >are not mentioned in the current RFC candidate but we do not need to do a last-minute change to that document.  The >needed error codes could be added in the new nfsv4 security document or in a document defining use of RPC-over-TLS by >NFS.   Having those updating the rpc-tls document is a possibility but appears not to be necessary.
Yes. I'd like to thank David for wading through the initial discussion and providing the above summary.
I'll add a little more, using what FreeBSD has implemented sofar.

In the current FreeBSD server implementation there are three variants checked:
1 - RPC-over-TLS is not required.
2 - RPC-over-TLS is using at least Server-only Host Authentication, is required.
3 - RPC-over-TLS using Mutual Host Authentication, is required
and as David describes above, these variants are applied two ways by the server configuration.
A - Globally to all NFSv4 operations.
B - Per file system, to NFSv4 operations that set and/or use file handle(s)

For B, since David thinks it is ok, I am fine with replying NFS4ERR_WRONGSEC when #2 or #3 is required
for the file/handle and is not being done.
I think that Secinfo/Secinfo_no_name should be exempt from #2, #3 and permitted to be performed.
(More on Secinfo/Secinfo_no_name later.)

It sounds like David thinks new "enum auth_stat" values can be defined for #2, #3 and do not need
to be specified in the current RPC-over-TLS draft.
As such, it sounds like the RPC layer MSG_REJECTED with AUTH_ERROR and one of these two new
"enum auth_stat" values is the preferred reply for A.
AUTH_NEEDS_TLS - for #2
AUTH_NEEDS_TLS_MUTUAL_HOST - for #3
(Please suggest better names. I am terrible at naming stuff.)

Now, what about Secinfo/Secinfo_no_name?
Although adding a case to secinfo4 might be nice, I don't think that can be done without
creating a new minor version?
NFSv4.2 could be extended with new Secinfo_plus/Secinfo_no_name_plus operations, but that seems
like a lot of work and does not handle NFSv4.0 or 4.1.

Another possibility would be to state the following:
- If a client not using RPC-over-TLS receives a NFS4ERR_WRONGSEC reply when using
   a credential flavor listed in the Secinfo/Secinfo_no_name reply for the file/handle,
   that implies #2 or #3.
- If a client using RPC-over-TLS with Server-only Host Authentication receives a NFS4ERR_WRONGSEC
  reply when using a credential flavor listed in the Secinfo/Secinfo_no_name reply for the file/handle,
  that implies #3.
I think this would allow the NFSv4 client to differential a NFS4ERR_WRONGSEC reply due to #2 or #3
from one sent due to use of the wrong credential flavor in the RPC header.

Finally, although this is the NFSv4 working group, I suppose we cannot completely ignore NFSv3.
FreeBSD uses B for all NFSv3 RPCs. (ie. Per file system based on file handle.)
However, since NFS4ERR_WRONGSEC does not exist in NFSv3, FreeBSD currently uses NFS3ERR_ACCES.
I think that using the RPC layer error replies used for A would be preferred to NFS3ERR_ACCES (or any
other NFS3ERR_xxx value), since it gives the client explicit information w.r.t. what caused the RPC to fail.

So, what do others think? rick

On Wed, May 26, 2021, 7:27 PM Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:
Chuck Lever III wrote:
>> On May 25, 2021, at 7:02 PM, Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> Hi,
>>

rick

> [email-masked]<mailto:[email-masked]>

--
Chuck Lever

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: David Noveck <[email-masked]>
Date: Sun, 30 May 2021 08:54:44 -0400

On Thu, May 27, 2021, 10:00 PM Rick Macklem <[email-masked]> wrote:

> server that requires use of RPC-over-TLS for accessing >particular server
> NFS4ERR_WRONGSEC, I believe that is a good >choice for that case and should
> nfsv4 security >document being done in connection with rfc5661bis.
> >
> use of RPC-over-TLS with client authentication >for all nfsv4 access.  That
> codes.  Those codes >are not mentioned in the current RFC candidate but we
> do not need to do a last-minute change to that document.  The >needed error
> defining use of RPC-over-TLS by >NFS.   Having those updating the rpc-tls
> For B, since David thinks it is ok, I am fine with replying
> NFS4ERR_WRONGSEC when #2 or #3 is required

Yes.  I'll start by defining them in draft-dnoveck-nfsv4-security-00, which
I expect to be out before the next wg meeting in July.

As such, it sounds like the RPC layer MSG_REJECTED with AUTH_ERROR and one
> of these two new
> HAUTH_NEEDS_TLS_MUTUAL_HOST - for #3

For now, I intend to use your suggestions and incorporate improvements
suggested by the wg in later drafts.

> creating a new minor version?

The XDR cannot be changed. However, We can add new values using the
existing default case.  It helps that AUTH_TLS is an already-defined flavor
value, but I think we will be able to define pseudo-flavors that could
appear in a secinfo4.

NFSv4.2 could be extended with new Secinfo_plus/Secinfo_no_name_plus
> operations, but that seems

I think we can do what is needed without a new operation.

>   reply when using a credential flavor listed in the
> Secinfo/Secinfo_no_name reply for the file/handle,
>   that implies #3.

I'd prefer we define how we use the secinfo results to make this
distinction and intend to do so.  I will keep your cases in mind but I also
want to address situations in which AUTH_SYS can be validly used but only
when mutual authentication is in effect.  This will be addressed in
security-00 but I intend to send out preliminary text for the secinfo piece
in the next few weeks.

> Finally, although this is the NFSv4 working group, I suppose we cannot

I suppose not. Sigh!  Our ability to change/fix it is quite limited,
although we can address its issues in RPC.

Chuck has been mentioning a possible NFS-over-RPC-TLS document which would
discuss NFSv3.   He and I will have to co-ordinate since some of the issues
we have been discussing could appear in either document.

FreeBSD uses B for all NFSv3 RPCs. (ie. Per file system based on file
> handle.)

I agree.  Given that an NFSv3 connection is single-fs, the A-style
treatment is appropriate.

> https://www.ietf.org/mailman/listinfo/nfsv4U_
>

---

From: Chuck Lever III <[email-masked]>
Date: Sun, 30 May 2021 18:04:38 +0000

> On May 27, 2021, at 10:00 PM, Rick Macklem <[email-masked]> wrote:
> 
> David Noveck wrote:

Section 9 of RFC 5531 already defines:

enum auth_stat {
    AUTH_OK             = 0, /* success */
    /*
     * failed at remote end
     */
    AUTH_BADCRED        = 1, /* bad credential (seal broken)   */
    AUTH_REJECTEDCRED   = 2, /* client must begin new session  */
    AUTH_BADVERF        = 3, /* bad verifier (seal broken)     */
    AUTH_REJECTEDVERF   = 4, /* verifier expired or replayed   */
    AUTH_TOOWEAK        = 5, /* rejected for security reasons  */
    /*
     * failed locally
     */
    AUTH_INVALIDRESP    = 6, /* bogus response verifier        */
    AUTH_FAILED         = 7, /* reason unknown                 */

Maybe one or two of these might be appropriated for cases #2 or #3
if the TLS handshake failure status doesn't provide this information.

> from one sent due to use of the wrong credential flavor in the RPC header.

Before proceeding too far, we should examine what kind of detail
can be provided by TLS itself when a handshake fails. It might
help the client distinguish between #2 and #3 with no changes
needed in RPC, but I'm not yet familiar with handshake failure
modes.

> Finally, although this is the NFSv4 working group, I suppose we cannot completely ignore NFSv3.

NFSv3's MNT procedure looks a bit like NFSv4's SECINFO.

mountres3 MOUNTPROC3_MNT(dirpath) = 1;

struct mountres3_ok {
    fhandle3 fhandle;
    int auth_flavors<>;
};

union mountres3 switch (mountstat3 fhs_status) {
    case MNT_OK:
        mountres3_ok mountinfo;
    default:
        void;
};

It should be possible to return RPC_AUTH_TLS in the counted array
of auth_flavors.

> other NFS3ERR_xxx value), since it gives the client explicit information w.r.t. what caused the RPC to fail.

And again, the TLS handshake failure mode might provide enough
information to enable the client to display an informative error
message so that the administrator can fix configuration or
certificate issues.

Maybe there's a way to model this after RPCGSS -- after all, a
client might use krb5 but access can fail anyway if the user isn't
authenticated as an authorized principal.

--
Chuck Lever

---

From: Chuck Lever III <[email-masked]>
Date: Sun, 30 May 2021 18:09:09 +0000

> On May 30, 2021, at 8:54 AM, David Noveck <[email-masked]> wrote:
> 
> I suppose not. Sigh!  Our ability to change/fix it is quite limited, although we can address its issues in RPC.
> 
> Chuck has been mentioning a possible NFS-over-RPC-TLS document which would discuss NFSv3.   He and I will have to co-ordinate since some of the issues we have been discussing could appear in either document.

I had envisioned that the NFS-over-TLS document would discuss both
NFSv4 and NFSv3, but I can see that there would be substantial
overlap with nfsv4-security. It might be sensible to discuss all
the NFSv4 TLS-related issues in nfsv4-security, and let the
NFS-over-TLS doc cover only the legacy NFS versions.

--
Chuck Lever