Subject: Re: [nfsv4] correct error reply when RPC-over-TLS is required and is not being used?

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Thu, 3 Jun 2021 10:36:44 +0200 (CEST)

Hi Rick et al.,

----- Original Message -----
> From: "Rick Macklem" <[email-masked]>
> To: "Dave Noveck" <[email-masked]>
> Cc: "NFSv4" <[email-masked]>
> Sent: Friday, 28 May, 2021 04:00:27
> Subject: Re: [nfsv4] correct error reply when RPC-over-TLS is required and is not being used?

> David Noveck wrote:
>>It seems to me that two separate server policies are being discussed.
>>
>>The first policy is the one that Rick initially introduced, that of a server
>>that requires use of RPC-over-TLS for accessing >particular server file
>>systems.  Despite Rick's reservations about returning NFS4ERR_WRONGSEC, I
>>believe that is a good >choice for that case and should be used.  This is not
>>currently documented but it should be in the new nfsv4 security >document being
>>done in connection with rfc5661bis.
>>
>>The second policy concerns a server that requires use of RPC-over-TLS or use of
>>RPC-over-TLS with client authentication >for all nfsv4 access.  That is dealt
>>with by dropping the connection connection, with appropriate error codes.
>>Those codes >are not mentioned in the current RFC candidate but we do not need
>>to do a last-minute change to that document.  The >needed error codes could be
>>added in the new nfsv4 security document or in a document defining use of
>>RPC-over-TLS by >NFS.   Having those updating the rpc-tls document is a
>>possibility but appears not to be necessary.
> Yes. I'd like to thank David for wading through the initial discussion and
> providing the above summary.
> I'll add a little more, using what FreeBSD has implemented sofar.
> 
> In the current FreeBSD server implementation there are three variants checked:
> 1 - RPC-over-TLS is not required.
> 2 - RPC-over-TLS is using at least Server-only Host Authentication, is required.
> 3 - RPC-over-TLS using Mutual Host Authentication, is required
> and as David describes above, these variants are applied two ways by the server
> configuration.
> A - Globally to all NFSv4 operations.
> B - Per file system, to NFSv4 operations that set and/or use file handle(s)

I am pretty sure that once TLS is established the client  can't send plain packets anymore.
So for 'Per file system' option client will need to have a per file system connection.

Tigran.

> 
> For B, since David thinks it is ok, I am fine with replying NFS4ERR_WRONGSEC
> when #2 or #3 is required
> for the file/handle and is not being done.
> I think that Secinfo/Secinfo_no_name should be exempt from #2, #3 and permitted
> to be performed.
> (More on Secinfo/Secinfo_no_name later.)
> 
> It sounds like David thinks new "enum auth_stat" values can be defined for #2,
> #3 and do not need
> to be specified in the current RPC-over-TLS draft.
> As such, it sounds like the RPC layer MSG_REJECTED with AUTH_ERROR and one of
> these two new
> "enum auth_stat" values is the preferred reply for A.
> AUTH_NEEDS_TLS - for #2
> AUTH_NEEDS_TLS_MUTUAL_HOST - for #3
> (Please suggest better names. I am terrible at naming stuff.)
> 
> Now, what about Secinfo/Secinfo_no_name?
> Although adding a case to secinfo4 might be nice, I don't think that can be done
> without
> creating a new minor version?
> NFSv4.2 could be extended with new Secinfo_plus/Secinfo_no_name_plus operations,
> but that seems
> like a lot of work and does not handle NFSv4.0 or 4.1.
> 
> Another possibility would be to state the following:
> - If a client not using RPC-over-TLS receives a NFS4ERR_WRONGSEC reply when
> using
>   a credential flavor listed in the Secinfo/Secinfo_no_name reply for the
>   file/handle,
>   that implies #2 or #3.
> - If a client using RPC-over-TLS with Server-only Host Authentication receives a
> NFS4ERR_WRONGSEC
>  reply when using a credential flavor listed in the Secinfo/Secinfo_no_name reply
>  for the file/handle,
>  that implies #3.
> I think this would allow the NFSv4 client to differential a NFS4ERR_WRONGSEC
> reply due to #2 or #3
> from one sent due to use of the wrong credential flavor in the RPC header.
> 
> Finally, although this is the NFSv4 working group, I suppose we cannot
> completely ignore NFSv3.
> FreeBSD uses B for all NFSv3 RPCs. (ie. Per file system based on file handle.)
> However, since NFS4ERR_WRONGSEC does not exist in NFSv3, FreeBSD currently uses
> NFS3ERR_ACCES.
> I think that using the RPC layer error replies used for A would be preferred to
> NFS3ERR_ACCES (or any
> other NFS3ERR_xxx value), since it gives the client explicit information w.r.t.
> what caused the RPC to fail.
> 
> So, what do others think? rick
> 
> On Wed, May 26, 2021, 7:27 PM Rick Macklem
> <[email-masked]<mailto:[email-masked]>> wrote:
> Chuck Lever III wrote:
>>> On May 25, 2021, at 7:02 PM, Rick Macklem
>>> <[email-masked]<mailto:[email-masked]>> wrote:
>>>
>>> Hi,
>>>
>>> When I implemented RPC-over-TLS for the NFSv4 server, I included
>>> exports options that allowed the server sysadmin to require use of
>>> RPC-over-TLS on a per-filesystem basis.
>>>
>>> At this time, if a client attempts to perform an RPC without using
>>> RPC-over-TLS for such a case, the NFSv4 server replies NFS4ERR_WRONGSEC.
>>> I now think returning NFS4ERR_WRONGSEC may not be a good choice.
>>> 1 - NFS4ERR_WRONGSEC has a fairly specific meaning that does not really
>>>     include this case.
>>> 2 - A client might attempt a Secinfo/Secinfo_no_name operation after
>>>     receiving NFS4ERR_WRONGSEC and that will fail as well.
>>> 3 - It will be replied for operations which do not list it as a reply error
>>>     in RFC5661.
>>>
>>> So, what should the server do?
>>> When testing yesterday, Chuck mentioned he expected the connection
>>> was going to be dropped. If the server does this, I am concerned that a
>>> non-RPC-over-TLS aware client will just do TCP reconnects over and over...
>>> (I think Chuck agrees that an error reply is preferred, but I'm sure he will
>>> clarify this if I am incorrect.)
>>
>>I was thinking of the case where the TLS handshake fails. I thought we had
>>some language that required the server to disconnect in that case, but I
>>could be misremembering.
> Ok, I see this in the draft...
>   If spurious
>   traffic appears on a TCP connection between the initial clear-text
>   AUTH_TLS probe and the TLS session handshake, receivers MUST discard
>   that data without response and then SHOULD drop the connection.
> 
> Ok, so for your test, the unencrypted ExchangeID would be considered
> "spurious traffic". As such, it looks like dropping the connection is the
> correct
> server response for this case.
> 
> To be honest, for my implementation, all this is buried inside OpenSSL's
> SSL_accept() call, which is called once the STARTTLS reply is sent to the
> client.
> --> I can implement "close the TCP connection" when SSL_accept() returns
>      an error in the server, although that applies to both "spurious traffic"
>      and other handshake failures.
> 
>> The client would be aware of the handshake failure,
>> and could report that to the consumer.
> Yes, and then it presumably should drop the TCP connection, so it shouldn't
> be a problem for the server to do so.
> 
>>
>>The client in this test isn't even doing a handshake. I hadn't considered
>>that case.
> Actually, the draft addresses what your test did, since it sent "spurious
> traffic"
> after the STARTTLS probe. I just didn't realize that.
> 
> The cases I was thinking of are related to Sec. 4.2 of the draft:
>   Server-only Host Authentication
>      In this type of deployment, the client can authenticate the server
>      host using the presented server peer TLS identity, but the server
>      cannot authenticate the client.  In this situation, RPC-over-TLS
>      clients are anonymous.  They present no globally unique identifier
>      to the server peer.
> Suppose the server wants Mutual Host Authentication, but the client
> provides Server-only Host Authentication. (Or, the client does not
> provide a X.509 certificate and the server wants on, if you prefer.)
> 
>   Mutual Host Authentication
>      In this type of deployment, the client possesses an identity that
>      is backed by a trusted entity (e.g. a pre-shared key or a
>      certificate validated with a certification path).  As part of the
>      TLS handshake, both peers authenticate using the presented TLS
>      identities.  If authentication of either peer fails, or if
>      authorization based on those identities blocks access to the
>      server, the peers MUST reject the association.
> Here it states "reject the association".
> My question could be rephrased as "how does the server do the reject?".
> 
> There is also the case where the client does not do any STARTTLS probe,
> but the server configuration requires that RPC-over-TLS be used.
> 
>>The SECINFO result is:
>>
>> typedef secinfo4 SECINFO4resok<>;
>>
>>Seems like it could include RPC_AUTH_TLS in that list?
> Well, for a case where the server requires TLS, it would
> have to allow at least ExchangeID, CreateSession, some PutFH variant and
> Secinfo or Secinfo_no_name to all be done before the STARTTLS probe.
> For the case where it requires Mutual Host Authentication, the above would
> have to be allowed for at least Server-only Host Authentication via TLS.
> 
> This does not seem to be a good plan to me. For most cases, I think the server
> will not want to allow any non-NULL RPCs to succeed for cases where the
> server configuration requires the client to use RPC-over-TLS.
> 
>>> I had thought that, since the NFS layer decides if RPC-over-TLS is required,
>>> that an NFS layer error was appropriate. I am now thinking that, since
>>> RPC-over-TLS is in the RPC layer, that an RPC layer error reply would be
>>> more appropriate (plus no current NFS4ERR_xxx seems to fit well).
>>> --> How about MSG_REJECTED with AUTH_ERR plus some value for
>>>       "enum auth_stat"?
>>>      - AUTH_TOOWEAK might be considered a reasonable fit.
>>>      Alternately, adding new entries to "enum auth_stat" might be better.
>>>      (I hesitate to suggest that the draft be revised at this point, but it does
>>>       seem that having a well defined way for a server to tell a client that
>>>       "RPC-over-TLS is required" could be useful.)
>>>     Multiple new values might be useful.
>>>     For example:
>>>     AUTH_NEEDS_TLS - client must use RPC-over-TLS
>>>     AUTH_NEEDS_TLSWITHX509 - client must use RPC-over-TLS plus it
>>>                                                       must provide a X.509 certificate during handshake
> I suppose this might be named something like
>         AUTH_NEEDS_TLS_MUTUAL_HOST - to use the terminology in the draft
> 
> Note that RFC5531 allows additional "enum auth_stat" values be added through
> the IANA registry.
> 
> rick
> 
>>
>> So, what do others think?
>>
>> rick
>>
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]<mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> --
> Chuck Lever
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]<mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]<mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Rick Macklem <[email-masked]>
Date: Thu, 3 Jun 2021 13:55:23 +0000

>So for 'Per file system' option client will need to have a per file system connection.
For the current FreeBSD server, the errors will be sent over TLS after the probe and
TLS handshake has succeeded.
--> FreeBSD uses OpenSSL for the handshake and the server doesn't know if Mutual
       Host Authentication (aka if certificate has verified) until after the handshake completes.
      --> The "requires Mutual Host Authentication" errors will often be sent over TLS.

If the error reply is received over TLS, then the client would have to create a new TCP
connection, then probe etc. to fix the problem.
--> I suspect this will require something like "acquire new certificate" for the client
      to resolve.

For the case where the client is not using TLS and then hits one of these errors
at a file system boundary, it wouldn't need a new connection, but would have to
do a TLS probe/handshake to gain access to the file system.

I don't see a need for a separate TCP connection in general, but for the upgrade
from Server-only Authentication->Mutual Host Authentication then, yes, the
client will have to create a new connection. It shouldn't need to be an additional
one, since requiring Server-only Authentication and not allowing Mutual Host
Authentication does not make sense.
--> The new connection can replace the current one.

Does this make sense? rick

Tigran.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 3 Jun 2021 14:10:57 +0000

Rick wrote:
[stuff snipped]
>Tigran wrote:
Just to clarify...
"for the cases where the error is detected after TLS probe/handshake" should be appended
to the above.

rick

Tigran.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 3 Jun 2021 14:31:18 +0000

I misplaced David's email w.r.t. returning AUTH_TLS plus possibly a pseudo-flavor
to indicates "needs TLS with Mutual Host Authentication" in the Secinfo/Secinfo_no_name
SECINFO4resok<> reply list, but I do think doing so is a good idea. (I see "flavor" is defined as the
values for auth_flavor.)

For my server code (considered experimental at this time), I'm adding
AUTH_TLS_NEEDS_MUTUAL_HOST = 8,
to the "enum auth_flavor".

Putting these in SECINFOresok is a little odd, since the other entries in the list are "choose one of these auth_flavors",
whereas having AUTH_TLS or AUTH_TLS_NEEDS_MUTUAL_HOST in the list indicates
TLS is required plus use one of the other entries in the list is to be used for RPC credentials.
However, I'm ok with this if others are?

rick

---

From: David Noveck <[email-masked]>
Date: Sat, 5 Jun 2021 06:46:15 -0400

On Thu, Jun 3, 2021, 10:31 AM Rick Macklem <[email-masked]> wrote:

> SECINFO4resok<> reply list, but I do think doing so is a good idea.

Having done a fair amount of it myself, I don't think misplacing emails is
a good idea :-)

(I see "flavor" is defined as the
> values for auth_flavor.)

IANA would prefer that they maintain this list and I don't have a problem
with that.

> to the "enum auth_flavor".

Because other potential transports have transport-level mutual
authentication facilities (e.g. rpc-over-rdma v2 has one), the approach I
am working on for my forthcoming security draft is somewhat different.

In my scheme, to do what you want, SECINFO would have entries with the
flavor AUTH_TLS and the pseudo-flavor AUTH_XP_SECURE (or AUTH_XP_PEERAUTH).

If the server wanted to indicate that AUTH_SYS is only usable on transports
providing peer authentication it would combine the pseudo-flavors
AUTH_SYS_XP and AUTH_XP_PEERAUTH.

> Putting these in SECINFOresok is a little odd,

Not a problem for me.

since the other entries in the list are "choose one of these auth_flavors",
> used for RPC credentials.

The not-so-odd alternative seems prone to an unfortunate combinatorial
explosion.

Good to know.  We'll come back to this and make a decision when we have a
draft to discuss.

---

From: Rick Macklem <[email-masked]>
Date: Sat, 12 Jun 2021 00:32:15 +0000

David Noveck wrote:
> On Thu, Jun 3, 2021, 10:31 AM Rick Macklem wrote:
>>I misplaced David's email w.r.t. returning AUTH_TLS plus possibly a
>>pseudo-flavor to indicates "needs TLS with Mutual Host Authentication"
>>in the Secinfo/Secinfo_no_name SECINFO4resok<> reply list,
>>but I do think doing so is a good idea.
>
>Having done a fair amount of it myself, I don't think misplacing
>emails is a good idea :-)
>
>>(I see "flavor" is defined as the values for auth_flavor.)
>
>IANA would prefer that they maintain this list and I don't have a problem with that.
>
>Because other potential transports have transport-level mutual authentication
>facilities (e.g. rpc-over-rdma v2 has one), the approach I am working on for my
>forthcoming security draft is somewhat different.
>
>In my scheme, to do what you want, SECINFO would have entries with the flavor
>AUTH_TLS and the pseudo-flavor AUTH_XP_SECURE (or AUTH_XP_PEERAUTH).
>
>providing peer authentication it would combine the pseudo-flavors AUTH_SYS_XP
>and AUTH_XP_PEERAUTH.
For this case, did you mean the flavors list?
AUTH_SYS
AUTH_XP_PEERAUTH

and that the client could use any transport that provides peer authentication
vs
AUTH_SYS
AUTH_TLS
AUTH_XP_PEERAUTH

would require RPC-over-TLS with Mutual host authentication?

Someday, there might also be QUIC, which I understand includes TLS,
although I'm not sure if it always uses TLS?

Now, since the "enum auth_stat" returned with an RPC message AUTH_ERR
reply is a single value (not a list), do you think we need to try and cover
most combinations as separate entries added to auth_stat?

Just trying to make sure I understand your plan, rick

Not a problem for me.

The not-so-odd alternative seems prone to an unfortunate combinatorial explosion.

Good to know.  We'll come back to this and make a decision when we have a draft to discuss.

rick

---

From: Lars Eggert <[email-masked]>
Date: Sat, 12 Jun 2021 07:58:20 +0300

> although I'm not sure if it always uses TLS?

The current version 1 of QUIC (RFC9000+) always uses TLS1.3. Future versions of QUIC may use different versions of TLS or even non-TLS security.

-- 
Sent from a mobile device; please excuse typos.

---

From: David Noveck <[email-masked]>
Date: Sat, 12 Jun 2021 02:40:07 -0400

On Fri, Jun 11, 2021, 8:32 PM Rick Macklem <[email-masked]> wrote:

> authentication
> flavor

No.

> and that the client could use any transport that provides peer

Yes.

> would require RPC-over-TLS with Mutual host authentication?

Yes.

> Someday, there might also be QUIC,

I think the RFC is published but I am not sure what else is necessary to
allow implementation of NFSV4 over RPC-over-TLS over QUIC.  Possibly only
publication of the RPC-OVER-TLS RFC (someday) is all that is required but
I'm not sure.

which I understand includes TLS,
> although I'm not sure if it always uses TLS?

"Always" covers a very long time.  BTW, I've been reminded of that dealing
with emails about the end of Unix time (in 2038) and the end of nfsv3 time
(early in the twenty-second century).  Sigh!

> most combinations as separate entries added to auth_stat?

Yes but I've tried to keep that limit by focus on capabilities rather than
specific implementations.

> Just trying to make sure I understand your plan, rick

To the degree I understand it, it is to focus on getting an initial
security draft out in time for IETF111.

---

From: Chuck Lever III <[email-masked]>
Date: Sat, 12 Jun 2021 16:12:41 +0000

Hi-

> On Jun 12, 2021, at 2:40 AM, David Noveck <[email-masked]> wrote:
> 
> I think the RFC is published but I am not sure what else is necessary to allow implementation of NFSV4 over RPC-over-TLS over QUIC.  Possibly only publication of the RPC-OVER-TLS RFC (someday) is all that is required but I'm not sure.

I think we've confused some people by calling it "RPC-over-TLS".
Those people now think that RPC-over-TLS is a transport. It's
not, it's a security model. "RPC-with-TLS" might have been a better
choice. I'd subsequently like to use "NFS-with-TLS" when discussing
NFS and TLS. We should note from the outset that NFS-with-TLS
would apply to any network transport protocol on which a TLS session
can be established: UDP, TCP, or QUIC.

To address Rick's question directly, draft-ietf-nfsv4-rpc-tls
provides a basis for using TLS when RPC actually runs over a QUIC
transport. That document provides certificate management guidance, an
ALPN, and good security policies that apply to any transport that
might use TLSv1.3 or later, including QUIC. What QUIC won't need
is the RPC_AUTH_TLS probe defined in that document, since I believe
there are already mechanisms built into QUIC to identify when TLS
is available and can to be used.

What draft-ietf-nfsv4-rpc-tls does not supply is the definitions
needed to use QUIC as an RPC transport. In other words, something
like RFC 8166, except it can be much much shorter. IMO we will need
a document that specifies these items:

 - RPC message framing (streams? datagrams / transactions?)
 - possibly how to deal with multiple streams per connection
 - maybe a GSS channel binding
 - an IANA request for one or two new netids

It's on my to-do list, but I haven't yet read the newly-published
RFC 9000 cluster to see if there is anything else needed. I hope
Lars can provide some advice to get us started.

I'm not sure we need to allocate meeting time for technical
discussion at this point, but it might be wise to get "official"
rough consensus that QUIC is now a transport that should convey
SunRPC. Does that need a WG charter update? We can probably cover
that in 10 minutes.

--
Chuck Lever