Subject: [nfsv4] client-side TLS security policies

From: Chuck Lever III <[email-masked]>
Date: Sun, 13 Jun 2021 16:16:35 +0000

Hi-

Our client prototypes are preparing to support several security
policies that might apply to using TLS. Implementers would like to
get some feedback about what might be reasonable policy choices to
provide to administrators, and perhaps start the conversation about
what needs to appear in TLS-related security considerations.

The base assumption is that the server is responsible for enforcing
security policies, and the client's job is to try to interoperate
with as few unexpected denials-of-service as possible. Some basic
policies the client might implement:

"none" -- no TLS is used on the connection, even if the server
supports TLS. This might be used when an administrator wants the
best performance.

"opportunistic" -- TLS is used whenever both the client and server
support it, but if the server does not, the client switches to
"none". This provides the best interoperability.

"conservative" -- TLS is used only when the client and server
support it _and_ the server requires TLS to access its content.
This provides good interoperability, but it also means a server
might be less overwhelmed if TLS connections are resource-intensive.
It would depend on the client being able to detect when the server
requires the use of TLS.

"required" -- if TLS is not supported by the server or a TLS
session cannot be established (based on TLS-related policies
configured elsewhere) then access is not permitted (eg, a mount is
not permitted to succeed).

I prefer opportunistic more than conservative:
 - TLS will only get less expensive over time
 - best-effort means TLS would be employed in more cases

Also, I kind of feel like opportunistic and conservative are simply
two implementation choices for what is effectively the same use
case: the default best-effort setting. A client implementor can
choose whichever suits their preferences. Do these need to be
separate choices?

Are there any common cases I've missed?

--
Chuck Lever

---

From: Rick Macklem <[email-masked]>
Date: Sun, 13 Jun 2021 22:08:07 +0000

> choose whichever suits their preferences. Do these need to be
I figured I'd add a little, related to implementation experience for
FreeBSD.

FreeBSD currently implements "none" and "required".
Adding support for "opportunistic" will be easy (for the FreeBSD
code), since it is the same as "required" except the mount is allowed
even when the TLS probe fails.
Adding support for "conservative" is harder. It's something like:
- Block new RPC requests.
- Wait for outstanding RPCs to receive replies or time out waiting
  for replies.
- Do the TLS probe/handshake.
- Allow new RPC requests. (Requires creation of a new TCP connection
  if RPCs that timed out need to be retried.)

Having said this, I tend to prefer "conservative", since I am concerned
about server overheads if TLS is used extensively.
I can't provide real benchmarking data, but here is what I observe
for the simple test hardware I have. (Two used laptops Dell call
Latitude 6420s. 4 cores with AESNI and 1Gbps net ports.)
- Copying a 1Gbyte file the first time (not cached in server nor client).
   Without TLS             With TLS
   65Mbytes/sec           50Mbytes/sec
- Same copy, but after umount/mount (cached in server memory).
  Without TLS              With TLS
  110Mbytes/sec          110Mbytes/sec
- Same copy, but mounted locally on server (cached on server side
  and avoids net wire).
  Without TLS              With TLS
  350Mbytes/sec         200Mbytes/sec
  --> For this test "With TLS", two of the 4 cores were 80-90% busy
        during the test. (The KTLS in FreeBSD has one kernel thread for
        each core and assigns sockets using KTLS to the threads in a
        round robin fashion. In this case one core/KTLS thread is doing
        the encryption/decryption for the server end and another
        core/KTLS thread is doing the encryption/decryption for the client end.)

In summary, running at 1Gbps, wire speed is feasible for basic
hardware, but a couple of these connections will use up a core
when actively transferring data, at least for my limited test case.
--> For heavy use of RPC-over-TLS, the server will either need
      lots of cores or offload hardware, I think?
      FreeBSD already has drivers for assorted hardware offloads,
      but none of these have been tested/benchmarked for
      RPC-over-TLS yet.  (I'm trying to coax people who have the
       hardware into doing so, but haven't yet succeeded.;-)

As such, I will probably implement "opportunistic" since it is
easy to do and, maybe, "conservative", since I believe it could
be useful. I do believe that "opportunistic" and "conservative"
are separate and potentially useful cases.

rick

--
Chuck Lever

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Mon, 14 Jun 2021 18:55:37 +0200 (CEST)

Hi Chuck et al.,

I think we should avoit any situation, data accidentally transferred
unprotected. Thus for me there are two options (with additional flavors):

- server enforces TLS, any other connections are rejected.
  The server might required client certificates, in addition.

- client requires TLS, any other connections are rejcted.

Any automatic/opportunistic downgrades to the security is
not an option.  I guess, this is the "required".

The "none" mode can per ip based export option:

/data 192.168.17.0/24(rw,sec=tls)
/data localhost(rw,no_root_squash,sec=sys)

Best Regards,
   Tigran.

----- Original Message -----
> From: "Chuck Lever III" <[email-masked]>
> To: "NFSv4" <[email-masked]>
> Sent: Sunday, 13 June, 2021 18:16:35
> Subject: [nfsv4] client-side TLS security policies

---

From: Rick Macklem <[email-masked]>
Date: Fri, 18 Jun 2021 23:53:24 +0000

>  The server might required client certificates, in addition.
This is done on the FreeBSD server, either globally or "per file
system", based on client IP address.
So, unless you feel this should always be enforced, I think we agree.

The problem I see with always requiring NFS-with-TLS is the increase in
server overheads/connection that might result in server performance
degradation.

I believe that the cases Chuck described were for a client mount configuration
and assumed that the server would enforce the above, as required by the
server configuration.

>not an option.  I guess, this is the "required".
The "conservative" option would not downgrade, but would only
upgrade to NFS-with-TLS when the server required it, as I understand it.

The "opportunistic" option would only upgrade, but it might
leave the connection non-upgraded without indicating that no
upgrade to NFS-with-TLS has been done.  This is why I do not
particularly like it.

>/data localhost(rw,no_root_squash,sec=sys)
I think Chuck intended the cases to be client mount options
(we had an off-list discussion before he posted to the list).

However, yes, the FreeBSD server uses export options
(different export file syntax) to indicate various cases:
- In FreeBSD, there is
  "tls" - TLS required
  "tlscert" - TLS with a verifiable client certificate required.
  "tlscertuser" - TLS with a verifiable certificate required.
      The certificate will be used for "TLS identity squashing",
      where the POSIX credentials used for all RPCs are
     derived from the certificate and the credentials in the
     RPC request message are ignored.
     
These are separate from the "sec=" option, since NFS-with-TLS
could require any of the security flavor credentials plus the above.

rick

Best Regards,
   Tigran.

---

From: Chuck Lever III <[email-masked]>
Date: Sat, 19 Jun 2021 15:23:58 +0000

> On Jun 18, 2021, at 7:53 PM, Rick Macklem <[email-masked]> wrote:
> 
> Mkrtchyan, Tigran wrote:
> upgrade to NFS-with-TLS has been done.

See Section 7.1 of rpc-tls-11. It REQUIRES an implementation
provide an audit log of security mode selection, and that
an RPC client exposes the security settings in effect to
all consumers.

> This is why I do not particularly like it.

Some implementations might choose to provide this as they
could be required to not break interoperability when adding
new features. "Hey my mounts suddenly stopped working.
Why are you telling me I have to introduce another mount
option I didn't need before?"

> (we had an off-list discussion before he posted to the list).

Yep, we are discussing client-side settings in this thread.

At the top, I stated that I presume that the server is
responsible for enforcing security policy. If a server houses
sensitive content, it is responsible for ensuring all access
to that content is protected. Thus, if the administrator wants
to require TLS, the server's security policy can force clients
to use it.

For QUIC, as I understand it, TLS will be used no matter what.
Since that's a new transport, we don't have backwards-
compatibility concerns as we do with existing transports like
TCP, and the only client policy that makes sense for QUIC is
tls=required (until QUIC supports modulation of transport
layer security policy).

> could require any of the security flavor credentials plus the above.

I expect the Linux NFS server will have a similar array of
tls policy settings, fwiw; "tlscertuser" might be added
later, though.

--
Chuck Lever

---

From: Rick Macklem <[email-masked]>
Date: Sat, 19 Jun 2021 16:31:42 +0000

> On Jun 18, 2021, at 7:53 PM, Rick Macklem <[email-masked]> wrote:
[stuff snipped]
>an RPC client exposes the security settings in effect to
Yes, I suppose that covers it. When I read this in the draft I'll
admit I thought "log file" and the "nfsstat -m" command.
Neither of these get much attention from joe user on their laptop.
However, if it's a big "WARNING..." splattered across the screen.
The above working is pretty strong.

>option I didn't need before?"
Good point. If you make either "opportunistic" or "conservative" the
default, you avoid this.

>>> /data 192.168.17.0/24(rw,sec=tls)
Btw, one additional question here is whether or not you can "trust
the client's IP address to identify the client".

At this time, the FreeBSD server has a global flag that enables a
"client must present a verifiable certificate and it must have a "DNS"
field within SubjectAltName which matches the reverseDNS name
for the client's IP address". I did not include a check for a matching
"IP" field within SubjectAltName, in part because the OpenSSL library
function did the former and not the latter. (I need to re-read the RFC
that describes how client's verify server hosts, since this is essentially
the same.)

Since, this is needed for cases where clients might spoof their IP address,
I could only figure out how to enable/disable it globally.

I haven't yet decided if this is a good approach.
What do others think?

rick

--
Chuck Lever