Subject: Re: [nfsv4] NFS over TLS for floating clients

From: Rick Macklem <[email-masked]>
Date: Wed, 1 Apr 2020 19:26:09 +0000

Benjamin Kaduk wrote:
>On Mon, Mar 30, 2020 at 02:16:46AM +0000, Rick Macklem wrote:
>> Benjamin Kaduk wrote:
>> >On Fri, Mar 06, 2020 at 10:47:29PM +0000, Rick Macklem wrote:
>> >> Craig Everhart wrote:
>> >> >I think the problem statement contains the issue here:
>> >> >> Here's an example scenario:
>> >> >> - The client is a laptop that wants to mount a server from "anywhere" using
>> >> >>   TLS, so that data is encrypted on the wire.
>> >> >>   The server understandably wants to use "mutual authentication" to determine
>> >> >>   that the client is indeed one that is allowed to mount the server.
>> >> >What attribute is the server looking for that will "allow" the server to be mounted?
>> >> >And can it be done anonymously, which this "anywhere" laptop really wants to be?
>> >> >
>> >> >Note that on-the-wire encryption will be done even if there is no client certificate--as long as >the server has a certificate (or if the client/server are using a pre-shared key, which is a little >further afield).
>> >> Yes, but I wouldn't be comfortable with "any" client mounting from anywhere.
>> >
>> >The NFS server can decide which X.509 CAs it will trust to issue
>> >certificates that it wants to use.  Whether this is a "well-known global
>> >CA" or a local one is just a matter of configuration, and doesn't really
>> >need to be nailed down at a protocol level.
>> >
>> >> >If the client owns a domain, somehow, then the PKI machinery is available to it; a certificate >will allow it to prove that it is able to speak for that domain name.  Is that what's being >checked as this stuff about "allowing" the mount?
>> >> Well, I didn't envision the laptop owning a domain, but I suppose that might work.
>> >> (Since the laptop's IP address/DNS host name wouldn't match this domain, it sounds
>> >>  a bit sketchy, but so is the use of a certificate signed by the NFS admin using a site
>> >>  local CA, see below.)
>> >
>> >There's a little bit of a mismatch between what TLS client certificates say
>> >and what TLS server certificate say, at least in the common class of usage.
>> >If you read RFC 6125, you note that it *only* talks about naming TLS server
>> >certificates, because the process for that is along the lines of "the
>> >client figures out in some mechanism specified by the application protocol
>> >a name that it's trying to contact, and the server has to present a
>> >certificate that matches that desired name".  For client certificates, on
>> >the other hand, the server may not have some preconceived notion of what
>> >identity the client should be proving, and so the X.509 certificate serves
>> >solely as an identifier, not a verification of identity.  In this sense,
>> >depending on the server's authentication policy, there's not necessarily a
>> >need for the client to have a well-known IP address or DNS name to validate
>> >against.
>> A case I've coded up that may or may not be allowed by the current draft is:
>> - Optionally, if the client presents a certificate to the server that verifies and
>>    where the CN is of the form "user@dns_domain", then this "user" is translated
>
>Oof, could I persuade you to go with a subjectAltName instead?  Putting
>structure within (or really, using at all) the legacy subject CN is
>generally disrecommended.
I've changed it to use the otherName field of subjectAltName.
I haven't figured out a nice way to input this to openssl, but inputting:
DNS:mylaptop.my.domain,otherName:1.2.4.5.6;UTF8:[email-masked]
works.

>>    to a set of credentials used for all RPCs on the TCP connection instead of
>>    what is   provided in the RPC request header. I put "user" in quotes because
>>    I think you can argue that this is the identity of the client host and the server
>>    chooses to assign credentials to that client identity.
>>    (Basically, I'd argue that, for laptops, the line between "user" and "host" is fuzzy
>>     and normally one and the same.)
>
>This is a unique-enough usage that I'd recommend a dedicated OID for the
"otherName" SAN type.

Any quick hints on how I get an OID?

>>> As for the current draft, it says:
>>> (end of sec. 4.2)
>>> In either of these modes, RPC user authentication is not affected by
>>>    the use of transport layer security.  When a client presents a TLS
>>>    peer identity to an RPC server, the protocol extension described in
>>>    the current document provides no way for the server to know whether
>>>    that identity represents one RPC user on that client, or is shared
>>>    amongst many RPC users.
>>
>> True, although there is the case where there is only one user on the client
>> and that is known by the server administrator.
>>
>>  Therefore, a server implementation must not
>>    utilize the remote TLS peer identity for RPC user authentication.
>>
>> I would argue that this is a server implementation choice and does not
>> affect the protocol.
>
>I'm inclined to agree with Chuck -- this is passing information across
>layers of the protocol stack in a way that is not well understood, and the
>spec forbids it because of the potential for hazardous consequences.
True. I am going to respond to Chuck's comments in a separate post
(since they are not here).

rick

> (in sec. 7.3)
> In light of the above, it is RECOMMENDED that when AUTH_SYS is used,
>    every RPC client should present host authentication material to RPC
>    servers to prove that the client is a known one.  The server can then
>    determine whether the UIDs and GIDs in AUTH_SYS requests from that
>    client can be accepted.
>
> I would argue this is what the above case does.

I would frame it differently, in that the client is authenticating itself
and the server is (possibly implicitly) restricting that client to only
allow AUTH_SYS authentication to a specific subset of users (possibly only
one).  But it's the client that authenticates with host credentials and
asserting user identity, and the server trusting the client to faithfully
represent the user identity.

> The use of TLS does not enable RPC clients to detect compromise that
>    leads to the impersonation of RPC users.  Also, there continues to be
>    a requirement that the mapping of 32-bit user and group ID values to
>    user identities is the same on both the RPC client and server.
>
> Doing the above solves/avoids this problem.
>
> I do feel the above case can be useful (optionally, not always) and I would
> appreciate comments w.r.t. it.
> If others think it is a reasonable approach, it would be nice if the draft allowed
> it, but I can leave it in the FreeBSD code, noting it is "non-RFC conformant".
>
> >> >I think that there's a problem with the NFS server wanting to know much about the client, >unless it's limited to a DNS name.
> >> Well, my thinking is that, if the NFS admin. is running a site local CA and the client has
> >> a certificate signed by that site local CA, then the certificate must have been issued by
> >> the NFS admin. I don't think the contents of the certificate is of much use in this case,
> >> just the fact it was signed by the NFS admin. using the site local CA.
> >> Creating the site local CA just seems easier and cheaper than trying to register a
> >> domain name for a laptop and getting a certificate signed by a trusted CA.
> >> After all, some of the motivation for doing NFS over TLS is that NFS admins. don't
> >> seem willing to run KDCs, etc and use sec=krb5p. To get it widely adopted, I think
> >> it needs to be relatively simple to set up and use.
> >>
> >> Obviously, if the laptop is compromised and the certificate signed by the site local CA
> >> is copied to a different system, then that system could mount the server.
> >> (I think exactly the same risk exists for a certificate for a domain owned by the laptop
> >>  and signed by a trusted CA. ie. Copy the certificate to a different system and it
> >>  would allow that system to mount the server.)
> >> Whether or not that is an acceptable risk is up to the NFS admin, I think?
> >
> >It's also possible to put the private key for the certificate in a
> >protected hardware module on the laptop, so that the key cannot be copied
> >elsewhere (and thus the copied certificate would not be useful).
> I'll admit I know nothing about protected hardware modules, but it is easy
> to have the client's private key encrypted so that the user must enter a
> passphrase when they start the daemon to unencrypt the key.
> --> A bit of a bother, but not that bad for a laptop.
>    (Of course, if the client laptop is compromised, the typed in passphrase could
>     be captured, but this handles the simple case of the key, certificate being copied
>     to a different laptop.)

"PKCS #11 hardware token" is probably the best search term to start with.
(There is at least one implementation that uses software to emulate this
interface as well, for testing.)

> Also, for my CN set to "user@dns_domain" case, the certificate only allows access
> to the NFS server as "user" and is less useful if compromised.

It's also possible to define a new certificate purpose via X.509 extension
for NFS use.  I'd have to reflect a bit more on whether that's the right
thing to do here (and possibly consult the LAMPS WG as well).

-Ben

> Thanks for your comments, rick
>
> > rick
> > Craig
> >
> > On Fri, Mar 6, 2020 at 2:08 PM Trond Myklebust <[email-masked]<mailto:[email-masked]>> wrote:
> > On Thu, 5 Mar 2020 at 22:06, Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:
> > >
> > > Hi,
> > >
> > > As I am working through implementation of NFS over TLS, I have run into
> > > a couple of things related to certificates.
> > > Here's an example scenario:
> > > - The client is a laptop that wants to mount a server from "anywhere" using
> > >   TLS, so that data is encrypted on the wire.
> > >   The server understandably wants to use "mutual authentication" to determine
> > >   that the client is indeed one that is allowed to mount the server.
> > >
> > > Ok, so now how do you get a certificate for the client that the server can
> > > reasonably verify?
> > > --> After a discussion over on a FreeBSD mailing list, it sounds like the easy
> > >       (maybe only?) way to do this is for the NFS server admin. to run a site local
> > >       CA and generate certificates against that.
> > >       - Although I'm sure there are other ways, you can create a site local CA
> > >          certificate with two openssl commands and sign a certificate for a client
> > >          with two more openssl commands.
> > >      Then the server can verify the certificate using the CAcert that was used to
> > >      sign the client's certificate.
> >
> > It really boils down to the question of who do you trust to assert
> > what information.
> >
> > If you own a domain, you can usually buy SSL certificates for it that
> > assert a given name within that domain. As long as you trust the major
> > CA vendors not to sell such a certificate to someone who does not own
> > the rights to the domain, then you might have your server use that
> > chain of trust to verify that this is indeed a trusted laptop. You
> > might decide to compare the full name appearing in the certificate to
> > a trusted list, or maybe just verify that the domain or subdomain info
> > matches a list of trusted domains or subdomains. Yes, you can do this
> > more cheaply by creating your own site-local CA, but it is essentially
> > the same process of setting up a chain of trust for your source of
> > information and then of asserting that information in a certificate.
> >
> > > Now, when I read the sections around Page 6 of the draft...
> > >    Mutual Host Authentication
> > >       In this type of deployment, the client possesses a unique global
> > >       identity (e.g., a certificate).  As part of the TLS handshake,
> > >       both peers authenticate using the presented TLS identities.  If
> > >       authentication of either peer fails, or if authorization based on
> > >       those identities blocks access to the server, the client
> > >       association MUST be rejected.
> > > For the above, the client does not possess a unique global identity,
> > > it might more correctly be called a "site local identity" that the server
> > > can authenticate.
> > > Is the "unique global identity" requirement necessary? It seems to me
> > > that a site local CA issued certificate might be appropriate.
> > > (RFC 5280 page 12, second (a) item seems to allow site local CA
> > >  certificates).
> >
> > It might be better to word in terms of the language of chains of
> > trust. "...the client possesses an identity (e.g. a certificate) that
> > is backed by a trusted entity."
>
> I think there's a few different ways to word this, and don't expect us to
> be particularly sensitive to the details of the wording.  As Tigram(IIRC?)
> noted upstream, the combination of issuer, key, and subject name really
> will be globally unique, so the current wording doesn't seem technically
> wrong, albeit an unusual way to phrase it.
>
> -Ben
>
> > > Also, w.r.t. server certificates, the draft says:
> > >    Each RPC server that supports RPC-over-TLS MUST possess a unique
> > >    global identity (e.g., a certificate that is signed by a well-known
> > >    trust anchor).  Such an RPC server MUST request a TLS peer identity...
> > > I wonder if the above must be a MUST?
> > > For example, I have an NFS server at home. It doe not have a well known
> > > fixed DNS address (residential internet connection, where it sits behind
> > > a NAT gateway where the address stays the same most of the time).
> > > --> If I want to mount this server from anywhere, I do want to use TLS
> > >       so that data is encrypted on the wire. Although it would be nice for
> > >       the laptop to be able to verify the server's identity, I don't see how I
> > >       can get a certificate for it from a well known trust anchor. I can live
> > >       with it having a self-signed certificate.
> > >
> > > Also, although an NFS server administrator can get a certificate from a
> > > well known trust anchor, it might cost $$ or it might not be easy. (Lets
> > > Encrypt expects to be able to use ACME on a web site or similar to issue
> > > a certificate, if I understand their setup?)
> > >
> > > Acquiring a certificate from a "well known trust anchor" might be a
> > > significant effort that will discourage use of TLS. (Again, you can easily
> > > create a self-signed certificate with a couple of openssl commands.)
> > > --> Maybe this could be a recommendation instead of a MUST and
> > >        the choice of accepting a self-signed certificate be left up to the
> > >        client via configuration?
> > >
> > > So, what do others think about this? rick
> > >
> > >
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [email-masked]<mailto:[email-masked]>
> > > https://www.ietf.org/mailman/listinfo/nfsv4
> >
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]<mailto:[email-masked]>
> > https://www.ietf.org/mailman/listinfo/nfsv4
> >
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Rick Macklem <[email-masked]>
Date: Wed, 1 Apr 2020 20:36:08 +0000

Chuck Lever wrote:
> On Mar 29, 2020, at 10:16 PM, Rick Macklem <[email-masked]> wrote:
>>
>I don't fully understand this statement, but the protocol described in
>draft-ietf-nfsv4-rpc-tls forces this choice to be "does not utilize
>the TLS peer identity for RPC user authentication."
Btw, when I said "does not affect the protocol" I was referring to what goes
on the wire. The only thing that changes in the content of the client's certificate,
which is not covered by this document (except for where the DNS name goes).
--> Actually I am surprised that Ben and others accept putting the DNS name in
      the CN field of subjectName and aren't insisting it be in the DNS field
      of the subjectAltName. (I, personally, don't care.)

>IMO we cannot
>relax this requirement in this protocol, though it can be revisited if
>RPC is made to handle user authentication with certificates: I will
>be discussing that possibility during our interim meetings in April.
>
As you'll see below, I am not talking about user authentication, I am
referring to what credentials are being used for an RPC, which I believe
is a server configuration issue.
FreeBSD NFS server have had a "-mapall" option which says "use these
credentials for all RPCs for a very long time (since the 1980s, I think?).

>A connection/TLS session uses exactly one pair of certificates for its
>entire lifetime: a server cert and a client cert (let's set aside the
>"no client cert" case for a moment).
>
>After the TLS session has been established, in general a client is free
>to send multiple RPC user identities on that connection. Users have no
>association with the certificate the client used to establish that
>session.
Yes, I understand all of the above.

>
>The certificates here are used to establish secure communication between
>two hosts. They are not intended to be used for authenticating RPC user
>identities, period.
Yes, I understand that you do not want to address user authentication, which
the draft clearly defines as the credentials in the RPC request header.

Another way to put what I am doing is:
- The server derives "machine credentials" from the client's certificate.
  The server then chooses via server configuration what RPCs are performed
  using these machine credentials instead of the user credentials derived from
  the RPC request header.

I know that Sun RPC has never had the concept of machine credentials and
that is one reason why it has not been a great fit for NFSv4, since NFSv4 does
use machine credentials.
- The workaround has been for the NFSv4 server to define the user credentials
   used for the first ExchangeID (SetClientID) operation as the machine credentials
   that must be used.
   For AUTH_SYS, this is typically the infamous uid == 0.
   For RPCSEC_GSS (Kerberos mechanism), it has typically required:
   - host based principal being created on the KDC.
   - keytab entry for this principal being created.
   - this keytab entry being copied to the client via some secure method.
(I won't bother to mention what credentials actually get used for callbacks.
 I'll admit I don't even understand how to do callback user credentials via
 RPCSEC_GSS. I think most just use AUTH_SYS with uid == 0 and clients mostly
 ignore the RPC request credentials for callbacks?)

It might make sense to look at using client certificates to create machine credentials
in the future?

I am fine with this not being allowed by this spec. I will simply document that.
However, you could state:
Creation of machine credentials from peer certificates is out of the scope of
this document.

>We do not want to alter RPC user authentication at this time. If using
>a certificate to identify an RPC user is something we _do_ want to do,
>and we most likely do want to allow it eventually, then that will
>require a different set of changes to the RPC protocol, and not just at
>the transport layer.
I have no idea what you are considering, but if it is along the lines of
clients providing separate X509 certificates for each user,
I think that would require:
- generation of certificates for every user
- copying these certificates into a database on each client in some secure
  manner.
  - You almost have to generate a separate certificate for each user for each
     client (ie. #clients * #users).
--> Too much administrative overhead, in my opinion.
It is easier (and a better fit, imho) to just use Kerberos for this case and it
is already implemented and works.

However, I do think that a certificate is a good fit for machine credentials,
which is not a good fit for Kerberos (the above exercise of host based principal...).
Having a way for clients to do a Kerberos NFSv4 mount without the host based
principal in a keytab on the client, would make Kerberized NFSv4 easier, I think?

>IMO you should wait until the RPC protocol has a purpose-built mechanism
>for user authentication via certificates, then implement a client that
>presents the same certificate to establish a TLS session as it does to
>identify its only user.
As above, I'm not convinced this is practical.

>However, you can probably make your current proposed implementation a
>little more future proof:
>
>An NFS client should use AUTH_NONE for your use case, since the NFS
>server ignores the client-provided RPC user identities. Maybe the server
>should enforce the use of AUTH_NONE. That would possibly navigate around
>the restriction stated in rpc-tls that the server cannot use the client's
>TLS identity to make RPC user-specific authentication and authorization
>decisions.
I don't believe extant NFSv4 clients can be configured to use AUTH_NONE
for all RPCs. What I am doing should work with an extant NFSv4 client once
it has RPC-over-TLS support (I'm assuming client's will consider the X509
certificate is an opaque blob that is only used for the TLS handshake).

>Add the following per-export controls on your NFS server implementation:
>
>- A control requiring a TLS connection to access the shared filesystem.
Yes, this exists in my implementation as the export option "-tls".

>- A control requiring the client to present a certificate to establish
>a TLS session (ie, the server should prevent TLS with anonymous clients).
Yes, this exists in my implementation as the export option "-tlscert".

>- A control that enables mapping between the client's TLS identity to a
>particular user identity that already exists on your server. The mapping
>should happen via NFSv4 id-mapping. Then the server can map both user-
>specific or client certificates to particular local user identities.
As above, I do not think user specific certificates is practical.
(Of course, feel free to prove me wrong;-)

>
>It's significantly more secure if the server is pre-configured with
>known user/client identities. Then, if the server does not recognize a
>client's certificate, it rejects the attempt to establish a TLS
>session.
I also have the "-h" option on the server side TLS handshake daemon,
which states that the client's certificate must have a DNS name that
matches the reverse DNS name for the client's IP address.
--> This is on the daemon and not in exports, since for this case,
      the IP address presented by the client is not trusted and, as such,
      the exports file cannot be applied to it until it has passed this
      requirement. (With "-h" it follows RFC 6125 style verification, but
      for on the server for a client certificate and not the other way around.)

rick

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Wed, 1 Apr 2020 16:48:12 -0400

> On Apr 1, 2020, at 4:36 PM, Rick Macklem <[email-masked]> wrote:
> 
> Chuck Lever wrote:
> for all RPCs.

Really? mount with sec=null/none does not work?

> What I am doing should work with an extant NFSv4 client once

For the moment, I'll just say that you might be misreading what I
wrote here: take out the "user-specific" and just make it "the server
maps the client-presented transport credential to a local user
identity on the server." That might be close to what you've already
implemented.

I agree that the preference in multi-user environments is a certificate
for the machine and Kerberos principals for user identities, for exactly
the reasons you stated.

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Wed, 1 Apr 2020 17:15:50 -0400

> On Apr 1, 2020, at 4:48 PM, Chuck Lever <[email-masked]> wrote:
> 
> 
> 

Duh. sec= does not control the authentication principal or flavor
of lease management operations. :-)

--
Chuck Lever

---

From: Benjamin Kaduk <[email-masked]>
Date: Wed, 1 Apr 2020 20:34:23 -0700

> Any quick hints on how I get an OID?

Well, it's a distributed (hierarchical) assignment policy, so in theory
it's supposed to be pretty easy, in that you know someone who controls a
sub-arc of numbers and can assign you one from that.  There are several
subtrees that are managed by IANA that a final RFC could get an allocation
from, but that doesn't help you for your experimental/local use.
I expect there's a few folks on this list that could find someone to make
an allocation if needed.

-Ben