Subject: [nfsv4] testing of RPC over TLS

From: Rick Macklem <[email-masked]>
Date: Fri, 27 Nov 2020 03:44:49 +0000

The implementation of RPC over TLS for freebsd is now basically
done and setting up a test system is fairly straightforward.
The main limitation is that it currently uses TLS1.2, since the
FreeBSD ktls doesn't do TLS1.3 yet.

If you are interested see
https://people.freebsd.org/~rmacklem/nfs-over-tls-setup.txt

If you have any questions about it, feel free to ask, rick

---

From: grarpamp <[email-masked]>
Date: Fri, 27 Nov 2020 00:18:16 -0500

> done and setting up a test system is fairly straightforward.

Nice.

> If you have any questions about it, feel free to ask, rick

How do users and administrators pin down the
public key fingerprint of the NFS server's TLS certificate
so that the hash is checked and thus a MITM
attack is prevented?

>From March on list...
Implementations must provide users with TLS public key
fingerprint pinning options. Implementations that do not offer
those options are exposing their users to MITM risks (those
risks are not mitigated by the legacy wrong answer of 'trust
the [rogue and or MITMable] "CA" system').
MITM attacks are in play by many parties over many networks.
*ALL* applications that speak TLS today must make these
pubkey fingerprint pinning options available for their users.

Cheers.

https://tools.ietf.org/html/draft-ietf-nfsv4-rpc-tls

People appear to be talking about using and
"authenticating / verifying" TLS certs now with at least
perhaps this NFSv4, and certainly with other apps.

If so, it's required critical thing for the admins and users to have
the option to pin the certificate pubkey fingerprints in four ways...

- Ignore the CA chain / expiry / etc, validate only the fingerprint.
- Validate the CA chain / expiry / etc, and validate the fingerprint.
- Validate the CA chain / expiry / etc, ignore the fingerprint.
- A TOFU mode, with some management by fingerprint options.

No application that uses TLS should be considered completely
featured and security capable without fingerprint pinning functions.

The "SHOULD implement" loophole in 5.5.2 will end up with many OS
and implementations lacking such basic security feature option.

For some background reasons on why pubkey fingerprint pinning
implementations are now showing up in softwares that speak TLS,
and for sample code, and related infos, see the links...

https://owasp.org/www-community/controls/Certificate_and_Public_Key_Pinning
https://cheatsheetseries.owasp.org/cheatsheets/Pinning_Cheat_Sheet.html

https://curl.haxx.se/libcurl/c/CURLOPT_PINNEDPUBLICKEY.html
--pinnedpubkey <hashes | file>
Tells curl to use the specified public key file (or hashes) to verify the
peer. This can be a path to a file which contains a single public key in PEM
or DER format, or any number of base64 encoded sha256 hashes preceded by
'sha256//' and separated by ';'.
When negotiating a TLS or SSL connection, the server sends a certificate
indicating its identity. A public key is extracted from this certificate and
if it does not exactly match the public key provided to this option, curl will
abort the connection before sending or receiving any data.

Please note this option is rightly very specific covering only the
isolated pubkey, not the DER form of the entire "CA signed" cert
(ie: not the typically referenced coverage of "openssl x509 -fingerprint").
This allows additional adaptability to some cert environments.

Complete fingerprint implementations need both modes: pubkey, and cert DER.

Also note the use of sha-256, not the broken and now deprecated sha-1,
sha-3 could function as a backstop.

When fully implemented, fingerprint pinning enables a local admin and
user environment of more flexible certificate validation service cababilities
and security model hardening when subject to various third party things
and adversaries like...

- Environment of rogue / forced / spy MITM CA's, TLS termination / proxy
cloud MITM, VPN / overlay / WiFi networks MITM, etc.
- Annoying "expired" certs awaiting tax revenue from their captured audience.
- Assigning pinned trust to intermediate CA's such as Lets Encrypt, Google,
and corporate schemes, to let edge server certs they sign be freely
rotated and or freshly signed without need to update pin.
- Avoid need to update pin every "expiry" period.
- Avoid CA's by using cert owners publicly available and out of band self
certified hash attestations found on keybase, social, observatories, PGP, etc.
- As mentioned above, optionally in combination with other CA / expiry / etc
checks, or ignoring the CA altogether.
- CRL checks are a massive metadata privacy and user monetization
leak that some users might not want exposed to.
- Pinning one or both of: pubkey (herein) and or CA (openssl x509 -fingerprint)

Another very useful security feature to have is a trust on first use TOFU
mode that stores, pins, and subsequently validates against those fingerprints,
similar to SSH model. This is useful for both known comms partners such as
client-server model, and in more distributed group or even p2p applications
to help keep things a bit more locked down by default.

Defense (like this pubkey fingerprint pinning) in depth... you can use it :)

References (obviously TLS_1.3 is todays version to use)...

https://www.netcraft.com/internet-data-mining/ssl-survey/
https://www.ssllabs.com/ssl-pulse/
https://arstechnica.com/gadgets/2018/10/browser-vendors-unite-to-end-support-for-20-year-old-tls-1-0/
https://www.bleepingcomputer.com/news/security/ietf-approves-tls-13-as-internet-standard/
https://en.wikipedia.org/wiki/Transport_Layer_Security
https://tools.ietf.org/html/rfc8446

https://github.com/OWASP/www-community/blob/master/pages/controls/Certificate_and_Public_Key_Pinning.md
https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/Pinning_Cheat_Sheet.md

https://github.com/curl/curl/blob/master/docs/cmdline-opts/pinnedpubkey.d
https://github.com/curl/curl/blob/deb9462ff2de8e955c67ed441f5f48619a31198d/docs/libcurl/opts/CURLOPT_PINNEDPUBLICKEY.3
https://github.com/curl/curl/blob/51fde337471c9125e7bf425e7ce0a0bf53691992/docs/TODO#L728

On 3/30/20, Black, David <[email-masked]> wrote:
> This looks interesting - is there a BCP (Best Current Practice) RFC or
> similar reference on what MUST/SHOULD to be implemented and how implementers
> ought to think about this to make good choices?  Or is there one being
> prepared?

Not aware of one, maybe the list knows of some more resources?
Would be good to have some more documents in the space.
In particular herein regarding fingerprinting... differentiating the
two different cert DER vs pubkey forms, their various features and
capabilities in real world cert schemes scenarios, and re MITM.

There is a general encryption proponent mandate RFC,
commensurate with Snowden events, the number escapes
at this moment.

Yes, more development of such a BCP as you mention regarding
TLS fingerprint capabilities, and TLS extent forward in general,
would be nice. That would then extend to many protocols
in a default guide of reference way as to what is possible,
including say NFS.

It is difficult to say "MUST" or "SHOULD" or "MAY"
regarding this fingerprinting business for NFS.
But it may be reasonable that 5.5.2 be expanded to mention
the two different fingerprint classes, the four modes, as reasonable
desireables. And to then reference out to the OWASP links at the
bottom of the RFC.

Maybe then leverage that effort into a BCP with other
application RFC groups regarding TLS.

> For example, such a document ought to cover the tradeoffs among
> the four validation options, and how a safe default be chosen.   In general,
> this looks like a good direction as trust anchors and trust anchor
> management have always been a weak point in the web PKI
>
> The OWASP pages are a good start, but leave revocation for future work and
> are weak on certificate replacement - in particular, the comment that the
> "application would need to be updated regularly" appears to punt the
> underlying app validation problem upward to the app store operators (e.g.,
> to defend against a rogue app that accesses the server via an MITM proxy
> server), which doesn't mesh well with "An application which pins a
> certificate or public key no longer needs to depend on others - such as DNS
> or CAs".  Some dependence on the app store operator may be unavoidable for
> first-use.

It is assumed, and guaranteed under public embarrassment
of news media, that revocation will results in a physical regen
and change of cert (triggering both cert DER and pubkey
hard physical swapouts). Thus revocation, and the whole
revocation "authority" repository CRL OCSP checks etc,
is not at all coming into an issue in this fingerprint context,
since they are entirely independant and fixed and local,
and detect and resolve all such revocation swaps by
very nature of fingerprint failure and fingerprint updates.

The "application being updated regularly" is of course a nice
option for such app shipping devs to do gratuitously for users.
But has no bearing on users subsequent needed capability
to pin whatever they want and feel is necessary independantly.

If I were an NFS users in particular infrastructures and risks,
I might want to pin the end server pubkey (or cert DER),
or intermediate instances of the same (leaving server cert
free to float) regardless of what my vertical or lateral silo,
or other peer model, or CA, or rogue, says to accept.

Is there a reasonable way to more strongly suggest implementation
of those two fingerprint methods, and even some of the four modes,
as options between server-client in the NFS TLS RFC?

> There is a general encryption proponent mandate RFC...

One might also try full text RFC search for "fingerprint"
references in context of "TLS" (or the older "SSL").

The following incomplete list all in manner of spirit do
generally give rise and encourage the offering and enablement
of such security option sets in apps / protocols utilizing TLS...

Recommendations for Secure Use of Transport Layer Security (TLS)
https://tools.ietf.org/html/rfc7525
Pervasive Monitoring Is an Attack
https://tools.ietf.org/html/rfc7258
Privacy Considerations for Internet Protocols
https://tools.ietf.org/html/rfc6973
Certificate Transparency
https://tools.ietf.org/html/rfc6962
Strong Security Requirements for Internet Engineering Task Force
Standard Protocols
https://tools.ietf.org/html/rfc3365
Guidelines for Writing RFC Text on Security Considerations
https://tools.ietf.org/html/rfc3552
IETF Policy on Wiretapping
https://tools.ietf.org/html/rfc2804
IAB and IESG Statement on Cryptographic Technology and the Internet
https://tools.ietf.org/html/rfc1984

"It is the consensus of the IETF that IETF standard protocols MUST
make use of appropriate strong security mechanisms."

Privacy Requirements for IETF Protocols
https://tools.ietf.org/html/draft-cooper-ietf-privacy-requirements-01
   It is the consensus of the IETF that our protocols be designed to
   avoid privacy violations to the extent possible.

Handling Pervasive Monitoring in the IETF (perpass) (WG)
https://www.ietf.org/proceedings/88/perpass.html
https://www.ietf.org/mailman/listinfo/perpass

Not requiring app (NFS) spec implementations to have some options
for some of the previously noted modes of fingerprint checking to help
avoid some TLS MITM attacks, would seem to be in conflict with all above.

> the premise that pinning by fingerprint is the
> strongest security/authentication mechanism possible in many cases.

It is certainly among the set of them that might be applied
in various ways depending on the users security model.

> In IETF documents we essentially universally leave the deployment's
> selection of trust anchor(s) as a matter of local policy, and thus it is a
> bit out of character for us to be recommending specific trust anchors or
> classes of trust anchor.  We provide robust security mechanisms and
> describe how they can be safely used, but leave policy to others.

Allow to better define below the context as being in
supporting the presence of options for policies...

> I'm not sure I understand your point here.  You seem to be saying that
> CA-driven revocation is orthogonal to key pinning, and thus that when a
> revocation event occurs, the legitimate counterparty will generate a new
> key (and fingerprint), with the resulting connection errors being detected
> and used as a trigger to update local configuration.
> But that doesn't make sense to me!  If there's a revocation event and
> presumed key compromise, then the attacker that got the key can continue to
> use it against the deployment that pinned that key until the configuration
> is changed.  This seems to be ignoring the signal from the CA that
> "something bad happened" and could lead to a long period of vulnerability.
> It's far from clear to me that key pinning provides superior properties to
> CA-based setups in the face of such revocation events.

It is risky and then embarrassing to continue using a compromised
private key. The compromise is the preceding and defining element,
and rightly triggers a security rekey event. The rekey hash mismatch is,
the and only required, signal when a users application is watching the
fingerprint, if that is how the user configured their options.
This is independant, and if so chosen "orthagonal", from any assertion
or not as such by or to any CA signal crl scheme, that assertion being
at that point, equally the same as pinning check, a separate function
from the security compromise and restoration itself.
Fingerprints, "CA signals", other checks, are independant cases, and
can be combined in some users environments, provided they have
options to do so.
Further, many environments use CA only to avoid software CA nag
warnings and have little knowledge or care for CRL systems, and
many users have little knowledge or care to check them.
People also often try to trot out this false argument of relying "CA signal"
or revocation as some means to moot or demur the implementing of
fingerprinting options in the app code, or to avoid configuring them.
That is not the case.

As to NFS... in this suggestion, nfsv4-rpc-tls in 5.2.2
needs to REQUIRE all conformant implementations
to have fingerprint configuration OPTIONS for the user
to use as they deem appropriate. This does NOT
require users to use it, only that it be present
as option in conformant implementations.

If CA method of 5.2.1 is a REQUIRED security thing, then
similarly, so can fingerprint security OPTION things of 5.2.2 be.

If nfsv4-rpc-tls anywhere REQUIRES any things
of any conformant implementations, then so can it
require some fingerprint OPTIONS things.

If nfsv4-rpc-tls 5.2.2 is already specifying a
SHOULD-hint as to a "via"-form of SHA-256 being
over the cert-DER form (with or without including
reference to just what spec that is), it really must
also fairly provide a SHOULD-hint as to the "via"-form
of pubkey. That is probably simple enough, leaving
the four modes and other option combinatrics up
to the application teams, the possibility for such
potentially hinted in note in section 7.

External references are already possible by example
of URI list in 9.3. Additionally, perhaps something in
RFC5280 or similar could be noted or pointed to
as RFC5280 was noted and or pointed to in 5.2.1.

RFC's even have places available for further discussion
and strong words encouragement, such as section 7.

> fingerprint pinning is hardly ubiquitous in those protocols.

Unfortunately that is largely because fingerprint pinning is a
hardly understood black magic of additional TLS defence, in
depth, hardly mentioned, referenced, documented, implemented,
or educated on, etc. Most don't really understand MITM modes.
And with all the noise droning on about CA, $fees, "validation"
of corporate entites, advertising, who can blame them.
This issue is a similarly common copout / chicken egg to the
above CA CRL excuse for not creating fingerprint options.

Beneficial ubiquity can be fostered by more rfc's,
internet application docs, etc becoming a part of the
set of same that begins and further integrates more
mentions and supports for pinning options in their
various works.

It would suck to effort to add this TLS to NFS but
not enjoy some fingerprint pinning options along with it,
because, well, it was only a SHOULD, so your OS
skipped them resulting in less than featureful. Oops.

> It would perhaps be compatible to focus on the choice of trust anchor and
> the possibility to pin intermediate certificates, but this again starts to
> stray into areas that the IETF historically leaves as a matter of local
> policy.

Requiring pinning OPTIONS would enable this making of local
policy choices among such options. No options, no choices.
Users like having options and choices.

Opportunistic Security: Some Protection Most of the Time
https://tools.ietf.org/html/rfc7435.html

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 26 Nov 2020 22:09:59 -0800

[freebsd-questions to bcc]

> so that the hash is checked and thus a MITM

As I recall, you were the only vocal proponent of this, and I do not recall
a declaration that there was WG consensus that such functionality is
needed.

I strongly encourage you to read the "PKI vs. Pinning" section of
https://datatracker.ietf.org/doc/minutes-108-saag/ (and possibly watch the
recording and read the associated email thread) and reframe your desiderata
in light of the discussions therein.

Thanks,

Ben

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: grarpamp <[email-masked]>
Date: Fri, 27 Nov 2020 06:02:13 -0500

> only vocal proponent of this

Often NFS users and admins...
Are in and from nice private LAN environments, not public services.
Aren't necessarily oriented to other nasty threat models.
Come from legacy schools of thought that still incorrectly
believe "CA" validation is all that's needed to neuter MITM.

That's of course not universal, but "only" is relative,
and in no way invalidates the considerations for pinning.

If something speaks TLS, it needs to have pubkey
pinning options.

If RFC compliant implementations must do some
certain things re TLS, then RFC can also say that...
"5.2.3 Fingerprint pinning
Implementations must offer options to configure
pinning of the TLS certificate pubkey fingerprint hash".

Some simple sentence like that is all that's needed.
Then all the implementations, such as FreeBSD's,
will have the option, users can read their manuals
and use it as desired or not.

If not put in the RFC, then some better implementions will
have them, other lesser ones won't, the space for nfsv4
interop will thus be fragmented, users who want to use
pinning will be constrained to certain OS's, comparisons
made, etc.

FreeBSD has a good degree of security consciousness,
so it would be great and not surprising for them to offer
pubkey pinning for their users and admins from the start.

Sections 4 and 5 already require lots of things re TLS... so there
is no reason it cannot also require the *options* to configure
pubkey and CA pinning... further info on those can be included
in references and appendix (while indeed at same time progressing
more formal material in IETF or elsewhere).

5.2.3 Fingerprint Pinning
[opts required]
5.2.3.1 Pubkey Hash
[defn refs]
5.2.3.2 Certificate Hash
[defn refs]

"
5.2. TLS Peer Authentication

   TLS can perform peer authentication using any of the following
   mechanisms.

5.2.1. X.509 Certificates Using PKIX Trust

   RPC-over-
   TLS implementations are REQUIRED to support the PKIX mechanism
   described in [RFC5280].
"

> "PKI vs. Pinning" section of
> https://datatracker.ietf.org/doc/minutes-108-saag/

The above sentence also rightly avoids such TLS debates...
- "scale"
- "decisions about which rogue / forced / compromised CA's to 'trust'"
- what "AV" "Google" companies think
- namespaces, grand pie sky schemes
- whether or not anyone thinks users need pinning or not
- etc

Pinning to the CA is not the sole and universal option for all users.
For one, CA / Let's Encrypt sigs expire, such expiry is meaningless,
the rotation causes excess "scale" maintenance under cert level pins,
CA $igs often don't mean anything to real security contexts... and it
has already been noted that any privkey compromise will result in the
pubkey hash changing anyway, so CA is either moot or secondary
to that pubkey fundamental. Pinning the pubkey "end entity cert"
is the underlying simple method to avoid any concerning about CA.

Users TLS environments, needs, models, validations...
are different from debaters... instead stick to how
to get the option into all implementations.

But to not even mention pinning in RFC, to have removed it,
not even having as 5.2.2's "optional option", or more fully
detailed in Section 7 security considerations... seems potentially
a bit irresponsible, particularly now that other TLS client-server
applications are making some more mention and implement of it.

"
   In 2014 the IETF published a document entitled "Pervasive Monitoring
   Is an Attack" [RFC7258], which recognized that unauthorized
   observation of network traffic had become widespread and was a
   subversive threat to all who make use of the Internet at large.  It
   strongly recommended that newly defined Internet protocols should
   make a genuine effort to mitigate monitoring attacks.
"
"
   A classic form of attack on network protocols that initiate an
   association in plain-text to discover support for TLS is a man-in-
   the-middle
"

> watch the recording and read the associated email thread

https://mailarchive.ietf.org/

IETF has configured CloudFlare (which btw is in fact a MITM,
for which too few have contemplated that problem) to block
users of VPN and other overlay services who would otherwise
be quite happy to watch or read such things at that URL.

Didn't see a video link in the minutes link.

Best regards.

---

From: David Noveck <[email-masked]>
Date: Fri, 27 Nov 2020 07:44:36 -0500

There is no way we can make the sort of changes in the existing RFC
candidate that you propose, at this point.   The document has already
successfully been through WG last call, IETF last call and IESG
consideration and has a good chance of resulting in an RFC within 2021   If
we were to decide that we wanted to significantly change it, we would be
delaying publication for at least a year.   As a result, we would be living
even longer with the current approach to NFSv4 security including the use
of AUTH_SYS without any client host authentication whatever :-(. Sigh!

The benefits that you anticipate from your proposed changes would have to
be balanced against the negative consequences of a delay in rpc-tls as
currently defined.    That is not worth it.   The proper time to raise and
address these sorts of issues was during WGLC, which ran from 11/24/2019 to
12/23/2019.  In any case, the document approved by the working group did
not take the view that you do regarding these issues.

It is possible for the working group to change its mind and start work on a
revised RFC.   You can propose that to the working group.   Another
possibility is to address these issues in an nfsv4-specific context (rather
than in a general rpc context) as part of the anticipated RFC deriving from
an eventual draft-ietf-nfsv4-security.    I'd appreciate it if you
provided material regarding these (and other important) security issues
that would fit in draft-dnoveck-nfsv4-security-needs.   I think it would be
best if we incorporated all these before submitting
draft-ietf-nfsv4-security-needs.