Subject: [Acme] Fixing the TLS-SNI challenge type

From: Jonathan Rudenberg <[email-masked]>
Date: Thu, 11 Jan 2018 17:41:23 -0500

As many of you are probably aware, Frans Rosén of Detectify discovered a method of exploiting many shared hosting providers to obtain unauthorized certificates using the ACME TLS-SNI-01/02 challenge types[0]. Let’s Encrypt is in the process of removing support for these challenge types[1].

Obviously this is not ideal, as the TLS-SNI challenge is useful in a variety of use cases. The good news is that TLS-SNI-02 appears to be fixable.

In order to get the ball rolling on a fixed version (aka TLS-SNI-03), here’s a straw proposal, based on something originally publicly suggested by Ryan Sleevi[2]:

Start with the TLS-SNI-02 challenge specification, and add the requirement that the ACME server MUST send an ALPN extension in the ClientHello with a single “acme” protocol name, and the ACME server MUST confirm that the ServerHello also includes an ALPN extension with a single “acme” protocol name.

The only concern I’ve seen about this is the theoretical possibility that servers might just repeat back the ALPN extension with the same protocol name, which I believe we can remedy by doing a scan of the Alexa Top 1M (or similar) to see if this behavior exists in the wild.

Jonathan

[0] https://community.letsencrypt.org/t/2018-01-09-issue-with-tls-sni-01-and-shared-hosting-infrastructure/49996
[1] https://community.letsencrypt.org/t/2018-01-11-update-regarding-acme-tls-sni-and-shared-hosting-infrastructure/50188
[2] https://www.mail-archive.com/[email-masked]/msg08984.html

---

From: Roland Bracewell Shoemaker <[email-masked]>
Date: Thu, 11 Jan 2018 16:49:45 -0800

This seems like a silver bullet for the problems we’ve been seeing. Given that blindly responding to an unknown ALPN value would be an RFC violation this seems safe (although, hey, who knows what servers/cloud providers actually do). Definitely interested in the results of the scan. There could still be some argument about ‘misuse’ of the SNI extension but unless we have a concrete reason to the host name being validated I’m not sure I’m convinced we should.

Does anyone have any objections/spot any major issues with doing this?

> On Jan 11, 2018, at 2:41 PM, Jonathan Rudenberg <[email-masked]> wrote:
> 
> _______________________________________________
> Acme mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/acme

---

From: Jonathan Rudenberg <[email-masked]>
Date: Thu, 11 Jan 2018 20:04:38 -0500

> On Jan 11, 2018, at 19:49, Roland Bracewell Shoemaker <[email-masked]> wrote:
> 
> This seems like a silver bullet for the problems we’ve been seeing. Given that blindly responding to an unknown ALPN value would be an RFC violation this seems safe (although, hey, who knows what servers/cloud providers actually do). Definitely interested in the results of the scan.

I’ve completed[0] a scan of the Alexa Top 1M list, and no servers repeated back the unknown ALPN protocol of “acme” that I used[1].

I also opened a PR that adds this change to the spec: https://github.com/ietf-wg-acme/acme/pull/389

Jonathan

[0] https://storage.googleapis.com/titanous-acme/alpn_scan.csv.gz
[1] https://gist.github.com/titanous/8daa24ed3375f3c690950e8a97c7527d

---

From: Roland Bracewell Shoemaker <[email-masked]>
Date: Thu, 11 Jan 2018 19:06:02 -0800

I'm actually going to roll back my thoughts on the SNI value. Thinking
about this more I think it does actually make sense to revert to using
the actual host name here.

In the initial design of the TLS-SNI challenge the XXX.acme.invalid
value was used as a way to allow servers to dynamically serve both
regular and validation traffic. Since we would be using ALPN here we no
longer need a special SNI value in order to differentiate validation
traffic from regular traffic so this token value is no longer needed.

If we change that I think we can also remove the need to compute a pair
of SAN values for the cert contents. The original purpose of this was to
prevent a 'dumb' client from simply constructing a certificate on the
fly by sticking the SNI value it received into the SAN and automatically
allowing validation of anything that hit it. If we switch to using the
host name in the SNI then this is no longer necessary as the token value
is no longer present in the request.

If we make these changes I think the challenge would be different enough
that at that point it should probably be given a completely new name in
order to differentiate it from the original TLS-SNI challenges. I'd
suggest the incredibly imaginative TLS-ALPN (or tls-alpn-01).

---

From: Peter Eckersley <[email-masked]>
Date: Thu, 11 Jan 2018 19:23:11 -0800

> Does anyone have any objections/spot any major issues with doing this?

I don't have any objections at this stage, but one thing to bear in mind is
what will be required to set the ALPN values from various places. There's some
support for using it to negotiatie HTTP/2 in Apache 2.4.17+ and Nginx
(apparently?) 1.9.5. Even if it's settable from their config files, we'll
still need to migrate a lot of Certbot Apache and Nginx plugin users over to
HTTP-x Challenge types.

And though this is unlikely we should think hard about any ways that malicious
actors might be able to set these values :)

Also CC'ing Stefan Eissing to get an opinion on the impact on built-in
Apache ACME support...

-- 
Peter Eckersley                            [email-masked]
Chief Computer Scientist          Tel  +1 415 436 9333 x131
Electronic Frontier Foundation    Fax  +1 415 436 9993

---

From: Martin Thomson <[email-masked]>
Date: Fri, 12 Jan 2018 14:26:10 +1100

If you are using a new protocol, all of the concerns Peter have come
into play.  If you are there, why wouldn't you do the challenge
validation within the new protocol rather than using the certificate
(which is sent in the clear in current TLS versions).  That might be
easier to implement than generating the self-signed certificate...

---

From: Jonathan Rudenberg <[email-masked]>
Date: Thu, 11 Jan 2018 22:29:58 -0500

> On Jan 11, 2018, at 22:26, Martin Thomson <[email-masked]> wrote:
> 
> easier to implement than generating the self-signed certificate…

I don’t think an entirely new protocol makes sense. In order to do the TLS/SNI/ALPN handshake a certificate is already sent in the ServerHello, so there’s no reason not to use it to transport the authorization token. Generating a self-signed certificate is trivial, and I’m not aware of any relevant security concerns around fetching the authorization in the clear (the dns-01 and http-01 challenge types are plaintext as well).

---

From: Martin Thomson <[email-masked]>
Date: Fri, 12 Jan 2018 14:35:29 +1100

The confidentiality point was a minor one, certainly, but it would
represent an improvement.

TLS-SNI is a protocol already, but it's more complicated than it needs
to be.  Certificate generation is a pretty big lift compared to the
other challenges.

---

From: Patrick Figel <[email-masked]>
Date: Fri, 12 Jan 2018 16:29:10 +0100

> traffic from regular traffic so this token value is no longer needed.

One minor advantage of keeping the current acme.invalid scheme is that
it allows people to use SNI-based routing. Web servers like HAProxy
allow you to proxy requests (on the TCP layer) based on the SNI value,
so you could match "*.acme.invalid" and forward it to a validation
server that supports the new challenge and the ALPN ACME protocol, even
if the web server itself doesn't. If we use the "real" identifier for
SNI, we'd limit that option to web servers that natively support the
ALPN value for ACME and can route based on it.

Not sure how common this kind of setup is, if it's just a small subset
of HAProxy deployments it'd probably not have much of an impact.

Patrick

[1]:
https://www.cloudoptimizedsmb.com/articles/20160409-00/using-haproxy-to-split-letsencrypt-acme-challenges-from-regular-traffic

---

From: "Matthew D. Hardeman" <[email-masked]>
Date: Fri, 12 Jan 2018 10:04:17 -0600

Breaking that on purpose is actually why I most believe that the SNI should be required to reflect the exact SNI of the domain label being validated.

The threat model I was concerned about evaporates if the SNI is the actual domain label and the decisioning of which TLS certificate to present to the validator versus the normal far-end browser or other endpoint is then dependent on the combination of the ALPN marker AND the SNI.

(i.e. use most recent real certificate/key for connection with real endpoints indicating SNI of the real domain label, but use the validation certificate/key when the real SNI + “acme” ALPN  tag is received.)

Imagine that the HAProxy you describe is run not by the website administrator, but by the shared hosting provider.  Imagine that their infrastructure was vulnerable to the TLS-SNI-01 and -02 problems because their outside facing HAProxy matched SNI of any number of customer configured unique labels with customer provided certificates…

If you keep letting that model be manipulated by customers of the hosting provider, and keep the SNI as the .acme.invalid pattern, the path of least resistance to return the web hosts’s prior level of function is simply to create a fake “acme” ALPN signal and proceed with old behavior.  That’s a danger.

The concerns I raised would more or less be fully resolved by standardizing on having the SNI label be required to be the domain label being validated and then the TLS termination endpoint on the server side has to look to the “acme” ALPN tag to know to switch to presenting the validation challenge-response cerfificate/keypair.

> On Jan 12, 2018, at 9:29 AM, Patrick Figel <[email-masked]> wrote:
>

---

From: "Matthew D. Hardeman" <[email-masked]>
Date: Fri, 12 Jan 2018 10:21:57 -0600

So the original source of the TLS-SNI-01 in-the-field vulnerability that caused Let’s Encrypt to disable TLS-SNI-01 has just openly published details of the exploit he demonstrated:

https://labs.detectify.com/2018/01/12/how-i-exploited-acme-tls-sni-01-issuing-lets-encrypt-ssl-certs-for-any-domain-using-shared-hosting/

In this case, he used AWS CloudFront and Heroku to achieve the issue.

And it pertains to a TLS load balancer being programmatically bound to whatever newly unique to the platform name they want.  Just like the threat I had mentioned.

Now, imagine if there was a web host configured like AWS CloudFront and/or Heroku.  Then imagine that they’re sleazier, less well funded, and looking for quick resolution to a new customer complaint…

Customer complaint: My code used to be able to do X,Y,Z and achieve TLS-SNI-01 validation.  Now, because your TLS balancer doesn’t send this “acme” ALPN signal, I can’t anymore.  Fix it!

The cheap quick fix to this — the one that makes the customer’s complaint go away while spending the least time and money possible — is to just patch in a fake echo of the “acme” ALPN signal.

Note: this is fully resolved if the TLS SNI name in the protocol is changed to specify that it must be the actual domain label being validated.  But if you don’t require that, you’ll create an opportunity for the web host to most expeditiously return to the prior status quo by making a minimal fake “acme” ALPN tag.

> On Jan 12, 2018, at 10:04 AM, Matthew D. Hardeman <[email-masked]> wrote:
>

---

From: Ilari Liusvaara <[email-masked]>
Date: Fri, 12 Jan 2018 21:01:48 +0200

On Thu, Jan 11, 2018 at 04:49:45PM -0800, Roland Bracewell Shoemaker wrote:
> This seems like a silver bullet for the problems we’ve been seeing.
> Given that blindly responding to an unknown ALPN value would be an
> RFC violation this seems safe (although, hey, who knows what servers/
> cloud providers actually do). Definitely interested in the results of
> the scan. There could still be some argument about ‘misuse’ of the
> SNI extension but unless we have a concrete reason to the host name
> being validated I’m not sure I’m convinced we should.

Note that method 10 definition seems to require the name to be
validated to appear in the certificate. The name appearing in either
the request or response is also required for validation to be
deputizable (and I regard any non-deputizable validation to not be
proper domain validation; both HTTP-01 and DNS-01 are deputizable).

And I guess the reason why there is no blind echoing of ALPN has more
to do with the horrible breakage rates it would cause (due to HTTP/2).

Also, even if one used the name to be validated in the SNI, there is
still a concern that there is some horribly broken hoster, that:

- Allows claiming arbitrary https name, even if there is existing http
  site there (apparently these exist). And,
- Allows claiming arbitrary ALPN values.

Note that there is no requirement to be able to serve different
certificate for given ALPN, because other ALPNs are not checked, so
those might have the spoofed validation certificate.

AFAICT, these are not common. Most provoders that let one claim
arbitrary names (which would make TLS-SNI vulernable) do not let
one claim name in use for http.

And I suppose provoders that let one claim arbitrary ALPN values
without forwarding the entiere connection are extremely rare
(also, any such setup would need off-band signaling for ALPN, e.g.,
via PROXY/HAPROXY protocol).

With "squatting" for SNI, one obtains weaker security bounds, as
this drops the "even if" condition. So any provoder that is
vulernable to TLS-SNI misissuance and lets one claim arbitrary
ALPNs (and then presumably signals those off-band).

-Ilari

---

From: Niklas Keller <[email-masked]>
Date: Sat, 13 Jan 2018 09:53:55 +0100

I'm not sure, but I guess a certificate / key selection based on the ALPN
value needs integrated support in webservers then. Currently these can be
configured with a simple additional virtual host. I'm pretty sure
application servers written in PHP wouldn't be able to do that currently.
They're probably pretty rare compared to a traditional PHP deployment, but
other languages might be similarly affected.

How about extending the HTTP challenge instead? Validation via HTTP+TLS on
port 443 has been disabled due to shared hosting, which might be configured
correctly on port 80, but choose the first virtual host in case of port
443, given not all hosts have a TLS configuration. If we mandate that port
80 must be tried first and result in connection refused / TCP connect
timeout (might be unbound port or DROP / REJECT in a firewall, an HTTP
timeout doesn't count) before validating via port 443, couldn't that work?
I think TLS-SNI mainly makes sense where port 80 is closed and not required
for redirects anyway, e.g. APIs like api.github.com.

Regards, Niklas

---

From: Daniel McCarney <[email-masked]>
Date: Fri, 19 Jan 2018 09:43:55 -0500

>
> If we use the "real" identifier for SNI, we'd limit that option to web
> servers that natively support the ALPN value for ACME and can route based
> on it.
> Not sure how common this kind of setup is, if it's just a small subset of
> HAProxy deployments it'd probably not have much of an impact.

I agree this approach will limit compatible TLS servers but luckily HAProxy
should be fully compatible.

HAProxy has a "ssl_fc_alpn"[0] layer 5 fetch method that can be referenced
in a "use_backend" directive to route based on the negotiated ALPN from the
handshake. It looks like the feature has been present since HAProxy version
1.5 which hopefully means most HAProxy deployments will have access to it.
As one data-point Ubuntu 16.04 LTS packages HAProxy 1.6.3 and should be
able to build a ALPN-based routing solution similar to what was done for
SNI.

[0] -
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.3.4-ssl_fc_alpn

On Fri, Jan 12, 2018 at 10:29 AM, Patrick Figel <[email-masked]> wrote:

> https://www.cloudoptimizedsmb.com/articles/20160409-00/
> using-haproxy-to-split-letsencrypt-acme-challenges-from-regular-traffic
>

---

From: Daniel McCarney <[email-masked]>
Date: Fri, 19 Jan 2018 09:51:33 -0500

>
> I agree this approach will limit compatible TLS servers but luckily
> HAProxy should be fully compatible.

Alas, there's nothing like hitting send on a message to a mailing list to
make you think a little bit harder.

I overlooked the fact that it doesn't seem possible to select the
certificate based on the ALPN, just the backend. That's not really the
primary difficulty at hand. I suspect matching the request to the `bind`
with the correct alpn is still going to be problematic if the server needs
to respond to requests for the real identifier with both a certificate and
a challenge response certificate.

I would be happy to be corrected :-)

On Fri, Jan 19, 2018 at 9:43 AM, Daniel McCarney <[email-masked]>
wrote:

>> servers that natively support the ALPN value for ACME and can route
>> based on it.
> I agree this approach will limit compatible TLS servers but luckily
> HAProxy should be fully compatible.
>
> [0] - http://cbonte.github.io/haproxy-dconv/1.8/
> configuration.html#7.3.4-ssl_fc_alpn
>
>> https://www.cloudoptimizedsmb.com/articles/20160409-00/using
>> -haproxy-to-split-letsencrypt-acme-challenges-from-regular-traffic
>>

---

From: Ilari Liusvaara <[email-masked]>
Date: Fri, 19 Jan 2018 17:38:32 +0200

> I would be happy to be corrected :-)

Basically, for security, one needs to put the domain to be validated to
the SNI field. Not doing that was the reason for the TLS-SNI-01/02
vulernability.

AFAICT, the hard constraints are:

- No taking site offline when validating.
- The validation certificates are not publically trusted.
- SNI needs to be set to the domain being validated.
- Compliant with reasonably strict reading of "10 methods" (which
  essentially impiles the third point and using certificate to
  transmit data).

(I do not consider compatiblity with present software a hard
requirment).

Few ways that come to mind:

- Have "acme" ALPN and send validation certificate for that.
- Have some new TLS extension and send validation certificate for that.
- Have "acme" ALPN, establish data link and exchange an exported
  authenticator request/response over it (response carries an X.509
  certificate).

One also wants as good assurance as possible that the entity
terminating the TLS link approves the certificate. This entity may be
the applicatnt themselves (e.g., applicant's VPS with direct IP-
layer connectivity, or with SNI-level routing of unterminated TLS
connections), or it might not be (shared hosting).

The first and second method are much better there than the third
(any provoder that lets users claim ALPN protocols runs into trouble).
And of the first two, the second would be slightly more safe, but
probably not enough for it to be worth trouble.

However, one should keep in mind that if hosting provoder allows
claiming https hosts regardless of http ones (some provoders have
problems of this sort[1]), they would probably also approve the
certificate requests for the owner of the https host.

[1] The Detectify article started with one unnamed such provoder
I think it would have been easy to find a CA that would have
validated the domain as-is, and it was only due to author's fixation at
using LE that the TLS-SNI issue was discovered.

-Ilari

---

From: Richard Barnes <[email-masked]>
Date: Fri, 19 Jan 2018 16:47:59 +0100

>   essentially impiles the third point and using certificate to

I'm less worried about this constraint.  If there's consensus for a change,
changes can be made to the BRs much more quickly than an RFC can be issued.

> - Have some new TLS extension and send validation certificate for that.

My inclination here would be to do the whole thing in a TLS extension: Have
the client send its value in an extension, and have the server respond in
an extension.  It's what they're there for.  You would need upgrades to TLS
stacks and new API for this, but IIRC, QUIC is going to require some
similar changes to make applications aware of extensions anyway. (Cf.
https://github.com/bifurcation/mint/pull/137 for EKR's minq implementation
of QUIC)

Any approach that relies on overloading an existing TLS field make me
nervous.  If the field is there for some other purpose, then it seems
inevitable that you'll run into bugs of more or less the character we've
hit two times already.

--Richard

---

From: Ilari Liusvaara <[email-masked]>
Date: Fri, 19 Jan 2018 18:01:53 +0200

> changes can be made to the BRs much more quickly than an RFC can be issued.

Oh yeah, the minimum process latency for changing BRs is ~7 weeks.

However, that would take well-fleshed proposal to do it even close to
that quick. Take note that "10 methods" took years.
 
> https://github.com/bifurcation/mint/pull/137 for EKR's minq implementation

Yes, that is one way, if BRs are easy enough to change. If they are not,
then just change where the reply is sent, and it is compliant.

Also, I consider TLS-SNI type methods to be UNSUITABLE for validating
usual webservers. Anything that the method is suitable for should have
special case support anyway (e.g., I had support in one TLS library, before
I ripped the code out).

> inevitable that you'll run into bugs of more or less the character we've

Yeah, as I said, custom extension provodes slightly better guarantees.

-Ilari

---

From: Tim Hollebeek <[email-masked]>
Date: Fri, 19 Jan 2018 17:17:11 +0000

> Basically, for security, one needs to put the domain to be validated to
the SNI
> field. Not doing that was the reason for the TLS-SNI-01/02 vulernability.

I agree.  Not only for security, but for compliance, both with the Baseline
Requirements [1] and the intended use of SNI.  Abusing SNI as an OOB 
communication channel was a bad idea and should not continue.

-Tim

[1] I am unpersuaded by tortured arguments used to claim TLS-SNI-01/02 
complies with the Method 10 requirements.

---

From: Tim Hollebeek <[email-masked]>
Date: Fri, 19 Jan 2018 17:24:01 +0000

> > I'm less worried about this constraint.  If there's consensus for a
> > change, changes can be made to the BRs much more quickly than an RFC can
> be issued.
> 
> However, that would take well-fleshed proposal to do it even close to that
> quick. Take note that "10 methods" took years.

Years is a bit of an exaggeration.  One and change, I think it was.  And
that
was to get agreement on all ten methods and to work out exactly how the
forum's IPR policy applied to validation methods (which was about 2/3
of the timeline ...).

I think a well-written proposal to add or improve an existing validation
method could easily get passed in close to the minimum timeline.

As the Validation WG chair, I'm even willing to help get secure, 
well-analyzed and well-specified proposals through the voting process.

-Tim

---

From: Ilari Liusvaara <[email-masked]>
Date: Fri, 19 Jan 2018 20:31:01 +0200

> well-analyzed and well-specified proposals through the voting process.

Just as an example of what the new method could be (I am assuming
back-transport via extension, because if it was via certificate, one
could claim method 10 as-is):

challenge: Random string of at least 128 bits of entropy.

Define new server_name NameType: network_address. The payload is IP
address of endpoint (4 bytes for IPv4, 16  bytes for IPv6).

Then define extension domain_validation:

Validator sends challenge_hash. This is 32 octet SHA-256 hash of the
challenge. server_name extension MUST be sent and set to the identifier
being validated (this is the reason for adding network_address, in
order to support validating IP addresses).

Server replies (in ServerHello or EncryptedExtensions) with its own
hash (using strong hash specified by the CA) over:

opaque challenge<1..2^16-1>;
opaque identifier<1..2^16-1>;
opaque applicant_id<1..2^16-1>;

The validation passes if:

- Response hash matches what CA computed, AND
- Server Finished MAC matches what CA computed, AND
- The exchange signature validates with the key from the certificate
  message.

In general, I would expect the BRs to refer to the TLS extension,
instead of specifying framework it fits into. This allows much more
precise security analysis, and much more compact BR text.

-Ilari

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 19 Jan 2018 18:33:47 +0000

➢ challenge: Random string of at least 128 bits of entropy.

What does that mean?  How can you measure if you’ve got 125 or 67 or 2 bits of entropy?

---

From: Tim Hollebeek <[email-masked]>
Date: Fri, 19 Jan 2018 18:41:08 +0000

Use: "at least 128 bits of output from a PRNG that is suitable for use in cryptographic operations", or similar.

The Baseline Requirements have a similar definition.

-Tim

> -----Original Message-----
> From: Salz, Rich [mailto:[email-masked]]
> Sent: Friday, January 19, 2018 11:34 AM
> To: Ilari Liusvaara <[email-masked]>; Tim Hollebeek
> <[email-masked]>
> Cc: IETF ACME <[email-masked]>
> Subject: Re: [Acme] Fixing the TLS-SNI challenge type
>