Subject: Re: [Acme] Fwd: I-D Action: draft-ietf-acme-acme-07.txt

From: Richard Barnes <[email-masked]>
Date: Tue, 18 Jul 2017 17:44:00 +0200

On Fri, Jun 23, 2017 at 3:33 PM, Rifaat Shekh-Yusef <[email-masked]>
wrote:

> I do not believe that I got an answer to my pre-authorization question
> here:
> https://www.ietf.org/mail-archive/web/acme/current/msg01991.html
>
> What would be the response of the server if the client sends the
> certificate issuance request after it sends the pre-authorization request
> but
> before it completes the pre-authorization process?
>
> Would the server reply with the same challenges it provided in the
> response to the pre-authorization request?
>

Hey Rifaat,

Sorry for missing this.  I think the short answer is that it's up to the
server.  Either way, the server is instructing the client to fulfill
certain instructions.  I don't think this really needs much specification;
the most would be a recommendation that the server should re-use existing
pending challenges.

--Richard

>
> Regards,
>  Rifaat
>
>
>
> On Wed, Jun 21, 2017 at 3:20 PM, Ted Hardie <[email-masked]> wrote:
>
>> Howdy,
>>
>> We'd like to start a short working group last call on the changes made in
>> response to the previous last call.  It will end June 28th, 2017, any time
>> zone (to accommodate any changes needed before the draft deadline the
>> following Monday).
>>
>> Please review carefully, but please respect the discussion which have
>> already been held by focusing on new issues or new information about the
>> issues that have been resolved.
>>
>> thanks,
>>
>> Ted, Rich, (and now) Yoav
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: <[email-masked]>
>> Date: Wed, Jun 21, 2017 at 12:00 PM
>> Subject: [Acme] I-D Action: draft-ietf-acme-acme-07.txt
>> To: [email-masked]
>> Cc: [email-masked]
>>
>>
>>
>> A New Internet-Draft is available from the on-line Internet-Drafts
>> directories.
>> This draft is a work item of the Automated Certificate Management
>> Environment of the IETF.
>>
>>         Title           : Automatic Certificate Management Environment
>> (ACME)
>>         Authors         : Richard Barnes
>>                           Jacob Hoffman-Andrews
>>                           James Kasten
>>         Filename        : draft-ietf-acme-acme-07.txt
>>         Pages           : 74
>>         Date            : 2017-06-21
>>
>> Abstract:
>>    Certificates in PKI using X.509 (PKIX) are used for a number of
>>    purposes, the most significant of which is the authentication of
>>    domain names.  Thus, certificate authorities in the Web PKI are
>>    trusted to verify that an applicant for a certificate legitimately
>>    represents the domain name(s) in the certificate.  Today, this
>>    verification is done through a collection of ad hoc mechanisms.  This
>>    document describes a protocol that a certification authority (CA) and
>>    an applicant can use to automate the process of verification and
>>    certificate issuance.  The protocol also provides facilities for
>>    other certificate management functions, such as certificate
>>    revocation.
>>
>>
>> The IETF datatracker status page for this draft is:
>> https://datatracker.ietf.org/doc/draft-ietf-acme-acme/
>>
>> There are also htmlized versions available at:
>> https://tools.ietf.org/html/draft-ietf-acme-acme-07
>> https://datatracker.ietf.org/doc/html/draft-ietf-acme-acme-07
>>
>> A diff from the previous version is available at:
>> https://www.ietf.org/rfcdiff?url2=draft-ietf-acme-acme-07
>>
>>
>> Please note that it may take a couple of minutes from the time of
>> submission
>> until the htmlized version and diff are available at tools.ietf.org.
>>
>> Internet-Drafts are also available by anonymous FTP at:
>> ftp://ftp.ietf.org/internet-drafts/
>>
>> _______________________________________________
>> Acme mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/acme
>>
>>
>> _______________________________________________
>> Acme mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/acme
>>
>>
>
> _______________________________________________
> Acme mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/acme
>
>

---

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Tue, 18 Jul 2017 13:13:54 -0400

Hi Richard,

Take a look at the following exchange with Jacob:
https://www.ietf.org/mail-archive/web/acme/current/msg02085.html

I think the draft should be clear on the response when the server reuses
existing pending authorization:
1. Is it still 201 Created? or should it be 200 OK?
2. Would the payload include the "challenges" object as per the response to
the /new-authz request, or would the payload still include the
"authorization" object as per the response to the /new-order?

Regards,
 Rifaat

On Tue, Jul 18, 2017 at 11:44 AM, Richard Barnes <[email-masked]> wrote:

>
>
> On Fri, Jun 23, 2017 at 3:33 PM, Rifaat Shekh-Yusef <[email-masked]
> > wrote:
>
>>> We'd like to start a short working group last call on the changes made
>>> in response to the previous last call.  It will end June 28th, 2017, any
>>> time zone (to accommodate any changes needed before the draft deadline the
>>> following Monday).
>>>

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Tue, 18 Jul 2017 17:49:40 -0700

On 07/18/2017 10:13 AM, Rifaat Shekh-Yusef wrote:
> <https://www.ietf.org/mail-archive/web/acme/current/msg02085.html>
>
>
> I think the draft should be clear on the response when the server
> reuses existing pending authorization:
FWIW, Let's Encrypt is planning to return the exact same code when
reusing an existing pending authorization as when creating a new one.

---

From: Rifaat Shekh-Yusef <[email-masked]>
Date: Wed, 19 Jul 2017 08:28:00 -0400

What about the payload? would you return "challenges" or "authorization?

Regards,
 Rifaat

On Tue, Jul 18, 2017 at 8:49 PM, Jacob Hoffman-Andrews <[email-masked]> wrote:

> FWIW, Let's Encrypt is planning to return the exact same code when reusing
> an existing pending authorization as when creating a new one.
>

---

From: Kevin Borgolte <[email-masked]>
Date: Thu, 20 Jul 2017 00:18:10 -0700

Hi everyone,

we are currently conducting a measurement study about DNS staleness issues
with a focus on IP address churning. We encountered the issue that Let's
Encrypt (and ACME in general) can be (ab)used to request and receive a
valid certificate for a domain, as long as the attacker obtains access to
the IP address to which the DNS record points. This can happen due to stale
DNS records (e.g., pointing to cloud providers), or through MitM attacks
(e.g., as recently observed when an (accidental) BGP hijack re-routed all
major payment processors via Moscow). The issued certificates are
independent of the actual IP address used for the fraudulent certificate
request, and can later be used for malicious purposes (local interception
etc.).

Although it is a valid certificate request, we think that the ACMEv2
standard should mention practical ways to prevent these cases. This is
especially the case, as the malicious certificate issuances may not
necessarily attract attention through certificate transparency logs (as the
domain points to the same IP address before/after).

For example: Consider that Let's Encrypt is used by a service running
a.example.com, where the A record of a.example.com points to 192.0.2.1 and
198.51.100.1, with the record pointing to 198.51.100.1 being a stale DNS
record, and 198.51.100.1 being an address for a VM instance in a cloud
provider's pool. If an attacker now finds a way to allocate 198.51.100.1,
she might also use Let's Encrypt to obtain a valid certificate. The
certificate request by the attacker is indistinguishable from the outside
and from interpreting the certificate transparency logs from what the
legitimate operator would do to fix the service behind 198.51.100.1, which
previously went stale.

Since ACMEv2 is currently in draft, we think now is a good idea to bring up
our proposal in the hope to incorporate parts of it into the draft. It
relates closely to Yaron's and Jacob's earlier discussion on HTTPS
authorization (acme-acme) and -to some degree- also to CAA (acme-caa).

While stale DNS records are technically a DNS issue, we think that adding
additional language to "11. Operational Considerations" is in the interest
of the standard given the impact that it can have in the real world.

We also propose, focusing on high-risk targets, a stricter issuance policy:

If a valid certificate (e.g., issued by the same operator or a set of
operators, checked via CT logs) exists for the requested domain, then the
current challenge should be signed by the key. If the last certificate has
expired, a grace period set by operators could apply (e.g., 1 month or 3
months). If the expiration date has passed a long time ago, or if no grace
period is used by the operator, then a second channel should be used to
verify the request (e.g., DNS CRP).

This stricter issuance policy has the following benefits:

* No burden in the normal and regular case
* No (significant) burden if a domain changes ownership legitimately
* Only minor and a one-time burden if a certificate has expired recently
(depending on a grace period being used)

At the same time, it prevents issuing certificates for domains that have
stale DNS records (due to caching or not updated yet). Furthermore, it is a
solution independent of CAA records, which is not a solution for all
problems arising from stale DNS records (e.g., if a DNS record is stale, it
points (A RR) to a cloud IP that an attacker gained control over, a CA is
designated as allowed to issue certificates via a CAA RR, and the
designated account at the CA is using an email address pointing to the same
domain; in this case, the attacker can request a certificate valid for some
months, giving ample time for an attack).

While one could argue that an attacker could use a different
domain-validating CA all together, HTTP Public Key Pinning could prevent
her attacks in practice.

In respect to high-risk targets, we are seeing significant staleness issues
for domains pointing to IP addresses at various cloud providers in our
study. Additionally, from our results, requesting the respective IP
addresses quickly and automatically is practical for an attacker.
Therefore, we would recommend to classify domains pointing to cloud
providers as high-risk targets; practically, this should be a decision made
by the operator and the standard should only inform the operator about the
issue and possible solution.

Please let us know what you think about the proposal. We would also be
happy to share a version of our paper once we have finalized it for
submission in the upcoming weeks.

Best,
Kevin

---

From: Ilari Liusvaara <[email-masked]>
Date: Thu, 20 Jul 2017 12:52:55 +0300

> domain points to the same IP address before/after).

The methods in base spec just fundamentially rely on security of DNS.
The only way to remove dependence on routing would be:

1) Deploy DNSSEC and
2) Add acme-CAA record restricting validation to DNS.

> legitimate operator would do to fix the service behind 198.51.100.1, which

Stale IP addresses cause problems when renewing (however, that does not
occur that often). Also, due to recent "fixes", IPv6 stale addresses can
be missed (there are just so many folks with busted DNS configuraiton).

And I think that would also cause large amounts of problems for users
if the stale record is IPv4 (if it is IPv6, then "happy eyeballs" hides
the problem).

> verify the request (e.g., DNS CRP).

There was PoP challenge in earlier versions of ACME spec for exactly
that. However, it was removed, because it seemed to be more trouble
than worth, and because it signed JSON messages with key intended for
something else, which is cryptographically not kosher (and contexts are
unusable in real world).

In the "order flow" change, did put the CSR first so one can look up
the requested key first. However, this would cause major problems if
trying to change keys (and many (I am not among them) believe that
rotating private keys often is important).

This would also complicate "disaster recovery" where the previous keys
are lost (usually due to admin mistake, and backups being AWOL like
usual). You would not believe how often that happens (and I hear just
a small subset).

> (depending on a grace period being used)

Err, what? These don't seem even remotely right to me.

> months, giving ample time for an attack).

Caching should not be a big issue: LE limits the cache validity to
pretty short values.

How many times has LE issued certificate to wrong actor due to stale
DNS data (I am aware of at least one DNS hijack incident, but
protecting against those is considered too high bar for CAs, even
DNSSEC would not have helped).

> domain-validating CA all together, HTTP Public Key Pinning could prevent

Unfortunately, HPKP is pretty hard to deploy:

CA HPKP has two major problems:

- CAs rotating intermediate keys in unpredictable ways (upcoming LE
  ECDSA intermediate anyone?).
- Chain-building problems, especially with Chrome. Sending the pinned
  intermediate in legal chain does not imply that HPKP will pass!

EE HPKP does not fail due to CAs rotating keys or chain-building
issues, however:

- One needs to be serious about backups.

And EE HPKP would take care of this problem by itself anyway.

> issue and possible solution.

Unfortunately domains pointing at cloud provoders is pretty common,
and classifying those as high-risk would cause major issues for
legimate users.

> submission in the upcoming weeks.

I think it is a bad bad idea.

-Ilari

---

From: Sean Leonard <[email-masked]>
Date: Thu, 20 Jul 2017 10:16:56 -0700

I reviewed this draft.

The concerns raised about the textual transmission of certificates, have not been fully addressed. But at least they have been partially addressed, so that progress is appreciated.

However since it’s my bad that I have not had the time to flesh out a response, we can let this go if the certificates are just transmitted “according to [RFC7468]”, and as text/plain. I would also advocate stronger language about the format, namely, that it is just -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----, separated only by newlines (no other text, supplemental or otherwise). I also don’t see why a client “‘SHOULD’ verify that the ‘file’ contains only encoded certificates. “MUST” would be better, and closes security holes. (MUST is also essentially stated by the sentence that follows.) If there is any “SHOULD”, the client “SHOULD” verify that the public key of the certificate matches the public key of the submission. Also it’s not a ‘file’, it’s unnamed content.

Given that the payload is text (and, in particular, no supplementary text), you may want to note that it’s charset=us-ascii. However that is the default anyway.

I can supply draft text if desirable.

Regards,

Sean

> On Jun 21, 2017, at 12:00 PM, [email-masked] wrote:
> 
> 
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
> This draft is a work item of the Automated Certificate Management Environment of the IETF.
> 
>        Title           : Automatic Certificate Management Environment (ACME)
>        Authors         : Richard Barnes
>                          Jacob Hoffman-Andrews
>                          James Kasten
> Please note that it may take a couple of minutes from the time of submission

---

From: Jacob Hoffman-Andrews <[email-masked]>
Date: Thu, 20 Jul 2017 11:46:22 -0700

On 07/20/2017 12:18 AM, Kevin Borgolte wrote:
> we are currently conducting a measurement study about DNS staleness
> issues with a focus on IP address churning.
Excellent, I look forward to reading it!
> We encountered the issue that Let's Encrypt (and ACME in general) can
> be (ab)used to request and receive a valid certificate for a domain,
> as long as the attacker obtains access to the IP address to which the
> DNS record points.
This is true of HTTP and TLS based validation in general. I think the
place to address these issues is not in the ACME standard, but
potentially as a separate IETF draft with the goal of later adoption by
the CA/Browser Forum. For instance, if HTTP and TLS validation were
required to contact every IP address they found in DNS, that would
mitigate the stale DNS issue you raise. On the other hand, it would hurt
usability and reliability. And my intuition is that truly "high risk"
domains (those with lots of users, or important data) do not tend to
leave stale entries in DNS, because it would harm performance. Hopefully
your study will shed light on that assumption!
> operators, checked via CT logs) exists for the requested domain, then
> the current challenge should be signed by the key. If the last
> certificate has expired, a grace period set by operators could apply
> (e.g., 1 month or 3 months). If the expiration date has passed a long
> time ago, or if no grace period is used by the operator, then a second
> channel should be used to verify the request (e.g., DNS CRP).
As Ilari mentioned, this was originally in the plan for ACME, but we
removed it because it was not fully baked. I think it's definitely
valuable to consider a successor as a separate document. Currently the
idea I have running around my head is to specify an HTTPS challenge
alongside the HTTP challenge, where the server expects to receive a
currently valid certificate that chains either to its own roots or to
some other roots it chooses to trust. This would be an optional extra
level of validation that CAs could apply to domains they consider high
risk. Upsides: Easy to implement for the both the CA and the Subscriber.
Downsides: If all certificate keys are lost, issuance would be blocked;
HTTPS validation still assumes that it's hard for third parties to
upload unauthorized files to certain paths on the web server.

---

From: Kevin Borgolte <[email-masked]>
Date: Thu, 20 Jul 2017 23:54:26 -0700

Hi Jacob,

> Excellent, I look forward to reading it!

Great! We will be happy to reach out to you in a separate thread if
you are interested in reading a draft version, and providing some
feedback?

> your study will shed light on that assumption!

Thanks for the suggestion about the IETF draft, we will look into. I
do not think anyone of us has any prior IETF draft experience, so
hopping onto an existing one would be the better option for us :-)

>From our other response to Ilari:

Our definition of stale DNS seems be slightly different or broader
than yours: specifically, we also consider domains as stale for which
there is only A record, and not only cases in which a single IP
address out of multiple might be stale. For instance, a practice we
encountered quite often is that VM instances on a cloud provider are
freed and the associated IP addresses are released, but the respective
DNS records are not updated for some days. In turn, someone else can
claim the same IP address and then request a certificate for the
domain. In practice, some hours are often sufficient to claim a
released address to successfully launch the attack (which might be
possible with caching depending on the used nameservers) and we have
seen some cases with customer-specific sub-domains being susceptible
to the attack (which might be a worthwhile phishing target).

We definitely agree with you that requiring all IP addresses they
found in DNS to complete HTTP/TLS challenges to combat stale DNS is
not a solution, simply for performance and reliability reasons as you
mentioned.

> upload unauthorized files to certain paths on the web server.

Possibly overly naive, but a solution here could be:

* Adapt the HTTP challenge to HTTPS, requiring a valid certificate chain.
* Upon receiving a certificate request, offer HTTPS and a second
challenge that is considered more trustworthy (DNS? out-of-band?).
* Default to attempting HTTPS, fail if the certificate chain is not
following the stricter policy.
* The remaining challenge is DNS/out-of-band.

It would introduce a stricter policy, but would not require changing
any language in the current draft, but only require adding two
sub-sections:

1. HTTPS challenge
2. How and why to use a stricter policy

Furthermore, a CA could default to HTTPS over HTTP if it has issued a
certificate to the same domain in the past.

Best,
Kevin

---

From: "Salz, Rich" <[email-masked]>
Date: Fri, 21 Jul 2017 11:24:06 +0000

Sean, at the meeting we agreed to do more in this area; look for new content from the editors.

--  
Senior Architect, Akamai Technologies
Member, OpenSSL Dev Team
IM: [email-masked] Twitter: RichSalz

> -----Original Message-----
> From: Sean Leonard [mailto:[email-masked]]
> Sent: Thursday, July 20, 2017 7:17 PM
> To: ACME WG <[email-masked]>
> Subject: Re: [Acme] I-D Action: draft-ietf-acme-acme-07.txt
> 
> I reviewed this draft.
> 
> The concerns raised about the textual transmission of certificates, have not
> been fully addressed. But at least they have been partially addressed, so that
> progress is appreciated.
> 
> However since it’s my bad that I have not had the time to flesh out a
> response, we can let this go if the certificates are just transmitted “according
> to [RFC7468]”, and as text/plain. I would also advocate stronger language
> about the format, namely, that it is just -----BEGIN CERTIFICATE----- and -----
> END CERTIFICATE-----, separated only by newlines (no other text,
> supplemental or otherwise). I also don’t see why a client “‘SHOULD’ verify
> that the ‘file’ contains only encoded certificates. “MUST” would be better,
> and closes security holes. (MUST is also essentially stated by the sentence
> that follows.) If there is any “SHOULD”, the client “SHOULD” verify that the
> public key of the certificate matches the public key of the submission. Also
> it’s not a ‘file’, it’s unnamed content.
> 
> Given that the payload is text (and, in particular, no supplementary text), you
> may want to note that it’s charset=us-ascii. However that is the default
> anyway.
> 
> > submission until the htmlized version and diff are available at tools.ietf.org.
> >