Subject: [TLS] ESNI and Multi-CDN

From: Ben Schwartz <[email-masked]>
Date: Tue, 18 Dec 2018 12:29:56 -0500

I'd like to propose a solution to the ESNI + Multi-CDN problem (which has
been discussed a lot on this list already).  My suggestion is that we
define the ESNI DNS record format as optionally including "stapled" A/AAAA
records.

Server operators would have the option to publish an ESNI record that only
contains an ESNIKeys structure (like the current TXT record), or to publish
an ESNI record that also includes IPv4 and/or IPv6 addresses.  (A
Sufficiently Advanced authoritative DNS server would generate such records
automatically.)  This kind of address stapling would only be required of
CDN operators who want to support multi-CDN deployments.

Clients would issue A, AAAA, and ESNI queries in parallel (as with the
current TXT record).  If an ESNI record exists, and it contains IP
addresses, the client discards the results of the A or AAAA query.

Advantages:
 - No possibility of mixing up which ESNIKeys goes with which server IP.
 - No changes needed to recursives (which would treat the entire record as
opaque).
 - Doesn't add roundtrips at any stage.
 - Fully compatible with existing CNAME delegations.
 - Only adds complexity to the most sophisticated party (CDNs who want to
enable multi-CDN).
 - Not limited to one protocol (e.g. HTTPS).

Disadvantages:
 - IP address data is duplicated (10-100 extra bytes in the ESNI record),
and single-stack (e.g. v4-only) clients would still get both sets of IP
addresses.
 - Clients must wait for the ESNI query to complete before sending a TCP
SYN.  (Any resulting delay should be very small.  Does not apply to TFO or
QUIC.)
 - Support requires implementation work in authoritatives, especially for
load-balancing or geotargeting.
 - If IP stapling is rarely used (or rarely _not_ used), some clients might
fail to implement the rare case properly.

--Ben Schwartz, Jigsaw

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 18 Dec 2018 19:56:03 +0000

*   I'd like to propose a solution to the ESNI + Multi-CDN problem (which has been discussed a lot on this list already).  My suggestion is that we define the ESNI DNS record format as optionally including "stapled" A/AAAA records.

As in a multiple response?  That might be interesting, but it allows an adversary to just strip those responses, right?

> This kind of address stapling would only be required of CDN operators who want to support multi-CDN deployments.

Or anyone who maintains DNS records for a site that wants multi-CDN.  Many of our customers, for example, maintain their own DNS.  It’d say its common because they want switch quickly (very short TTL).  Yes, this usually is okay because the initial redirection is done via CNAME, but it is worth calling out that explicitly.

                /r$

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 18 Dec 2018 15:09:52 -0500

On Tue, Dec 18, 2018 at 2:56 PM Salz, Rich <[email-masked]> wrote:

>
>    - I'd like to propose a solution to the ESNI + Multi-CDN problem
>    (which has been discussed a lot on this list already).  My suggestion is
>    that we define the ESNI DNS record format as optionally including "stapled"
>    A/AAAA records.
>
>
>
> As in a multiple response?  That might be interesting, but it allows an
> adversary to just strip those responses, right?
>

I don't think I mean "multiple response" in the DNS sense.  What I mean by
"stapling" is "the ESNI RRTYPE has a defined format that can convey both
the ESNIKeys structure and some IP addresses".  This is all RFC
3597-compliant, and is a unified opaque blob to DNSSEC.

> > This kind of address stapling would only be required of CDN operators
> who want to support multi-CDN deployments.
>
>
>
> Or anyone who maintains DNS records for a site that wants multi-CDN.  Many
> of our customers, for example, maintain their own DNS.  It’d say its common
> because they want switch quickly (very short TTL).  Yes, this usually is
> okay because the initial redirection is done via CNAME, but it is worth
> calling out that explicitly.
>

OK, to be precise, the stapling would be required of multi-CDN customers
_if_ they do not use CNAME, and instead maintain the various CDNs' A/AAAA
records in their own zone.  As I think you are noting, this is unusual;
most multi-CDN customers use CNAME, which (in this design) means that they
do not have to think about ESNI at all.

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 18 Dec 2018 23:14:35 +0200

> addresses, the client discards the results of the A or AAAA query.

I do not think this will work:

- The CDNs need control of ESNIkeys
- If you hand them this control, you can hand over address control at
  the same time.
- Now you are in HTTP service discovery territory.

I am not saying that HTTP service discovery is undesirable (it is one
of those perennial topics that are not seemingly bad ideas but never
seem to get done).

Conversely, if one had HTTP service discovery, almost everything about
the CDN problems with ESNI would be solved instantly (what is not
solved by the same-owner rule).

-Ilari

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 18 Dec 2018 16:26:45 -0500

> of those perennial topics that are not seemingly bad ideas but never

Sorry, I don't understand.  What are you saying will not work?

This proposal is not HTTP-related, nor is it any kind of service
discovery.  A CDN necessarily controls both the addresses and ESNIKeys;
this proposal just stores that info in one RR instead of two.

---

From: Ilari Liusvaara <[email-masked]>
Date: Tue, 18 Dec 2018 23:49:28 +0200

> this proposal just stores that info in one RR instead of two.
 
What I meant is that one must be able to redirect the ESNI lookup
somehow to the CDN itself. Otherwise updating the ESNI keys is too
hard due to excessive coordination required. And if you can somehow
redirect the ESNI lookup, you could redirect the address lookup using
the same mechanism.

And what that results is a service discovery mechanism. And turns
out that HTTP is pretty much the only widely used protocol that does
not have service discovery mechansim (e.g., SMTP uses MX, XMPP uses
SRV, etc..).

-Ilari

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 18 Dec 2018 16:59:35 -0500

> redirect the ESNI lookup, you could redirect the address lookup using

Yes.  This proposal assumes that these lookups are in fact the same, i.e.
they are both for the same name (www.example.com), and this entire name is
redirected (i.e. delegated) to the CDN via CNAME (which is not
type-specific).  This is the typical deployment pattern for CDN customers.

> And what that results is a service discovery mechanism.

No, it's just CNAME.

> And turns

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 18 Dec 2018 22:46:29 +0000

Here is how it usually works.  Not everyone necessarily does it this way, but in our experience almost all of them do.

1. Sites use a CNAME, usually using a host-specific name.  E.g., www.akamai.com is a CNAME to www.akamai.com.edgekey.net  The DNS entry for the origin may be controlled by Akamai (under direction of our customer, of course), or it may be controlled by them or by something like a CDN-brokerage company.  For example, www.paypal.com CNAMEs to www.glob.paypal.com, where GLB presumably stands for Global Load Balancer.

2. Sometimes the CDN owns the DNS entry. In my experience this is more common for "anycast" types of services. For example, uureading.org returns A/QuadA records pointing into a CloudFlare address block.

Having the ESNI RRtype include optional a/quadA records is something we have talked about internally. If so, it should be part of the RRtype definition, of course. We came to the same idea. Of course, we will then have to fight the intent to make this "generic server record" data such as draft-nygren-service-bindings. :)

DNS is a strange and wondrous beast, like a bear riding a bicycle.  We should make sure that DNS folks are heavily involved in this draft.

---

From: Ilari Liusvaara <[email-masked]>
Date: Wed, 19 Dec 2018 13:27:48 +0200

On Tue, Dec 18, 2018 at 10:46:29PM +0000, Salz, Rich wrote:
> Here is how it usually works.  Not everyone necessarily does it this
> way, but in our experience almost all of them do.
> 
> 1. Sites use a CNAME, usually using a host-specific name.
> 
> 2. Sometimes the CDN owns the DNS entry. 

There is also:

3. The site owns the zone apex DNS entry. And since it is zone apex, no
CNAME can be used (yes, everything falls apart in practice if you do).
So zone owner inserts a bunch of A/AAAA records pointing at the CDN (or
worse, CDNs).

Fixing the above is a perennial topic. Proposals include CNAME-like
mechanisms scoped to just addresses, and HTTP service discovery
mechanisms (as most popular things other than HTTP already do service
discovery).

> Having the ESNI RRtype include optional a/quadA records is something
> we have talked about internally. If so, it should be part of the RRtype
> definition, of course. We came to the same idea. Of course, we will
> then have to fight the intent to make this "generic server record"
> data such as draft-nygren-service-bindings. :)

The only use for addresses in ESNI records I can come up with is
scoping keys to practicular addresses. But one could do that by
including address masks in ESNI records so clients can match ESNI
keys to addresses without breaking database normalization.

And if one is dealing with the third case above, the solutions are
basically service discovery. And then one presumably would want it to
be service-specific (as there are other rarer services that also lack
discovery besides HTTP).

> DNS is a strange and wondrous beast, like a bear riding a bicycle. 
> We should make sure that DNS folks are heavily involved in this draft.

Here are some tips:

- If you have name references, encode those in DNS wire format and
  preferably put them first, so that recursives can follow them and
  include records they think are helpful.
- Do not assume the above happens (but do assume it may happen). Client
  has to act as a backstop.
- DNS is 8-bit clean just fine with new types.
- Records do not need explicit terminators.
- There can be many records for each type on the same owner.
- The unit of atomicity is set of records with the same (owner,
  class, type) combo. 
- The 64kB limit is not just for individual record but whole set of
  records with (owner, class, type). And to be safe, limit it to 63.5kB
  minus 12 bytes per record.

The following are Bad Idea:

- Using type or class ANY (a.k.a. *). 
  - Type ANY does the wrong thing.
  - Class ANY does completely nonsensical thing.
- Defining a new class.
- Using class other than IN (Internet).
- If you use labels, using lots of different labels at outermost level.
- Putting anything into new type that DNS can not treat as a blob.
- Assuming authoritatives do anything smart.
- Assuming authoritatives follow any references.
- Assuming different types are in any way atomic w.r.t. each other.
- Copying TXT record wire format.

-Ilari

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 19 Dec 2018 13:47:35 -0500

> mechanisms (as most popular things other than HTTP already do service

Agreed.  This proposal is not meant to address the CNAME-at-apex problem,
and does not make that situation better or worse.

> scoping keys to practicular addresses.

Yes, that is the multi-CDN ESNI problem that we are trying to solve.  Each
CDN has its own ESNIKeys and its own IP addresses.

> But one could do that by

No, this doesn't work.  If the client has a AAAA RRSET and an ESNI RRSET,
and the ESNI RRSET contains a mask that is not compatible with the AAAA
RRSET, then the client can tell that it has the wrong IP addresses, but it
has no way to acquire the right IP addresses.

> _______________________________________________
> TLS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tls
>

---

From: Ilari Liusvaara <[email-masked]>
Date: Wed, 19 Dec 2018 21:23:43 +0200

> has no way to acquire the right IP addresses.

At least the client can tell the result is not going to work and
disable ESNI. Whereas with addresses there is no indication anything
is wrong, leading to potentially unrecoverable failure.

-Ilari

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 19 Dec 2018 19:33:14 +0000

So on the one hand, fallback to public ESNI seems likely and on the other hand, communication fails.  Not much difference?