Subject: Re: [TLS] Multi-CDN and ESNI

From: "Salz, Rich" <[email-masked]>
Date: Tue, 23 Oct 2018 17:58:42 +0000

I think perhaps we need to take a step back and explain something that might not be well-known outside the community of CDN’s and their customers.  It is not uncommon for (admittedly larger) origins to use multiple CDN’s, and to switch among them. This can be done on a per-request basis, because of things like contractual arrangements that make one preferable, or it can be done globally but switched in a matter of seconds because of a short TTL on the www.example.com<http://www.example.com> entry.

The issues that Mike discusses impact on this, somewhat negatively.

A quick hack thought is to allow multiple entries in the TXT record, forcing a wee bit more work on the CDN.

---

From: Patrick McManus <[email-masked]>
Date: Tue, 23 Oct 2018 17:27:50 -0400

Definitely agree this is something that needs to be addressed..

As Mike notes, the fundamental issue is that there are 2 pieces of
information that are statefully related (the key and address) but obtained
statelessly from each other and can therefore come from un-coordinated
sources. 2 CDNs are commonly un-coordinated sources.. but I've seen this
with other kinds of cloud providers too. All of which should be target
audiences of ESNI (because they terminate a variety of hostnames on a
smaller number of addresses).

fwiw there is a muddled github issue open on this in the old
individual-draft repo:
https://github.com/ekr/draft-rescorla-tls-esni/issues/35 .. which I'll
re-open when the tls-wg repo starts being used for this draft.

I see three solution spaces.

1] As Rich suggests, you can coordinate the uncoordinated sources to have
one keying record to rule them all. I suspect that's actually not good for
anonymity unless it somehow had a normalized global set .. and given the
one way nature of a CNAME pointer, I feel like this would be really fragile.

2] you could scope the keys to only be valid for certain addresses by
putting the valid addresses in the key record. This wouldn't help you do
ESNI when they didn't match, but at least you could avoid hard failure. (or
you could ignore the addressing record but that might be a bridge too far?)

3] My preferred approach, you could scope the keys to the same intermediate
name. So if www.example.com -> cname www.cdn-a.com -> ESNI record
www.cdn-a.com we could make a rule that the ESNI record would only apply to
address records with a final intermediate of www.cdn-a.com. (I'm presuming
we'll shift from TXT to ESNI RRTYPE without a prefix, but I don't think it
matters for this..).. when there is a mismatch you could use the addressing
intermediate as a place to try a new ESNI query from.. That's a perf
penalty, but its only paid in the hopefully rare case where these things
are unsyncd.

#3 is admittedly a bit non traditional, but I spent a bit of time looking
at the features of DNS APIs that already have the necessary surface to make
a query for a new RRTYPE... and the cname intermediates are indeed
generally available.. its true of getdns, res_query, the IOS and MacOS
interfaces, DnsQuery* on windows, and of course the custom dns stacks in
cronet/android and firefox/doh could do it.

I hope to work this into a PR.. my first attempt wasn't very readable, but
I'll try again tomorrow.

-P

On Tue, Oct 23, 2018 at 1:59 PM Salz, Rich <[email-masked]> wrote:

> I think perhaps we need to take a step back and explain something that
> might not be well-known outside the community of CDN’s and their
> customers.  It is not uncommon for (admittedly larger) origins to use
> multiple CDN’s, and to switch among them. This can be done on a per-request
> basis, because of things like contractual arrangements that make one
> preferable, or it can be done globally but switched in a matter of seconds
> because of a short TTL on the www.example.com entry.
>
>
>
> _______________________________________________
> TLS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tls
>

---

From: Nick Sullivan <[email-masked]>
Date: Tue, 23 Oct 2018 17:45:28 -0700

This line of commentary describes one instance of a more general situation
that is unrelated to the multi-provider case: what happens when you connect
to a server that doesn't know the ESNI key you're using? This can even
happen on a single provider due to DNS caching issues, for example.

The two cases are related and there are two features missing from the
current ESNI draft that might be useful to mitigate these scenarios:
1) Allow multiple simultaneous ESNI keys. The DNS provider could present
ESNI keys from multiple HTTPS providers (perhaps as different DNS records)
and the client could choose to send multiple entries in its ESNI extension,
each encrypted to a different key. This would likely solve the multi-CDN
scenario at the cost of some clienthello bloat.
2) Allow in-band distribution of ESNI keys if the server does not know the
ESNI key sent by the client. The server could prove ownership over a
specific long-term "fallback key" advertised in the ESNI record (or maybe a
fallback domain) when sent a TLS connection for which no ESNI extensions
can be decrypted, then in-band send back a new ESNI public key that the
client could use to send the SNI in a follow-up message. It's also possible
that handshake encryption alone is enough. This might require some TLS
message gymnastics, but doesn't seem impossible. I've heard more well-baked
versions of this idea batted around by others. (note that this incurs at
least one additional round-trip, but considering that a server not having
the active ESNI key should be rare, it is likely ok)

To comment specifically on your proposals, Patrick, I don't see #3 as a
workable solution since many authoritative DNS providers hide the fact that
a CNAME is used when returning A/AAAA records. This obfuscates the
indirection that would be necessary to do what you propose, and it does it
for a good reason (saving round trips on DNS resolution). Furthermore,
intermediate values of CNAME chains seems like a brittle mechanism to rely
on for policy enforcement.

Nick

On Tue, Oct 23, 2018 at 2:28 PM Patrick McManus <[email-masked]>
wrote:

> 3] My preferred approach, you could scope the keys to the same
> intermediate name. So if www.example.com -> cname www.cdn-a.com -> ESNI
> record www.cdn-a.com we could make a rule that the ESNI record would only
> apply to address records with a final intermediate of www.cdn-a.com. (I'm
> presuming we'll shift from TXT to ESNI RRTYPE without a prefix, but I don't
> think it matters for this..).. when there is a mismatch you could use the
> addressing intermediate as a place to try a new ESNI query from.. That's a
> perf penalty, but its only paid in the hopefully rare case where these
> things are unsyncd.
>

---

From: Patrick McManus <[email-masked]>
Date: Wed, 24 Oct 2018 11:37:03 -0400

Hey Nick,

> scenario at the cost of some clienthello bloat.

I would like to see this feature, but I don't think it addresses this
problem..

consider load balancer L, CDN A and CDN B. In a world in which L is aware
of A and B (duh) but not ESNI, and A wants to deploy ESNI but B has not yet
done so. You have the likely situation where at some point you've got an
address record for B but a key for only A and you run into hard failure.

even if A and B both want to do ESNI they need to be aware of each other
and coordinate key rotations somehow to synthesize records - or else you
end up spf like includes (please no). Requiring cross provider coordination
seems like assuming knowledge the providers don't reliably have and
probably could't reliably maintain. You could boot the problem to cusomter
configuration but.. 1] esni is so much better as automatic infrastructure
so CDNs can just turn it on and 2] origin names may not wish to disclose
all of their providers to each other.

> the active ESNI key should be rare, it is likely ok)
My concern would be a legacy TLS stack. The problematic case is when the
lookup for ESNI yields a key, but the host at the end of the A record has
never heard of it (or perhaps even heard of TLS 1.3). These are different
orgs, so its totally possible and inevitable in the sense that there are no
flag days. How does that work with this scheme?

This is tricky because the failure mode is so painful. provider A can
essentially break provider B by deploying ESNI - externalizing some of its
own risk. A and B don't have a super reliable way of knowing about each
other to even make sane risk judgments.

> on for policy enforcement.
Roughly, this is inspired by what krb does with canonical names in
practice. And it makes logical sense - the key applies to the canonical
name, not the alias. That said - this is a pain point I agree.

I'm glad you brought up flattening - I know of a handful of orgs that do
that. But do any of them do it across administrative domains or just as
optimizations to their own 'zone files'?  I understood it to be the latter,
and I don't think that would really be a problem here as long as it was
consistent across rrtypes.

another thought, Kazuho had mentioned reverse lookups at some point.
Definitely too painful for the normal case, but perhaps plausible as a
fallback? Doing a reverse lookup of the address record to obtain the
canonical key for the address (and absence of a reverse-lookup key means
that host doesn't really do ESNI so proceed the old way). it would be
backwards compatible but I worry both that its too fragile itself (IP
addressing can be pretty disjoint from http operations) and perhaps too
much of a time penalty to bear given the 'externalization' problem. But
let's throw it out there.

>> record www.cdn-a.com we could make a rule that the ESNI record would
>> only apply to address records with a final intermediate of www.cdn-a.com.
>> (I'm presuming we'll shift from TXT to ESNI RRTYPE without a prefix, but I
>> don't think it matters for this..).. when there is a mismatch you could use
>> the addressing intermediate as a place to try a new ESNI query from..
>> That's a perf penalty, but its only paid in the hopefully rare case where
>> these things are unsyncd.
>>
>> I hope to work this into a PR.. my first attempt wasn't very readable,
>> but I'll try again tomorrow.
>>
>> -P
>>
>>
>>
>>
>>
>>

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 24 Oct 2018 17:05:23 +0000

I think Mike mentioned the one keying record; I was suggesting multiple keying records.  But perhaps the one key record is a key-wrapping key?  Need to think about that a bit.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 24 Oct 2018 17:04:14 -0400

Here's a PR on one way to skin this cat.
https://github.com/ekr/draft-rescorla-tls-esni/pull/104/files