Subject: Re: [dns-privacy] Root Server Operators Statement on DNS Encryption

From: Tomas Krizek <[email-masked]>
Date: Thu, 1 Apr 2021 11:59:26 +0200

On 31/03/2021 23.28, Rob Sayre wrote:
> On Wed, Mar 31, 2021 at 2:16 PM Bill Woodcock <[email-masked]> wrote:
> 
>>
>> …and it’s measuring latency rather than server-side load.  I just checked
>> with our engineers, and it sounds like the server load per-query is more
>> like 3x-5x higher for the encrypted queries.
>>
> 
> Plenty of folks have evaluated the costs here. I'd prefer to discuss data
> rather than "checking with engineers". It's not really reasonable to
> measure "server load per-query" without a bunch of other data on how the
> TLS sessions are being created and maintained.
> 
> So, if you have some data you'd like to share with the list, that would 
be
> most welcome.

We've done some measurements comparing the server-load overhead of DoT
and DoH and while the exact results vary greatly with the client
behaviour, connection management and other parameters, I think 3x-5x
server-load overhead (compared to UDP) is a reasonable expectation. [1]

Our measurements were for resolvers, but the same methodology and tool
could also used for authoritative server measurements.

From latency point of view, it again depends on connection management
etc., but inevitably, some queries will have the additional cost of
establishing a TCP/TLS handshake. If anyone's interested in measuring
the impact in various network conditions, it's possible to simulate
latency with kernels' Network Emulator [2]. When we've tried that, the
results looked as I've described [3], but the exact outcome depends on
many variables.

[1] - https://indico.dns-oarc.net/event/34/contributions/782/
[2] -
https://dns-shotgun.readthedocs.io/en/stable/replaying-traffic/#emulating-link-latency
[3] - https://dns-shotgun.readthedocs.io/en/stable/latency-histogram/

-- 
Tomas Krizek
PGP: 4A8B A48C 2AED 933B D495  C509 A1FB A5F7 EF8C 4869

---

From: Petr Špaček <[email-masked]>
Date: Thu, 1 Apr 2021 14:04:31 +0200

> server-load overhead (compared to UDP) is a reasonable expectation. [1]

I will add couple more details about these measurements:

Data show that TLS algorithms in use *and* connection reuse parameters 
are crucial. Well-behaved clients over DoT with P256 certificate can be 
"only" ~ 2.9 x more expensive (on server) than plain UDP, thanks to the 
fact that TCP+TLS connection setup costs get amortized for longer 
connections.

For well-behaved clients who do not keep idle connections open the cost 
jumps up to 10x.

To generalize for the root server scenario, we need to consider that a 
particular well-behaved client should contact root servers rarely, which 
means that connection reuse is unlikely. For this reason, I believe that 
costs on the server side are likely to be closer to the 10x mark for CPU 
costs.

There are limitations to this measurement and generalization, however:
- Measured on stub<->recursive so the generalization might not apply, 
especially if recursive<->root has much better connection reuse 
conditions (seems extremely unlikely to me)
- No information about bandwidth
- No information about latency
- No information about memory consumption
- No information about DoS potential - we measured well-behaved clients

Petr Špaček  @  ISC

> [3] - https://dns-shotgun.readthedocs.io/en/stable/latency-histogram

---

From: Brian Haberman <[email-masked]>
Date: Thu, 1 Apr 2021 08:08:14 -0400

On 3/31/21 5:51 PM, Bill Woodcock wrote:
> 
> 
>> On Mar 31, 2021, at 11:49 PM, Stephen Farrell <[email-masked]> wrote:
>> The real issue IMO is not querying the root servers but
>> the TLDs. There are still performance issues to consider
>> of course but the business model and the value to the
>> person somewhere behind the recursive are quite different.
>>
>> I really wish we could stop all mixing up the roots with
>> the TLDs in this discussion.
> 
> Yep, exactly.
> 

The WG seems to fluctuate between wanting to treat all authoritatives
the same and thinking of the root as being different from TLDs. If you
recall during our interim meeting last year, we tried to keep them
separate and some folks complained.

I (with no hats on) would take the Root Server Operators statement as a
strong indication that they believe the root does need to be considered
separately from the TLDs.

Regards,
Brian

---

From: Bill Woodcock <[email-masked]>
Date: Thu, 1 Apr 2021 14:29:15 +0200

> On Apr 1, 2021, at 2:08 PM, Brian Haberman <[email-masked]> wrote:
>>> The real issue IMO is not querying the root servers but the TLDs. There are still performance issues to consider of course but the business model and the value to the person somewhere behind the recursive are quite different. I really wish we could stop all mixing up the roots with the TLDs in this discussion.
> 

That would be my preference.  And, to be clear, this is not a DNS technical issue, this is an Internet governance issue.  There is one, unitary root, which we all, perforce, share.  Which means that it has to work for everyone, all the time, and serve all purposes reasonably well.  It’s a common, shared resource, and so needs to be governed very carefully, very conservatively, and with a very light touch.

TLDs come in all shapes and sizes and flavors to suit all purposes.  If you don’t like one, try another.  They’re also governed in different ways…  some legacy, some under binding ICANN agreements, some country-code.  Although we’re in danger of all future stuff being homogenized into the “binding ICANN agreements” category, which has been the worst and most restrictive form of governance of the three.

So they’re definitely different, with different purposes and needs.

The standards-making process needs to recognize that, and not tar them with the same brush, just because they use (some of) the same protocols and are complementary parts of the same system.

                                -Bill

---

From: Vittorio Bertola <[email-masked]>
Date: Thu, 1 Apr 2021 14:38:05 +0200 (CEST)

> Il 01/04/2021 14:08 Brian Haberman <[email-masked]> ha scritto:
> 

This discussion is more in the realm of policy than in that of protocol development, though it affects future decisions on ADoTQ. We started to discuss the root part of the issue, but the TLD one is also not that straightforward. If ADoTQ imposed significant additional costs on TLD operators, one could imagine that at least some operators could try - through whatever governance process applies to each of them - to raise the domain name price to compensate for these costs. Would ICANN and national regulators be receptive to this request? Would users be willing to pay for encrypted TLD connections? What would be the reaction of the rest of the domain name market? These are policy questions that would possibly determine a decision to deploy ADoTQ or not, and thus the success or failure of the protocol at the TLD level.

By the way, there are past examples of use of the DNS root by web browsers that generated very significant load, and thus cost, for root operators, without a matching advantage for end-users - e.g. https://blog.apnic.net/2020/08/21/chromiums-impact-on-root-dns-traffic/ . It looks like there is the need for better coordination and reciprocal understanding between the three communities - browsers, root operators and TLD operators. Perhaps this could be the subject of a session at one of the upcoming DNS community/policy conferences, trying to solicit views and/or experiments on how ADoTQ would impact root and TLD operations.

-- 
Vittorio Bertola | Head of Policy & Innovation, Open-Xchange
[email-masked] 
Office @ Via Treviso 12, 10144 Torino, Italy

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 1 Apr 2021 14:51:33 +0200

On Wed, Mar 31, 2021 at 11:43:57PM +0200,
 Bill Woodcock <[email-masked]> wrote 
 a message of 74 lines which said:

> This isn’t a place for pointless thrashing around as a byproduct of
> someone’s unrelated agenda.  Which was, I think, the point of the
> statement.

But I wonder again who this "someone" is? I don't remember ever seeing
some one or some organization saying (at least publically) "root name
servers must support encryption of DNS queries now".

That's why the root server operator statement surprised me. They very
rarely issue collective statements. What were they replying to?

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 1 Apr 2021 14:55:33 +0200

On Wed, Mar 31, 2021 at 10:12:52PM +0000,
 Andrew Campling <[email-masked]> wrote 
 a message of 30 lines which said:

> My apologies if the stance of TLD operators is well known to most in
> this group,

I don't think so, this they are a very diverse group, working under
very different conditions.

> do sufficient TLD operators believe that it needs to be solved

Also, it is not just a matter of the operators of a TLD, every TLD has
a consistuency [add various words like "stakeholders" here] which may
have opinions, too.

---

From: Shane Kerr <[email-masked]>
Date: Thu, 1 Apr 2021 17:12:51 +0200

Bill,

On 31/03/2021 22.29, Bill Woodcock wrote:
> 
> 
>> On Mar 31, 2021, at 9:55 PM, Rob Sayre <[email-masked]> wrote:
>> I still don't understand the resistance here. Some data on what the impact would be still seems like the most helpful thing to move the conversation forward.
> 
> We have that:
> 
> https://vaibhavbajpai.com/documents/papers/proceedings/dot-pam-2021.pdf
> 
> Short story is that it’s 3x - 15x higher load on the servers, more delay, fewer people served, easier DDoS.
> 
> That’s a price that can be paid, if there’s a suitably corresponding benefit, and no more effective way to solve the problem.

One thing about DDoS is that any connection-oriented protocol (TCP, TLS, 
HTTPS, or QUIC) is that you don't need to overbuild your infrastructure 
to the same degree because spoofing source address is a lot harder and 
you can drop unsophisticated traffic floods at or near your edge.

In my mind this probably means changing from the 10x peak-traffic (or 
more!) over-build that authoritative providers have today to a much 
smaller factor.

For unencrypted TCP you can probably get back the 4x or so higher cost 
of running TCP vs. UDP. For encrypted traffic probably not, but I have 
no idea what the additional costs there are (I heard that earlier 
implementations of QUIC were CPU hogs, for example).

Cheers,

--
Shane

---

From: John Heidemann <[email-masked]>
Date: Thu, 01 Apr 2021 21:23:47 -0700

>server-load overhead (compared to UDP) is a reasonable expectation. [1]

We've published results of server-side CPU and memory at

Liang Zhu and John Heidemann.
  LDplayer: DNS Experimentation at Scale.
  In _Proceedings of the ACM Internet Measurement Conference_, Boston, Massachusetts, USA, ACM.
  October, 2018.
  <https://doi.org/10.1145/3278532.3278544>, <https://www.isi.edu/%7ejohnh/PAPERS/Zhu18b.html>.
  
Including replay of root DNS traffic (DITL) with it's current mix of UDP
and TCP and all-TLS.

We found (DNS over) UDP and TLS took about the same CPU on our hardware,
and pure TCP (without TLS) was actually lower CPU than UDP.  See Figure
11 and section 5.2.3 in the paper.

(Our code and data is available if you want to reproduce our experiments
on your hardware.)

IMHO, a bigger operational concern than CPU is actually memory---it's
easy to burn through a lot of RAM with TCP and TLS state---see section
5.2.2 and figures 13 and 14 in the paper.  It depends a lot on the TCP
and TLS timeout you use.  Fortunately it's less than 24GB at steady
state, which is not a problem for today's hardware.  (Although the
server should be prepared to early-terminate connections if it's under
memory pressure.)

Our experiments assume well configured clients and servers.  Although
not so important for load, optimizations make a huge difference
difference for latency.  Options like TCP fast open and TLS connection
resumption get some of the improvements in QUIC, but over TCP (see
<http://dx.doi.org/10.1109/SP.2015.18>,
<https://www.isi.edu/%7ejohnh/PAPERS/Zhu15b.html> for the details).  I
hope people evaluating DNS over QUIC compare against best-configured TCP
and not just basic TCP without optimizations.

   -John Heidemann