Subject: [Doh] DNS over HTTP/3?

From: bert hubert <[email-masked]>
Date: Mon, 19 Nov 2018 11:09:54 +0100

Hi everyone,

Now that we are still here, perhaps some work is left for us, but I'm not
sure. 

I've observed that the thousands of users of doh.powerdns.org (I also do not
know how this happened) take around 22 packets per DNS query/response. 

Larger scale adoption of TLSv1.3 might improve this somewhat, but it is a
big number.

I've also personally observed that a "slightly suboptimal" network
absolutely kills browsing performance in Firefox Nightly using DoH.  A naive
calculation shows that 0.5% packet loss turns into a 5% failure rate per DoH
query, which then can cause Head of Line blocking for further queries, which
cascades into "blank pages" getting rendered. 

Of course, once we have HTTP/3, DNS over HTTP/3 would suffer way less from
incidental packet loss, and in general there would be a lot less packets
too.

My question now is, is there any specific work to be done for DoH/3? Are
there number, priorities, features you'd want or not want to use to make
things work well?

In other words, to the sound of a groaning camel, do we need a draft? Or a
section in the HTTP/3 I-D?

And, perhaps somewhat more provocatively, should we maybe not start pushing
DoH/2 if it leaves people with a sub-standard experience, causing them to
disable DoH?  DoH/3 might be somewhat of a wait but it might prevent that
sour taste from developing.

I'd love to hear your thoughts!

	Bert

---

From: "Ralf Weber" <[email-masked]>
Date: Mon, 19 Nov 2018 12:01:01 +0100

Moin!

> know how this happened) take around 22 packets per DNS query/response.
Wow. Good for manufactures of 10GB ethernet cards ;-).

> cascades into "blank pages" getting rendered.
Hmm doh.powerdns.org resolves to one IP address while 
mozilla.cloudflare-dns.com resolves to two IP addresses. Not sure if 
that would make a difference as I don’t know what the Mozilla stub 
does. I also didn’t find anything in the RFC wrt to the 
failure/failover behaviour other than DoH follows standard HTTP 
mechanisms, but I have no idea what that means for unresponsive servers, 
and more important when a server is declared unresponsive.

[…]
> sour taste from developing.
Maybe there is a need for dohop (DNS over HTTP Operations) working 
group. I think there are still a lot of unknowns in the operations of 
DoH clients and servers, but the same is true for regular DNS and I’ve 
been working on that for decades now. Maybe a draft or some measurements 
would also help to get a discussion started.

So long
-Ralf
—--
Ralf Weber

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 19 Nov 2018 14:12:18 +0000

> know how this happened) take around 22 packets per DNS query/response. 

It would be valuable to know if this is this because your DoH server kills the entire connection after every DNS query, or because the clients are doing so.

> Larger scale adoption of TLSv1.3 might improve this somewhat, but it is a

DoH was designed for long-lived connections, so the number would normally be smaller after the initial connection.

> cascades into "blank pages" getting rendered. 

When you say "naive calculation", do you mean this is observed in the clients hitting your DoH server? That would indeed be scary. However, the Mozilla numbers tell a very different story. It would be good to know which one is true.

> there number, priorities, features you'd want or not want to use to make

DoH is just a specification for how to do a normal HTTP interaction, so there should be litter or no changes to DoH if the transport under HTTP changes. We'll know more when HTTP/3 is finished, but probably not before then.

> section in the HTTP/3 I-D?

We might, but we can't tell now.

> disable DoH?

If the implementations on the client or server side make using DoH bad, those implementations should be fixed. If they don't get fixed, we'll certainly hear about it.

> DoH/3 might be somewhat of a wait but it might prevent that

Without sufficient data, it's hard to say if there really is a sour taste.

--Paul Hoffman

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 19 Nov 2018 16:02:19 +0100 (CET)

> cascades into "blank pages" getting rendered.

Isn't this mostly because it uses (multiplexing over) TCP and TLS? Meaning 
that DoT will basically see roughly the same degredation?

(And regarding "blank pages", that's also a matter of what the 
browser/user-agnes does or doesn't do when it gets DoH timeouts...)

-- 

  / daniel.haxx.se

---

From: bert hubert <[email-masked]>
Date: Mon, 19 Nov 2018 16:12:09 +0100

On Mon, Nov 19, 2018 at 02:12:18PM +0000, Paul Hoffman wrote:
> It would be valuable to know if this is this because your DoH server kills
> the entire connection after every DNS query, or because the clients are
> doing so.

It does around 2 DNS queries per HTTPS connection on average dnsdist-doh
keeps open an HTTP/2 connection for 10 seconds if it becomes idle.

> DoH was designed for long-lived connections, so the number would normally be smaller after the initial connection.

I can only report what I measure based on several thousands of users.

> When you say "naive calculation", do you mean this is observed in the
> clients hitting your DoH server?  That would indeed be scary.  However,
> the Mozilla numbers tell a very different story.  It would be good to know
> which one is true.

My personal observation is that on a slightly bad network, turning off DoH
speeds up things massively and makes the web useable again.  This naively
makes sense, DoH uses way more packets and suffers from Head of Line
blocking, unlike plain DNS.

We have asked Mozilla to provide more details of their measurements, but
these have not been forthcoming.

Note that the graph published on the blog misses the 95% and 99% bins,
https://blog.nightly.mozilla.org/files/2018/08/DNS-over-HTTPS-Performance-Improvement.png
and does not include details on timeouts.

> If the implementations on the client or server side make using DoH bad,
> those implementations should be fixed.  If they don't get fixed, we'll
> certainly hear about it.

There is already a Firefox ticket I think, but with Patrick and Daniel gone
or gone soon, I am not sure who owns it now

> Without sufficient data, it's hard to say if there really is a sour taste.

Perhaps Mozilla can clarify further what they measured beyond the 7
milliseconds slowdown.

I may do some induced packetloss experiments to see if the "experience" can
be reproduced.

	Bert

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 19 Nov 2018 15:39:16 +0000

> keeps open an HTTP/2 connection for 10 seconds if it becomes idle.

Thanks, that seems to be your problem. Can the dnsdist DoH server package be configured to have an idle timeout that is more representative of typical browser use, such as on the order of minutes?

> blocking, unlike plain DNS.

Is that still true with a server timeout that is not so short?

> these have not been forthcoming.

Noted. It would be *really* useful to the community if there was better communication from them.

> and does not include details on timeouts.

Yep, and that would be useful too.

> or gone soon, I am not sure who owns it now.

Hopefully he or she will decloak soon.

> I may do some induced packetloss experiments to see if the "experience" can

That would be grand, but it would also be good to have such experiments when the server has a more sensible timeout value.

--Paul Hoffman

---

From: bert hubert <[email-masked]>
Date: Mon, 19 Nov 2018 17:04:14 +0100

On Mon, Nov 19, 2018 at 03:39:16PM +0000, Paul Hoffman wrote:
> Thanks, that seems to be your problem.  Can the dnsdist DoH server package
> be configured to have an idle timeout that is more representative of
> typical browser use, such as on the order of minutes?

I have measured with timeout of 100 seconds and 300 seconds. In both cases,
the packets per query ratio drops to 12, from 22. This is still around 6
times more than UDP, and I now carry around hundreds of additional
filedescriptors. But ok.

> Is that still true with a server timeout that is not so short?

I will need to measure again. Linux can inject packet loss, so this should
be doable.

	Bert

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Mon, 19 Nov 2018 11:17:45 -0800

> On Nov 19, 2018, at 7:12 , bert hubert <[email-masked]> wrote:
> 

Wouldn’t comparing to DNS-over-TLS be a more relevant/fair comparison?

DoH makes sense, I believe, if you need a long-lived/persistent connection because we get to take advantage of improvements in HTTP/3, etc, without “extra DNS work”.

If you don’t need a session (or encryption) or compare with clients that don’t, that’s never going to look favorable.

Many new DNS features require a session so as those gain adoption DoH should look more favorable.

Ask

---

From: Ted Lemon <[email-masked]>
Date: Mon, 19 Nov 2018 15:12:25 -0500

What new DNS features require a session?

On Mon, Nov 19, 2018 at 2:18 PM Ask Bjørn Hansen <[email-masked]> wrote:

> DoH makes sense, I believe, if you need a long-lived/persistent connection
> because we get to take advantage of improvements in HTTP/3, etc, without
> “extra DNS work”.
>
> If you don’t need a session (or encryption) or compare with clients that
> don’t, that’s never going to look favorable.
>
> _______________________________________________
> Doh mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/doh
>

---

From: bert hubert <[email-masked]>
Date: Mon, 19 Nov 2018 21:14:07 +0100

> Wouldn’t comparing to DNS-over-TLS be a more relevant/fair comparison?

The reason I compare DoH to 'plain DNS' is that it was proposed as the new
default for Firefox. And if it turns out that this new default will in fact
cause a performance degradation, it may be worth pondering skipping 'DOH/2'
as default and waiting for 'DOH/3'.

(note that Mozilla has let the "we'd like to turn on DoH by default"
wording stand, but also stated there are currently no plans yet to do so).

> If you don’t need a session (or encryption) or compare with clients that don’t, that’s never going to look favorable.

DNSCrypt still offers encryption without sessions, and it works well.
Perhaps DNSCrypt/DoH is the fairest comparison.

	Bert

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 19 Nov 2018 22:34:05 +0100 (CET)

> or gone soon, I am not sure who owns it now

One of my more recent changes to Firefox's DoH code was exactly a system that 
makes it switch off "TRR" when the DoH server repeatedly fails to respond 
correctly (in time).

But we're digressing, that's really a Firefox/webbrowser issue, not a general 
DoH issue.

-- 

  / daniel.haxx.se

---

From: Mukund Sivaraman <[email-masked]>
Date: Tue, 20 Nov 2018 17:00:56 +0530

> times more than UDP, and I now carry around hundreds of additional

This is not specifically about DoH, but I don't like the path DNS is
taking. I don't like the general push to heavy protocols for DNS and
have often commented about it on dprive and dnsop.

DNS historically has been mostly a light-weight single-request
single-response stateless protocol (TCP is used as a last resort vs. UDP
that's used in preference). The addition of a privacy layer could add
more weight, but we seem to be uninterested in exploring other stateless
or low-state ideas, outside TLS and even QUIC.

After the resolver work, the phase-2 introduction for resolver<->auth
communications[1] started with TLS which seemed illogical to me. There
must be empirical studies of consequences before protocols push through
to RFC. People such as Sara Dickinson at Sinodun have been studying the
performance and scalability of the privacy layer in DNS implementations
which is very commendable.

[1] https://tools.ietf.org/html/draft-bortzmeyer-dprive-resolver-to-auth-01

Some people such as Brian Haberman have taken an initiative to gather
requirements for resolver-auth privacy instead of jumping directly to
design which is again commendable.

dnsop and dprive should study this before pushing drafts to RFCs. What
is the worst case on resource utilization, number of round-trips and
number of packets to implement these layers? What could happen on
average in practice? Large operators who plan ahead ask these
questions. Can <implementation> handle TCP and DNS over TLS and DNS over
HTTPS in a future world at rates similar to UDP now?  I can picture the
overhead for UDP and TCP, but it is a lot more complicated with the
extra layers.

It's not ok to assume and hand-wave away that optimizations such as TCP
fastopen will avoid roundtrips and make up for performance. A TCP client
may not have a cookie for the average nameserver case for the SYN except
if it is for a "CDN/cloud" concentrated nameserver. I'm afraid these
things will encourage more concentration for performance vs. keeping the
internet distributed.

Facilities such as TCP fastopen are also experimental and we should not
rely on them until more time has passed. Server-side support for it is
still turned off by default in current versions of popular Linux
distributions (a system-level setting that can't be enabled by
applications such as a nameserver). Resolvers such as BIND have not yet
implemented TCP fastopen for making connections. fastopen also suggests
downgrades globally to regular 3-way TCP handshake from fastopen when
there is a flood, which is not an attack in the regular sense, but loses
the fastopen optimization if DNS relies on it to perform satisfactorily.

DNS is not a special protocol, but it is at the head of every internet
activity. It has to perform well and scale well. The chairs should
encourage studies into stateless methods to do privacy with low
roundtrip overhead. Other protocols pay heed to stateless methods, e.g.,
DNS COOKIE is stateless on the server side. TCP fastopen is also
stateless on the server side.

There is unexpected advice in RFCs too. When TCP fastopen is available
for the DNS over TCP case, it seems better for the TCP connection to be
closed than it be kept open because the open connection would revert to
slow-start phase after an idle period with typical DNS query traffic
patterns (unlike replies to a browser's HTTP requests that are typically
much larger and more in sequence) unless it is continually active. In a
greedy world, closing idle connections is something that authoritative
servers would prefer to do to get rid of state - resolvers can't rely on
having long-lived connections.

This has become a rant but I'm unhappy about the process that is pushing
new DNS features without sufficient evaluation. Modifying the transport
of DNS without studies may affect performance and scalability that we
may come to regret.

		Mukund

---

From: Az Mankin <[email-masked]>
Date: Thu, 22 Nov 2018 22:43:30 -0500

Bert,

>In other words, to the sound of a groaning >camel, do we need a draft? Or a

Please take a look at
draft-huitema-quic-dnsoquic-05

It was introduced in the QUIC WG and revised as QUIC evolved (and as some
prototype stacks by the authors gave results).  It needs a refresh now.
QUIC made it low priority while working on the HTTP transport.