Subject: [DNSOP] How Slack didn't turn on DNSSEC

From: "John Levine" <[email-masked]>
Date: 30 Nov 2021 13:38:07 -0500

This blog post has been making the rounds. Since it is about a
sequence of DNS operational failures, it seems somewhat relevant here.

https://slack.engineering/what-happened-during-slacks-dnssec-rollout/

tl;dr first try was rolled back due to what turned out to be an unrelated failure at some ISP

second try was rolled back when they found they had a CNAME at a zone
apex, which they had never noticed until it caused DNSSEC validation
errors.

third try was rolled back when they found random-looking failures that
they eventually tracked down to bugs in Amazon's Route 53 DNS server.
They had a wildcard with A but not AAAA records. When someone did an
AAAA query, the response was wrong and said there were no records at
all, not just no AAAA records. This caused failures at 8.8.8.8 clients
since Google does aggressive NSEC, not at 1.1.1.1 because Cloudflare
doesn't.

They also got some bad advice, e.g., yes the .COM zone adds and
deletes records very quickly, but that doesn't mean you can unpublish
a DS and just turn off DNSSEC because its TTL is a day. Their tooling
somehow didn't let them republish the DNSKEY at the zone apex that
matched the DS, only a new one that didn't.

It is clear from the blog post that this is a fairly sophisticated
group of ops people, who had a reasonable test plan, a bunch of test
points set up in dnsviz and so forth.  Neither of these bugs seem
very exotic, and could have been caught by routine tests.

Can or should we offer advice on how to do this better, sort of like
RFC 8901 but one level of DNS expertise down?

R's,
John

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 1 Dec 2021 01:39:19 -0500

> On 30 Nov 2021, at 1:38 pm, John Levine <[email-masked]> wrote:
> 

The main advice that comes to mind is to use a DNS hosting provider
with a proven (multi-year) record of doing DNSSEC reliably.

If the DNS hosting provider where Cloudflare, Google, OVH, one.com,
TransIP, ... the implementation would have been correct.

Clearly Route 53 wasn't quite ready for prime time.  It is not clear
how we can help customers know which providers have solid implementations.
I don't know of any mainstream certification programs that would do the
job.

-- 
	Viktor.

---

From: Philip Homburg <[email-masked]>
Date: Wed, 01 Dec 2021 08:35:47 +0100

>very exotic, and could have been caught by routine tests.

It not clear to whether or not they did ZSK and KSK key rollovers
on test zones and on minor zones. If they didn't, thats a good way to
get in trouble later on.

The main lesson learned from this incident seems to be to always create
a test zone with content identical to that of the main zone and fully
test that zone.

A common lesson, also not mentioned, is to have low TTLs for stuff you
control. It would not have helped with the DS record. But the discussion
about the ZSK being lost would have been helped with a low TTL in the DNSKEY
RR set.

---

From: Mark Andrews <[email-masked]>
Date: Wed, 1 Dec 2021 18:30:27 +1100

Add tests to DNSVIZ to catch this sort of error. There is an open issue for this. 

-- 
Mark Andrews

> On 1 Dec 2021, at 05:38, John Levine <[email-masked]> wrote:
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop