Subject: Re: [DNSOP] How Slack didn't turn on DNSSEC

From: Andrew Sullivan <[email-masked]>
Date: Wed, 1 Dec 2021 13:49:09 -0500

ObDisclaimer: work for Internet Society, speaking for me.

On Wed, Dec 01, 2021 at 01:39:19AM -0500, Viktor Dukhovni wrote:
>
>The main advice that comes to mind is to use a DNS hosting provider
>with a proven (multi-year) record of doing DNSSEC reliably.

Wouldn't that create a vicious circle in which the only way to start operating DNSSEC is already to have operated DNSSEC?

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Jim Reid <[email-masked]>
Date: Wed, 1 Dec 2021 19:37:12 +0000

> On 1 Dec 2021, at 18:49, Andrew Sullivan <[email-masked]> wrote:
> 

I think we’ve been in that vicious circle (or downward spiral) for several years now.

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 1 Dec 2021 16:06:24 -0500

> On 1 Dec 2021, at 2:37 pm, Jim Reid <[email-masked]> wrote:
> 

The graph at: https://stats.dnssec-tools.org/images/totalds.svg
does not look like a downward spiral to me.

But I also don't agree with Paul that one needs to be an expert to play
the game.  Tools are improving, and spinning up working DNSSEC with Knot,
BIND 9.16+, ... is increasingly easier.

Where things get more complex is in API integration with cloud providers,
bugs in the provider implementation that's recent and not fully baked, ...

These too will likely improved, but there will occasionally be issues when
some new managed service is introduced and users struggle to consume it,
and have complex unanticipated requirements.

-- 
	Viktor.

---

From: Paul Vixie <[email-masked]>
Date: Wed, 1 Dec 2021 13:19:05 -0800

Viktor Dukhovni wrote on 2021-12-01 13:06:
>> ...
> ...
> ...
with DNSSEC For Humans, we (ISC, at the time) made BIND9 as automatic 
and as user friendly as could be done with regard to DNSSEC operations. 
other implementations did likewise, especially including OpenDNSSEC.

and if you're an enterprise or hobbyist, that's quite good enough (no 
wizards are needed).

but if you're Slack or similar (cloud provider, ISP, MSSP, social 
network, xAAS provider), no automation will ever be good enough by 
itself (wizards will be needed.)

vixie

-- 
Sent from Postbox
<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 2 Dec 2021 01:30:11 -0500

> On 1 Dec 2021, at 4:19 pm, Paul Vixie <[email-masked]> wrote:
> 
> ... but if you're Slack or similar (cloud provider, ISP, MSSP, social network,
> xAAS provider), no automation will ever be good enough by itself (wizards will
> be needed.)

With that qualification, I think we're much less far apart.  Yes,
the operations team in a complex environment needs to be more
highly skilled, and not just in DNSSEC.

However, I do think that the skill level required need not always
be or remain "wizard".  They had some bad luck running into a not
fully baked stack on the provider end.

Unless of course the complexity of what's being attempted will
always rise to near or just past the skill level of the team,
boldly going to where no team has gone before.

In that case, even wizards will at times out-wizard themselves,
and this is then no longer about DNSSEC really.  Just that as
tools get better we're never content to keep solving the old
problems more reliably, we start to feel cocky about tackling
new bigger problems...

-- 
-- 
	Viktor.

---

From: "John Levine" <[email-masked]>
Date: 2 Dec 2021 21:41:49 -0500

>fully baked stack on the provider end.

While they should have caught the apex CNAME, I can't really blame
them for not anticipating that AWS Route 53 would have buggy
DNSSEC handling.  Wildcards are unusual but they're not *that* unusual.

I realize the problem with test suites that the test suite tends to
become the de-facto spec, but it might be nice to come up with a list
of tests that check for famous bugs.

R's,
John

---

From: Petr Špaček <[email-masked]>
Date: Fri, 3 Dec 2021 10:35:41 +0100

On 30. 11. 21 19:38, John Levine wrote:
> This blog post has been making the rounds. Since it is about a
> sequence of DNS operational failures, it seems somewhat relevant here.
> 
> https://slack.engineering/what-happened-during-slacks-dnssec-rollout/
> 
> tl;dr first try was rolled back due to what turned out to be an unrelated failure at some ISP
> 
> second try was rolled back when they found they had a CNAME at a zone
> apex, which they had never noticed until it caused DNSSEC validation
> errors.
> 
> third try was rolled back when they found random-looking failures that
> they eventually tracked down to bugs in Amazon's Route 53 DNS server.
> They had a wildcard with A but not AAAA records. When someone did an
> AAAA query, the response was wrong and said there were no records at
> all, not just no AAAA records. This caused failures at 8.8.8.8 clients
> since Google does aggressive NSEC, not at 1.1.1.1 because Cloudflare
> doesn't.
> 
> They also got some bad advice, e.g., yes the .COM zone adds and
> deletes records very quickly, but that doesn't mean you can unpublish
> a DS and just turn off DNSSEC because its TTL is a day. Their tooling
> somehow didn't let them republish the DNSKEY at the zone apex that
> matched the DS, only a new one that didn't.
> 
> It is clear from the blog post that this is a fairly sophisticated
> group of ops people, who had a reasonable test plan, a bunch of test
> points set up in dnsviz and so forth.  Neither of these bugs seem
> very exotic, and could have been caught by routine tests.
> 
> Can or should we offer advice on how to do this better, sort of like
> RFC 8901 but one level of DNS expertise down?

During DNS OARC 36, there were talks about an opportunity to write 
"Clarifications" RFC.

Personally I think the current RFC are clear how NSEC(3) bitmaps should 
look like, but I can also see that making sense various types of "NSEC 
lies", "type shotguns", etc. and their consequences requires way more 
research then reading RFC 4034.

Are there people on this list interested working on this?

I can contribute ideas and some proto-text, but as you can see for 
yourself someone needs to translate my texts to proper English so I 
should not be a document editor.

-- 
Petr Špaček