Subject: Re: [DNSOP] How Slack didn't turn on DNSSEC

From: Mark Andrews <[email-masked]>
Date: Wed, 1 Dec 2021 19:35:52 +1100

Also stop hiding this breakage. Knot and unbound ignore the NSEC records which trigger this when synthesising.  All it does is push the problem down the road and makes it harder for others to do proper synthesis based on the records returned. 

-- 
Mark Andrews

> On 1 Dec 2021, at 18:36, Philip Homburg <[email-masked]> wrote:
> 
> 
>> 
>> It is clear from the blog post that this is a fairly sophisticated
>> group of ops people, who had a reasonable test plan, a bunch of test
>> points set up in dnsviz and so forth.  Neither of these bugs seem
>> very exotic, and could have been caught by routine tests.
> 
> It not clear to whether or not they did ZSK and KSK key rollovers
> on test zones and on minor zones. If they didn't, thats a good way to
> get in trouble later on.
> 
> The main lesson learned from this incident seems to be to always create
> a test zone with content identical to that of the main zone and fully
> test that zone.
> 
> A common lesson, also not mentioned, is to have low TTLs for stuff you
> control. It would not have helped with the DS record. But the discussion
> about the ZSK being lost would have been helped with a low TTL in the DNSKEY
> RR set.
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Vladimír Čunát <[email-masked]>
Date: Wed, 1 Dec 2021 09:54:55 +0100

On 01/12/2021 09.35, Mark Andrews wrote:
> Also stop hiding this breakage. Knot and unbound ignore the NSEC records which trigger this when synthesising.

Knot Resolver stopped treating minimally-covering NSEC* aggressively, 
but there are *two* different reasons.

1) breakages like this.  We hard-enabled aggressivity for NSEC and NSEC3 
in 2018; at that point we felt very much in minority, and it was hard to 
convince others that it's them who's doing it wrong (say, F5 customers).

2) low benefits of aggressive caching in this case.  When the range 
covers basically a single name, the possible positive effect is very 
limited.  There are negative non-breaking effects as well, e.g. caching 
of approaches like [black-lies].  You also need to weight the 
(negligible) benefits against (small-ish) costs of aggressive 
cache-searching.

[black-lies] https://blog.cloudflare.com/black-lies/

--Vladimir

---

From: Philip Homburg <[email-masked]>
Date: Wed, 01 Dec 2021 11:46:47 +0100

> Also stop hiding this
> breakage. Knot and unbound ignore the NSEC records which trigger
> this when synthesising.  All it does is push the problem down the
> road and makes it harder for others to do proper synthesis based
> on the records returned.

I'm confused what this means. In the report from Slack about the incident
I found that the problem started with a bad NSEC record, shown in their
debug output as:

qqq.slackexperts.com.   2370    IN      NSEC    \000.qqq.slackexperts.com. RRSIG NSEC

This is returned in response to a AAAA query. The intent was that the NSEC
record should have the 'A' bit as well.

What exactly do Knot and Unbound ignore in this case?

Is it that they should have special processing for an NSEC that has only
RRSIG and NSEC and nothing more?

---

From: "libor.peltan" <[email-masked]>
Date: Wed, 1 Dec 2021 11:59:17 +0100

Hi Philip,

> What exactly do Knot and Unbound ignore in this case?
they do not ignore the record when processing the answer.

What they don't is to store this NSEC record in their cache and use it 
for crafting other negative responses for subsequent queries without 
re-querying the authoritative server.

Libor

---

From: Tim Wicinski <[email-masked]>
Date: Wed, 1 Dec 2021 06:07:47 -0500

What I noticed in reading this nice write up was the warning image they
missed in the Route53 console
because of the automation they use.  But most folks use
automation/tooling/etc in their workflow, and
catching those warnings via automation is a bit tricky.

Right after this happened several different teams looking to sign their
zones got a little nervous.
This writeup helps to show people how things can break, but also, there is
a great set of testing
methods to assist others.

Mark's comments about adding tests in DNSVIZ would be pretty great.

tim

---

From: Mark Andrews <[email-masked]>
Date: Wed, 1 Dec 2021 23:43:36 +1100

> On 1 Dec 2021, at 19:54, Vladimír Čunát <[email-masked]> wrote:
> 
> Knot Resolver stopped treating minimally-covering NSEC* aggressively, but there are *two* different reasons.
> 
> 1) breakages like this.  We hard-enabled aggressivity for NSEC and NSEC3 in 2018; at that point we felt very much in minority, and it was hard to convince others that it's them who's doing it wrong (say, F5 customers).
> 
> 2) low benefits of aggressive caching in this case.  When the range covers basically a single name, the possible positive effect is very limited.  There are negative non-breaking effects as well, e.g. caching of approaches like [black-lies].  You also need to weight the (negligible) benefits against (small-ish) costs of aggressive cache-searching.
> 

Accepting 'QNAME NSEC \000.QNAME NSEC RRSIG’ (and other type maps) protects you from random QTYPE attacks.  It also makes 'black lies' work as intended.

The existing synth-from-dnssec code in BIND doesn’t treat this pattern as special.  I’m arguing that special code for this isn’t needed and if it put in it would be removed in the next production release cycle.

As for F5 we had been reporting issues like this as well.  That said I believe they now have published work around instructions for the old servers (add A and/or AAAA records to the backing zones to generate the correct NSEC typemap) and the new code you specify the type list for NSEC / NSEC3.  Presumably F5 add A and AAAA as appropriate
to this if they are missing.  Hopefully HTTPS soon.

The issues with zone enumeration have been grossly exaggerated.  For most zones on the Internet there is no benefit in preventing enumeration and only costs as you leave yourself exposed to reflected random subdomain attacks.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Vladimír Čunát <[email-masked]>
Date: Wed, 1 Dec 2021 13:57:30 +0100

> Accepting 'QNAME NSEC \000.QNAME NSEC RRSIG’ (and other type maps) protects you from random QTYPE attacks.  It also makes 'black lies' work as intended.

Black lies got bad caching if Knot Resolver accepted this aggressively.  
Asking two different QTYPEs repeatedly got uncacheable, as those answers 
need different NSEC* record contents (for the same NSEC* owner).  Of 
course, with different caching design this might be different, but it 
just didn't seem worthwhile to me.  If someone cares for good caching, 
they shouldn't use minimal ranges.

--Vladimir

---

From: Mark Andrews <[email-masked]>
Date: Thu, 2 Dec 2021 01:49:16 +1100

> On 1 Dec 2021, at 23:57, Vladimír Čunát <[email-masked]> wrote:
> 
> Black lies got bad caching if Knot Resolver accepted this aggressively.  Asking two different QTYPEs repeatedly got uncacheable, as those answers need different NSEC* record contents (for the same NSEC* owner).  Of course, with different caching design this might be different, but it just didn't seem worthwhile to me.  If someone cares for good caching, they shouldn't use minimal ranges.

Black lies is “QNAME NSEC \000.QNAME NSEC RRSIG”.  There is no churn for "black lies”.  Black lies turns NXDOMAIN into NODATA.

"DNS Shotgun" can produce churn of NSEC if you ask for a type that is listed as existing but it doesn’t actually exist.  The NSEC returned is still valid for DNSSEC synthesis.

---

From: Vladimír Čunát <[email-masked]>
Date: Wed, 1 Dec 2021 15:57:30 +0100

> "DNS Shotgun" can produce churn of NSEC if you ask for a type that is listed as existing but it doesn’t actually exist.  The NSEC returned is still valid for DNSSEC synthesis.

Oh, I'm sorry; a terminological problem.  I used "black-lies" for the 
overall behavior of Cloudflare auths, as described in that blog 
article.  Maybe we could extend the current terminology draft :-D

(Nit: about random QTYPE attacks, I can't see a point when you leave 
random QNAME attacks undefended.)

---

From: Mark Andrews <[email-masked]>
Date: Thu, 2 Dec 2021 02:40:56 +1100

> On 2 Dec 2021, at 01:57, Vladimír Čunát <[email-masked]> wrote:
> 
> Oh, I'm sorry; a terminological problem.  I used "black-lies" for the overall behavior of Cloudflare auths, as described in that blog article.  Maybe we could extend the current terminology draft :-D
> 
> (Nit: about random QTYPE attacks, I can't see a point when you leave random QNAME attacks undefended.)

Dropping them also sets a bad precedent as one then has to deal with “but foobar works with the bad type map” complaints.
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: Paul Vixie <[email-masked]>
Date: Wed, 1 Dec 2021 10:32:36 -0800

two replies here.

Mark Andrews wrote on 2021-12-01 00:35:
> Also stop hiding this breakage. Knot and unbound ignore the NSEC
> records which trigger this when synthesising.  All it does is push
> the problem down the road and makes it harder for others to do proper
> synthesis based on the records returned.

+1. fail early and fail often.

Tim Wicinski wrote on 2021-12-01 03:07:>
> What I noticed in reading this nice write up was the warning image
> they missed in the Route53 console because of the automation they
> use. But most folks use automation/tooling/etc in their workflow,
> and catching those warnings via automation is a bit tricky.
> 
> ...

sadly for the dnssec adoption curve, there is no substitute for knowing 
what you're doing, and automators who wish to scale dnssec management to 
include well-trained non-wizards are going to create disappointment.

-- 
vixie

---

From: Petr Špaček <[email-masked]>
Date: Fri, 3 Dec 2021 10:01:48 +0100

On 01. 12. 21 12:07, Tim Wicinski wrote:
> This writeup helps to show people how things can break, but also, there 
> is a great set of testing

It is already implemented in DNSViz since December 2020 (so a year+ 
before the Slack outage), but it is not shown by default:

https://github.com/dnsviz/dnsviz/issues/76#issuecomment-985331543

-- 
Petr Špaček

---

From: Philip Homburg <[email-masked]>
Date: Wed, 08 Dec 2021 14:36:28 +0100

> road and makes it harder for others to do proper synthesis based

I did some tests with unbound (version 1.13.1-1 on Debian Bullseye). 

For types other than 'A', the behavior is quite simple: if both
DNSSEC validation (auto-trust-anchor-file) and aggressive-nsec are enabled
then unbound will synthesize NODATA based on a cached NSEC record.
Both are off by default.

For A records the situation is more complex. If qname-minimisation is off,
then the same applies to A records. However if qname-minimisation is on (and
it is on the default) then unbound will internally generate A record
queries. So the A record will be cached before the NSEC record. 

So in the case of Slack, anybody who enabled both DNSSEC validation and
aggressive-nsec would probably not have seen a failure due to the
broken NSEC records because qname-minimisation is on by default.

---

From: Mark Andrews <[email-masked]>
Date: Thu, 9 Dec 2021 13:04:09 +1100

> On 9 Dec 2021, at 00:36, Philip Homburg <[email-masked]> wrote:
> 
>> Also stop hiding this

Actually they still can.  Just wait until the A response expires while 
there are still cached NSEC present.  This will happen naturally with a
big enough client pool.  Incorrect negative proofs lead to intermittent
“incorrect” responses with aggressive synthesis.

Mark