Subject: [DNSOP] wrapping up draft-ietf-dnsop-nsec3-guidance

From: Wes Hardaker <[email-masked]>
Date: Wed, 20 Oct 2021 11:24:47 -0700

Good folks,

I think that draft-ietf-dnsop-nsec3-guidance is fairly well boiled, so
I'm asking for a last call on moving toward a last call.  The draft is
intentionally short and to the point but at the same time we've waited a
while to see what the industry would do with the guidance.  Viktor's
recent measurements have very much shown acceptance of the document as a
whole, and certainly of the proposed 150 iteration cap.

But, as Viktor indicated in his posts, we could move even lower (100
being the next obvious step, but even lower is possible to still retain
a reasonable percentage).  But there is of course a risk of we'll never
get to a definitive value, and may operators by constantly lowering it
and they have to keep changing values.

So, the question: what's the right FINAL value to put in the draft
before LC?

-- 
Wes Hardaker
USC/ISI

---

From: Peter van Dijk <[email-masked]>
Date: Thu, 21 Oct 2021 13:22:25 +0200

> So, the question: what's the right FINAL value to put in the draft

I don't know what the -right- value is, but I know what I want: 0 iterations, empty salt, otherwise the NSEC3 gets ignored, presumably leading to SERVFAIL. This removes the 'insecure' window completely.

So, I'll support any push to lower the numbers.

Editorial nit, already hinted at above: the text currently has "Validating resolvers MAY return SERVFAIL when processing NSEC3 records with iterations larger than 500." - I suggest changing this to "validating resolvers MAY ignore NSEC3 records with iterations larger than 500". That way, zones in the middle of a transition from 1000 to 0 iterations do not get punished. Zones at 1000, not in a transition, will still get SERVFAIL by virtue of the NSEC3 proof missing (because it is ignored).

Kind regards,
-- 
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

---

From: Matthijs Mekking <[email-masked]>
Date: Thu, 21 Oct 2021 14:52:47 +0200

> Editorial nit, already hinted at above: the text currently has "Validating resolvers MAY return SERVFAIL when processing NSEC3 records with iterations larger than 500." - I suggest changing this to…

In addition, the line just before that says "Validating resolvers SHOULD
return an insecure response when processing NSEC3 records with 
iterations larger than 100."

And I suggest to change it to "larger than 150", a value that open 
source DNS vendors have been adopting over the last couple of months:

https://nlnetlabs.nl/news/2021/Aug/12/unbound-1.13.2-released/

https://blog.powerdns.com/2021/06/09/powerdns-recursor-4-4-4-and-4-5-2-released/

https://www.knot-resolver.cz/2021-03-31-knot-resolver-5.3.1.html

https://bind9.readthedocs.io/en/v9_16_21/notes.html#notes-for-bind-9-16-16

(sorry that this is not pushing for lower numbers)

Best regards,

Matthijs

---

From: Miek Gieben <[email-masked]>
Date: Thu, 21 Oct 2021 15:28:26 +0200

[ Quoting <[email-masked]> in "Re: [DNSOP] wrapping up draft-ietf-..." ]
>source DNS vendors have been adopting over the last couple of months:

I would recommend against using a limit that happens to be in use at the current time, and
would just use 100 (or even lower). Resolvers will continue to work fine and can lower their
limit at their leisure.

/Miek

--
Miek Gieben

---

From: Matthijs Mekking <[email-masked]>
Date: Thu, 21 Oct 2021 15:49:13 +0200

IIRC the vendors agreed on 150 for two reasons:

1. There are still a fair amount of zones using this value. Only a 
handful of zones where using above 150.

2. Resolvers could still cope with such numbers pretty confidently.

I agree lower is better, but let's not pick a number randomly, but have 
data to back up that number.

- Matthijs

On 21-10-2021 15:28, Miek Gieben wrote:
>>> I don't know what the -right- value is, but I know what I want: 0 
>>> iterations, empty salt, otherwise the NSEC3 gets ignored, presumably 
>>> leading to SERVFAIL. This removes the 'insecure' window completely.
>>>
>>> Editorial nit, already hinted at above: the text currently has 
>>> "Validating resolvers MAY return SERVFAIL when processing NSEC3 
>>> records with iterations larger than 500." - I suggest changing this 
>>> to "validating resolvers MAY ignore NSEC3 records with iterations 
>>> larger than 500". That way, zones in the middle of a transition from 
>>> 1000 to 0 iterations do not get punished. Zones at 1000, not in a 
>>> transition, will still get SERVFAIL by virtue of the NSEC3 proof 
>>> missing (because it is ignored).
>>

---

From: Paul Vixie <[email-masked]>
Date: Thu, 21 Oct 2021 07:24:21 -0700

Matthijs Mekking wrote on 2021-10-21 06:49:
> ...
> 
> I agree lower is better, but let's not pick a number randomly, but
> have data to back up that number.

if we need a number that has objective merit, it is zero (0).

>>>> ...
>>> ...
>> I would recommend against using a limit that happens to be in use
>> at the current time, and would just use 100 (or even lower).
>> Resolvers will continue to work fine and can lower their limit at
>> their leisure.

+1.

vixie

-- 
Sent from Postbox
<https://www.postbox-inc.com/?utm_source=email&utm_medium=siglink&utm_campaign=reach>

---

From: Vladimír Čunát <[email-masked]>
Date: Thu, 21 Oct 2021 19:36:37 +0200

> Editorial nit, already hinted at above: the text currently has "Validating resolvers MAY return SERVFAIL when processing NSEC3 records with iterations larger than 500." - I suggest changing this to…

I'm fine with either.  Ignoring such NSEC3 records seems to be clearly 
the easiest way of achieving "MAY return SERVFAIL".  On the other hand, 
the downgrading can turn out relatively complex and (security-)error-prone.

--Vladimir | knot-resolver.cz

---

From: Viktor Dukhovni <[email-masked]>
Date: Thu, 21 Oct 2021 17:20:47 -0400

> So, the question: what's the right FINAL value to put in the draft

Some observations to help the decision process:

   1. A validating resolver that prefers to SERVFAIL on all responses
      with excessive iterations, avoiding downgrades to "insecure",
      can simply ignore such NSEC3 records, and if no appropriate NSEC
      or NSEC3 records remain can then treat the reply as bogus.

   2. The downside of insecure downgrade is that any affected zones are
      subject to forgery of all names strictly below the zone apex, via
      fake insecure delegations (the denial of existence of DS records
      will be accepted), and also NODATA forgery for all qtypes at the
      zone apex (except NSEC3 and RRSIG).

   3. The downside of SERVFAIL for excess iterations is that if the
      target zone handles names of SMTP hosts without DANE TLSA records,
      then TLSA denial of existence failure with render these mail
      servers unavailable to DANE-enabled SMTP clients.

      Also any wildcard replies that are based of non-existence proofs
      of the qname, ... will be bogus, thus e.g. wildcard A/AAAA answers
      are likely to SERVFAIL.

   4. The cost of P256 signature verification is (on a now somewhat dated
      Xeon Skylake system) ~300 times that of a SHA1 hash.  Thus south
      of 150 iterations, further reductions in the iteration count offer
      only a modest benefit to validating resolvers that are also
      validating the signature.

   5. However, no signature verification applies on the authoritative
      server (perhaps a secondary that did not specifically "volunteer"
      to serve zones with a high iteration count).

      Also when doing aggressive negative caching via previously
      received NSEC3 records, once again only SHA1 hashing is involve,
      the signature verification happened when the records were cached.

Therefore, while a softfail to insecure makes it possible to avoid
immediate pain, the SERVFAIL alternative is simpler to implement
correctly, but may require setting the bar somewhat higher.

Ideally, all zone operators would get the message, apply a realistic
threat model, and set the iteration counts to 0.

Much progress has been made in a comparatively short time, but pockets
of "resistance" remain, with a large majority of domains in the [1-20]
range, and low but perhaps non-negligible zone counts (out of 12.46M
zones) for:

    50 iterations: ~13k
    100 iterations: ~20k (7.9k netcup.de, 2.2k nlhosting.net, 2.1k core-networks.de)
    150 iterations: ~6k  (5.8k mijnhostingpartner.nl)
    500 iterations: 101  (85 raytheon.com)

With a bit more nagging we could probably convince the small number of
operators that dominate the counts in question to make adjustments.

Otherwise, we can declare victory at either 100 or 150, and recommend
SERVFAIL above 500, but MAY SERVFAIL at the lower cutoff.

I'd like to see more responses with specific numbers, and thoughts on
whether a range in which downgrade to insecure happens is a good or
bad idea.

That is, is it always either AD=1 or SERVFAIL, or is there merit in
AD=0 for a range of values above a soft cutoff before a higher hard
cutoff is reached.

At this point, my inclination is to hardfail at 150 and avoid softfail.
Raytheon may be briefly incovenienced, but otherwise this feels the most
robust option, unless rough consensus is to try to set the bar lower and
softfail from there to some suitable upper bound in the 150 to 500
range.

> So, I'll support any push to lower the numbers.

Please be specific, even if you feel you'll land in the rough.

> records with iterations larger than 500." - I suggest changing this to
> "validating resolvers MAY ignore NSEC3 records with iterations larger
> than 500". That way, zones in the middle of a transition from 1000 to
> 0 iterations do not get punished. Zones at 1000, not in a transition,
> will still get SERVFAIL by virtue of the NSEC3 proof missing (because
> it is ignored).

Thanks, I think I agree. Ignore records with counts that would SERVFAIL
if best-available, sounds sensible.

> (sorry that this is not pushing for lower numbers)

Thanks for the specific number!

> would just use 100 (or even lower). Resolvers will continue to work fine and can lower their

Please be specific, or do you mean that a resolver should be free to
choose any number above 0 (I'd suggest 1 in that case, as some operators
appear to to be confident that 0 *is* one iteration, and so choose 1
just in case).

That means that operators who choose non-zero values would have to keep
adjusting the numbers down as resolvers gradually set the bar lower, and
the only sane settings would then be 0 and 1.

> handful of zones where using above 150.

The numbers came down a lot, and publication of a lower number in the
RFC could similarly drive these down further (pretty much just
mijnhostingpartner.nl left at 150).

> 2. Resolvers could still cope with such numbers pretty confidently.

This is where I'm looking for experienced feedback from resolver
maintainers and operators.  I have deployment stats, but not performance
stats.

> have data to back up that number.

I've provided the deployment numbers, others have relevant numbers
on performance impact, please share.

> >> Resolvers will continue to work fine and can lower their limit at

So all resolver behaviour is fair above 0 (and then I'd suggest also 1)?
With resolver ceilings drifting down over time?  Or something else?

-- 
    Viktor.

---

From: Vladimír Čunát <[email-masked]>
Date: Fri, 22 Oct 2021 10:48:16 +0200

On 21/10/2021 23.20, Viktor Dukhovni wrote:
> maintainers and operators.

I don't think that NSEC3 hashing could consume significant resources in 
*normal* traffic.  The only real risk I see is that expensive parameters 
might get utilized in some DoS attack, so I'd focus on worst-case scenarios.

Salt length: that's what I'd additionally restrict, because 255 bytes 
make even less sense than high iterations and it's quite expensive - 
salt gets added on every iteration, so the expenses multiply each other.

Example micro-benchmark doing just the NSEC3 hashing shows that even 
quite long 32B salt has little effect but 255B adds a noticeable 
multiplicative factor.  Therefore I'd suggest that NSEC3 records with 
salt > 32B may be ignored or something similar; I don't expect they 
really exist in practice and we still shave some factor from attacks.  
That's all assuming we can't afford pushing the validator limits very 
close to zero iterations.

iter, salt, us / hash

    0,   0,   0.14
    0,  32,   0.16
    0, 255,   0.47

   50,   0,   3.72
   50,  32,   4.66
   50, 255,  21

  150,   0,  10.7
  150,  32,  13.7
  150, 255,  62

(gnutls using SHA extensions on zen1 CPU, but the conclusion was also 
confirmed by Peter van Dijk on a different stack)

---

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 22 Oct 2021 10:48:24 -0400

> On 22 Oct 2021, at 4:48 am, Vladimír Čunát <[email-masked]> wrote:
> 
> Example micro-benchmark doing just the NSEC3 hashing shows that even quite long 32B salt has little effect but 255B adds a noticeable multiplicative factor.  Therefore I'd suggest that NSEC3 records with salt > 32B may be ignored or something similar; I don't expect they really exist in practice and we still shave some factor from attacks.  That's all assuming we can't afford pushing the validator limits very close to zero iterations.

The observed salt lengths from my NSEC3 scan were, so there's a large cluster
of ~291k zones with 40 byte salts.  It looks like the recent burst of signing
by hostpoint.ch used that salt length.  We could suggest they reconsider that
choice.

The small cluster of 125 zones with 64 byte salts looks dominated by
(but not exclusive to) kilobajt.sk.

The 3 outliers at 255 were: matejgroma.com, saren.sa and saren.org.sa.

0 167477
1 2010575
2 237576
3 15971
4 965075
5 1164236
6 11130
7 1224
8 7465501
9 909
10 1821
11 928
12 17848
13 884
14 982
15 158
16 87950
17 25
18 59
19 136
20 7221
21 216
22 193
23 220
24 9192
25 125
26 123
27 96
28 51
29 44
30 38
31 33
32 584
33 11
34 10
35 5
36 3
37 1
39 1
40 291294
44 1
64 125
120 1
128 1
255 3

-- 
	Viktor.