Subject: Re: [dnsext] EDNS0 fallback "revisited"

From: bert hubert <[email-masked]>
Date: Tue, 1 Sep 2009 09:21:52 +0200

2009/9/1 Patrik Fältström <[email-masked]>:
> The question is who the complaint is going to. Some DNSSEC problems will
> target the large ISPs that run the large resolvers. They will detect people
> with broken signed zones -- but they will most certainly _not_ have the
> fragmentation issues (or they have serious problems anyway). The complaints

I'm not sure about the latter. What will probably happen a lot is the
following scenario:

1) ISP/Corporation turns on DNSSEC validation
2) complaints come in that goodsecuritybank.org does not resolve
anymore (because of fragments not arriving or TCP/IP not working), or
is slow to resolve. badsecuritybank.org resolves normally!
3) ISP/Corporation turns off DNSSEC validation, goodsecuritybank.org
is fast again.
3a) goodsecuritybank.org reads about itself online, and its domain not
resolving, and thinks twice about its DNSSEC signing..
3b) badsecuritybank.org thinks they were right all along
4) Wait.. and perhaps go back to step 1

A well-motivated ISP or corporation with time on its hand should of
course launch a good investigation after step 2, and fix their own
(corporate) firewalls or access lists.

The steps outlined above become worse if they only happen over time.
Say, at first things fit within a single packet, and people consider
the 'turn on DNSSEC validation' project a success. Few weeks later
things start breaking down (move to NSEC3, or larger keys, or key
rollover..), no one has time to investigate, and as a workaround,
validation is shut off.

Do realise that Nicholas' numbers indicate that 13% of all resolvers
today that understand EDNS appear to be *unable* to receive an answer
that requires fragmentation or TCP to be working.

So if we roll out DNSSEC that needs fragmentation (even if only for
one DNSKEY in the trust chain), which appears likely, 1 in 7 places
would get into problems if they turned on DNSSEC validation.

Anyone not impressed by this number might want to talk to people that
carry responsibility for system administration, and see what they
think of it.

       Bert

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 1 Sep 2009 09:31:59 +0200

> that requires fragmentation or TCP to be working.

I do understand your scenario description, and I do know about  
Nicholas number. But I do not understand why we have not seen any  
problems in newspapers in Sweden where we have been running DNSSEC for  
quite some time.

Is that because the large resolvers (with many queries) "of course"  
are not behind a firewall (something I find being a pretty weird  
thing) or behind a firewall that do the right thing (those exists,  
remember), and in reality the people that have participated in Nic's  
tests are the ones that really are behind broken things that "just  
want to test"? I.e. you simply do not run Nic's tests if it is not the  
case that you already do see some problems.

Sure, I agree the 13% number is large, very large, and even if the  
real number of affected queries is 0.1% it might be too large. I would  
even be surprised if 13% of the resolvers in the world that send  
queries to the TLDs etc have this problem. But even 1% might be too  
large!

> carry responsibility for system administration, and see what they

Bert, I do. I really do. And that is what confuses me. People that do  
carry responsibility for DNSSEC validating resolvers in Sweden are  
worried about DNSSEC, but not about these findings. They do not say  
these issues exist.

I am not saying you are wrong. I am just saying I hear different  
things from two camps. Which makes me confused!

    Patrik

---

From: [email-masked]
Date: Tue, 1 Sep 2009 08:31:45 +0000

> case that you already do see some problems.
	some very interesting advice from Roland Dobbins - Don't 
	put your webservers/dnsservers behind firewalls.  - this to 
	the assembled folks @ ausnog yesterday.  -  so maybe not so weird.

> queries to the TLDs etc have this problem. But even 1% might be too  

	a much smaller sample was run by me today - testing against
	Duanes MTU-checker.  7 of 31 have an MTU smaller than 2048.
	would need more testing to see if the path can handle UDP frags
	or TCP.  ...  even if they do, thats not looking good for
	signed data.  Lots of extra traffic - even if it does work.
	Geo. Michaelson noticed and commented on the fact that the need
	to handle NXDOMAIN replies and the amnt of traffic was unexpected
	and not planned for.  This is going ot be exciting and a little
	bit precarious.  I would not want ot be front line helpdesk next year

---

From: "Aki Tuomi" <[email-masked]>
Date: Tue, 1 Sep 2009 12:00:45 +0300

> -----Original Message-----
> From: [email-masked] [mailto:owner-
> [email-masked]] On Behalf Of [email-masked]
> Sent: Tuesday, September 01, 2009 11:32 AM
> To: Patrik Fältström
> Cc: bert hubert; Nicholas Weaver; Alex Bligh; Olafur Gudmundsson/DNSEXT
> chair; [email-masked]
> Subject: Re: [dnsext] EDNS0 fallback "revisited"
> 
 
<snip />

There is rarely any need to put border authoritative servers (or resolvers, for that matter) behind firewall. Unfortunately this is not known by many administrators, but perhaps Swedish ones know this quite well? 

It was argued in a previous mail that Nicholas numbers could be biased by people doing the test only when there is need to test. Could be, but I don't know if we can use Sweden as statistically representative group either. 

Aki Tuomi

---

From: Olafur Gudmundsson <[email-masked]>
Date: Tue, 01 Sep 2009 09:19:02 -0400

<no-hat>
>1) ISP/Corporation turns on DNSSEC validation

I think before they perform this step ALL HOWTO guides should say something
0) Test your DNS:
         Run test XXXXX
         Run test YYYYY
         Update configurations see http:.....

Blindly turning on DNSSEC validation should only be done if
a) the DNSSEC validator is capable of discovering its path and check it works
    for strange corner cases
b) the operator has checked the path 
characteristics and possibly performed some
    tuning to make DNSSEC answers get through.

         Olafur

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 1 Sep 2009 06:48:33 -0700

On Sep 1, 2009, at 6:19 AM, Olafur Gudmundsson wrote:

> b) the operator has checked the path characteristics and possibly  
> performed some

Except the problem is not blindly turning on DNSSEC validation, but  
blindly turning on requesting DNSSEC records.

Which bind does by default.  This hasn't been a problem yet, because  
nobody is really signing anything.  But it may be once major TLDs  
start signing things eagerly.

---

From: [email-masked]
Date: Tue, 1 Sep 2009 14:06:00 +0000

On Tue, Sep 01, 2009 at 06:48:33AM -0700, Nicholas Weaver wrote:
> >>2009/9/1 Patrik Fdltstrvm <[email-masked]>:

	or ... once the root gets signed - then every priming query
	will have DNSSEC data in the response.

--bill

---

From: Doug Barton <[email-masked]>
Date: Tue, 01 Sep 2009 08:54:44 -0700

> 	the assembled folks @ ausnog yesterday.  -  so maybe not so weird.

I've advocated this for years. The only thing running on those servers
that is visible to the outside world is the key service (DNS or
HTTP[S]). You can run ssh with tcp wrappers so that it's only
accessible from appropriate locations.

Things get a little more complicated if your name server needs access
to a database, but even that can be wrapped up appropriately.

Of course YMMV, but this configuration has worked well for me for years.

Doug

---

From: Paul Vixie <[email-masked]>
Date: Tue, 01 Sep 2009 17:06:27 +0000

> From: Nicholas Weaver <[email-masked]>
> Date: Tue, 1 Sep 2009 06:48:33 -0700
> ...
> nobody is really signing anything.  But it may be once major TLDs  start
> signing things eagerly.

by that reasoning, BIND folks (85% market share) would be seeing a lot of
timeouts on nonexistent ORG subdomains, and there are no reports of same.

	; <<>> DiG 9.4.1-P1 <<>> +dnssec lkjhdsflkjhdsf.org mx
	;; global options:  printcmd
	;; Got answer:
	;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 48597
	;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 8, ADDITIONAL: 1

	;; OPT PSEUDOSECTION:
	; EDNS: version: 0, flags: do; udp: 4096
	;; QUESTION SECTION:
	;lkjhdsflkjhdsf.org.            IN      MX

	...

	;; Query time: 27 msec
	;; SERVER: 204.152.187.17#53(204.152.187.17)
	;; WHEN: Tue Sep  1 17:03:36 2009
	;; MSG SIZE  rcvd: 1004

i think we may be overblowing this.

---

From: bert hubert <[email-masked]>
Date: Tue, 1 Sep 2009 09:55:29 +0200

[ Moderators note: Post was moderated, either because it was posted by
   a non-subscriber, or because it was over 20K.  
   With the massive amount of spam, it is easy to miss and therefore 
   delete relevant posts by non-subscribers. 
   Please fix your subscription addresses. ]

2009/9/1 Patrik Fältström <[email-masked]>:
> I do understand your scenario description, and I do know about Nicholas
> number. But I do not understand why we have not seen any problems in
> newspapers in Sweden where we have been running DNSSEC for quite some time.

(to the list) I continued this conversation with Patrik off-list, will
let you know if we've discovered why Sweden sees no problems.

Maybe everything IS better in Sweden? ;-)

> I am not saying you are wrong. I am just saying I hear different things from
> two camps. Which makes me confused!
>
>   Patrik

---

From: bert hubert <[email-masked]>
Date: Tue, 1 Sep 2009 20:05:06 +0200

> timeouts on nonexistent ORG subdomains, and there are no reports of same.

We should keep the numbers straight. Only 4% of people can only do <=
512, plus BIND then tries TCP, and if that does not work it drops EDNS
all together. So almost nobody will be seeing timeouts at this stage.

We did see the 250-fold increase in .org TCP queries - those were not imagined.

> i think we may be overblowing this.

Maybe. But so far all you know is that unfragmented responses do not
cause noticeable problems in a population that drops back to TCP and
then to do=0.

This is not a datapoint on fragmented responses - and the only
fragmented responses going out in bulk are .GOV NXDOMAINS.

Any validating resolver will still need to have a clear >1470 path to
authoritative servers - either using fragmented answers, or over TCP.

      Bert

---

From: Joe Abley <[email-masked]>
Date: Tue, 1 Sep 2009 16:58:20 -0400

> We did see the 250-fold increase in .org TCP queries - those were  

600-fold, I think. But those were likely exaccerbated by the lack of  
negative caching and the zero-second TTL on the apex DNSKEY RRSet.

Do we know what the steady-state increase in UDP:TCP transport for ORG  
is, compared to a pre-signed zone, now that those problems have been  
fixed? Maybe there has been analysis done, but if so I have missed it.

Joe

---

From: Duane Wessels <[email-masked]>
Date: Tue, 1 Sep 2009 21:29:19 +0000 (UTC)

On Tue, 1 Sep 2009, Joe Abley wrote:

> Do we know what the steady-state increase in UDP:TCP transport for ORG is, 
> compared to a pre-signed zone, now that those problems have been fixed? Maybe 
> there has been analysis done, but if so I have missed it.

Not yet.  Afilias has provided some additional data to OARC (i.e.
from last week), but I haven't had time to crunch it yet.

DW

---

From: Wayne Beaver <[email-masked]>
Date: Wed, 2 Sep 2009 09:31:27 -0400

On Sep 1, 2009, at 3:55 AM, bert hubert wrote:

>> Nicholas

The following is shared pursuant to permission obtained from Niclas at  
IIS (The Ops for .SE). Sounds like IIS did some outreach and education  
to mitigate TCP increase from its operator community. May not scale  
for some TLDs

 > Tue,  9 Jun 2009 14:54:17 +0000 (UTC)
 > From: Niclas Rosell <[email-masked]>
 >
 > Yes we had a increase in TCP when we introduced DNSSEC and it was  
due to firewalls not accepting dns-packages greater than 512 byte and  
thereby forcing TCP. ("fixup-dns-packages" or something in > Cisco PIX  
conf was causing this). We fixed this and TCP/UDP is not much higher  
than before. Rougly 0.5% of all traffic.

wb

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 2 Sep 2009 08:30:50 -0700

On Sep 2, 2009, at 6:31 AM, Wayne Beaver wrote:

> The following is shared pursuant to permission obtained from Niclas  
> at IIS (The Ops for .SE). Sounds like IIS did some outreach and  
> education to mitigate TCP increase from its operator community. May  
> not scale for some TLDs
>
> due to firewalls not accepting dns-packages greater than 512 byte  
> and thereby forcing TCP. ("fixup-dns-packages" or something in >  
> Cisco PIX conf was causing this). We fixed this and TCP/UDP is not  
> much higher than before. Rougly 0.5% of all traffic.

Could you ask the followup question?  How many of the responses were  
 >512B but less that 1500B, and how many were more than 1500B?