Subject: [dnsext] DNS over SCTP problems, part 1, theory

From: William Allen Simpson <[email-masked]>
Date: Mon, 10 Aug 2009 12:05:15 -0400

In general, I'm opposed to trying to run DNS over SCTP.  I've started
composing this message several times (previously for the NANOG list), as
this is a complicated issue.  I've even downgraded the "Subject:" from
"considered harmful" to "problems" to try to be less alarmist....

I'd hoped that the IETF would have paid attention and made the changes to
the SCTP protocol suggested circa 2000.  Sadly, that has not happened.

SCTP is an overly complicated protocol, with different kinds of TLV
parameters, some of which are called "chunks" instead.  It probably
would have been better to only have chunks.

But DNS packets already have their own variable length parameters, so
there's 4 or more layers of variable length checking, and overlapping
layers of error returns.

It would be easier and better to add cookie extensions to EDNS0 (or TCP).

===

SCTP asserts that is has "[a] cookie mechanism, similar to" the Photuris
security protocol "described by Karn and Simpson in [RFC2522]".
Unfortunately, the authors don't seem to have an actual understanding of
the underlying theory of cookies and Message Authentication Codes (MACs).
The resulting cookie specification is at best half-baked.

[NB: There are many chunky and cookie puns in the specification.]

In SCTP, the Verification Tag field(s) performs a similar function to the
Photuris Cookie pair.  The Verification Tags are only 32 bits.  Only 1 tag
is present in the common header.

Photuris Cookies are 128 bits each.  The *PAIR* of cookies forms the
256-bit message identification found in every Photuris common header.

Why 256-bits?  According to Ron Rivest's calculations, that's enough for
a multi-terabit per second attack.  Even 15 years ago, we were planning
for higher link speeds and long delay paths.

32 bits is not remotely enough today.  Not for gigabit links, not for
satellite links, nor for any other broadcast/multi-path/radio links.

The Verification Tags seem to be nonces (generated randomly, used once).
"Moreover, the Verification Tag value used by either endpoint in a given
association MUST NOT change during the life time of an association."
[page 72, section 5.3.1]

I've not been able to discern how this happens, as the Verification Tag
sent by the initiator is merely sent back by the responder (who discards
it).  And the responder is depending on the initiator to send back its
own (discarded) random nonce, so garbling could cause the entire
exchange to fail without notice.  Or not fail, depending entirely on the
implementation....

A random number that is different in every message, merely to identify
its response, might have some utility intended to prevent mid-connection
guessing attacks.  Even then, 32-bits is not really adequate today.

In SCTP, the exchange of its single "Cookie", created entirely by the
responder, begins one packet later than Photuris.  For some odd reason,
it suggests using a MAC.  In cryptologic terminology, a MAC requires a
secret verifiable by both parties.  There's not any such secret in SCTP,
so there's no "authentication" (the "A" in "MAC").

Unlike Photuris, there are no requirements for its generation.  It's not
specified to be dependent on the initiator's Verification Tag, nor the
responder's Verification Tag, nor the IP addresses, nor the ports.  It's
simply opaque.

Hopefully, it's not merely dependent on the internal secret used at the
time of generation.  That would be an easy replay attack.  The attacker
would merely send a valid request, get the current "cookie" and its
time remaining, and then send spoofed packets.  The time can be used to
estimate the number of attempts before making another valid request....

Moreover, the specification explicitly leaks the length of time the
internal secret used to generate the responder "cookie" will remain valid!
This is very bad.  If that wasn't bad enough, the specification allows
the initiator to suggest lengthening the "stale cookie" time by a
"Cookie Preservative" parameter.  Fortunately, that can be ignored.

Finally, for any protocol (let alone a "security" protocol), there should
be some method for determining and recovering from an attack.  There are
plenty of error messages, but the single most important lack is signaling
between the parties that there has been unknown intermediate tampering,
or that the other party has lost all its prior state.

Photuris has the simplest method (brilliantly suggested by Keromytis): a
Counter of the number of previously established connections.

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/namedroppers/>

---

From: Paul Vixie <[email-masked]>
Date: Mon, 10 Aug 2009 16:42:23 +0000

> Date: Mon, 10 Aug 2009 12:05:15 -0400
> From: William Allen Simpson <[email-masked]>
> 

edns0 and tcp have other problems that make datagram-friendly transport
alternatives worth looking at.  large messages, state load at high volume.
if sctp isn't suitable then we may be looking at extensions to udp, or at
some completely new transport.

btw, would photuris be a candidate?

---

From: Doug Otis <[email-masked]>
Date: Mon, 10 Aug 2009 10:47:59 -0700

> It would be easier and better to add cookie extensions to EDNS0 (or TCP).

Concerns prompting a recommendation of SCTP regard large UDP packets 
attempting to return DNSSEC answers, and if or when that becomes 
blocked, subsequent reliance upon TCP Fallback.

Some have suggested partial UDP responses be enabled by an EDNS0 option. 
In essence, this develops a sequential transport protocol lacking 
effective congestion avoidance, since this relies on application layer 
behaviors.  When it was pointed out to SPF proponents the time their 
sequential DNS queries might take, they imposed timeouts at the 
application layer which was free to restart the process against the same 
target immediately. :^(

Applications wanting faster DNS responses might simply request all pages 
at once to reduce latency.  Perhaps this could be avoided by returning 
tokens needed to request subsequent pages.  This scheme becomes 
increasingly fragile with packet loss.

Your comments about needing better Cookies appears to refer to the 
experimental RFC 2522 that establishes short lived session keys. You 
appear to not understand what SCTP cookies provide.  A cookie represents 
the entire INIT chunk, tie-tags (two unsigned 32 bit variables), 
time-stamps, and time-stamp life.  When combined with the two 
verification tags, this depends upon four 32 bit association variables. 
  These values can then be protected by a secret held by the server and 
a secure hash.

The SCTP cookie validates the source of the client.  There are SCTP AUTH 
RFC 4895, and SCTP TLS RFC 3436 to increase the level of protection that 
SCTP affords.  Frankly, DNSSEC has that covered.  Why are are worried 
about SCTP verification tags being modified?

-Doug

---

From: William Allen Simpson <[email-masked]>
Date: Mon, 10 Aug 2009 14:35:06 -0400

> some completely new transport.
SCTP has greater state load than TCP.  It only has the advantage that the
state is delayed until after the initial handshake, by passing the context
back to the initiator, and expecting the initiator to pass it back again.

Clever, but that could have easily been added as a TCP option.

SCTP is much more complex than either EDNS0 or TCP.

Whatever happened to the EDNS0 shortcut, where only part of the answer
was passed back, and the rest was requested in a second exchange?

That would have generated a lot less packets and RTT and state.

> btw, would photuris be a candidate?
No, that's an entire protocol (like DNS) that was specifically designed to
generate session keys for other protocols to use (just as DNS looks up
addresses and stuff for other protocols to use).

Photuris ran over UDP (and PPP, although never published by IETF).

---

From: Paul Vixie <[email-masked]>
Date: Mon, 10 Aug 2009 19:07:59 +0000

> Date: Mon, 10 Aug 2009 14:35:06 -0400

there was an MD bit ("more data") in the original EDNS0 proposal but it
was removed by the working group in the name of protocol simplification.

there's a current proposal ("page mode") by george barwood that i don't
like because then state has to be held in the application rather than in
the stack, and i don't want to pay in context switches to resist a ddos.

> That would have generated a lot less packets and RTT and state.

if N>1 packets have to remain associated, then somebody somewhere's got
state.  but we need something that isn't like current udp or current tcp.
from what you and florian have said, sctp isn't going to be "it".

---

From: William Allen Simpson <[email-masked]>
Date: Mon, 10 Aug 2009 15:35:04 -0400

> blocked, subsequent reliance upon TCP Fallback.
Thank you for restating the obvious.

Hint: I wrote one of the first widely deployed DNS implementations, circa
1987 (yes, that's an 8).  And was peripherally involved in the first
DNSsec go 'round....

> Some have suggested partial UDP responses be enabled by an EDNS0 option.

Long ago, I was in this camp.  What happened to the option?

> application layer which was free to restart the process against the same 
OK, prove that SCTP (4 handshakes plus data plus close, probably 6-8 RTT)
will take less time than 2-3 RTT of UDP?

Yes, there are programmers that do stupid things, and applications that do
stupid things, and operating systems that do stupid things.  Your point?

> increasingly fragile with packet loss.
True.  But an appropriate design can handle it.

> experimental RFC 2522 that establishes short lived session keys.

Yes, I quoted and cited the SCTP references to RFC-2522.  Arguably, some
ideas from Phorutis were appropriated without understanding them.

> You 
>  These values can then be protected by a secret held by the server and a 
> secure hash.
> 
Interesting, that /one/ of us might "not understand what SCTP cookies
provide."  Did you take a peek at the authors of RFC-2522?

Now, please explain exactly where in RFC-4960 your interpretation of that
whole megillah was defined?

All I see is the technically incorrect and woefully insufficient:

5.1.3.  Generating State Cookie

...

    3)  From the TCB, identify and collect the minimal subset of
        information needed to re-create the TCB, and generate a MAC using
        this subset of information and a secret key....

...

Note there's no mention of "the entire INIT chunk, tie-tags (two unsigned
32 bit variables), time-stamps, and time-stamp life."  None of those are
in the traditional TCB.

And that's my point.  Those should all be hashed in the cookie.  That
should be specified.  Anything else should be non-conformant.

Even with "four 32 bit association variables", that's *much* too small.

Compare with Photuris:

3.3.2.  Responder Cookie

    The Responder secret value that affects its cookies MAY remain the
    same for many different Initiators.  However, this secret SHOULD be
    changed periodically to limit the time for use of its cookies
    (typically each 60 seconds).

    The Responder-Cookie SHOULD include the Initiator-Cookie.  The
    Responder-Cookie MUST include the Counter (that is returned in the
    Cookie_Response).  This provides improved synchronization and
    protection against replay attacks.

    It is recommended that the cookie be calculated over the secret
    value, the IP Source and Destination addresses, its own UDP
    Destination port, the Counter, the Initiator-Cookie, and the
    currently Offered-Schemes.

...

That's a proper one-way hash of a *thousand* bits or more.

> The SCTP cookie validates the source of the client. ... Why are are worried 
> about SCTP verification tags being modified? [sic]
> 
Well, actually as a technical matter it *doesn't* validate any source.
There is a nice proof that verification cannot occur without generating a
shared session-key.  (By memory, van Oorshot and somebody.)

All a properly formed cookie can ever do is provide resistance to replay
and spoofing attacks, as long as the attacker isn't in the middle.

---

From: William Allen Simpson <[email-masked]>
Date: Mon, 10 Aug 2009 16:06:53 -0400

> was removed by the working group in the name of protocol simplification.
Ah, the joy of working groups.

Now somebody's proposing simplifying even further by falling back from DNS
to SCTP to TCP....  Some simplification!

> the stack, and i don't want to pay in context switches to resist a ddos.
I cannot find it.  The words "page mode" don't appear in the archive, and
the only draft-barwood I could find was resolver-mitigations-08.

However, I certainly agree about context switches.

> state.  but we need something that isn't like current udp or current tcp.

Like Photuris (and SCTP to a lesser extent), keep the state in the UDP
packets in transit.

I'll start a new design thread later (probably tomorrow).

---

From: "George Barwood" <[email-masked]>
Date: Mon, 10 Aug 2009 23:26:54 +0100

> the only draft-barwood I could find was resolver-mitigations-08.

Sorry, no draft as yet, just an informal proposal, see

http://ops.ietf.org/lists/namedroppers/namedroppers.2009/msg01278.html

George

---

From: Tony Finch <[email-masked]>
Date: Tue, 11 Aug 2009 13:43:50 +0100

> Clever, but that could have easily been added as a TCP option.

I was under the impression that TCP stacks already have this feature (SYN
cookies) or have other SYN-flood protection mechanisms. So unless SCTP is
a win for established associations, it isn't a win.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

---

From: William Allen Simpson <[email-masked]>
Date: Tue, 11 Aug 2009 10:17:18 -0400

> a win for established associations, it isn't a win.
It seems strange to me that this wasn't already specified, as I remember
discussion back in the mid-90s.  It also seems strange that it took 10+
years for Bernstein's clever (small) SYN cookies hack to be documented in
the RFC series (RFC-4987).

Apparently, the TCP community is even more conservative than the DNS
community in making changes....

The proposal, as I remember it circa 1996, was to add an explicit Cookie
option (128-bit cookie) to the original SYN, signaling to the responder
that no TCB was necessary.

If the Cookie option was returned (256-bit cookie) in the SYN+ACK, no TCB
would be created.  Then, the initiator would re-send all initial options
(including the bigger 256-bit Cookie) back to the responder in the first
data packet, and the responder TCB would be created.

Maybe that's the original genesis of SCTP?

One of the counter arguments at the time was that a robust IPv6 and IPsec
would render this entirely unnecessary.  Admittedly, at the time I was
part of that chorus.  In retrospect, I was wrong.

Anyway, given the general conservative nature of TCP deployment, it would
be better to work on methods for DNS to use UDP efficiently.

Unless there's a *lot* of interest....

---

From: Paul Vixie <[email-masked]>
Date: Tue, 11 Aug 2009 22:30:11 +0000

> Date: Tue, 11 Aug 2009 10:17:18 -0400

this sounds really quite nice.  is it dead?

> part of that chorus.  In retrospect, I was wrong.

"toldja so!"  :-)

> be better to work on methods for DNS to use UDP efficiently.

a horrific number of badly implemented middleboxes think UDP/53 is fair game
and thus rewrite messages according to policy.  given that most messages do
not get rewritten, this should not be a problem, but it is, since these badly
implemented middleboxes make the single-packet assumption (no fragmentation).

udp datagrams that are fragmented into multiple ip packets don't have udp
headers or udp port numbers except in the first ip fragment.  so these badly
implemented middleboxes either drop all the fragments, or pass the first one
and block the rest of them.

none of us consider these badly implemented middleboxes to be repairable since
they are all over the internet now and many of their manufacturers are out of
business and many of their original installers/operators are retired or dead.

so, the burden is back on the endpoints, to use a protocol that no badly
implemented middleboxes are looking at.  at the moment, endpoints don't get
a reliable indication that edns0 with large datagrams won't work, since the
symptom of brokenness tends to be a timeout, which is ambiguous.

udp is not fragmentation-aware in the way that tcp is.  tcp can set DF and
wait for ICMP to tell it to use smaller segments.  udp has no segmentation
since it relies on ip's fragmentation, which as i mentioned above, doesn't
work reliably and more importantly doesn't fail reliably.

t/tcp is dead.  sctp sounds like it remains unborn.  what i worry about in
what you say about bernstein's tcp cookie proposal is that we might not get
reliable failure indication, other than timeouts, which are ambiguous.

a fragmentation-aware udp-like protocol with session state based on
cookies, that had its own ip protocol number, is starting to seem like a
good idea.  when it doesn't work it'll be unambiguous -- even a timeout is
unambiguous if the protocol number is neither tcp nor udp... a timeout
means this new protocol isn't going to work and that some kind of fallback
(like udp/53 or tcp/53 or a VPN) should be used.

i had thought sctp was possibly going to be that protocol but i guess not?

---

From: Tony Finch <[email-masked]>
Date: Wed, 12 Aug 2009 00:04:34 +0100

> community in making changes....

TCP RFCs are often waaay behind deployment. See VJ congestion control for
a major example.

Tony.

---

From: Mark Andrews <[email-masked]>
Date: Wed, 12 Aug 2009 09:50:28 +1000

> business and many of their original installers/operators are retired or dead.

We don't need to repair them.  The net can cope with the what is
already out there.  We just need to make sure new boxes being
installed are not defective in particular all the IPv6 capable CPE
equipement that will need to start rolling out the door in the next
couple of years.

That means doing the hop-by-hop processing that is in the DNS if
you advertise yourself as a nameserver rather than forwarding the
packet and hoping that it is ok.  We know EDNS UDP sizes in the
forwarded packets, both request and reply, may need to be reduced
to reflect the actual buffer sizes in the CPE equipment.  DO should
be able to be passed through, similarly AD and CD.

The client advertises a 4096 byte EDNS UDP size option, the CPE the
advertises a 1460 byte EDNS UDP size option when it "forwards" the
packet.  The reply with a 4096 byte EDNS UDP size will then be able
to be processed by the CPE equipment, when the reply is forwarded
to the original client the CPE will put the miniumum of the received
EDNS UDP size and its buffer size in the EDNS UDP size returned to
the original client.

The problem with CPE equipement is that it pretends to be a DNS
server without fulfilling enough of the requirements of a DNS server.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: William Allen Simpson <[email-masked]>
Date: Tue, 11 Aug 2009 19:55:12 -0400

> this sounds really quite nice.  is it dead?
I'd have to ask on the good old end2end-interest list.  Do we know
enough server developers?

>> part of that chorus.  In retrospect, I was wrong.
Well, the old grey IETF, she ain't what she used to be....

> and thus rewrite messages according to policy.  [...]
> 
Most of what you mentioned above has all been discussed at the meetings
for years.  I'd thought/hoped that the edns0 with a size of 1420 was
working, and the IANA signed root tests for the past year had worked well?

> reliable failure indication, other than timeouts, which are ambiguous.
And doesn't happen in any case until the box is under attack, so it doesn't
get tested often enough.  Still, it was clever and helped at a bad time.

> i had thought sctp was possibly going to be that protocol but i guess not?
Besides the various problems with SCTP, there's the major one that only UDP
and TCP get through most IPv4 NAT routers.

Anyway, send me a private message with a phone and a time, and I'll call.

---

From: Paul Vixie <[email-masked]>
Date: Wed, 12 Aug 2009 00:30:15 +0000

> Date: Tue, 11 Aug 2009 19:55:12 -0400

i don't think of 1420 as working, nor do i think of 1420 as enough, nor
do i think of dnssec metadata as the only thing that makes or can make
or will make common/frequent dns messages exceed PMTU or that root zone
signature sizes are the only signatures we need to test.  so, "mumble."

> > cookies, that had its own ip protocol number, is starting to seem like
> > a good idea.  when it doesn't work it'll be unambiguous -- even a
> > timeout is unambiguous if the protocol number is neither tcp nor
> > udp... a timeout means this new protocol isn't going to work and that
> > some kind of fallback (like udp/53 or tcp/53 or a VPN) should be used.
> >

that's a good thing and i'm counting on it.  we need a reliable unambiguous
indicator that fallback should be tried.  trying to get through a middlebox
that doesn't understand a new transport protocol at all, fits the bill.

> Anyway, send me a private message with a phone and a time, and I'll call.

on it.

---

From: David Conrad <[email-masked]>
Date: Tue, 11 Aug 2009 17:37:53 -0700

> working, and the IANA signed root tests for the past year had worked  

Unfortunately, the IANA testbed doesn't generally see resolver-side  
failures.  We did (or rather we hired OARC to) do some testing related  
to L root server and discovered that BIND's retry with a buffer size  
of 512 bytes with DO=1 (arguably in violation of 3226) significantly  
increases the fallback to TCP (since a signed NXDOMAIN is bigger than  
512 bytes), but that's different.

Regards,
-drc

---

From: Alex Bligh <[email-masked]>
Date: Wed, 12 Aug 2009 07:54:25 +0100

--On 11 August 2009 22:30:11 +0000 Paul Vixie <[email-masked]> wrote:

> a horrific number of badly implemented middleboxes think UDP/53 is fair
> game and thus rewrite messages according to policy.

This suggests another fallback policy: fall back to (e.g.) UDP/54 (yes, I
know that's XNS Clearinghouse or something, but I'm sure we can find
another well known port number). The spec for this port would be
"DNS that's not broken" (i.e. EDNS0 support compulsory, "MUST NOT"
on interfering middleboxen, etc.).

Any middlebox is going to do one of three things: pass the packets
through verbatim (fine), drop the packets entirely (in which case
SCTP etc. would be unlikely to get through either), or drop only
fragments (still a problem). What they wouldn't do is pretend to be a
UDP/54 DNS server when they aren't, or mangle the packets.

-- 
Alex Bligh

---

From: Paul Vixie <[email-masked]>
Date: Wed, 12 Aug 2009 07:03:43 +0000

> From: Mark Andrews <[email-masked]>
> Date: Wed, 12 Aug 2009 09:50:28 +1000
> ...
> > none of us consider these badly implemented middleboxes to be
> > repairable since they are all over the internet now and many of their
> > manufacturers are out of business and many of their original
> > installers/operators are retired or dead.
> 
> We don't need to repair them.  The net can cope with the what is already
> out there.  We just need to make sure new boxes being installed are not
> defective in particular all the IPv6 capable CPE equipement that will
> need to start rolling out the door in the next couple of years.

there is nothing we can do to stop new bad middleboxes from being bad,
either because they'll inherit codebases from bad ones, or because being
bad increases a few revenue opportunities.

i am particularly worried about stateless firewalls that don't remember
the <SRC,DST,IPID> of first fragments they forward, and thus cannot tell
the difference between a non-first fragment (having no UDP port numbers)
they ought to forward vs. one that they ought not forward.  i don't see
firewalls become more stateful, it's a hell of a lot of state and the
only folks clamouring for it are DNSSEC pushers who might sound insane.

> server without fulfilling enough of the requirements of a DNS server.

i'd say the problem with CPE equipment is that it was tested only against
current traffic, not against theoretical traffic, and so stuff that wasn't
in the test dataset usually didn't get implemented.  since initial revenue
didn't suffer, everybody moved up or moved on or cashed out and were heroes,
so stories from namedroppers@ about how these products are somehow bad, are
completely off-topic for most product designers.

---

From: Paul Vixie <[email-masked]>
Date: Wed, 12 Aug 2009 07:05:33 +0000

> Date: Wed, 12 Aug 2009 07:54:25 +0100
> From: Alex Bligh <[email-masked]>
> 
> another well known port number). The spec for this port would be "DNS
> that's not broken" (i.e. EDNS0 support compulsory, "MUST NOT" on
> interfering middleboxen, etc.).

i don't really see a way to enforce rules on third parties like middlebox
product designers.  but if i could i would still want something other than
UDP because firewalls aren't stateful and non-first fragments don't have
port numbers.  TCP is fragmentation-aware and thus many successful products
can stomp all over fragmentation and feel no market/revenue pain, and have.

---

From: Alex Bligh <[email-masked]>
Date: Wed, 12 Aug 2009 08:55:03 +0100

--On 12 August 2009 07:05:33 +0000 Paul Vixie <[email-masked]> wrote:

> product designers.

I meant "MUST NOT" in an RFC sense. I am of the impression that many
middleboxen do not intend to be obnoxious. They are either making
a misguided attempt to be helpful (e.g. proxying UDP/53 apparently
transparently) or are just importing code that was used in the
previous^n release. This cruft would be blown away on UDP/54.

Middlebox designers can always deliberately interfere with UDP/54, with
SCTP or with indeed anything else which isn't cryptographically signed.
So short of HTTPS+XML / IPsec type solutions, I can't see how
anything solves this at the transport layer.

However, the original problem space that we concerned about is that
DNSSEC is likely to make packets larger (and thus run into problems).
If it's DNSSEC packets we are talking about, they will themselves be
a lot harder to /productively/ interfere with, as the records themselves
will be signed. Protecting against such interference was, after all, one
of the design goals of DNSSEC. I agree UDP/54 would give no protection
to vanilla DNS, but I wasn't really proposing it for that.

> but if i could i would still want something other than
> port numbers.  TCP is fragmentation-aware and thus many successful
> products can stomp all over fragmentation and feel no market/revenue
> pain, and have.

Indeed, it doesn't solve the fragmentation problem.

BTW I think it's not just firewalls, but also NAT boxes.

However, most firewalls/NATs (at least on the middleboxen currently
causing us pain) don't appear to pass anything other than UDP,
TCP and ICMP, so we are stuck with choosing between them if the
goal is to traverse existing middleboxen better (I recommend
not using ICMP...).

-- 
Alex Bligh

---

From: Jim Reid <[email-masked]>
Date: Wed, 12 Aug 2009 09:30:50 +0100

> UDP/54 DNS server when they aren't, or mangle the packets.

I wouldn't want to rely on that assumption. If/when DNS uses an  
alternate or fallback port as you suggest, new middleboxes will almost  
certainly get firmware which breaks packets to that port number in the  
same way they break stuff to port 53. The same will no doubt hold if/ 
when an alternate transport is used for DNS traffic.

---

From: [email-masked]
Date: Wed, 12 Aug 2009 09:44:46 +0100

> port numbers.  TCP is fragmentation-aware and thus many successful 
products
> can stomp all over fragmentation and feel no market/revenue pain, and 
have.

Paul,

Please note that in the testing for SAC035 we experienced no significant 
issues with _routing_ fragmented DNS packets through any of the 
middleboxes[*].  All of them were able to forward fragmented responses to 
the original client.  FWIW - NAT and firewalling was enabled for every 
test - the state simply wasn't a problem for them.

The fragmentation problems were with the _proxies_ in those boxes, where 
those proxies were only used because that's what their DHCP servers 
offered.

So, in the SOHO case, yes, there are potential problems, but most of them 
will only be evident iff an end user box is requesting large responses 
(e.g. for DNSSEC validation) and the requests are directed at their local 
SOHO gateway.  If the end user is sending their DNS requests directly to 
their ISP resolver they won't normally have _any_ problems with UDP or 
EDNS0/4096.

Since I did my research Eric Osterweil has shown that there are apparently 
also fragmentation related problems on the recursor -> authority path. 
However I for one would like to know what the specific network 
architecture was for each test site so that we can tell what class of 
device actually caused the problems and whether in effect he's just found 
the same problems that I did.

kind regards,

Ray

[*] I'm only aware of a couple of rare cases where SOHO middleboxes
 _transparently_ modify DNS traffic.  Specifically on Zyxel DSL routers
 their zero-configuration logic spoofs a specific A record such that
 end-user HTTP requests go to a "your internet connection is offline"
 page when the DSL WAN port is down.

 There was also one router which would only permit outbound DNS traffic
 if it was to the same DNS servers that were in the ISP's DHCP offer.
 That was fixed in a subsequent firmware update.

-- 
Ray Bellis, MA(Oxon) MIET
Senior Researcher in Advanced Projects, Nominet
e: [email-masked], t: +44 1865 332211

---

From: [email-masked]
Date: Wed, 12 Aug 2009 09:53:22 +0100

> when an alternate transport is used for DNS traffic.

To re-iterate what I just wrote to Paul:

The overwhelming majority of middleboxes *DO NOT* break DNS traffic that's 
_routed_ through them.

However most SOHO boxes *DO* break DNS requests that are _directed_ at 
them, particularly those with "large" responses.

Unfortunately that's a very common case because that's the "standard" DHCP 
offer from their built-in DHCP servers, as apparently recommended by the 
Broadband Forum.

What's really needed is a better way to bootstrap end-device DNS settings 
so that clients can find their *real* upstream recursive servers and not 
send queries to their SOHO gateway's proxy.

Ray

---

From: [email-masked]
Date: Wed, 12 Aug 2009 10:07:46 +0100

> previous^n release.

Alex,

What sort of boxes are you talking about?

Don't forget that the sort of breakage I found and reported on was not 
caused by "transparent proxying".

Sure, there's the "helpful" PIX configuration that says all DNS is <= 
512B, but that's comparatively rare.

Ray

---

From: Alex Bligh <[email-masked]>
Date: Wed, 12 Aug 2009 10:39:36 +0100

> caused by "transparent proxying".

Sorry, slack terminology. I am including (a) the case where they
advertise themselves as the DNS server through DHCP, to the local LAN, then
one way or another handle packets addressed to them (I recognise there are
various ways they can do that), (b) the "transparent proxy" case, which as
I understand it happens in some WiFi hotspot / hotel room configurations,
at least until authentication has been performed (that's the "e.g." case
above) and (c) firewalls which appear to proxy at the DNS layer (they
appear to work as forwarders, decoding the query etc.)

(from another mail)
> middleboxes

So to be clear, passing fragmented UDP packets through NAT/PNAT is
not in general a problem (whether on UDP/53 or otherwise)?

(from a third email)
> send queries to their SOHO gateway's proxy.

The problem appears to have been framed in terms where modifying the
middleboxen (at least those in case (a) above) is difficult to impossible.

Given that this iteration of the problem appears to be addressed at
solving the problem of large DNS answers to DNSSEC queries, we can
I suppose assume that the client software is being updated, or it
would be asking for DNSSEC records at all.

If what you are saying is correct (i.e. the problem only arises when the
queries are addressed to the router as per a DHCP response), then perhaps
the answer is for the client stub resolver to make some early query (when
it receives the DHCP option) for a well known A record (that ISPs could
anycast), from which they'd either pull the addresses of "proper"
name-servers directly, or perhaps more secure pull them over https from one
the A records provided.

-- 
Alex Bligh

---

From: [email-masked]
Date: Wed, 12 Aug 2009 10:56:40 +0100

> appear to work as forwarders, decoding the query etc.)

OK,

So, case (a) is what I tested.

All bets are off in case (b), but as I understand it this sort of DNS 
interception is generally being phased out in favour of HTTP interception 
because of the client cacheing problem.

I don't have hard data on case (c), but I believe it's relatively 
uncommon.

> not in general a problem (whether on UDP/53 or otherwise)?

In my experience, that's correct, there's no problem with fragmentation 
and NAT/PNAT.
 
> name-servers directly, or perhaps more secure pull them over https from 
one

I've previously had discussions with Wouter about how a smart stub might 
try and find the address of the local recursive servers instead of using 
those from DHCP.

Part of the solution might be an authoritative DNS server that returns the 
IP address of the recursive server that sent the query.  Funnily enough, 
there's one of those in my recently published evldns software ;-)

(see http://code.google.com/p/evldns/)

Ray

---

From: Eric Osterweil <[email-masked]>
Date: Wed, 12 Aug 2009 09:50:39 -0600

> In message <[email-masked]>, Paul Vixie writes:

<snip>

>> repairable sinc
>> e

<snip>

> EDNS UDP size and its buffer size in the EDNS UDP size returned to

You know, rather than trying to imagine how middleboxes can be made to  
help, we could just talk about the relatively simple fix of letting  
resolvers discover the buffer size that works for them.  afaict, we  
still have control over resolver software ("we" meaning those resolver  
maintainers on the list).  This would allow resolvers to advertise the  
proper sizes and it would allow _them_ to keep state if different  
buffer sizes were needed for different paths.  This ought to be  
possible with fewer RTTs than TCP and certainly fewer than SCTP.

Hop-by-hop rewrite is a slippery slope.  If we really needed it, that  
would be one thing, but I think it's quite unnecessary here, and  
likely infeasible anyway.

Eric
é

---

From: [email-masked]
Date: Wed, 12 Aug 2009 17:15:54 +0100

> I'm not 100% clear on what you're asking, so please let me know if 
> this is helpful.
> 
> The testing we do is from each of our 8 pollers.  Each of these is 
> hosted in someone else's network.  The list can be found at
>    http://secspider.cs.ucla.edu/clients.html
> 
> I am not familiar with the actual architecture of each network. 

Hi Eric,

What I'm hoping to establish is whether at each probe site the DNS queries 
are sent:

1.  directly to each zone's authoritative servers, or
2.  directly to specified recursive servers, or
3.  to whatever recursive servers the local site offered by DHCP

(where option 3 in some cases may be a local device's DNS proxy).

> I was curious if you think there is some merit to doing some additional 
> forensics from those pollers when we get a failure, and if you have 
> any thoughts there?  We currently to not catch any ICMP responses, so 
> that has been an idea we've kicked around to help with correlating 
> failures.

We didn't look for ICMP responses either.  What we did do was look at 
packet traces either side of the the proxy to determine whether it was the 
request or (fragments of) the response being dropped.

kind regards,

Ray

---

From: Eric Osterweil <[email-masked]>
Date: Wed, 12 Aug 2009 10:29:25 -0600

On Aug 12, 2009, at 10:15 AM, [email-masked] wrote:

> What I'm hoping to establish is whether at each probe site the DNS  
> queries are sent:
>

Yes.

> 3.  to whatever recursive servers the local site offered by DHCP

Only for NS+A.  After that, the DNSSEC queries are sent to auth name  
servers only.

> We didn't look for ICMP responses either.  What we did do was look  
> at packet traces either side of the the proxy to determine whether  
> it was the request or (fragments of) the response being dropped.

Then, this should be an illuminating extension I hope. :)

Eric
é

---

From: Mark Andrews <[email-masked]>
Date: Thu, 13 Aug 2009 09:31:59 +1000

In message <[email-masked]>, Eric Osterweil writes:
> This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
> --Apple-Mail-15--945383664
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes
> Content-Transfer-Encoding: quoted-printable
> 
> 
> >> none of us consider these badly implemented middleboxes to be =20
> >> repairable sinc
> >> e
> >> they are all over the internet now and many of their manufacturers =20=
> 
> >> are out of
> >> business and many of their original installers/operators are =20
> >> retired or dead.
> >
> 
> <snip>
> 
> >
> You know, rather than trying to imagine how middleboxes can be made to =20=
> help, we could just talk about the relatively simple fix of letting =20
> resolvers discover the buffer size that works for them.  afaict, we =20
> still have control over resolver software ("we" meaning those resolver =20=
> maintainers on the list).  This would allow resolvers to advertise the =20=
> proper sizes and it would allow _them_ to keep state if different =20
> buffer sizes were needed for different paths.  This ought to be =20
> Hop-by-hop rewrite is a slippery slope.  If we really needed it, that =20=
> would be one thing, but I think it's quite unnecessary here, and =20

So you don't think we should log bug reports with the vendors of
products which advertise themselves as being recursive DNS servers
but fail to perform correctly in that role?

How about those that don't support DNS over TCP?  Fixing the resolver
to use optimal UDP sizes won't fix that problem.  If I remember
Ray's report there were a number of vendors that stuffed up TCP.

The vendors of these boxes need to be held accountable.

I've got a NETGEAR WGR614v6 (firmware V2.0.19_1.0.19NA) wireless
router which doesn't do either of these things correctly.  Should
I log a fault report or not?

For the record the EDNS problem is logged as case 9937612 and the
TCP problem is logged as case 9937776.

Mark

---

From: Eric Osterweil <[email-masked]>
Date: Wed, 12 Aug 2009 18:44:55 -0600

On Aug 12, 2009, at 5:31 PM, Mark Andrews wrote:

>> =20
>> =20

Hey Mark,

I don't think namedroppers is a bug-report list... is it? ;)

Of course you should report these things, but I don't think that  
holding one's breath for a flag-day when everyone patches is a good  
plan.

> Ray's report there were a number of vendors that stuffed up TCP.

If you don't fallback to TCP because of overzealous 512 queries, you  
might be just fine.  If a middlebox doesn't support TCP it should be  
fixed.  In the meantime, the e2e argument would seem to indicate that  
the _ends_ should try to look after themselves.  There's growing  
evidence (which I'm happy to rehash here for the `n'th time if u  
really want) that this is not only possible, but reasonably  
straightforward in many cases.  A few well chosen UDP queries (once  
per name server at the cost of a little state), and often times you're  
all set.

> The vendors of these boxes need to be held accountable.

Sure, but that is very clearly a separate argument.  One which I do  
not in any way object to.

> I log a fault report or not?

Nice flame bait.

> TCP problem is logged as case 9937776.

Great, so are the fixes being belayed until the middleboxes are  
fixed?  I think _that_ would be a shame.

Eric
é

---

From: Michael Richardson <[email-masked]>
Date: Wed, 12 Aug 2009 15:48:19 -0400

[ Moderators note: Post was moderated, either because it was posted by
   a non-subscriber, or because it was over 20K.  
   With the massive amount of spam, it is easy to miss and therefore 
   delete relevant posts by non-subscribers. 
   Please fix your subscription addresses. ]

[email-masked] wrote:
> middleboxes[*].  All of them were able to forward fragmented responses 
> to the original client.  FWIW - NAT and firewalling was enabled for 
> every test - the state simply wasn't a problem for them.
> 

By "proxy", you mean the sort-a recursive resolvers that their DHCP 
servers offered? (some of them aren't really recursive resolvers, they 
just punt to the ISP, some of them can answer queries about ".local" 
though, ...)

> So, in the SOHO case, yes, there are potential problems, but most of 
> them will only be evident iff an end user box is requesting large 
> responses (e.g. for DNSSEC validation) and the requests are directed at 
> their local SOHO gateway.  If the end user is sending their DNS requests 
> directly to their ISP resolver they won't normally have _any_ problems 
> with UDP or EDNS0/4096.

That is an interesting case of yet-again, the middle box is too smart 
for anyone's good.

---

From: Paul Vixie <[email-masked]>
Date: Thu, 13 Aug 2009 15:08:35 +0000

none of the analyses or even theories shown on this thread explain why
edns0@4096 fails often enough that we have to fall back to edns0@1420
or even edns@512, except to bolster the case for "fragmentation is
broken in a lot of places, and edns0 should not have depended on it."

whether it's firewalls who can only forward udp first-frags that have port
numbers, or proxies who don't speak edns at all, or IDS boxes who won't
forward udp/53 if the payload size is larger than 512 or has a non-empty
additional data section, doesn't matter as much as the basic fact that
edns0's large-response mechanism isn't working.

some folks who want dnssec are now proposing a way to make responses
smaller, and this isn't because they want to save bytes on the wire, it's
because large responses basically don't work.  noting that there are other
reasons why large responses could be valuable, i prefer another approach.
but the need these folks feel further bolsters the case that edns0's large
response mechanism isn't working.

i'm not going to wade back in right now on why i don't consider tcp/53 a
solution.  i just thought i'd better summarize this thread in my own eyes
which is to say, edns0's large-response mechanism isn't working, often
enough that TLD operators who turn on dnssec see a surge in tcp traffic,
often enough that some dnssec folks are trying to find ways to make their
messages smaller.  none of these problems are in the endpoints, it's all
boxes in the middle who are causing these problems.

---

From: Florian Weimer <[email-masked]>
Date: Thu, 13 Aug 2009 15:35:24 +0000

> enough that TLD operators who turn on dnssec see a surge in tcp traffic,

This is just based on a single TLD's experience which had implemented
not one, but two peculiar configuration choices, right?

-- 
Florian Weimer                <[email-masked]>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

---

From: Paul Vixie <[email-masked]>
Date: Thu, 13 Aug 2009 15:39:30 +0000

> From: Florian Weimer <[email-masked]>
> Date: Thu, 13 Aug 2009 15:35:24 +0000
> 
> > i'm not going to wade back in right now on why i don't consider tcp/53
> > a solution.  i just thought i'd better summarize this thread in my own
> > eyes which is to say, edns0's large-response mechanism isn't working,
> > often enough that TLD operators who turn on dnssec see a surge in tcp
> > traffic,
> 

dunno.  can we get summaries of tld experiences posted here?

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 13 Aug 2009 11:58:35 -0400

> dunno.  can we get summaries of tld experiences posted here?

Your friendly neighbourhodd moderator notes that this (like some other
recent threads) is drifting precariously close to discussions of
operations, and there is another working group to discuss those
issues.  In this case, I don't object because it's desirable input to
discussions of whether we need protocol work to add a transport option
for DNS.  Just let's please not drift too far from protocol changes,
ok?

Thanks,

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: [email-masked]
Date: Thu, 13 Aug 2009 17:15:22 +0100

> broken in a lot of places, and edns0 should not have depended on it."

I for one really would like to know why EDNS0 @ 4096 apparently doesn't 
work in same cases (per Eric's research) and be able to explain it.

I already know why for the SOHO middlebox case, but to be honest that's 
not really a big problem (IMHO) for most DNS packets.  Those users can 
probably bypass their SOHO DNS proxies without too much effort.

what we don't have is an explanation for the potential causes further 
inside the enterprise / ISP network.

If anyone would like to help me research that (Eric, Nick W ?) I'm up for 
it.

Ray

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 13 Aug 2009 10:19:39 -0600

> edns0's large-response mechanism isn't working.

Hey Paul,

I've previously mentioned my analyses that directly speak to this.   
Rather than rehash them all, take a look at this one figure (as the  
most concise example I can think of to make):
	http://secspider.cs.ucla.edu/doc/pmtu-poller.pdf

This is one figure from the presentation I gave at RIPE 58.  This  
shows (among other things), that EDNS0 is (in fact) working.  Many of  
the queries that fail w/ buffer sizes of 4096 would work just fine  
with EDNS0 over UDP if the buffer size were more cluefully set.  There  
are, of course, some that don't.  These numbers do vary by location,  
and some locations show far less trouble than others.  So, I don't  
think it's reasonable or even true to say "none of the analyses" have  
addressed this.  Further, this single graph refutes the absolute  
statement that EDNS0 is broken.

There is a path to addressing this problem, and it doesn't have to be  
anywhere near as drastic as starting with SCTP.

> because large responses basically don't work.

Not true.  Sometimes there are problems, but often it is possible to  
overcome them.

> noting that there are other

I strongly disagree based on what I've said above.

> enough that TLD operators who turn on dnssec see a surge in tcp  

I suspect (and would love to try and verify with data) that this has  
something to do with an overly aggressive fallback to 512.  This does  
not have to happen, and can (likely) be addresses on the resolver side.

> boxes in the middle who are causing these problems.

The same could be said for network congestion, but the endpoints have  
been pretty successful addressing it with back-offs.  I mention this  
only cite precedence for successfully addressing path problems at the  
ends.

Eric
é

---

From: Paul Vixie <[email-masked]>
Date: Thu, 13 Aug 2009 16:54:17 +0000

> From: Eric Osterweil <[email-masked]>
> Date: Thu, 13 Aug 2009 10:19:39 -0600
> ...
> This is one figure from the presentation I gave at RIPE 58.  This shows
> (among other things), that EDNS0 is (in fact) working.  Many of the
> queries that fail w/ buffer sizes of 4096 would work just fine with EDNS0
> over UDP if the buffer size were more cluefully set.

"more cluefully"?  if edns0 at 4096 is not working, then is fragmentation
at fault?  if there's a "more cluefully" that would make it all work, then
shouldn't we be working on 2671-bis rather than dnssec-algo-signal?

> anywhere near as drastic as starting with SCTP.

based on florian's and william's explainations i now understand that sctp
has too many moving parts, is not mature, and may not be secure by design.

> Sometimes there are problems, but often it is possible to overcome them.

is this a recommendation that work be done on 2671-bis rather than on
alternative transports or dnssec-algo-signal?

---

From: Michael Graff <[email-masked]>
Date: Thu, 13 Aug 2009 12:04:44 -0500

I've always thought the edns0 buffer size option was in fact too
limiting.  It's a single hard limit.  What I mean is, if a client says
"I can handle 4096 bytes" the server feels this is great, and will
happily add all glue and other bits in to make it as large as it can
given data it knows.

This seems OK from a protocol point of view, but I think it makes some
analysis of what is currently seen "in the wild" suspect.  For instance,
suppose a query to a ORG server resulted in a 3000 byte packet.  The
question I have here is what is the SMALLEST size this response would
have fit into, if only the data essential to the query at hand were
included?  I think we can all agree that smaller means "more likely to
make it to the client."

Perhaps a lot of this packet size issue can be solved by better
responder behavior.  A server might consider the ENDS0 as the upper
bounds, but try to keep the response below some soft threshold, say 1400
bytes.  The number is not critical for this discussion, just that there
is a number.

This means the server might need to decide between different types of
additional data to include rather than just "first-found."  Some glue is
necessary for certain queries, for instance, but the associated A
records for the query name might not be quite as urgent.  The goal here
is to make it more likely to get stuff through.

If this were done, an EDNS0 option could be added to say "I know I have
a clean path, set my soft limit to 4096, really."  This would be used
for people who really know what they are doing, and really do have
things set up properly and are willing to ensure it remains working.

- --Michael

---

From: Nicholas Weaver <[email-masked]>
Date: Thu, 13 Aug 2009 13:21:47 -0400

On Aug 13, 2009, at 12:15 PM, [email-masked] wrote:

> I for one really would like to know why EDNS0 @ 4096 apparently  
> doesn't work in same cases (per Eric's research) and be able to  
> explain it.
>
> I already know why for the SOHO middlebox case, but to be honest  
> that's not really a big problem (IMHO) for most DNS packets.  Those  
> users can probably bypass their SOHO DNS proxies without too much  
> effort.

Not if you want DNSSEC to really work, since DNSSEC really working  
requires that the validation occur at the stub resolver.

> what we don't have is an explanation for the potential causes  
> further inside the enterprise / ISP network.
>
> If anyone would like to help me research that (Eric, Nick W ?) I'm  
> up for it.

There are the following possible failure cases:

1) A middlebox or firewall blocks UDP fragments.

1') A middlebox or firewall which is active on DNS does not reassemble  
fragments.

2) A middlebox or firewall which is active on DNS assumes that DNS <=  
512B

3) A middlebox or firewall which is active on DNS does not understand  
and blocks EDNS

Testing in Netalyzr showed that for recursive resolvers which claim  
EDNS and support for EDNS>2000B, case 1 or 1' (we can't tell the  
difference [1]) occurs on the path between the recursive resolver and  
UC Berkeley for roughly 15% of the tested recursive resolvers, with  
case 2 or 3 being a very slim minority.

For end users, its still roughly 15% failure, but 2/3rds are 1 or 1',  
1/3rd is case 2 or 3, well, roughly speaking.

[1] We could theoretically tell the difference between 1 and 1' for  
the end user if we added a test for generic UDP fragments, but this  
would not work for the resolver themselves as we can't test the  
resolver's ability to process arbitrary UDP, only DNS packets.

---

From: Eric Osterweil <[email-masked]>
Date: Thu, 13 Aug 2009 12:14:09 -0600

On Aug 13, 2009, at 11:04 AM, Michael Graff wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>

Or the resolver could just more accurately represent what buffer size  
is likely to make it, and the name server can continue to be simple/ 
fast.

Rather than have the name server guess what might fit, it might be  
better to keep the protocol's current expressiveness, but teach  
resolvers not to false-advertise to the name severs.  Again, this  
ought to be doable in a very few RTTs and a small constant increase in  
the state that resolvers currently keep.

Eric
é

---

From: Alex Bligh <[email-masked]>
Date: Thu, 13 Aug 2009 19:15:36 +0100

> work in same cases (per Eric's research) and be able to explain it.

I'm guessing the /real/ question is why EDNS0@4096 apparently doesn't
work but EDNS@1420 does. That goes to large packet support.

EDNS@4096 might just fail through lack of EDNS support.

-- 
Alex Bligh

---

From: Michael Graff <[email-masked]>
Date: Thu, 13 Aug 2009 13:59:44 -0500

Eric Osterweil wrote:
> Or the resolver could just more accurately represent what buffer size is
> likely to make it, and the name server can continue to be simple/fast.

Which is likely to happen this month, I wonder?  :)

- --Michael

---

From: Doug Barton <[email-masked]>
Date: Thu, 13 Aug 2009 13:50:32 -0700

> broken in a lot of places, and edns0 should not have depended on it."

My understanding for why 1420 might work when 4096 doesn't is that one
or both of (bad) PMTUD or tunneling is the culprit.

My understanding of why 512 works when nothing bigger works is that
there are still a lot of legacy "firewalls" and/or nat devices that
don't believe any answer coming back with greater than 512 bytes can
be valid. I know we ran into this a lot when I was at Yahoo! to the
extent that we hacked BIND to make sure our answers fit into 512 bytes.

Not sure either of these issues are "news" (although I don't think
anyone has mentioned PMTUD problems yet) but I mention them both to
indicate why I am opposed to "solving" the "problem" with a new
transport protocol. I don't see any chance that adding a new form of
DNS transport will solve any of the existing problems given the origin
of those problems.

Before going any further though I would like to see a clear statement
of the problem we're trying to solve. I've heard some vague references
to "fallback to tcp is a problem" without data to back it up.

That said, my past experience tells me that when deployment of DNSSEC
becomes more widespread we _will_ see problems of a "packet too big"
nature, at least partly because so many dns and/or network
"administrators" have foolishly closed off tcp access for their name
servers in spite of the efforts of those of us who are now blue in the
face from telling people why that's a bad idea. My argument in
response to that is that this is an _existing_ problem that will
simply be made worse by DNSSEC, and needs to be solved anyway. More
than 80% of the problem exists on the client/configuration side, and
needs to be solved on the client side without any protocol changes.

In regards to actual protocol stuff, I agree with those who have said
that making EDNS packet size backoff more intelligent is the way to
move forward NOW, regardless of how large the tcp fallback problem
turns out to be (or how long it takes to get data on it). Since we
already know that EDNS isn't working as we had hoped, and since I
imagine there would be general agreement that we would like to keep as
much traffic on udp as we can, making EDNS work better solves both
problems we can see, and ones we can't (yet).

In the absence of anything more specific on either end, some sort of
process like:
1. Try 4096, if that doesn't work
2. Try 2048, if that doesn't work
3. Try 1420, if that doesn't work
4. Stick with 512

The number for 2 is arbitrarily proposed at 4096/2 for strawman
purposes. It would be interesting to see how large a "larger than
average" packet is really going to be in the DNSSEC world, and if
there is a more intelligent value between 4096 and 1420 that really is
useful. If not, we could eliminate step 2 altogether. The 1420 number
is my understanding of the largest number that will safely avoid
problems with both PMTUD and tunnels. Those more knowledgeable about
those topics are welcome to suggest a different number.

hope this helps,

Doug

---

From: [email-masked]
Date: Fri, 14 Aug 2009 00:49:52 +0100

> requires that the validation occur at the stub resolver.

Agreed.

> ....

SAC035 results were 100% success for 1, and ~50% for 1', admittedly only 
on a sample of 24 SOHO units.

Ray

---

From: Mark Andrews <[email-masked]>
Date: Fri, 14 Aug 2009 10:34:19 +1000

In message <[email-masked]
> If anyone would like to help me research that (Eric, Nick W ?) I'm up for 

Overly conservative or not conservative enough firewall vendors
basically explain the firewall things.

If firewall vendors examined the queries then EDNS would not have
passed out through the firewall and there would be no problem with
EDNS replies.   Firewall vendors would have had presure put on them
to support EDNS and it would have been planned support.

Instead firewall vendors only examined the replies which is what
caused the problems.  The allowed EDNS out without allowing EDNS
in.

Now we are in the state were that lack of checking is causing issues
and everyone want the issues resolved *NOW*.   If we (resolver
vendors) just leave this alone the middleware (SOHO and Firewall)
vendors will fix their boxes or have new boxes without the problem.
The problem boxes will be upgraded/replaced.  Yes it will take a
few years.

EDNS hasn't failed.  It just hasn't been on the middle box verdors
radar up until now.  We just need to make it more present on their
radar.

We (IETF) needs to start dialogs with the DSL and Cable Forums, as
most of the vendors involved here will be members of one or other
of those forums, to point out how a common faults in their products
are causing real problems.

Mark

---

From: Mark Andrews <[email-masked]>
Date: Fri, 14 Aug 2009 10:55:42 +1000

In message <[email-masked]>, Eric Osterweil w
> not have to happen, and can (likely) be addresses on the resolver side.

There isn't time to add more fallback steps to recursive servers
as they need to do all this probing within the stub resolver timeout.

For stub resolvers there should be the ability to try EDNS@1400 and
EDNS@1200 as the stub resolver controls the timeout.

Note servers can also impose their own limits on the EDNS response
sizes.  Cable and DSL vendors could configure their recursive servers
to not send answers that exceed the ethernet sizes to their recursive
clients.  This is really a workaround until CPE equipment is upgraded.

With a bit more intelligence they could use a IDS and tune this on
a per customer basis.  They already do this sort of thing for other
services by looking at the vendor codes in the ethernet MAC.  They
get it wrong sometimes which is how I know they do this.

Mark

---

From: Nicholas Weaver <[email-masked]>
Date: Thu, 13 Aug 2009 21:00:49 -0400

> extent that we hacked BIND to make sure our answers fit into 512  

Its not PMTUD, its "inability to handle fragments".  DNS UDP is  
generally not sent with DF set, so it will fragment just fine, just as  
soon as it hits the ethernet.  And 85% of the measured cases will  
reassemble the fragments quite happily.

Its just some device in path either can't handle fragments at all, or  
wants to handle DNS but can't handle fragmented DNS packets.

> 2. Try 2048, if that doesn't work

Skip this.  If 4096 fails, it is because of SOME process that won't  
like anything bigger than the de-facto Internet MTU defined by Ethernet.

The problem in most cases (for resolvers) occurs when packet  
fragmentation begins.

---

From: Doug Otis <[email-masked]>
Date: Thu, 13 Aug 2009 18:26:53 -0700

> radar up until now.  We just need to make it more present on their

Middleware with EDNS0@4096 is not only a problem.  There are issues 
related to network amplifications, and whether fragmentation reassembly 
can endure potential abuses caused by UDP's lack of PMTU.  In other 
words, EDNS0@4096 is not inherently safe, in addition to problems 
created by dependence upon the acceptance of jumbo frames.

Ethernet frame size maximum is 1522 per IEEE 802.Q where the CRC 
polynomial suffers reduced error detection rates beyond this limit.  Use 
of the CRC32c polynomial in SCTP restores detection rates for jumbo 
frames, while also ensuring detection is end-to-end, rather than NIC TO 
NIC where systems might inject errors caused by faulty bus drivers or 
memory modules.  Lacking end-to-end protection leaves just the TCP or 
UDP checksums which fail to detect as much as 2% of the bus (bit 
specific) errors since they tend to self cancel.

-Doug

---

From: Florian Weimer <[email-masked]>
Date: Fri, 14 Aug 2009 22:45:55 +0200

* Nicholas Weaver:

> DNS UDP is generally not sent with DF set,

The picture is mixed.  There's a lost of DF at the TLD and root level.
End users typically use mainstream systems in more recent versions
which generally don't set DF anymore.

---

From: Mark Andrews <[email-masked]>
Date: Sat, 15 Aug 2009 08:30:09 +1000

> which generally don't set DF anymore.

Only because OS vendors turned it on without thinking about the
applications.  Named turns it off it if can on a per socket basis.
The only OS where it is on for UDP and named can't turn it off on
a per socket basis that I am aware of is Solaris.

Mark

---

From: "George Barwood" <[email-masked]>
Date: Sun, 16 Aug 2009 09:49:26 +0100

----- Original Message ----- 
From: "Mark Andrews" <[email-masked]>
To: "Florian Weimer" <[email-masked]>
Cc: "Nicholas Weaver" <[email-masked]>; "Doug Barton" 
<[email-masked]>; "Paul Vixie" <[email-masked]>; 
<[email-masked]>
Sent: Friday, August 14, 2009 11:30 PM
Subject: Re: [dnsext] DNS over SCTP problems, part 1, theory

> a per socket basis that I am aware of is Solaris.

Since fragmented responses are vulnerable to spoofing, would it be
better to set DF=1 where possible, and accept TCP fallback?

Does Named react to ICMP messages  when DF=1 is in effect
( by transmitting a truncated response ) ?  I don't know if support
for this is widespread. A quick google search turned up this

http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.commadmn/doc/commadmndita/sctp_mtu.htm

which suggests that it is possible in principle.

George