Subject: Re: [DNSOP] I-D Action: draft-ietf-dnsop-avoid-fragmentation-00.txt

From: [email-masked]
Date: Wed, 08 Jul 2020 17:01:23 +0900 (JST)

DNSOP WG,

Paul Vixie and I submitted draft-ietf-dnsop-avoid-fragmentation-00.
Please review it.

> https://tools.ietf.org/html/draft-ietf-dnsop-avoid-fragmentation-00

I may have some mistakes, I could not find links to show differences
from draft-fujiwara-dnsop-avoid-fragmentation-03.
Please see differences from this URL.
  https://tools.ietf.org/rfcdiff?url1=https://tools.ietf.org/id/draft-fujiwara-dnsop-avoid-fragmentation-03.txt&url2=https://tools.ietf.org/id/draft-ietf-dnsop-avoid-fragmentation-00.txt

Differences from -03 to 00 are
  Added "DNSSEC is a countermeasure .." in Intro.
  Removed 7.2 DNS packet size.
  Moved details of Minimal-responses to appendix B
  Added reference to draft-ietf-tsvwg-datagram-plpmtud

And more, we would like to make some changes in -01.

  * Adding new text in abstract.

    "EDNS0 enables a DNS server to send large responses using UDP
     and is widely deployed."

  * Change text related to TCP in Introduction because TCP changes MSS
    value to avoid IP fragmentation under ICMP NEEDFRAG attacks.

    OLD
      By comparison, TCP is considered resistant against IP
      fragmentation attacks because TCP has a 32-bit sequence number
      and 32-bit acknowledgment number in each segment.

    NEW
      By comparison, TCP protocol stack controls packet size and
      avoid IP fragmentation under ICMP NEEDFRAG attacks.

  In TCP, fragmentation should be avoided for performance reasons, whereas for
  UDP, fragmentation should be avoided for resiliency and authenticity reasons.

  * I would like to use "in-domain" (defined in RFC 8499)

    OLD: and in-zone and below-zone glue in the additional data section.
    NEW: and in-domain (in-zone and below-zone) glue in the additional data section.

Regards,

--
Kazunori Fujiwara, JPRS <[email-masked]>

> From: [email-masked]
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
> This draft is a work item of the Domain Name System Operations WG of the IETF.
> 
>         Title           : Fragmentation Avoidance in DNS
>         Authors         : Kazunori Fujiwara
>                           Paul Vixie
> 	Filename        : draft-ietf-dnsop-avoid-fragmentation-00.txt
> 	Pages           : 10
> 	Date            : 2020-06-30
> 
> Abstract:
>    Path MTU discovery remains widely undeployed due to security issues,
>    and IP fragmentation has exposed weaknesses in application protocols.
>    Currently, DNS is known to be the largest user of IP fragmentation.
>    It is possible to avoid IP fragmentation in DNS by limiting response
>    size where possible, and signaling the need to upgrade from UDP to
>    TCP transport where necessary.  This document proposes to avoid IP
>    fragmentation in DNS.
> 
> 
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-ietf-dnsop-avoid-fragmentation/
> 
> There are also htmlized versions available at:
> https://tools.ietf.org/html/draft-ietf-dnsop-avoid-fragmentation-00
> https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-avoid-fragmentation-00
> 
> 
> Please note that it may take a couple of minutes from the time of submission
> until the htmlized version and diff are available at tools.ietf.org.
> 
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
> 
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: Marek Majkowski <[email-masked]>
Date: Wed, 8 Jul 2020 16:50:30 +0200

> Paul Vixie and I submitted draft-ietf-dnsop-avoid-fragmentation-00.

Hi!

> UDP requestors and responders SHOULD send DNS responses with
> IP_DONTFRAG / IPV6_DONTFRAG [RFC3542] options, which will yield
> either a silent timeout, or a network (ICMP) error, if the path
> MTU is exceeded.

When MTU is exceeded the sender might also receive plain old EMSGSIZE
error on sendto(). I would love to see an example on what
IP_MTU_DISCOVER settings authors expect. This option is notoriously
hard to get right.

> The maximum buffer size offered by an EDNS0 initiator SHOULD be
> no larger than the estimated maximum DNS/UDP payload size...

This seems to indicate that EDNS0 over TCP should have a small buffer
size as well. Consider wording like "...buffer size offered by an
EDNS0 initator over UDP...".

> Fragmented DNS/UDP messages may be dropped without IP reassembly

Not sure what it has to do with the draft. Are we worried about
request fragmentation and allowing the DNS server to drop fragmented
requests? Are we worried about response fragmentation?

I have two problems with this proposal. First, it doesn't mention IPv4
vs IPv6 differences at all. In IPv4 landscape fragmentation, while a
security issue, is generally fine. In the IPv6 world, fragmentation is
disastrous - packets with extension headers are known to be dropped.

Second, this proposal assumes that path MTU detection works correctly.
This is surprisingly optimistic. Let's consider IPv6 - in IPv6 the
smaller path MTU < 1500 is very common.

Let's say a DNS auth server sent an IPv6 DNS response packet exceeding
path MTU. An intermediate router will drop the offending packet and
one of three scenarios will happen:

- (A) No ICMP PTB message is sent back.

- (B) ICMP PTB message is sent back, but fails to be delivered.

- (C) ICMP PTB message is sent back and delivered correctly to the server.

All three scenarios are disastrous on the practical internet. The
proposal assumes (A) and (B) will rarely happen, and puts the
responsibility on the DNS client to retry over TCP. This will cause
unnecessary timeouts and degrade the overall quality of the service.

But perhaps most importantly even option (C) will *not* result in good
service. Consider a setup with multiple DNS servers behind an ECMP
router, or another L4 load balancer. Even if the return ICMP will hit
back the correct server - which is far from obvious - the ICMP will
update the Path MTU on *one server*. If a client attempts to retry the
query, as suggested by the proposal, it will most likely hit another
server, which is not aware of non standard Path MTU.

These days DNS Auth installations use ECMP routing for load balancing.
A single physical box serving important DNS is a rare occurrence.

In this proposal all three (A), (B), and (C) scenarios will result in
dropped responses. DNS client needs to wait for timeout, retry over
UDP, wait more and eventually retry over TCP. This is bad.

We could fix (C) by making the DNS server to capture the ICMP PTB in
DNS server code. The ICMP payload often has enough context for the DNS
server to prepare another reply. This reply of course should be sent
with lowered MTU.

In other words, I'm asking for capturing ICMP PTB in DNS servers,
unpacking the paylad and treating it as another request to be handled.
This would solve (C). Also, note this opens an interesting DDoS
vector, but this is another story.

On Linux it is possible to capture the ICMP PTB without privileges, by
setting IP_RECVERR and inspecting MSG_ERRQUEUE. In IPv4 the PTB
messages often have 520 bytes of payload and in IPv6 1184 bytes. This
is enough context to build another response, without having to wait
for any timeout.

Cheers,
Marek

---

From: Mukund Sivaraman <[email-masked]>
Date: Wed, 8 Jul 2020 20:48:30 +0530

> EDNS0 initator over UDP...".

The field has no effect in DNS over TCP. The EDNS "requestor's UDP
payload size" field that is encoded in the CLASS field of the OPT RR is
only used for DNS responses over UDP. It is ignored for DNS over TCP.

		Mukund

---

From: Mark Andrews <[email-masked]>
Date: Thu, 9 Jul 2020 11:35:05 +1000

> On 9 Jul 2020, at 00:50, Marek Majkowski <[email-masked]> wrote:
> 

Not really. UNKNOWN extensions tend to get dropped but the fragmentation
header is a KNOWN extension header.

> smaller path MTU < 1500 is very common.

Which is why IPV6_USE_MIN_MTU exists (RFC 3542).  USE THE SOCKET OPTION.
It was put there specifically to support DNS over UDP and other applications
like that.  I know this as I proposed the predecessor option back in 1999
which became IPV6_USE_MIN_MTU.

If the OS hosting your DNS server doesn’t support this option 17 years
after is was defined throw it in the bin.

IPV6_USE_MIN_MTU also helps with TCP.  DNS does not need to suffer from
PMTUD issues.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: [email-masked]

---

From: [email-masked]
Date: Thu, 09 Jul 2020 17:28:35 +0900 (JST)

> From: Marek Majkowski <[email-masked]>

Is IP_MTU_DISCOVER a Linux-only option ?

Authors don't consider discouraging Path MTU discovery.
We refer RFC1191, RFC8201 and ietf-tsvwg-datagram-plpmtud.
If Path MTU discovery works well, we can use large UDP datagram size.

> requests? Are we worried about response fragmentation?

I would like to change the text as:

  The DNS Requestor MAY drop Fragmented DNS/UDP responses without IP
  reassembly. (before IP reassembly?)

  (Texts related ICMP error may be dropped, I think.)

> disastrous - packets with extension headers are known to be dropped.

On IPv6, "every link in the Internet have an MTU of 1280 octets or
greater" (RFC 8200).
Then, if Path MTU discovery works, we can use real MTU value.
Otherwise, we can use 1280 as MTU.
We can easily avoid IPv6 fragmentation. (Fragmented IPv6 packets are bogus.)

On IPv4, it's terrible because IPv4 minimal MTU is 68, but most of
links support 1500 octet MTU.

> smaller path MTU < 1500 is very common.

Without IPv6 over IPv4 tunnels, most of IPv6 links support 1500 octet
MTU size.

We can detect path MTU discovery failure, then we can use 1280 MTU on IPv6.

> one of three scenarios will happen:

  Leaf client ----Tunnel-----[Tunnel Router]-----Routers----Auth Server
                 MTU 1280                   1500       1500
                             drop ------------------------>
			     ICMP PTB

> - (B) ICMP PTB message is sent back, but fails to be delivered.

A and B is the same.
The first response is simply dropped.

Leaf client (full-service resolvers) may retry queries to other Auth
servers by UDP, or retry queries to the Auth server by TCP or UDP/EDNS
with small size.

> - (C) ICMP PTB message is sent back and delivered correctly to the server.

First, the first response is simply dropped.
Auth server knows that path MTU to the Leaf client is 1280.

Leaf client (full-service resolvers) may retry queries to other Auth
servers by UDP, or retry queries to the Auth server by TCP or UDP/EDNS
with small size. (The same as A and B.)

After some time, if the leaf node send next queries to the Auth server
with UDP and same parameter, the Auth server knows path MTU to the
leaf.  Then, the auth server need to compose response packets fit in
the path MTU size, or set TC=1.

> unnecessary timeouts and degrade the overall quality of the service.

To avoid these cases, we can make new recommendations.  Authoritative
servers and full-service resolvers SHOULD support 1500-octet path MTU
to major parts of the Internet.
(we need to define major parts of the Internet)

> UDP, wait more and eventually retry over TCP. This is bad.

Do you mean that DNS client is stub resolvers in clients ?
Stub resolvers may not set DO (DNSSEC OK) bit and responses can be small.

Or full-service resolvers ? Full-service resolvers should be located
at 1500-octet MTU world.

> server to prepare another reply. This reply of course should be sent

I think that we can know ICMP PTB result from applications by IP_MTU
socket options on Linux.

> is enough context to build another response, without having to wait

In my opinion, the captured PTB data is similar to IP_MTU/IPV6_MTU
socket options.

---

From: Marek Majkowski <[email-masked]>
Date: Thu, 9 Jul 2020 17:34:55 +0200

> header is a KNOWN extension header.

https://labs.apnic.net/presentations/store/2019-09-11-ipv6-fail.pdf

Slide 34 and 35 indicate, with IPv6:
 - 38% recursive resolvers fail to reassemble packets with
fragmentation extension header
 - 22% of browsers fail to process TCP packets with fragmentation header

This data is from 2017 - around that time I did similar experiments
with similar results. Delivery of fragmented packets in IPv6 sometimes
work, but this is more of an exception than a rule.

> which became IPV6_USE_MIN_MTU.

You are suggesting to always use at most 1280 byte packets in IPv6.
The draft we are discussing does not suggest that. Did I misunderstand
something?

Marek

---

From: Marek Majkowski <[email-masked]>
Date: Thu, 9 Jul 2020 18:31:48 +0200

> Is IP_MTU_DISCOVER a Linux-only option ?

Oh, I see. Still, my point remains. As a developer I'm not exactly
sure what should I set on Linux.

> We can easily avoid IPv6 fragmentation. (Fragmented IPv6 packets are bogus.)

How do you define if Path MTU discovery works? In my case, it works
fine. It's just on ICMP exactly one server of a large fleet will learn
the new path MTU. It's also worth noting that Path MTU expires
quickly, I think < 10 minutes. So the whole dance of request, timeout,
second request must be repeated often.

> We can detect path MTU discovery failure, then we can use 1280 MTU on IPv6.

How do we detect path MTU discovery failure in the context of UDP and
DNS protocols?

> servers by UDP, or retry queries to the Auth server by TCP or UDP/EDNS

Right. Does the proposal address only resolver-auth communication? Is
the draft also addressing stub-resolver case?

It's not clear for me what resolver should do on a timeouted query:
 - retry to the same DNS auth server again over UDP
 - retry to the same DNS auth over UDP using different UDP source port
 - retry over TCP
 - retry using another IP protocol (v4 vs v6) to auth server
 - retry over UDP with tweaked EDNS
- retry using another DNS auth server

Or a mixture of above?

> leaf.

Negative. *one of potentially a hundred* of ECMP Auth servers learns
the ephemeral Path MTU. There is no guarantee that the retry will use
the same source port, and routing path. The retry may end up at a
completely different machine due to ECMP routing or another L4 load
balancing mechanism. Even perfectly working Path MTU discovery is
limited in scope to only one server. There may be more than one DNS
auth server sharing the same IP.

> Then, the auth server need to compose response packets fit in

Right, so you are saying:
 - in communication between DNS auth and DNS recursive
 - in order to avoid transmitting fragments
 - no answer from auth should exceed 1500
 - or detected path MTU if such is present

This is fair. But then if you are suggesting that any decent internet
host should have MTU 1500 anyway you can remove the last point and the
whole path-MTU-discovery discussion.

Maybe there are two different discussions. What you are saying is:
don't be ridiculous - don't send udp packets > 1500 which for sure
will require fragmentation. While I am saying - when you do get PTB
ICMP back, react immediately and save the requestor a wait.

> In my opinion, the captured PTB data is similar to IP_MTU/IPV6_MTU

With captured ICMP the dns server is able to react and send something
out immediately. This will improve the service quality and save the
requestor the wait.

Marek

---

From: Mark Andrews <[email-masked]>
Date: Thu, 16 Jul 2020 13:12:45 +1000

> On 10 Jul 2020, at 01:34, Marek Majkowski <[email-masked]> wrote:
> 

Which means 62% of the world properly handles fragmented UDP packets.

Also I forget how Geoff was actually performing the test.  Where any of
the responses greater than the advertised EDNS buffer size sent?  Was
fragmentation introduced when the UDP response size was less that 1280?

> - 22% of browsers fail to process TCP packets with fragmentation header

Which seems to be a very strange test to perform given that TCP segments
are supposed to be atomic on IPv6.

> The draft we are discussing does not suggest that. Did I misunderstand

I’m suggesting that if you send UDP/IPv6 packets > 1280 that you force them to be
fragmented into 1280 octet fragments.

I’m suggesting that we force DNS/TCP/IPv6 packets to never be bigger that 1280 for
regular requests.  AXFR/IXFR would be a potential exception.  Clients and servers
can do this by setting the MSS to an appropriate value (1280-IPv6 header size-TCP
header size).  Setting IPV6_USE_MIN_MTU should also achieve this as TCP is supposed
to take into account path MTU which is 1280 when this option is set.

The IETF has designed IPv6 such that it is possible to run a DNS server that
will never receive a ICMPv6 PTB to traffic it sends even if other applications
on the server do.

---

From: Paul Vixie <[email-masked]>
Date: Thu, 16 Jul 2020 07:08:09 +0000

On Thursday, 16 July 2020 03:12:45 UTC Mark Andrews wrote:
> > ...

there's another form of tough love available to us, which is to first send 
fragments. that is, even in the questions, fragment so that the additional 
data section containing OPT is in its own fragment. and likewise in responses. 
don't send any unfragmented datagrams to anybody until they have demonstrated 
that they can reassemble one. make the problem belong to the 38%, and let them 
scream in pain.

when i suggested this approach to the community at DNS-OARC BKK, i was shot 
down. therefore, i now suggest an alternate form of tough love: always set DF, 
and never send (or accept) fragments.

for decades we have been liberal in what we accepted. that scaled negatively. 
so, just as we did last year by making EDNS mandatory, it's time we shifted 
the costs of badly configured networks onto their operators and away from the 
rest of us. either assume fragmentation's going to work, or assume it's not 
going to work.

> I’m suggesting that if you send UDP/IPv6 packets > 1280 that you force them
> to be fragmented into 1280 octet fragments.

as the avoid-fragmentation draft explains, it's almost always true that the 
actual PMTU is higher than 1280, even if this has not been determined using 
PLPMTUD or similar.

also explained: why TCP's segment elasticity allows it to precisely meet an 
MTU requirement (whether regulatory like 1280, measured like PMTUD, or 
guessed.)

also: why 1280 for IPv6 corresponds to 68 in IPv4, which are both absurd.

and finally: why a UDP transmitter cannot know the maximum UDP payload size 
due to IP headers that it cannot always predict or control, and why 1232 is a 
bad guess even if the regulatory minimum (1280) is assumed.

those are the topics i'd like to see discussed. we are drifting.

-- 
Paul

---

From: Tony Finch <[email-masked]>
Date: Thu, 16 Jul 2020 14:37:37 +0100

> fragmentation introduced when the UDP response size was less that 1280?

Just this morning I saw another report from Geoff mostly about IPv6 but
with a fair amount of discussion about fragmentation
https://www.potaroo.net/ispcol/2020-07/dns6.html

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Lyme Regis to Lands End including the Isles of Scilly: Westerly or
northwesterly 3 or 4, occasionally 5 near south Devon, becoming variable 2 to
4 later. Smooth or slight in east, slight or moderate in west. Fair. Good.