Subject: Re: [tsvwg] UDP-Options: UDP has two ”maximums”

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 2 Apr 2021 12:29:55 +0100

In the last IETF TSVWG meeting you asked for throughts about what MSS 
should mean?

I have now thought - the key unknown here is how large a datagram 
segment the remote receiver is willing to accept - i.e., what is the 
largest fragmented datagram I can send that would be reassembled. This 
might be any number, depending on the way in which the receiver is 
designed or configured. Whereas a sender could anticipate a common 
fragment size - e.g. 1200B, or use dplpmtud to discover this, there is 
no real way of determining the largest datagram that would be the option.

I think you suggested this as: Max reassembly size, a hard upper bound, 
similar to MSS_R.

Does this help?

Gorry

---

From: Joseph Touch <[email-masked]>
Date: Fri, 2 Apr 2021 10:59:12 -0700

Hi, Gorry (et al.),

> On Apr 2, 2021, at 4:29 AM, Gorry Fairhurst <[email-masked]> wrote:
> 
> In the last IETF TSVWG meeting you asked for throughts about what MSS should mean?
> 
> I have now thought - the key unknown here is how large a datagram segment the remote receiver is willing to accept - i.e., what is the largest fragmented datagram I can send that would be reassembled. This might be any number, depending on the way in which the receiver is designed or configured. Whereas a sender could anticipate a common fragment size - e.g. 1200B, or use dplpmtud to discover this, there is no real way of determining the largest datagram that would be the option.
> 
> I think you suggested this as: Max reassembly size, a hard upper bound, similar to MSS_R.
> 
> Does this help?

I had assumed that would be at useful.

The question that remains is whether it is useful to also have an MSS option in the same spirit as TCP. That would argue for two different MSS values:

	UDP path MSS
	UDP EMTU_R

Are both actually useful? Is the TCP one useful? (esp. given it really isn’t path info)?

Joe

---

From: Paul Vixie <[email-masked]>
Date: Fri, 2 Apr 2021 23:01:20 +0000

> The question that remains is whether it is useful to also have an MSS
> option in the same spirit as TCP. That would argue for two different
> MSS values:
> 
> 	UDP path MSS
> 	UDP EMTU_R
> 
> Are both actually useful? Is the TCP one useful? (esp. given it really
> isn???t path info)?

is everyone here familiar with RFC 8899 and RFC 8900? while RFC 8900 talks
about IP layer fragmentation, one of its stated concerns would apply to
app-layer (QUIC) fragmentation (see section 3.5), and there's a well known
micro-burst problem (shows up in NFS) where app-layer message transmission
isochrony won't be maintained for packets issued by the fragmenter.

RFC 8899 specifies an ICMP-free method for packetization layer path MTU
discovery (PLPMTUD), and i think making QUIC dependant on PLPMTUD would
avoid either duplication or waste of effort, as well as informing the
UDP MSS discussion of this thread.

-- 
Paul Vixie

---

From: Paul Vixie <[email-masked]>
Date: Fri, 2 Apr 2021 23:12:00 +0000

On Fri, Apr 02, 2021 at 12:29:55PM +0100, Gorry Fairhurst wrote:
> ... Whereas a sender could anticipate a common fragment size - e.g.
> 1200B, or use dplpmtud to discover this, there is no real way of
> determining the largest datagram that would be the option. ...

i think some form of discovery/negotiation ought to be required here.
the internet has lived with ethernet-sized packets since the mid-1980's
even while underlying link speeds have grown O(4^10). we're now seeing
host interfaces with their own CPU's and operating systems just to cope
with the extraordinary packet rates it takes at that MTU to fill those
pipes. we won't get to the 22nd century without larger MTUs, and it's
nothing like too soon to begin laying the ground work for that. many of
us are using MTU ~9000 in our campuses, and i'm aware of transit networks
using an MTU ~4100 in their cores (and thus available at their edges.)

anticipating "a common fragment size - e.g., 1200B" is how we got here,
but it doesn't have to be how we move on from here. FDDI used MTU ~4000
to avoid having PPS take the full cost of the 10X bit rate improvement
over ethernet. in a non-bridged ethernet market, where each new speed did
not have to be L2-reachable by hosts operating at the previous speed(s),
we would likely have been at MTU ~32K by the gigabit era, and MTU ~64K by
the tenGig era. it's worth recalling the lesson of ATM: that 53b was
considered too small for an OC3C link, and that this was one of the
reasons ATM stopped attracting new customers after a while.

-- 
Paul Vixie

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 3 Apr 2021 10:17:34 +0100

> Are both actually useful? Is the TCP one useful? (esp. given it really isn’t path info)?
Hi Joe,

I'll argue also that the maximum size a remote transport receiver can 
process, isn't so helpful, because the transport endpoint often does not 
know the set of encaps/tunnel/link-MTU along the path. That's why I 
argue we should have a "default" (as in "For IPv4, EMTU_S is the smaller 
of 576 bytes and the first-hop MTU [RFC1122]. For IPv6, EMTU_S is 1280 
bytes [RFC2460]"), that could be used - and might be expected to usually 
work, *or* a "probe" to find out what the path actually supports: 
knowing the remote receiver can process X bytes does not really help 
that much, and actually could be unhelpful when information if it is not 
correct. So I'll argue against sharing this from the receiver.

I'll also follow-up on Paul's email in a moment...

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 3 Apr 2021 10:20:31 +0100

> reasons ATM stopped attracting new customers after a while.
I think I agree, is this something like you were suggesting:

https://tools.ietf.org/html/draft-fairhurst-tsvwg-udp-options-dplpmtud-04

Gorry

---

From: Joseph Touch <[email-masked]>
Date: Sat, 3 Apr 2021 12:07:21 -0700

> On Apr 3, 2021, at 2:17 AM, Gorry Fairhurst <[email-masked]> wrote:
> 
> I'll argue also that the maximum size a remote transport receiver can process, isn't so helpful, because the transport endpoint often does not know the set of encaps/tunnel/link-MTU along the path.

Yes - except for the PLPMTUD case. In that case, the receiver COULD report back the largest MTU received in the “UDP path MSS”.

> That's why I argue we should have a "default" (as in "For IPv4, EMTU_S is the smaller of 576 bytes and the first-hop MTU [RFC1122]. For IPv6, EMTU_S is 1280 bytes [RFC2460]"), that could be used - and might be expected to usually work, *or* a "probe" to find out what the path actually supports: knowing the remote receiver can process X bytes does not really help that much, and actually could be unhelpful when information if it is not correct. So I'll argue against sharing this from the receiver.

If it’s not needed for PLPMTUD and has no corollary to how it’s used in TCP, then agreed.

Anyone else have a position??

Joe

---

From: Paul Vixie <[email-masked]>
Date: Sat, 3 Apr 2021 20:23:13 +0000

> https://tools.ietf.org/html/draft-fairhurst-tsvwg-udp-options-dplpmtud-04

as long as it's not an april fools' joke, and to the extent that it does
not accidentally preclude TCP from being treated as a packetization layer
(so that TCP MSS can also scale with PMTU rather than first-hop MTU), yes.

PLPMTUD is exactly what i wanted for DNS over UDP, and we refer to it here:

https://www.ietf.org/archive/id/draft-ietf-dnsop-avoid-fragmentation-04.txt

note, a lot of others expect DNS to move to HTTP/3 or TCP, but even in
those cases i would like to use the largest discrete PDUs that will fit.

-- 
Paul Vixie

---

From: Joseph Touch <[email-masked]>
Date: Sat, 3 Apr 2021 16:07:22 -0700

> On Apr 3, 2021, at 1:23 PM, Paul Vixie <[email-masked]> wrote:
> 
> https://www.ietf.org/archive/id/draft-ietf-dnsop-avoid-fragmentation-04.txt <https://www.ietf.org/archive/id/draft-ietf-dnsop-avoid-fragmentation-04.txt>
> 

You might take a closer look at UDP options:
https://tools.ietf.org/html/draft-ietf-tsvwg-udp-options-09

It supports UDP-layer fragmentation and reassembly with a 32-bit ID field.

Joe

---

From: Paul Vixie <[email-masked]>
Date: Sun, 4 Apr 2021 01:29:03 +0000

> > ... draft-ietf-dnsop-avoid-fragmentation-04.txt
> > 

> It supports UDP-layer fragmentation and reassembly with a 32-bit ID field.

you're implicitly positing a situation wherein a DNS speaker could know that
the far endpoint knew about UDP options and could reassemble, and that the
local endpoint (kernel) knows about UDP options and could fragment, and that
using UDP Fragmentation would be seen as a better choice than leaving out
optional data or signalling a need to retry with TCP.

in practice this means if a DNS UDP initiator asked its networking stack to
include a UDP option list (perhaps empty, but present), this signal could be
made available to the DNS UDP responder by its networking stack, so that the
capability of UDP fragmentation and reassembly could be considered when
deciding how to respond.

i worry about microbursts. the DNS UDP responder knows how much data it has
committed to the network recently, and how much was committed to a given
initiator, and can slow down a little to avoid back-to-back transmissions,
but certainly will slow down a little and reach such avoidance if the
initiator is not pipelining its requests.

what we've learned from NFS, and high volume authoriative UDP DNS, is that
the network doesn't love minimum-spaced back-to-back packets, and that if
an 8KiB NFS result gets chopped into ~1500B chunks, tail drop is likely.
this is the biggest source of operator pain from IP fragmentation, fwiw.

(bizarrely, this was not an issue in 10base5 or 10baseT due to CSMA/CD, even
where repeaters were present. but once we add bridging, such as "switches",
there's no reliable interface-driver signal for non-local congestion.)

candidly, i would not want to have to teach a kernel network stack how to
pace its transmitted fragments. this observation was one of my few
contributions to RFC6013, in which a TCB could revert to embryonic state
but retain its CWND until either reused or pruned. alas, TCPM felt they
had to make a choice, and chose TCPFO, which has since proved unworkable,
thus leading to QUIC.

further digression: the framer of messages (like TCP, or DNS, or NFS) ought
to know the PMTU, which is why PMTUD was originally a non-optional feature
of IPv6 until we learned that ICMPv6 was dangerous as hell and threw out
PMTUD, thus leaving us with the pessimal and never-expected-to-be-used 1280
and 1232 numbers. if we can get PLPMTUD then we can make IPv6 better than
IPv4 in terms of header amortization rather than (as it currently is) worse.

-- 
Paul Vixie

---

From: Joseph Touch <[email-masked]>
Date: Sat, 3 Apr 2021 18:49:46 -0700

Hi, Paul,

> On Apr 3, 2021, at 6:29 PM, Paul Vixie <[email-masked]> wrote:
> 

Yes. Note though that the fragmentation in UDP can be used safely; legacy endpoints just see (at most) packets with zero data.

> capability of UDP fragmentation and reassembly could be considered when

Yes.

> this is the biggest source of operator pain from IP fragmentation, fwiw.

Although I appreciate this concern, TCP does the same kind of bursts - which seems like DNS would exacerbate, either by sending up to 10-packet bursts at the start of every new connection or by sending potentially larger bursts if persistent connections are used.

I had thought we knew about this long enough that vendors didn’t use tail drop; they should have been doing AQM or at least something akin to RED.

> had to make a choice, and chose TCPFO, which has since proved unworkable,

A few of us at ISI explored ways to adjust TCP slow start restart to avoid this issue too:
https://tools.ietf.org/html/draft-hughes-restart-00 <https://tools.ietf.org/html/draft-hughes-restart-00>

> IPv4 in terms of header amortization rather than (as it currently is) worse.

Agreed; that’s aided in UDP with options as per Gorry’s draft.

Joe

---

From: Paul Vixie <[email-masked]>
Date: Sun, 4 Apr 2021 03:31:28 +0000

On Sat, Apr 03, 2021 at 06:49:46PM -0700, Joseph Touch wrote:
> > you're implicitly positing a situation wherein a DNS speaker could know
> > that the far endpoint knew about UDP options and could reassemble, and
> > that the local endpoint (kernel) knows about UDP options and could
> > fragment, and that using UDP Fragmentation would be seen as a better
> > choice than leaving out optional data or signalling a need to retry
> > with TCP.
> 
> Yes. Note though that the fragmentation in UDP can be used safely;
> legacy endpoints just see (at most) packets with zero data.

i wasn't considering it unsafe. some kind of initiation signal would
be required though, or else the initiator would see a small set of
apparently empty UDP payloads coming back, and wonder why.

> > i worry about microbursts. ...
> > 
> > what we've learned from NFS, and high volume authoriative UDP DNS,
> > is that the network doesn't love minimum-spaced back-to-back packets,
> > and that if an 8KiB NFS result gets chopped into ~1500B chunks, tail
> > drop is likely. this is the biggest source of operator pain from IP
> > fragmentation, fwiw.
> 
> Although I appreciate this concern, TCP does the same kind of bursts --

TCP has a congestion window that commonly keeps burst size within "range".

> I had thought we knew about this long enough that vendors didn???t use
> tail drop; they should have been doing AQM or at least something akin
> to RED.

in routers and endpoints, yes. in switches, no. to a switch (multiport
bridge), the problem is felt too late and too near to the copper. in a fan-in
topology there can be too many gozinta for the gozouta, and this doesn't even
depend on link-layer flow control or whether it works, just ten gallons of
water trying to fit into a five gallon hat.

by the time a switch knows this is happening, the ethernet chip has already
DMA'd the contents of the queue. the driver's only recourse may be to reset
the chip to empty the queue, then re-enter the things not-wanted-lost, which
may already have been garbage collected because their refcount went to zero
after the last DMA. so, the things that aren't sent are those which come
later.

but it doesn't matter whether it's tail drop or RED, what's important is, the
loss won't be noticed until timeout and transmission. in NFS, most of us just
moved over to TCP, but those still using UDP know to set an explicit
rsize/wsize that we know is comfortably smaller than the PMTU. (if you
wondered why modern switches allow 9100B "jumbograms", it's likely NFS.)

> https://tools.ietf.org/html/draft-hughes-restart-00

good times.

> > further digression: the framer of messages (like TCP, or DNS, or NFS)
> > ought to know the PMTU, which is why PMTUD was originally a non-optional
> > feature of IPv6 until we learned that ICMPv6 was dangerous as hell and
> > threw out PMTUD, thus leaving us with the pessimal and never-expected-
> > to-be-used 1280 and 1232 numbers. if we can get PLPMTUD then we can make
> > IPv6 better than IPv4 in terms of header amortization rather than (as it
> > currently is) worse.
> 
> Agreed; that???s aided in UDP with options as per Gorry???s draft.

if we implement PLPMTUD/UDP for DNS, we're going to have a decades-long
period during which the far end doesn't understand UDP options, and so
we will have to do fancy guesswork based on transaction timeouts, just as
we did for EDNS in the first 19 years. i hope these time scales are clear.

the original IPng PMTUD model whereby the endpoint's routing table would
remember a discovered MTU for each endpoint was a shinier city than this
on a better hill. i hope to see this outcome in the PLPMTUD world, so that
(for example) TCP, QUIC, NFS, and UDP can all set their MSS accordingly,
without each service having to do its own discovery work per endpoint.

-- 
Paul Vixie

---

From: Joseph Touch <[email-masked]>
Date: Sat, 3 Apr 2021 21:25:00 -0700

> On Apr 3, 2021, at 8:31 PM, Paul Vixie <[email-masked]> wrote:
> 

The same could happen for nearly any new variant of DNS, too.

> TCP has a congestion window that commonly keeps burst size within "range”.

The window isn’t based on the ability to burst an entire window’s worth of messages back-to-back.

It represents an entire round trip's worth of transmissions; when the source goes idle, the next active period can burst up to that entire window.

> water trying to fit into a five gallon hat.

If you’re speaking of Ethernet, there is link layer flow control, e.g., congestion notification messages or explicit on/off flow control intended to push this feedback to routers and endpoints. 

> ….
> period during which the far end doesn't understand UDP options,

s/UDP options/{TCP, QUIC, etc.}/

Yes, all new mechanisms take time. 

> ...
> on a better hill. i hope to see this outcome in the PLPMTUD world,

PLPMTUD intends to cache MTUs per-endpoint exactly the same way that PMTUD did.

> so that

Sure, but someone has to figure it out for others to use it. That’s why we’re working on a mechanism native to UDP - so it doesn’t rely on TCP or QUIC connections that precede its use.

Joe

---

From: Paul Vixie <[email-masked]>
Date: Sun, 4 Apr 2021 04:57:07 +0000

> The same could happen for nearly any new variant of DNS, too.

well, yes, which is why RFC 2671 says:

+---
| 1.2. Existing clients will not know how to interpret the protocol
|      extensions detailed here.  In practice, these clients will be
|      upgraded when they have need of a new feature, and only new
|      features will make use of the extensions.  We must however take
|      account of client behaviour in the face of extra fields, and design
|      a fallback scheme for interoperability with these clients.
+---

> > TCP has a congestion window that commonly keeps burst size within range.
> 
> The window isn???t based on the ability to burst an entire window???s worth
> of messages back-to-back.

of course not. but it will be clamped by that limit if the limit is present.

> If you???re speaking of Ethernet, there is link layer flow control, e.g.,
> congestion notification messages or explicit on/off flow control intended
> to push this feedback to routers and endpoints.

that's what the spec says, sure. in practice, congestion is often a silent
killer.

> Yes, all new mechanisms take time. 

by "Gorry's draft" i took you to mean this one:

draft-fairhurst-tsvwg-udp-options-dplpmtud-04

which talks about "UDP Options" and i was talking about "PLPLTUD/UDP for DNS"
so please interpret my over-specificity (/UDP options/) as well-intentioned.

> PLPMTUD intends to cache MTUs per-endpoint exactly the same way

that's interesting! so, a DNS protocol speaker (framer of messages operating
at the packetization layer) who uses DNS/UDP messages of various sizes in
order to discover the PMTU, because the kernel routing table had no hint of
the MTU for the endpoint under consideration, will be expected to report its
finding to the kernel, which can store it in the routing table, where TCP,
QUIC, NFS and so on can find it?

i think this is a powerful idea and i'm in favour. but it's the kind of idea
that usually i like and others don't. so i'll be surprised if you don't have
to correct a misimpression here.

-- 
Paul Vixie

---

From: Joseph Touch <[email-masked]>
Date: Sun, 4 Apr 2021 09:07:13 -0700

Hi, Paul,

> On Apr 3, 2021, at 9:57 PM, Paul Vixie <[email-masked]> wrote:
> 

From RFC4821:

   ...Furthermore, the natural mechanism to
   share Path MTU information between concurrent or subsequent
   connections is a path information cache in the IP layer.  The various
   Packetization Protocols need to have the means to access and update
   the shared cache in the IP layer.  

Joe

---

From: Jeremy Harris <[email-masked]>
Date: Mon, 5 Apr 2021 21:11:25 +0100

On 04/04/2021 02:29, Paul Vixie wrote:
> TCPM [...]  chose TCPFO, which has since proved unworkable

Do you have some refs for the unworkable part?
-- 
Thanks,
   Jeremy

---

From: Paul Vixie <[email-masked]>
Date: Wed, 7 Apr 2021 00:11:35 +0000

> Do you have some refs for the unworkable part?

well, i was thinking of this article when i penned the above:

https://squeeze.isobar.com/2019/04/11/the-sad-story-of-tcp-fast-open/

however, a web search turned up a +1 that i hadn't previously known of:

https://bridge.grumpy-troll.org/2013/12/tcp-fastopen-security/

noting, if i'd known google was also working on this, i'd've stayed home:

https://www.usenix.org/system/files/login/articles/126-metzger.pdf

(also noting, RFC 6013 was designed to fail fast through middleboxes.)

-- 
Paul Vixie