Subject: Re: [Int-area] Side meeting follow-up: What exact features do we want from the Internet?

From: "Templin (US), Fred L" <[email-masked]>
Date: Sat, 4 Dec 2021 01:22:22 +0000

Joe, go read these and tell me what you think is missing because I assure you that
nothing is missing:

https://datatracker.ietf.org/doc/draft-templin-6man-omni/
https://datatracker.ietf.org/doc/draft-templin-dtn-ltpfrag/

And please quit sending the funky html emails - they are corrupting the list.

---

From: "[email-masked]" <[email-masked]>
Date: Fri, 3 Dec 2021 19:21:19 -0800

Hi, Fred,

These both address a way to support *fragmentation*. I clearly said what we need is a way to stop fragmenting (it was a wish list after all).

Ethernet never fragments because it just keeps adding headers. Nothing in the docs below does that.

THAT, in a nutshell, is what’s missing, as I said *in my first post* on this thread.

Joe

—
Joe Touch, temporal epistemologist
www.strayalpha.com

> On Dec 3, 2021, at 5:22 PM, Templin (US), Fred L <[email-masked]> wrote:
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 3 Dec 2021 22:35:51 -0500

Joe, I am missing something in your wish.
Ethernet, when it defines new heders, also revises the definition of the 
actual frame format on the wire.  The IETF does not own the frame format 
on the wire.  We do not own the link MTU.

Unless we want to reinvent X.25 with linkwise fragmentation and 
reassembly at each step, and all the problems that implies, I do not see 
how the IETF could get out of dealing with link MTU variation.  (Doing 
this on a tunnel basis is one of the features I think Fred Templin is 
arguing for.)

Yours,
Joel

On 12/3/2021 10:21 PM, [email-masked] wrote:
> Hi, Fred,
> 
> These both address a way to support *fragmentation*. I clearly said what 
> we need is a way to stop fragmenting (it was a wish list after all).
> 
> www.strayalpha.com <http://www.strayalpha.com>
> 
>> On Dec 3, 2021, at 5:22 PM, Templin (US), Fred L 
>> <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> <https://datatracker.ietf.org/doc/draft-templin-6man-omni/>
> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area
>

---

From: "[email-masked]" <[email-masked]>
Date: Fri, 3 Dec 2021 19:47:11 -0800

I know that Ethernet gets to do the redefinition.

My point, which appears not to be tracking, is I *wish* protocol layers didn’t have such strict MTUs, but rather expanded as headers were added *at all layers*, in the same *spirit* as Ethernet does.

The question was “what features do we want”, not “are they feasible right now”.

What I want is expandable packets, specifically to get around the need for fragmentation. I know how to do the latter and the problems it keeps causing - even when we get it right, everyone seems to think the solution is a “hint” and they do what they want, which invariably breaks.

If we didn’t need fragmentation in the first place, that goes away.

Again, it’s a WISH list.

Joe

—

> On Dec 3, 2021, at 7:35 PM, Joel M. Halpern <[email-masked]> wrote:
> 
> Ethernet, when it defines new heders, also revises the definition of the actual frame format on the wire.  The IETF does not own the frame format on the wire.  We do not own the link MTU.
> 
> Unless we want to reinvent X.25 with linkwise fragmentation and reassembly at each step, and all the problems that implies, I do not see how the IETF could get out of dealing with link MTU variation.  (Doing this on a tunnel basis is one of the features I think Fred Templin is arguing for.)
> 
> Yours,
> Joel
> 
>>> On Dec 3, 2021, at 5:22 PM, Templin (US), Fred L <[email-masked] <mailto:[email-masked]>> wrote:
>>> 
>>> https://datatracker.ietf.org/doc/draft-templin-6man-omni/ <https://datatracker.ietf.org/doc/draft-templin-6man-omni/>

---

From: Dino Farinacci <[email-masked]>
Date: Fri, 3 Dec 2021 20:38:43 -0800

> My point, which appears not to be tracking, is I *wish* protocol layers didn’t have such strict MTUs, but rather expanded as headers were added *at all layers*, in the same *spirit* as Ethernet doe…

The Internet can do this. Just make the MTU 1400. Then you can add up to 100 bytes of header. 

I have the opinion we don’t have an MTU problem. 

Dino

---

From: Luigi Iannone <[email-masked]>
Date: Mon, 6 Dec 2021 11:28:40 +0100

Hello,

I find the MTU discussion interesting. It pops up from time to time in different mailing list, a clear sign that we lack a solution and the discussion here show that there are different ideas of how the solution should look like.

Having said that, this is not caused by addressing itself, right? 
Certainly  large addresses eat a lot of that MTU space.  

I wonder if we are able to describe this as a possible way to add features. Assuming we are able somehow to get rid of the MTU issue, it seems we gain a degree off freedom, how this translates specifically for the addressing?

Ciao

L.

> On 4 Dec 2021, at 05:38, Dino Farinacci <[email-masked]> wrote:
> 
>

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 6 Dec 2021 15:48:28 -0600

That is why if you want robustness of connectivity with shortest paths, you need a static MTU of 1400 and an IPv4 underlay. 

Dino

> On Dec 6, 2021, at 4:28 AM, Luigi Iannone <[email-masked]> wrote:
> 
> Hello,
>

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 6 Dec 2021 15:52:26 -0600

Last email was the main point I wants to get across. Now to answer your questions inline. 

> Having said that, this is not caused by addressing itself, right? 

Right, IMHO. 

> Certainly  large addresses eat a lot of that MTU space.  

Well true as well as overlay encapsulation. 

> I wonder if we are able to describe this as a possible way to add features. Assuming we are able somehow to get rid of the MTU issue, it seems we gain a degree off freedom, how this translates spec…

I think the question is Luigi is, does the user notice an improvement in network performance by sending larger packets?

If so then the MTU issues we see as geeks at the network layer should be fixed. But note it’s hard (read: really hard if not impossible) to upgrade the underlay. 

Dino

---

From: Tom Herbert <[email-masked]>
Date: Mon, 6 Dec 2021 15:06:20 -0800

> I think the question is Luigi is, does the user notice an improvement in network performance by sending larger packets?

Dino,

Definitely at least for a limited domain. For instance, AFAIK Google
is using 9K MTUs in their internal networks. Whether the benefits of a
larger MTU scales to the whole Internet is probably still an open
question, however QUIC seems to require at least an MTU of 1280 bytes
so there are some attempts to enforce a baseline MTU for the Internet
greater than the specified minimums (at least greater than 64 bytes or
576 bytes for IPv4 MTU minimums.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 7 Dec 2021 13:29:17 +1300

On 07-Dec-21 12:06, Tom Herbert wrote:
>>> I wonder if we are able to describe this as a possible way to add features. Assuming we are able somehow to get rid of the MTU issue, it seems 
we gain a degree off freedom, how this translates specifically for the addressing?
> 576 bytes for IPv4 MTU minimums.

Google is allegedly using more than 9000 for performance reasons.
Potentially, much more:
https://netdevconf.info/0x15/session.html?BIG-TCP

RFC2675 rocks.

    Brian

---

From: Dino Farinacci <[email-masked]>
Date: Mon, 6 Dec 2021 20:17:55 -0600

> Dino,

Hey Tom. I should make it clear that I am replying to email in the context of "user requirements", that means end user requirements. Hence my comment about 1400.

> is using 9K MTUs in their internal networks. Whether the benefits of a

I am well aware of this but this is server to server and isn't related to end-user requirements. And note, internal networks don't have an MTU problem because they can force 9K MTUs.

> larger MTU scales to the whole Internet is probably still an open

Right.

> question, however QUIC seems to require at least an MTU of 1280 bytes

I have tested QUIC over LISP and when interface MTUs are 1400, you can send 1280-byte packets over IPv4 (1280+8+8+20)=1316 and over IPv6 (1280+8+8+40)=1336 which keeps the packets <= 1400. So any protocol above IP could send more than 1280 (120 bytes more to reach 1400-byte MTUs) with plenty of room for tunnels (even nested tunnels to 2 levels).

> 576 bytes for IPv4 MTU minimums.

Yes, and I vote for 1400. Just because I have tested it in many cases over many years on a LISP overlay. In cases for data center MTU, I would suggest 8900 (9000-100) and be consistent saying we save 100 bytes of packet head-room for overlays.

> Tom

Thanks,
Dino

---

From: "[email-masked]" <[email-masked]>
Date: Mon, 6 Dec 2021 21:18:13 -0800

> 
> On Dec 3, 2021, at 8:38 PM, Dino Farinacci <[email-masked]> wrote:
> 
> 

That may help, but only in limited cases.

E.g., let’s say you run IPsec tunnel mode for IPv6, which eats the majority of that space. Now that traffic runs over a second IPsec tunnel that you don’t know about.

That’s the problem - and why MTU (i.e., having a max in the first place) is itself the problem.

Joe

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 7 Dec 2021 09:36:23 -0600

> That’s the problem - and why MTU (i.e., having a max in the first place) is itself the problem.

Yep, definitely true. But one should use IPsec tunnel mode sparingly. It just costs too much. And having 2 levels would be regarded as execessive. Having said that, products may do this because security trumps all.

But you make another point which is pretty fundamental and foundational. Should data links be MTU-less, so to speak? And can they really do that. I won't hold my breath.

Dino

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 7 Dec 2021 08:30:20 -0800

> 
> On Dec 7, 2021, at 7:36 AM, Dino Farinacci <[email-masked]> wrote:
> 
> Yep, definitely true. But one should use IPsec tunnel mode sparingly.

Users don’t get to decide that.

> It just costs too much. And having 2 levels would be regarded as execessive.

They also just use tunnels that have similar impact. Multiple levels are common - because no single operator manages the entire E2E path.

> Having said that, products may do this because security trumps all.
> 

I don’t know yet, but I do know that’s what I *want* and why it’s different than simply assuming a smaller MTU anywhere in the system.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Tue, 7 Dec 2021 08:42:19 -0800

> Yes, and I vote for 1400. Just because I have tested it in many cases over many years on a LISP overlay. In cases for data center MTU, I would suggest 8900 (9000-100) and be consistent saying we sa…

QUIC has already set the de facto standard for minimum MTU on the
Internet as well as the minimum reserved space in the MTU for
extension headers or other overhead. From RFC9000:

QUIC assumes a minimum IP packet size of at least 1280 bytes.  This is
the IPv6 minimum size [IPv6] and is also supported by most modern IPv4
networks.

-and-

Note: This requirement to support a UDP payload of 1200 bytes limits
the space available for IPv6 extension headers to 32 bytes or IPv4
options to 52 bytes if the path only supports the IPv6 minimum MTU of
1280 bytes.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 8 Dec 2021 09:15:02 +1300

On 08-Dec-21 05:30, [email-masked] wrote:
...
> I don’t know yet, but I do know that’s what I *want* and why it’s different than simply assuming a smaller MTU anywhere in the system.

I think you'll need to discuss this point with the inventors of packet switching and queueing theory. Since the Internet is still basically a great big statistical multiplexer, getting rid of MTU at every layer seems like an impossibility.

    Brian

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 7 Dec 2021 12:48:03 -0800

> I think you'll need to discuss this point with the inventors of packet switching and queueing theory. Since the Internet is still basically a great big statistical multiplexer, getting rid of MTU a…

Perhaps, or maybe MTUs are an artifact of limited memory and BW.

I’m not suggesting infinite packets - though that’s another interesting exercise in protocol design (what if packets were infinite, what if protocols used only one packet, etc.). I’m suggesting an environment in which applications always assume some “MTU” and the headers are just “eaten” as overhead throughout the rest of the system, sort of the way that Ethernet has a 1500 fixed payload but no strict packet length limit (arguably, you can add Q-tags or MAC-in-MAC all day long).

Joe

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 7 Dec 2021 17:42:40 -0600

> I don’t know yet, but I do know that’s what I *want* and why it’s different than simply assuming a smaller MTU anywhere in the system.

Its the only thing that works over a hetrogeneous network of a myriad of products.

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 7 Dec 2021 17:46:00 -0600

Right, I understand this. I said *it could8 use larger headers. So sounds like we are in agremement. That is, if we can't agree on 1400 because protocols have been spec'ed for smaller ones, that is where we are headed. In the opposite direction we wantn to go.

Hence, there is no way to technically fix this. And fragmentation is *not the solution*. It will cause more packet loss and time-out buffers in receivers.

But what apps can do, and I know many that do is to send a packet train of MTU sized packets with FEC. So when there is loss, the receiver can build the packets via erasure codes.

Dino

> On Dec 7, 2021, at 10:42 AM, Tom Herbert <[email-masked]> wrote:
>

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 7 Dec 2021 17:47:48 -0600

And note if you get rid of data link MTUs, your head-of-line-blocking issue gets worse. Also note 1280 is not 53, and hence we have an international large scale network running, unlike ATM.

Dino

> On Dec 7, 2021, at 2:48 PM, [email-masked] wrote:
>

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 7 Dec 2021 16:54:06 -0800

I think we’re generally in agreement.

My view is that fragmentation is currently a necessary evil. Evil that should be avoided where possible, but necessary that MUST be supported.

Joe

—

> On Dec 7, 2021, at 3:46 PM, Dino Farinacci <[email-masked]> wrote:
>

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 7 Dec 2021 18:56:46 -0600

Since you can't FEC IP fragments, the apps have to do it. And since the apps do it, they fragment on IP MTU boundaries.

Dino

> On Dec 7, 2021, at 6:54 PM, [email-masked] wrote:
>

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 7 Dec 2021 16:56:46 -0800

> And note if you get rid of data link MTUs, your head-of-line-blocking issue gets worse. Also note 1280 is not 53, and hence we have an international large scale network running, unlike ATM.

It’s more like I want to get rid of the reduction of user payload space to make room for headers.

I don’t think link MTUs should be unlimited per se, but rather that we COULD treat the innermost user payload as the thing we limit, where everything else is “overhead” that makes the segment longer, rather than the payload shorter.

Schedulers would take that into account, e.g., scheduling to be fair “per innermost payload”.

That gets around the HOL blocking but doesn’t restrict us so badly as we currently are - again, this is basically how Ethernet *already* handles things. We COULD allow that to propagate up the stack, if we wanted to.

Joe

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 7 Dec 2021 16:58:44 -0800

Apps could (and would still need to) send individual payloads - but not have those payloads decreased based on how many headers are used. Again, like Ethernet.

Think of it this way: Ethernet user payload is 1500B, always. There’s no such thing as an Ethernet-layer MTU because headers can vary.

Joe

—

> On Dec 7, 2021, at 4:56 PM, Dino Farinacci <[email-masked]> wrote:
>