Subject: Re: [Int-area] IP parcels

From: Christian Huitema <[email-masked]>
Date: Thu, 27 Jan 2022 09:23:45 -1000

On 12/20/2021 10:03 AM, Templin (US), Fred L wrote:

> Tom, sorry I will try to use my words more carefully; I am using GSO/GRO also for
> a UDP-based transport protocol – not QUIC but something similar. I like GSO/GRO
> very much; I am glad the service is available and I want to see it continue. But, my
> understanding of the services is that they leverage the IP ID field in whole IPv4
> packets that are not eligible for fragmentation and those are limitations I am
> seeking to improve on.

Lots of QUIC implementation use UDP GSO/GRO. They do that to achieve 
good performance. Without GSO, we have observed that almost 2/3rd of the 
CPU time was spent in socket calls, mostly sendmsg on servers. Using GSO 
to send batches of packets results in big gains. For example, the same 
single-core implementation that achieves 500 Mbps without GSO gets to 
4-5 Gbps while sending batches of packets of about 64K using GSO.

Note however that this is *UDP* GSO. The application prepares a batch of 
UDP packets, all sharing the same source and destination address and 
ports, and all but the last having the same length. The GSO call 
provides these header parameters, the common payload length, and a set 
of UDP payloads, concatenated in a large buffer. Each payload is a fully 
formed QUIC packet, encrypted and sealed using AEAD. There is no 
expectation that these packets would be somehow re-segmented or 
re-assembled in the network -- if the network did that, decryption would 
fail. There is also no expectation that the batch of packets will be 
delivered as a single batch to the receiver. It might happen by chance, 
but mostly it does not because GRO and GSO are not synchronized. On the 
other hand, the handling of the small packets by the QUIC stack is 
rarely considered a problem.

> I want to enable a facility similar to GSO/GRO that works for both IPv4 and IPv6
> packets and allows for lower layers to fragment if necessary. And, I want to use
> a well-behaved 32-bit IPv6 ID instead of the 16-bit IPv4 one where the use is not
> well defined when DF=1.

Consider encryption. QUIC certainly expects that packets sent will be 
received unchanged, or they would fail decryption. Also consider packet 
loss. In the GSO example, the sender might send a batch of 42 packets. 
If any of them is lost, it will be resent individually. QUIC could of 
course send a 64KB packet if the network allowed. But if a 64KB parcel 
is segmented in 42 segments, any packet loss will cause the loss of the 
whole parcel, and that would cause a significant drop in performance 
compared to GSO. Which is indeed the core of the "segmentation 
considered harmful" argument.

-- Christian Huitema

---

From: Behcet Sarikaya <[email-masked]>
Date: Thu, 27 Jan 2022 15:29:01 -0600

Hi Folks,

Thanks Christian for explaining how GSO/GRO are used by Quic
implementations. So the use is not mandated in Quic RFCs but rather used in
implementations.

I found this presentation by Intel:
https://www.dpdk.org/wp-content/uploads/sites/35/2018/06/GRO-GSO-Libraries-Bring-Significant-Performance-Gains-to-DPDK-based-Applications.pdf
way back in 2017 quite useful.
So what I understand is GSO/GRO were originally used by TCP/UDP
and Quic implementations use UDP GSO/GRO because of Quic is UDP based.

Behcet

On Thu, Jan 27, 2022 at 1:24 PM Christian Huitema <[email-masked]>
wrote:

> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area
>

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 27 Jan 2022 14:16:54 -0800

FWIW, GRO/GSO give no end of headaches to the idea of new TCP options, esp. the current ones to extend option space after the SYN (draft-ietf-tcpm-tcp-edo).

Although I appreciate their zeal for optimization, implementers of GRO/GSO still need to play by the rules of TCP and UDP. It’s not clear they are (e.g., coalescing packets with different unknown options), and when they don’t, I want to be clear that it is they that are the problem.

I agree that the common Unix socket interface has performance flaws, but perhaps it’s that interface that needs to evolve.

Joe

—
Joe Touch, temporal epistemologist
www.strayalpha.com

> On Jan 27, 2022, at 1:29 PM, Behcet Sarikaya <[email-masked]> wrote:
> 
> Hi Folks,
> 
> Thanks Christian for explaining how GSO/GRO are used by Quic implementations. So the use is not mandated in Quic RFCs but rather used in implementations.

---

From: Tom Herbert <[email-masked]>
Date: Thu, 27 Jan 2022 14:46:31 -0800

> FWIW, GRO/GSO give no end of headaches to the idea of new TCP options, esp. the current ones to extend option space after the SYN (draft-ietf-tcpm-tcp-edo).

GRO and GSO are software implementations and in most deployments
> Although I appreciate their zeal for optimization, implementers of GRO/GSO still need to play by the rules of TCP and UDP. It’s not clear they are (e.g., coalescing packets with different unknown o…
Joe,

GRO and GSO are software implementations in kernel networking stacks
and in most cases these are open source projects of Linux or FreeBSD.
If they have flaws or there's areas for improvement, then by all means
submit patches to the respective project-- that, after all, is the
whole premise of an open source project. The hardware cognates, TSO
and LRO, do tend to be more closed and for this reason they haven't
gotten much traction-- TSO has seen a some amount of deployment, but
LRO hasn't because of the problems of putting fairly complex
algorithms in hardware. That problem is addressed once we have
sufficiently programmable hardware so the stack can program things
like GSO and GRO in hardware as easily in hardware and of course this
should be done in ubiquitous open source that works across all
hardware vendors.

Tom

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 27 Jan 2022 15:43:47 -0800

Hi, Tom,

> On Jan 27, 2022, at 2:46 PM, Tom Herbert <[email-masked]> wrote:
> 

Ubiquitously deployed Linux kernel software at one point had a bug that failed to lock the inode structure during modification. Uniquity didn’t make that magically correct.

> whole premise of an open source project.

That’s not how open source works. 

The onus is on those who currently maintain the code to ensure it complies with current protocols. I noted that I and others have experience that it doesn’t.

It is not the responsibility of the user community to fix their bugs or ensure that their approach remains viable.

> The hardware cognates, TSO

None of that has anything to do with the issue I raised.

Both hardware and software implementations of these optimizations MUST strictly comply with protocol specs. When they encounter options they don’t know, it’s not their prerogative to do anything beyond “disable” for that stream. That’s not our experience, though.

Joe

---

From: Tom Herbert <[email-masked]>
Date: Thu, 27 Jan 2022 16:06:53 -0800

> It is not the responsibility of the user community to fix their bugs or ensure that their approach remains viable.
Actually it is. We have users fixing bugs all the time on Linux.
There's no entry fee or membership, and anyone is free to submit
patches. It is similar to IETF where anyone can submit I-Ds. Of course
in both cases, acceptance often comes down to how much effort and
perseverance the submitter is willing to put into it.

Tom