Subject: [iccrg] Off loading "for dummies"

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 7 May 2018 06:30:49 +0000

Hi

Is there any reasonably good overview paper on different offloading techniques like GRO available somewhere ?

Regards
Ingemar
==================================
Ingemar Johansson  M.Sc.
Master Researcher

Ericsson Research
Network Protocols & E2E Performance
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked]<mailto:[email-masked]>
www.ericsson.com

The course gets set you got to see it through
             That's why you were born
                    David Baerwald
==================================

---

From: Sowmini Varadhan <[email-masked]>
Date: Mon, 7 May 2018 05:51:01 -0400

On (05/07/18 06:30), Ingemar Johansson S wrote:
> 
>    Is there any reasonably good overview paper on different offloading
>    techniques like GRO available somewhere ?

What level of detail are you looking for? Are you looking
for a description of generic concepts, or something stack specific?

There are various types of offloads: checksome offload, TSO/GSO,
GRO, IPsec offloads, filtering offloads, so your it would help
if you asked for something specific.

GRO on linux: https://lwn.net/Articles/358910/

--Sowmini

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 7 May 2018 12:00:57 +0200 (CEST)

On Mon, 7 May 2018, Sowmini Varadhan wrote:

> There are various types of offloads: checksome offload, TSO/GSO, GRO, 
> IPsec offloads, filtering offloads, so your it would help if you asked 
> for something specific.

It would probably help everybody if this "for dummies" document started by 
exactly giving an overview of what's available. It's hard to ask specifics 
if one doesn't know the generics, and in the "for dummies" part of the 
question, I'd say it's just that, somewhere to start. Overview document.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Sowmini Varadhan <[email-masked]>
Date: Mon, 7 May 2018 06:32:46 -0400

> question, I'd say it's just that, somewhere to start. Overview document.

As I mentioned before, you can offload anything you want (including
the entire network stack!), and usually CPU intensive operations (like
checksum calculation, crypto), or well-defined transforms like TCP
segmentation (where TCP state machine is still managed in the software
stack), are offloaded. The tradeoffs are usually speed vs algorithm-agility
(e.g. see the wiki page for TOE)

So a lot depends on the answer to the question "what problem are you
trying to solve?"

--Sowmini

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 7 May 2018 10:40:48 +0000

Hi

Thanks for the response so far. 
I am in particular interested in some insight as regards to how high the rate of ACK packets becomes in present network stacks, schoolbook says 1 ACK per two segments but as I understand things, this may not always apply. GRO plays a big role here I guess, one specific question is for instance if incoming MTU sized packets are collected over a (short) time interval before being concatenated and forwarded up to the CPU.
I understand that behavior may differ a lot depending on link speed and device. Is there for instance some GRO like behavior in Android and IPhones  ?

/Ingemar

> -----Original Message-----
> From: Sowmini Varadhan [mailto:[email-masked]]
> Sent: den 7 maj 2018 12:33
> To: Mikael Abrahamsson <[email-masked]>
> Cc: Ingemar Johansson S <[email-masked]>;
> [email-masked]' <[email-masked]>
> Subject: Re: [iccrg] Off loading "for dummies"
> 
> > It would probably help everybody if this "for dummies" document
> > started by exactly giving an overview of what's available. It's hard
> > to ask specifics if one doesn't know the generics, and in the "for
> > dummies" part of the question, I'd say it's just that, somewhere to start.
> Overview document.
> 
> As I mentioned before, you can offload anything you want (including the
> entire network stack!), and usually CPU intensive operations (like checksum
> calculation, crypto), or well-defined transforms like TCP segmentation
> (where TCP state machine is still managed in the software stack), are
> offloaded. The tradeoffs are usually speed vs algorithm-agility (e.g. see the
> wiki page for TOE)
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 7 May 2018 12:49:24 +0200 (CEST)

On Mon, 7 May 2018, Ingemar Johansson S wrote:

> I am in particular interested in some insight as regards to how high the 
> rate of ACK packets becomes in present network stacks, schoolbook says 1 
> ACK per two segments but as I understand things, this may not always 
> apply. GRO plays a big role here I guess, one specific question is for 
> instance if incoming MTU sized packets are collected over a (short) time 
> interval before being concatenated and forwarded up to the CPU.

One other observation: Some network types also do ACK suppression, so this 
isn't just host behaviour:

https://www.ietf.org/mail-archive/web/aqm/current/msg01482.html

---

From: Praveen Balasubramanian <[email-masked]>
Date: Mon, 7 May 2018 16:11:53 +0000

It's not just LRO/GRO, stretch ACKs are common and you cannot assume one ACK per two segments. I hope that since hosts are already reducing ACK burden, networks stop doing any more suppression on top. Most deployed congestion control is ACK clocked and too much suppression can affect perf. Also, dup ACKs should not be suppressed as otherwise loss recovery could be impaired. 

-----Original Message-----
From: iccrg [mailto:[email-masked]] On Behalf Of Mikael Abrahamsson
Sent: Monday, May 7, 2018 3:49 AM
To: Ingemar Johansson S <[email-masked]>
Cc: Sowmini Varadhan <[email-masked]>; [email-masked]' <[email-masked]>
Subject: Re: [iccrg] Off loading "for dummies"

On Mon, 7 May 2018, Ingemar Johansson S wrote:

> I am in particular interested in some insight as regards to how high 
> the rate of ACK packets becomes in present network stacks, schoolbook 
> says 1 ACK per two segments but as I understand things, this may not 
> always apply. GRO plays a big role here I guess, one specific question 
> is for instance if incoming MTU sized packets are collected over a 
> (short) time interval before being concatenated and forwarded up to the CPU.

One other observation: Some network types also do ACK suppression, so this isn't just host behaviour:

https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmail-archive%2Fweb%2Faqm%2Fcurrent%2Fmsg01482.html&data=02%7C01%7Cpravb%40microsoft.com%7Cb3f5f16cf5e94b63733c08d5b4083884%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636612869825428610&sdata=PfANJKdqYboHY2ayIIp4E5%2F2CxSLOCuJoaIkwOEd%2BrY%3D&reserved=0

_______________________________________________
iccrg mailing list
[email-masked]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.irtf.org%2Fmailman%2Flistinfo%2Ficcrg&data=02%7C01%7Cpravb%40microsoft.com%7Cb3f5f16cf5e94b63733c08d5b4083884%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636612869825428610&sdata=%2FiQnekW0ni%2F25xiW7qvMyi078liD0gYV3Nz%2FS%2BO25nA%3D&reserved=0

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 7 May 2018 18:45:14 +0200 (CEST)

On Mon, 7 May 2018, Praveen Balasubramanian wrote:

> congestion control is ACK clocked and too much suppression can affect

In for instance DOCSIS, ACK suppression seems to be done within the same 
"transmit window". So the whole "clocked" approach kind of goes out the 
window anyway, because you'll get all packets in bursts as the network 
gives the DOCSIS modem a transmit opportunity and then all buffered the 
ACKs are sent in one quick burst. This kind of "micro buffering" in the 
few ms quantum is quite commonly used for time divisioned networks. I 
imagine one would see similar behaviour in 3GPP networks if one were to do 
that kind of analysis. I am not aware of network level ACK suppression 
there though.

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 8 May 2018 06:30:16 +0000

Hi Mikael, Sowmini, Praveen.

And thanks for the feedback. 
Yes, it is true that you'll experience similar "microbuffering" properties with LTE. When a UE (User equipment) is scheduled for transmission, the allocated transport block can fit one or more IP packets, depending on radio bandwidth, network load and/or channel  quality, and also how much data that is present in the transmission buffer. This essentially means that you can get several TCP segments in on go in the network interface. How the cellphones handle this (one or several ACKs?) is however largely unknown to me.
And you can have the same buffering effects on the ACKs that go in the reverse path.
Some more details in https://datatracker.ietf.org/meeting/96/materials/slides-96-iccrg-1 

Regards
/Ingemar

> From: Mikael Abrahamsson [mailto:[email-masked]]
> Sent: den 7 maj 2018 18:45
> To: Praveen Balasubramanian <[email-masked]>
> Cc: Ingemar Johansson S <[email-masked]>; Sowmini
> Varadhan <[email-masked]>; [email-masked]' <[email-masked]>
> window anyway, because you'll get all packets in bursts as the network gives
> the DOCSIS modem a transmit opportunity and then all buffered the ACKs
> are sent in one quick burst. This kind of "micro buffering" in the few ms
> quantum is quite commonly used for time divisioned networks. I imagine one
> would see similar behaviour in 3GPP networks if one were to do that kind of
> analysis. I am not aware of network level ACK suppression there though.
> 
> --