Subject: [iccrg] Disadvantages of TCP connection splitters

From: Michael Welzl <[email-masked]>
Date: Fri, 10 Jan 2020 09:54:07 +0100

Hi,

I’ve been thinking a lot about TCP connection splitters lately ( https://tools.ietf.org/html/rfc3135#section-2.4 ).

I’m curious: what are the real practical disadvantages of this type of PEPs that people have seen?
I'll appreciate any kind of feedback, also anecdotes, but pointers to citable papers would be best.

BTW, let’s keep multi-path apart from this discussion please. My question is about single path TCP.

Cheers,
Michael

PS: I’m not trying to indirectly hint that such devices would be *always good*. However, the scenarios where they are not strike me as surprisingly narrow, so I wonder if I’m missing more.

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 10 Jan 2020 11:14:57 +0100

Main downside is that its limited to TCP, and i think it might
be easier nowadays to implement split/merge generically for
5 tuple flows and not for TCP alone, and the customer gets
the benefits for TCP, DCTCP, SCTP, QUIC, RTP and any other flow.

In most situations you can't simply split the packets without
encap or NAT anyhow because the alternative paths may likely
require different source or dest addresses (dual SP attachment
via broadband/DSL or the like). Once you do have such encap
its typically easier to run your own congestion/retransmission
between merge/split instead of just trying to passivle
observe and analyze the end-to-end TCP signals and figure
out how to deduce the appropriate load-split from them.

Toerless

On Fri, Jan 10, 2020 at 09:54:07AM +0100, Michael Welzl wrote:
> Hi,
> 
> I???ve been thinking a lot about TCP connection splitters lately ( https://tools.ietf.org/html/rfc3135#section-2.4 ).
> 
> I???m curious: what are the real practical disadvantages of this type of PEPs that people have seen?
> BTW, let???s keep multi-path apart from this discussion please. My question is about single path TCP.
> 
> Cheers,
> Michael
> 
> PS: I???m not trying to indirectly hint that such devices would be *always good*. However, the scenarios where they are not strike me as surprisingly narrow, so I wonder if I???m missing more.
> 
> _______________________________________________
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: Michael Welzl <[email-masked]>
Date: Fri, 10 Jan 2020 11:21:28 +0100

> On Jan 10, 2020, at 11:14 AM, Toerless Eckert <[email-masked]> wrote:
> 

Sure, you’re obviously right, but that’s also potentially solvable with app-level proxying (see draft-kuehlewind-quic-substrate, for example) - I’m more interested in the general congestion control & transport aspects of this.

> out how to deduce the appropriate load-split from them.

“run your own congestion/retransmission” -  yes, the added flexibility is a big plus here, that’s the plus that I’m trying to weigh against a minus…

Cheers,
Michael

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 10 Jan 2020 11:52:53 +0100

On Fri, Jan 10, 2020 at 11:21:28AM +0100, Michael Welzl wrote:
> Sure, you???re obviously right, but that???s also potentially solvable with app-level proxying (see draft-kuehlewind-quic-substrate, for example) - I???m more interested in the general congestion control & transport aspects of this.

Todays transport protocols expect tuple transparent end-to-end
connection, you can not cleanly design solutions with application level
proxies, so we're always talking about network/transport layer proxies
or encaps.  At least when you are talking about "application" in
the TCP/IP layering of the meaning. If you are just talking about
"userland" implementable proxies, thats a different issue. But given
enough good userland APIs, you can build any proxy there.

> ???run your own congestion/retransmission??? -  yes, the added flexibility is a big plus here, that???s the plus that I???m trying to weigh against a minus???

I can only see theoretical minuses that in todays real world would never
be important enough to be solved. 

Lets say you have differently CC-aggressive TCP stacks on your
hosts. Which by itself is usually more of a problem than a benefit, 
which is why you wouldn't necessarily invest big developmnet costs
in supporting split/merge for this situation.

But lets say you somehow do, and now you want to maintain this difference in
aggressiveness across the split paths. If you do your own CC on
split/merge you would need to try to explicitly recognize and emulate
those differences in CC-agggressiveness. Which would be hard given the
likely insufficient exposure of information in TCP and less so in e.g.: QUIC.

If you where to try to make a very passive tcp split/merge (maybe just a
dual-leaky bucket shaper per flow/path), you could probably
maintain difference in CC-aggressiveness of different flows easier, but
you would likely have a bigger problem in correctly predicting the
bandwidth (serving rate) and capacity (bucket size) of each split/merge path.

Aka: in the encap case you like will take out CC feedback for all flows
resulting in overall less and equal tthroughput for all flows (more
perfect split/merge path but lower bandwidth maybe higher latency),
in the passive split you can likely expose more the of the split/merge
path issues to the end-to-end CC but get a lot more difficult to predict
and well manage situaions.

Given how widely TCP is used, i am pretty sure, commercial vendors have
explored this space as well as commercially beneficial, and its likely all very hacky.

IMHO the better research question is how you would design a future
transport protocol (sublayer) that explicity supports coupled on-path
proxies with their various functions. Especially in a way where the
endpoints are in control of them.

Cheers
Toerless

-- 
---
[email-masked]

---

From: Keith Winstein <[email-masked]>
Date: Fri, 10 Jan 2020 11:41:00 -0800

In practice, I suspect some of the main downsides to these TCP
(transparent) connection splitters are probably the ones cited by Google in
their QUIC paper at SIGCOMM 2017: they have led to ossification of the TCP
protocol by enforcing various assumptions about transport behavior on
traffic that passes through.

See, e.g., "Is it Still Possible to Extend TCP?" (IMC 2011), "Fitting
Square Pegs Through Round Pipes" (NSDI 2012), or "How Hard Can It Be?
Designing and Implementing a Deployable Multipath TCP" (NSDI 2012).

A whole bunch of interesting TCP behavior seems to be frustrated by a
non-negligable percentage of these middleboxes. I've personally experienced
several middleboxes (for me, I've seen this on local virus scanners and
airplane WiFi services) that will kill a connection as soon as one side
sends FIN. This frustrates any application that uses half-open connections
in an interesting way. Middleboxes will forget about an idle connection
even if the endpoints still have the state. Middleboxes will freak out if
there is payload alongside SYN or SYN/ACK. SomemMiddleboxes will freak out
if they see TCP options they don't know about (including ENO/tcpcrypt).
Middleboxes will freak out if a segment appears unacked to them but never
gets retransmitted, or if a segment is acked but they didn't see it on the
forward path. And, as you say, the TCP connection splitters frustrate any
attempt by the endpoints to deploy newer/better/more path-appropriate
congestion-control schemes.

Just the FUD itself about what some lazy middlebox might freak out about
probably contributes substantially to the ossification of TCP.

-Keith

On Fri, Jan 10, 2020 at 12:54 AM Michael Welzl <[email-masked]> wrote:

> Hi,
>
> I’ve been thinking a lot about TCP connection splitters lately (
> https://tools.ietf.org/html/rfc3135#section-2.4 ).
>
> PS: I’m not trying to indirectly hint that such devices would be *always
> good*. However, the scenarios where they are not strike me as surprisingly
> narrow, so I wonder if I’m missing more.
>

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 10 Jan 2020 11:51:45 -0800

On Fri, Jan 10, 2020 at 11:41 AM Keith Winstein <[email-masked]> wrote:
>
> In practice, I suspect some of the main downsides to these TCP (transparent) connection splitters are probably the ones cited by Google in their QUIC paper at SIGCOMM 2017: they have led to ossification of the TCP protocol by enforcing various assumptions about transport behavior on traffic that passes through.
>
> See, e.g., "Is it Still Possible to Extend TCP?" (IMC 2011), "Fitting Square Pegs Through Round Pipes" (NSDI 2012), or "How Hard Can It Be? Designing and Implementing a Deployable Multipath TCP" (NSDI 2012).
>
> A whole bunch of interesting TCP behavior seems to be frustrated by a non-negligable percentage of these middleboxes. I've personally experienced several middleboxes (for me, I've seen this on local virus scanners and airplane WiFi services) that will kill a connection as soon as one side sends FIN. This frustrates any application that uses half-open connections in an interesting way. Middleboxes will forget about an idle connection even if the endpoints still have the state. Middleboxes will freak out if there is payload alongside SYN or SYN/ACK. SomemMiddleboxes will freak out if they see TCP options they don't know about (including ENO/tcpcrypt). Middleboxes will freak out if a segment appears unacked to them but never gets retransmitted, or if a segment is acked but they didn't see it on the forward path. And, as you say, the TCP connection splitters frustrate any attempt by the endpoints to deploy newer/better/more path-appropriate congestion-control schemes.
>
> Just the FUD itself about what some lazy middlebox might freak out about probably contributes substantially to the ossification of TCP.

Similar sentiments here on real practical disadvantages as a TCP
developer dealing with Internet issues over a decade.

There are good PEPs. The real disadvantages are the poorly implemented
ones and the upkeep. In my personal experience I've worked with a
cellular provider to disable their PEPS that ended up delivering much
better (YouTube video) performance. They were very happy to get rid of
those boxes to save both latency and money.

---

From: Michael Welzl <[email-masked]>
Date: Fri, 10 Jan 2020 22:33:30 +0100

Hi all,

This is all interesting, but it’s not going in the direction that I hoped. I mean, these problems are well known and obvious due to the nature of PEPs.
Imagine that connection-splitting PEPs would be a part of the architecture - known, signaled to, and officially doing what they’re doing, rather than secretly “cheating”.
Think of something more like an application-layer proxy, for example.

Then, some problems would remain, due to the way these devices operate - what they do with buffering, what they do with congestion control. That’s what I was interested in.

I’m getting the impression that problems in the style of those mentioned below are the ONLY types of problems that people have noticed….   is that true?

Cheers,
Michael

> On Jan 10, 2020, at 8:51 PM, Yuchung Cheng <[email-masked]> wrote:
>

---

From: Marie-Jose Montpetit <[email-masked]>
Date: Sat, 11 Jan 2020 00:41:09 +0100

(NWCRG chair off) We did peps years ago when satellite networks were
impacted by TCP - it used to be our main sales pitch for Teledesic that we
did not need it. But of course other networks needed them for like was all
mentioned in this thread.

Open source PEPs now are really rudimentary and application layer PEPs like
in "I am an app needing fast ACK” are more or less non existent.

What should we do to make progress?

Marie-José Montpetit, Ph.D.
Research Affiliate, MIT Media Laboratory
[email-masked]
[email-masked]

On January 10, 2020 at 4:33:51 PM, Michael Welzl ([email-masked]) wrote:

Hi all,

This is all interesting, but it’s not going in the direction that I hoped.
I mean, these problems are well known and obvious due to the nature of PEPs.
Imagine that connection-splitting PEPs would be a part of the architecture
- known, signaled to, and officially doing what they’re doing, rather than
secretly “cheating”.
Think of something more like an application-layer proxy, for example.

Then, some problems would remain, due to the way these devices operate -
what they do with buffering, what they do with congestion control. That’s
what I was interested in.

I’m getting the impression that problems in the style of those mentioned
below are the ONLY types of problems that people have noticed…. is that
true?

Cheers,
Michael

>>> PS: I’m not trying to indirectly hint that such devices would be
*always good*. However, the scenarios where they are not strike me as
surprisingly narrow, so I wonder if I’m missing more.

---

From: "[email-masked]" <[email-masked]>
Date: Sun, 12 Jan 2020 08:19:03 +0000 (UTC)

My take is that transparent TCP interception and optimisation(window sizing and buffering) will continue. It's usefulfor remote sites including over geostationary communications,and after decades of experience some of the commercialaccelerators are... well, pretty good at it. Transportacceleration is here to stay.

But above that, *transparent* application acceleration (read:interception, and 'helpful we know better, really' rewriting)will simply go away thanks to the increased use of transportsecurity that means it doesn't get used as much -- if at all.

Application acceleration is simply not worth the effort,and the interactions that remain (bugs, "secure" websitescalling non-secure website content and getting surprisedwith race conditions when it turns up at the wrong time)are too hard to debug and are increasingly corner cases --bad/unexpected website coding/behaviour meets bad/unexpectedPEP coding/behaviour over bad (insecure) connections, butthe number of insecure connections is decreasing, anddebugging this is not where you want to spend time.

We're looking at disabling application-level accelerationfor that reason. Vendor doesn't have to maintain theircode, we don't have to troubleshoot and test their code,most users don't even notice.

However, the use case that I can see remaining is thereturn of the non-transparent explicit proxy doingdeliberate caching and filtering, as opposedto the transparency we (don't) see everywherewith e.g. Akamai, content caching and local datafarms etc.

A practical example: business on remote islandconnected via geo satellite gets all its emailthrough an accelerator, but it's 99% spam thatthe accelerator is just delivering faster,eating the link capacity, so a mail proxy,filter and forwarder is set up on the otherside of the link, explicitly breaking theend-to-end path and allowing transportacceleration in cleanly too where bothends of the TCP connection are undercontrol. That's leveraging SMTP the wayit was designed, too.

Hypothetical thought: the return of visible
explicit web caching, but caching anddelivering signed assets to decrease tampering.

And there might be some recommendations documentto write around such use cases and how tohave applications allow and authorise offloadedfunctions into a proxy to do as much processingas possible before traffic hits the constrainedlink.

But there's a clear distinction to be madebetween layer-4 TCP stuff and higher-layerstuff.

L.

Lloyd Wood [email-masked] http://about.me/lloydwood 

    On Saturday, 11 January 2020, 10:41:32 GMT+11, Marie-Jose Montpetit <[email-masked]> wrote:  
 
 (NWCRG chair off) We did peps years ago when satellite networks were impacted by TCP - it used to be our main sales pitch for Teledesic that we did not need it. But of course other networks needed them for like was all mentioned in this thread.
Open source PEPs now are really rudimentary and application layer PEPs like in "I am an app needing fast ACK” are more or less non existent. 
What should we do to make progress? 
 Marie-José Montpetit, Ph.D.Research Affiliate, MIT Media Laboratory
[email-masked]@mit.edu 

Hi all,

Cheers,
Michael

---

From: Kuhn Nicolas <[email-masked]>
Date: Mon, 13 Jan 2020 14:40:01 +0000

Advantages :
TCP split have lots of performance advantage (getting up to speed, tuned congestion control, etc). They can divide by two the time needed to download a web page.
FYI, an open-source PEP for testing : https://wiki.net4sat.org/doku.php?id=pepsal:index

Issues:
                The issue with application layer PEP is that you have to split the encrypted communication – making it hard to deploy at large scale.
Transport layer PEP can be deployed for years in an operator infrastructure and may not follow the pace at which the Linux stack (and/or end points) is moving. It may not be relevant to keep the equipment in the network. That being said, PEP equipment can provide more than “just” TCP split (e.g. compression, QoS management, etc) and deactivating the split function can be tricky. I would say a practical issue is the flexibility of the operation of a PEP equipment in adjunction with end point evolutions. Another one would be their integration in complexed network architecture or constrained hardware components.

IMHO to make progress, we should induce more knowledge of the network underneath to end points so that adequate congestion control parameters can be applied.
There is an opportunity with QUIC here.

I hope this helps,
Cheers,

Nico

De : iccrg <[email-masked]> De la part de Marie-Jose Montpetit
Envoyé : samedi 11 janvier 2020 00:41
À : Yuchung Cheng <[email-masked]>; Michael Welzl <[email-masked]>
Cc : [email-masked]; iccrg IRTF list <[email-masked]>; Keith Winstein <[email-masked]>
Objet : Re: [iccrg] Disadvantages of TCP connection splitters

Marie-José Montpetit, Ph.D.
Research Affiliate, MIT Media Laboratory
[email-masked]<mailto:[email-masked]>
[email-masked]<mailto:[email-masked]>

On January 10, 2020 at 4:33:51 PM, Michael Welzl ([email-masked]<mailto:[email-masked]>) wrote:
Hi all,

Cheers,
Michael

> On Jan 10, 2020, at 8:51 PM, Yuchung Cheng <[email-masked]<mailto:[email-masked]>> wrote:
>
> On Fri, Jan 10, 2020 at 11:41 AM Keith Winstein <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> On Fri, Jan 10, 2020 at 12:54 AM Michael Welzl <[email-masked]<mailto:[email-masked]>> wrote:
>>>
>>> Hi,
>>>
>>> [email-masked]<mailto:[email-masked]>
>> [email-masked]<mailto:[email-masked]>

_______________________________________________
iccrg mailing list
[email-masked]<mailto:[email-masked]>
https://www.irtf.org/mailman/listinfo/iccrg