Subject: [gaia] Poor performance of Windows TCP Compound in high delay networks

From: David Johnson <[email-masked]>
Date: Thu, 12 Feb 2015 11:12:22 +0200

Has anybody come across the problem of the poor performance of Windows
using TCP compound vs the better performance of TCP Cubic used by
Linux / BSD / Apple Mac in networks with high delay - such as those
connecting over satellites or using multi-hop networks.

I came across this issue when I was analysing the performance of rural
networks connecting over satellite links with round trip delays over
500 ms. I found that outgoing throughput was three times better in
Linux Compared to Linux when analyzing the tcpdump traces. If I
simulated two parallel Linux and Window TCP flows with a 1s round trip
delay, Linux was 8x faster on the uplink than Windows.

The effect was most obvious when the local community was using Skype.
Windows users found that Skype just didn't work whereas Linux users
were having no problem with Skype. The solution is to use a split TCP
or TCP proxy at the gateway and I see this has been discussed in 5.2
of the IETF document but there is no mention of the major difference
between the TCPs used in Windows and Linux. I think this is quite an
important issue and I would be happy to work with the author of that
section to add this analysis. I have published these results in my PhD
but not at a conference yet - I think I should try and aim to get this
presented at the next ACMDEV possibly.

Interestingly one of the students that worked with me at UCSB while I
was uncovering this problem now works for Microsoft in their
networking section and says that he will be working to fix this -
perhaps there will be a release of Windows 10 for  regions with poor /
high-delay connectivity.

Regards
David

---

From: Arjuna Sathiaseelan <[email-masked]>
Date: Thu, 12 Feb 2015 09:21:22 +0000

This is an excellent question and I am sure will excite many here :)
On 12 Feb 2015 09:16, "David Johnson" <[email-masked]> wrote:

> _______________________________________________
> gaia mailing list
> [email-masked]
> https://irtf.org/mailman/listinfo/gaia
>

---

From: David Johnson <[email-masked]>
Date: Thu, 12 Feb 2015 11:49:11 +0200

Oops typo in my text ** I found that outgoing throughput was three
times better in Linux Compared to Windows when analyzing the tcpdump
traces **

---

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 12 Feb 2015 10:38:21 +0000

CC'ing the ICCRG, which may have insights here. Reply-to set to keep the discussion on the GAIA list.

---

From: Javier Simó <[email-masked]>
Date: Thu, 12 Feb 2015 12:19:13 +0100

Yes David, that's interesting, though I think that this is the type of 
thing that should be referenced better than explained in the draft.

That happens very clearly for satellite links, but also in other 
broadband long links. The delay can limit the throughput  in TCP 
connections over 802.11ac or even 802.11n, and over long fiber optics. 
And the Linux TCP is known to be better for that. I lost a lot of time a 
few years ago before discovering this when I was trying to understand 
our results in a 802.11n MIMO2x2 deployment in the Amazon forest!

Best wishes

Javier

El 12/02/15 a las 10:12, David Johnson escribió:
> .

-- 

---------------------------------------------------
Fco. Javier Simó Reigadas <[email-masked]>
Subdirector de Ord. Docente
ETS de Ingeniería de Telecomunicación
D-204, Departamental III
Camino Del Molino, s/n - 28943 Fuenlabrada (Madrid)
Tel: 914888428, Fax: 914887500
Web personal: http://www.tsc.urjc.es/~javier.simo

---

From: Michael Welzl <[email-masked]>
Date: Thu, 12 Feb 2015 12:33:26 +0100

Thanks Lars!

About the actual problem described:
Well, Cubic being more aggressive than Compound doesn’t surprise me much. However, this:

>> If I

seems to contradict that:

>> were having no problem with Skype.

The more aggressive protocol should be more likely to push Skype aside, not the other way round?!

Sounds like you should test this in a local testbed with artificial influence on capacity and delay to be able to zoom in and see what’s going on…

Cheers,
Michael

---

From: David Ros <[email-masked]>
Date: Thu, 12 Feb 2015 16:00:03 +0100

Forwarding to the gaia ML.

David

On 12 Feb 2015, at 15:56, Luca De Cicco <[email-masked]> wrote:

> Hi all,
> 
> another fact to be considered is that Skype tries to use UDP when possible
> and falls back on TCP when UDP is blocked.
> 
> Best,
> Luca
> On Thu, Feb 12, 2015 at 3:54 PM, Stefan Hirschmann <[email-masked]> wrote:
> 
> On 12.02.2015 12:33, Michael Welzl wrote:
> Hm. Skype for windows has the version 7.1.0.105
> Skype for Linux has the version 4.3.0.37
> 
> The current Skype for windows version does not use any P2P technology
> any more (to improve the quality of surveilance). But I don't know if
> there is still pieces of P2P code in the Linux code.
> 
> That confuses me a lot. "Local community" sounds like more clients than
> server and TCP congestion control is decided by the sender. So I am not
> sure if the PC running Skype have enough upload to really influence the
> network. It is more likely that the connection behavior is due to the
> differences in the Skype versions than it is due to different congestion
> control of the Skype running computer.
> 
> Cheers, Stefan
> 
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg
> 
> iccrg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/iccrg

---

From: David Johnson <[email-masked]>
Date: Thu, 12 Feb 2015 18:20:51 +0200

Hi all

Thanks all for the interesting inputs

If anybody wants to scrutinize the data - here is a link to an extract 
from my PhD that summarized the results of my analysis of the traffic 
traces as well as the simulation. This is definitely very preliminary 
data and I would need to run a simulation for longer than 20 seconds for 
a throughput test but I think the basic principle holds and doesn't seem 
to surprise many on this list.

https://www.dropbox.com/s/ax9lusyihjb5nqw/linux_vs_windows_tcp.pdf?dl=0

For those who don't want to read I'll summarize the table of my simulations:

For a simulated 1Mbps link between a windows 7 machine and a Linux 
server and a Linux machine and a Linux server, a tcp throughput test was 
carried out for 20 seconds using the iperf tool

No delay introduced (10 ms systemic delay present)
Linux TCP flow only: 892 kbps
Linux TCP flow (Windows flow added): 822 kbps (between Linux machine and 
Linux server)
Windows TCP flow only: 978 kbps
Windows TCP flow (Linux flow added): 151 kbps (between windows machine 
and Linux server)

1s delay introduced
Linux TCP flow only: 860 kbps
Linux TCP flow (Windows flow added): 858 kbps (between Linux machine and 
Linux server)
Windows TCP flow only: 110 kbps
Windows TCP flow (Linux flow added): 57 kbps (between windows machine 
and Linux server)

What really surprised me was that even with negligible delay, when a 
simultaneous Linux and Windows flow is present TCP cubic used by Linux 
is much more aggressive and hogs most of the bandwidth - this would be a 
universal problem - or is my simulation broken? With a 1s delay 
introduced, even when there is only a Windows flow present it only 
reached 110 kbps average throughput over the 20 second window.

I think the surprising issue for me is that Microsoft have not provided 
an option in the operating system which allows you to use a version of 
TCP that is better suited to high-delay networks. With wide scale roll 
outs of windows tablets etc. in many African countries this would seem 
like an urgent issue to address.

David

On 12/02/2015 17:00, David Ros wrote:
> On 12 Feb 2015, at 15:56, Luca De Cicco <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>> Hi all,
>>
>> On Thu, Feb 12, 2015 at 3:54 PM, Stefan Hirschmann <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>
>>     [email-masked] <mailto:[email-masked]>
>> [email-masked] <mailto:[email-masked]>

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 12 Feb 2015 21:08:44 +0100

Hi,

I would also assume that this effect is rather due to Skype than congestion control.

However, TCP Cubic is more aggressive than Compound. If you run one Cubic flow 
against one Compound flow, the Cubic flow will get a larger share of the capacity.

If you only have Cubic or Compound flows and see different performance that 
might depends on the buffer configuration. If you have one BDP of buffering both 
algos should provide 100% utilization. However, on high delay link one BDP of 
buffering might be quite large, so I can imaging that you'll find less. In this 
case Cubic can provide 100% utilization at least until a buffer size of half a 
BDP (because of a decrease factor of 0.3 instead of halving). While Cubic avoids 
underutilization in this scenario in the first place, Compound has to first 
detect the underutilization and will then try to increase the sending rate more 
and more. Moreover, Compound relies on delay measurements to figure out how 
quickly it can increase. That might also influence the performance. Actually 
I've never done any performance evaluation on Compound, so I can't really tell 
you how big the difference is.

As a side comment: fast is not always better because you have to share the link 
with others and if you get more capacity others might get less. In the case 
where skype is working in one case and not working in the other of course that 
does not seems to be an capacity sharing issue. But as I said I also don't think 
that this a congestion control issue as Skype is doing a lot of smart things.

Mirja

-- 
------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Communication Systems Group
Institute TIK, ETH Zürich
Gloriastrasse 35, 8092 Zürich, Switzerland

Room ETZ G93
phone: +41 44 63 26932
email: [email-masked]
------------------------------------------

---

From: "wang guodong" <[email-masked]>
Date: Fri, 13 Feb 2015 02:13:54 +0800 (GMT+08:00)

Hi David

Glad to hear researchers are doing performance optimization in Windows. TCP congestion control researches have been conducted for many years and there are great performance improvement in Linux. However, Windows users do not get many benefits from these researches. One of the problems is that Windows is not open sourced, so we can not easily change the TCP variants. Besides increasing the buffer size, is there any other methods to optimize the performance of TCP in Windows? If someone can conclude the methods, it will be beneficial to other researchers as well as Windows users.

Thanks

Guodong wang

> -----原始邮件-----
> 发件人:"David Ros" <[email-masked]>
> 发送时间:2015年2月12日 星期四
> 收件人:[email-masked]
> 抄送:"[email-masked]" <[email-masked]>, "Luca De Cicco" <[email-masked]>, "Stefan Hirschmann" <[email-masked]>
> 主题:Re: [iccrg] [gaia] Poor performance of Windows TCP Compound in high delay networks
> 
> 
> > On Thu, Feb 12, 2015 at 3:54 PM, Stefan Hirschmann <[email-masked]>wrote:
> > 
> > >

---

From: Javier Simó <[email-masked]>
Date: Fri, 13 Feb 2015 09:37:25 +0100

> I would also assume that this effect is rather due to Skype than 

Sorry if I say something stupid, because I am not a TCP expert, but I 
think that the problem of delay in TCP is not associated to congestion 
control, but with flow control. Without congestion, ACKs get back 
correctly but take time, and the transmitter has to wait for the 
reception of ACKs prior to transmitting new packets. Delay is wrongly 
interpreted as low link capacity.

I would not be able to diagnose what happens with Skype in the situation 
described by David, but in my opinion his analysis may be right.

>
> As a side comment: fast is not always better because you have to share 
> the link with others and if you get more capacity others might get less.

Well, I disagree, fast is better, because one can always do some sort of 
traffic differentiation and apply some traffic shaping technique to 
limit the throughput introduced per user or per flow, but you are not 
limited due to protocol missbehaviour.

Best

Javier

> In the case where skype is working in one case and not working in the 
> other of course that does not seems to be an capacity sharing issue. 
> But as I said I also don't think that this a congestion control issue 
> as Skype is doing a lot of smart things.
>
> Mirja
>
>

--

---

From: "Jose Saldana" <[email-masked]>
Date: Fri, 13 Feb 2015 10:15:49 +0100

Just one stupid question:

Skype works normally over UDP (I have just checked with Wireshark in my
Windows desktop). It seems that it switches to TCP when UDP does not work:

http://www.cs.toronto.edu/iptps2008/final/31.pdf

"For what concerns the transport layer, Skype relies on
both TCP and UDP. Both signaling and communication
are preferentially carried over UDP – although when
UDP communication is impossible, Skype falls back to
TCP, listening to the same random port whenever possible
(or eventually port 80 or 443, which are normally left
open by network administrators to allow Web browsing)"

Are we considering this? Perhaps the problem of skype in that scenarios is
not related to the TCP flavor of the computer.

Jose

> -----Mensaje original-----
> De: gaia [mailto:[email-masked]] En nombre de Mirja Kühlewind
> Enviado el: jueves, 12 de febrero de 2015 21:09
> Para: [email-masked]
> CC: [email-masked]; [email-masked]
> Asunto: Re: [gaia] [iccrg] Poor performance of Windows TCP Compound in
high
> delay networks
> However, TCP Cubic is more aggressive than Compound. If you run one Cubic
> flow against one Compound flow, the Cubic flow will get a larger share of
the
> capacity.
> buffering might be quite large, so I can imaging that you'll find less. In
this case
> Cubic can provide 100% utilization at least until a buffer size of half a
BDP (because
> of a decrease factor of 0.3 instead of halving). While Cubic avoids
underutilization in
> this scenario in the first place, Compound has to first detect the
underutilization and
> will then try to increase the sending rate more and more. Moreover,
Compound
> relies on delay measurements to figure out how quickly it can increase.
That might
> also influence the performance. Actually I've never done any performance
> evaluation on Compound, so I can't really tell you how big the difference
is.
> with others and if you get more capacity others might get less. In the
case where
> skype is working in one case and not working in the other of course that
does not
> seems to be an capacity sharing issue. But as I said I also don't think
that this a
> congestion control issue as Skype is doing a lot of smart things.
> 
> Mirja
> 
> 
> >> another fact to be considered is that Skype tries to use UDP when
> >> possible and falls back on TCP when UDP is blocked.
> >>
> >> Best,
> >> Luca
> >>     > The more aggressive protocol should be more likely to push Skype
aside,
> not the other way round?!

---

From: Rohan Mahy <[email-masked]>
Date: Sun, 15 Feb 2015 12:35:25 +0800

Hi Jose, David,
Yes, Skype usually works very well over UDP, even over geosynchronous
satellite Internet services. Skype usually only needs to use TCP to get
through a particularly restrictive firewall, a misbehaving NAT
implementation (especially related to address and port mapping behavior)
[1] or Twice NAT [2]. Many satellite modems have a built in NAT and many
satellite Internet services provide a single private IP address for the
customer. Sadly NATs built into satellite modems have much less market
pressure to update their NAT implementations than vendors or standalone
home gateways or wireless access points.

Another factor that David may want to consider is that some satellite
services (ex: HughesNet) implement TCP "acceleration" "features" in their
NOC. These often interfere with modern TCP implementations in order to get
around TCP slow start.

Both of these issues could cause a substantial difference from what you are
testing in a lab and what we see in the field.

Thanks,
-rohan

[1] https://datatracker.ietf.org/doc/rfc4787/ (NAT Behavioral Requirements
for Unicast UDP, especially Section 4.1)
[2] https://datatracker.ietf.org/doc/rfc2663/  (NAT Terminology and
Definitions, Section 4.3)

On Fri, Feb 13, 2015 at 5:15 PM, Jose Saldana <[email-masked]> wrote:

> > algos should provide 100% utilization. However, on high delay link one
> BDP
> of
> link
> > >>     server and TCP congestion control is decided by the sender. So I
> am
> not
> > >>     sure if the PC running Skype have enough upload to really
> influence
> the