Subject: Re: Unfairness of parallel TCP connections

From: Shai Herzog <[email-masked]>
Date: Fri, 06 Feb 1998 06:53:52 -0500

Hi,

It may sound funny (or trivial), but have you checked that you weren't 
passing through a 64K link (or share) on the way. It certainly may 
explain these counter-intuitive results ;-)  Clearly, no matter how many 
TCP connection you'll create, you'll never go beyond the phisical speed 
of the link.

As an experiment, I why not try sending aggressive UDP at different 
times and see what happens (given that UDP is not subject to any 
fairness but only to available link capacity).

Perhaps a lab experiment may be more controlled and produce 
more irrefutible data.

Cheers,

Shai

At 01:38 AM 2/5/98 -0600, Konstantinos Dovrolis wrote:
>* Transferred file size: 3843948 bytes
>* All numbers are throughput in Kbytes/sec.
>* Three experiment types:
>	a) one connection
>	b) three connections
>	c) six connections
>* Every experiment was performed 3 times.
>* All transfers were performed within 1 hour 
>(between 8:45 and 9:45pm, Madison time),
>so the time-of-day traffic variations should not
>be very important.
>
>Results:
>
>a) One active connection:
>	29.7	34.1	36.1
>
>
>b) Three active connections:
>	15.9	15.7	19.6
>	15.9	15.7	19.4
>	15.0	14.8 	13.3
>	---------------------
>Total:  46.8	46.2	52.3
>
>With three connections you hardly get 1.5 times the throughput of one 
>connection.
>
>
>c) Six active connections:
>	14.9  	11.4	9.4	
>	10.6	10.7	9.0	
>	 9.4	11.4    9.1	
>	10.2    14.8    9.0	
>	14.3	 7.8    7.8 	
>	10.3	 9.0    7.1		
>	-------------------
>Total:  69.7 	56.1   51.4
>
>With six connections you get about 1.8 times the throughput of one 
>connection.
>
>
>In other words, the aggregate throughput that you get with
>N parallel connections scales much slower than linearly with N. 

________________________________________________________________
Shai Herzog   http://www.iphighway.com/herzog.html     IPHighway
Tel: (408) 390-3045                2055 Gateway Place, Suite 400
Fax: (408) 490-2886                           San Jose, CA 95110

---

From: Jonathan Stone <[email-masked]>
Date: Fri, 06 Feb 1998 09:20:26 -0800

>TCP connection you'll create, you'll never go beyond the phisical speed 

And (again trivially), if there are already M connections using
bottleneck link, the totally naive like me expect I'd get aggregate
throughput of N/(N+M) for N new connections of my own, if the bottlenecklink
becomes saturated.

The data aren't a good fit for a 128k link with 3-5 additional
sessions, but it's not utterly implausible, either. Or it could be a
very loaded FTP server.  Or Konstantinos could be right.

---

From: [email-masked] (Konstantinos Dovrolis)
Date: Fri, 6 Feb 1998 16:58:37 -0600

Hi folks,

I dislike sending back-to-back e-mails to a large mailing-list,
so I am sending a common reply to the points that Masataka, Zheng,
Shai, and Jonathan made (in this way, people that are not
interested will push the delete button only once...).

Regarding Masataka's comments:
------------------------------

> The phenomena is not so complex.
> 
> If the network configuration around the bottleneck is:
> 
> 	    2      1      2
> 	R0-----R1-----R2-----R3
> 
> and there are N other TCPs, the performance of P parallel TCPs is
> 
> 	P/(N+P)
> 
> It's not linear, from the beginning.
> 

Your equation would apply to fluids that fill the bottleneck link,
and only if the individual "fluids" are pushed from their sources
with the same "pressure". The packet-level granularity, the congestion 
avoidance mechanisms of TCP, as well as the fact that the connections 
have different RTDs makes the effect quite more complex and interesting.

Regarding Zheng's comments:
---------------------------

> If company A and company B are connected to the same ISP,
> and company A paid twice as much as company B, the ISP
> should be able to assure company A that it is getting
> what it paid for. Otherwise how do you get people to
> pay more.

Zheng, perhaps the reason that we are not communicating is that
you start from the assumption that a certain entity called "user"
is interested in getting at all times a certain bandwidth share
at all the links of an ISP and "it" pays according to this share 
(i.e., the USD scheme that you suggested at the last IETF gathering). 
On the other hand, I start from the assumption of flat-rate pricing
(at another thread we may discuss why this seem to be the only
viable approach for the Internet and why the USD scheme may not be
appropriate, with the exception of some corporations that want to 
build their own virtual networks over certain parts of the Internet). 

In such a context (flat-rate pricing and complete aggregation between
users with same congestion-related mechanisms)  the question
is: "How unfair can the resource distribution become when a "user"
is pushing more traffic at a bottleneck link"?  There is no such
concept as "my share is x%"; your share is what is determined by
the current load of the link, and this load is affected by you
also. If you push a lot of TCP traffic in the (FCFS) bottleneck,
you will cause at some point losses at your own connection(s)
and you will backoff.

The difference in what we say becomes more clear in your example:
> If A own 1 TCP and B owns 3 TCP connections, the relative
> (throughput) ratio is 1:3. Now is this unfair? We cannot say without
> knowing about how much A and B paid. It is fair if 
> B paid three times as much as A.

In a flat-rate priced Internet, it is not. If A and B try to compete
by opening more connections, they will eventually get the same
share (assuming that their RTDs are more or less similar).

Regarding Shai's comments:
--------------------------

> TCP connection you'll create, you'll never go beyond the phisical speed 

Even if the traffic was passing through a limited bandwidth link, what's
the difference? I am talking in general about connections that pass through
a "bottleneck link", and this link determines their final throughput.
A link can be a bottleneck in two ways: first, because of its peak bandwidth
(no matter what you do, you can get only 28.8kbps from a common dial-up
connection), and second, because of its current load. An OC-3 can also
be a bottleneck if the aggregate cross traffic entering the link 
is large enough.

> Perhaps a lab experiment may be more controlled and produce 

Completely agree. The results that I showed came from a "quick and dirty",
one-hour, after-dinner, fun-based, experiment, just to show my point to Zheng. 
It would be a (big!) mistake for anyone to view those numbers as "research
results".

Regarding Jonathan's comments:
------------------------------

> very loaded FTP server.  Or Konstantinos could be right.

Please read my previous comments.
Regarding the points that you made for Zheng's point of view:

> Suppose both company A and company B want to send flows via their ISP,
> I1, to company C.  C uses a second ISP, I2. Suppose the bottleneck
> link between A and C, inside I2, is getting congested.
> 
> Are you proposing that I2 is supposed to satisfy I1's relative
> bandwidth shares for A and B?  If it is, then don't you need this
> `relative share' information throughout the Internet? 

This is exactly the problem with USD. If it is not applied in the 
"user"-level granularity along any path that any connection from any
"user" that subscribed can follow, it doesn't work (at least reliably).

Cheers to all and have fun in this pre-Sigcomm-deadline weekend!

Constantinos

---

From: [email-masked]
Date: Sun, 8 Feb 1998 00:12:43 -0800

I'll second you on a framework here...   In addition, this framework should
also need to define how each different network component (i.e., Hosts, Edge
Routers, Core Routers and maybe even how Layer2 devices) in the network
will participate as well in the diff-server model.   As Issues like IPSec
and scalability will tend to force the enforcement points as far to the
edge of the network (even into the hosts) as possible. Whereas managment
point would tend to be more centralized.

In addition, I think we need to not only think about ISPs but also how this
will interoperate and work in corporate Intranets too.

/jsb

Zheng Wang <[email-masked]> on 02/07/98 01:36:14 PM

To:   Geoff Huston <gih @telstra.net>
cc:   [email-masked] (James Binder/HQ/3Com)
Subject:  Re: Unfairness of parallel TCP connections

Geoff Huston wrote:
> A framework document for Diff-Serv is really going to help nail
> these things down - we really do have somewhat differnet
> frameworks in mind as we discuss this.
I totally agree with you on this:-)
We need to be clear about the serivce models for diff-serv
before we can discuss ways of achieving the objective.
Cheers
Zheng