Subject: Unfairness of parallel TCP connections

From: [email-masked] (Konstantinos Dovrolis)
Date: Thu, 5 Feb 1998 01:38:27 -0600

> Well, you can trust TCP implementation but what do you do 
> with people who open multiple parallel TCP connections in order to 
> gain unfair share of the bandwidth?
> 
> Cheers
> Zheng
> 

Zheng,

It is quite interesting the issue that you raised. 
First, I agree with Geoff Huston that when we talk
about fairness, we don't really mean fairness
between human users, but fairness between TCP
or other flows. The notion of "network user" is
kind of ill-defined (perhaps I also agree with
the point that Jon Crowcroft made, but I am still
trying to parse his sentence... :-) ).

However, as I show below with a simple experiment,
the idea of having many parallel connections is
not really so great. I did some FTP transfers 
from a distant FTP site in Japan (Keio University)
measuring the final throughput. In the
first case there was only one connection, then
three parallel connections downloading the same
file, and then six parallel connections.
Here are the details and results:

* Transferred file size: 3843948 bytes
* All numbers are throughput in Kbytes/sec.
* Three experiment types:
	a) one connection
	b) three connections
	c) six connections
* Every experiment was performed 3 times.
* All transfers were performed within 1 hour 
(between 8:45 and 9:45pm, Madison time),
so the time-of-day traffic variations should not
be very important.

Results:

a) One active connection:
	29.7	34.1	36.1

b) Three active connections:
	15.9	15.7	19.6
	15.9	15.7	19.4
	15.0	14.8 	13.3
	---------------------
Total:  46.8	46.2	52.3

With three connections you hardly get 1.5 times the throughput of one 
connection.

c) Six active connections:
	14.9  	11.4	9.4	
	10.6	10.7	9.0	
	 9.4	11.4    9.1	
	10.2    14.8    9.0	
	14.3	 7.8    7.8 	
	10.3	 9.0    7.1		
	-------------------
Total:  69.7 	56.1   51.4

With six connections you get about 1.8 times the throughput of one 
connection.

In other words, the aggregate throughput that you get with
N parallel connections scales much slower than linearly with N. 
The reason is that these parallel connections compete with each
other (as well as with the rest of the cross traffic) for
the available bandwidth along the path. We are currently working on
a more general aspect of this problem. I guess that sometime
in early Spring we will have more specific things to say, 
as well as something formal written down.

I suggest you to run similar experiments. Also, the "hash" command
of FTP is quite useful to visualize how the connection's throughput
evolves over time.

However, there is also a second, practical issue that makes the use 
of multiple connections a problematic idea: 
suppose that you write a video-conference application,
and you decide to have one TCP connection for the audio,
and three more for the three video layers. You need to get
the four streams synchronously; the fact that the four connections
are competing each other causes periodically a connection(s)
to stall while the others progress faster (something that was
often clear in the above FTP transfers). 

Cheers

Constantinos

PS: I keep trying to change the subject of this thread...

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 6 Feb 98 14:10:10 JST

Constantinos;

The problem is that

> > gain unfair share of the bandwidth?

no that

> not really so great.

> as well as something formal written down.

The phenomena is not so complex.

If the network configuration around the bottleneck is:

	    2      1      2
	R0-----R1-----R2-----R3

and there are N other TCPs, the performance of P parallel TCPs is

	P/(N+P)

It's not linear, from the beginning.

Moreover, as is often the case, if the network configuration around
the bottleneck is:

	    1      1      1
	R0-----R1-----R2-----R3
              /         \
          1 /             \ 1
          /                 \
	R4                   R5

and there are N/2 TCPs through R0-R1-R2-R3 and R4-R1-R2-R5 respectively
and routers R1 and R2 treat all the incoming interfaces evenly, the
performance will be:

	P/(N+2*P)

and there is less linearity.

This is a, and perhaps the only, way to implement the dream (:-) of
Jon Crowcroft but works only for the aggregation around the
bottleneck.

							Masataka Ohta

---

From: Zheng Wang <[email-masked]>
Date: Fri, 06 Feb 1998 09:48:31 -0500

Konstantinos Dovrolis wrote:
 
> or other flows.  

I am *not* talking about the fairness between indivudial
human users - I am talking about the fairness between
user organizations (i.e. aggregated flows).

If company A and company B are connected to the same ISP,
and company A paid twice as much as company B, the ISP
should be able to assure company A that it is getting
what it paid for. Otherwise how do you get people to
pay more.

> Here are the details and results:
 
I am *not* saying that you will get twice of more 
bandwidth by openning two connections. The bandwidth
is fixed so you get less than twice when more people
join. It is the *relative* sharing that matters.

You dont need to perform experiments to show that.
Suppose that there are 4 TCP connections over a 
bottleneck. If all conditions are identical,  we
can assume that TCP will give each connection equal share 
of bandwidth (well on average over a relatively long period). 
If A own 1 TCP and B owns 3 TCP connections, the relative
ratio is 1:3. Now is this unfair? We cannot say without
knowing about how much A and B paid. It is fair if 
B paid three times as much as A.

Cheers
Zheng

---

From: Jonathan Stone <[email-masked]>
Date: Fri, 06 Feb 1998 08:57:20 -0800

>user organizations (i.e. aggregated flows).

is there a consensus that differential service between individual
`users' in a given organization is beyond the scope of diff-serv?

>what it paid for. Otherwise how do you get people to

(By giving them a bigger pipe to the ISP?)

Suppose both company A and company B want to send flows via their ISP,
I1, to company C.  C uses a second ISP, I2. Suppose the bottleneck
link between A and C, inside I2, is getting congested.

Are you proposing that I2 is supposed to satisfy I1's relative
bandwidth shares for A and B?  If it is, then don't you need this
`relative share' information throughout the Internet? And if not, and
A and B's traffic is treated equally when it leaves their ISP, where
exactly is the incentive for A and B to pay for `fairness', as opposed
to the current pricing and service models?

I could be wrong here, but reading between the lines it sounds as if
you're proposing a mechanism which lets a single ISP do a fair-share
partitioning of their internal backbone amognst traffic originating
within the ISP's customers, and no more. Is that right?

That's clearly a reasonble way for ISPs to segment their market, with
the proviso that most customer traffic is intranet traffic that
doesn't leave the ISP. But is that all diff-serv is about?

Again, perhaps I'm wrong, but this seems totally orthogonal to the
issues Steve Deering raised, of creating incentives for applications
to be better behaved (more adaptive).

---

From: Geoff Huston <[email-masked]>
Date: Sat, 07 Feb 1998 08:08:35 +1100

>what it paid for. Otherwise how do you get people to

Your point about having the network identify 'companies'
are about as practical as having the network identify 'users'.
It is not a concept which is tractable at the level of the network.

If you want absolute fairness look at a model which provides network
resources using a  quantum of end to end 64Kbps circuits at  a price
relative to the distance between the end points using a time based tariff.

Whoops - _that's_ the PSTN! :-) _This_ discussion is diff-serv on the Internet

Geoff

-- 
Mailto: [email-masked] 
http://www.telstra.net/gih

See you at INET'98, Geneva 21-24,July 98 http://www.isoc.org/inet98/

---

From: Zheng Wang <[email-masked]>
Date: Fri, 06 Feb 1998 22:01:28 -0500

> It is not a concept which is tractable at the level of the network.

Well, let me make two points:

1) Even with the Premuim Service or Assured Service,
the routers that set the bits at the ISP-ISP boundaries will still
need to know the user profiles in order to set the bits properly.

A-----ISP1---R1---ISP2---R2--ISP3---B

When traffic is coming from ISP2 towards A, the router R1 needs to
re-set the bit according to the amount of premium bandwidth A has
paid for. So router R1 needs to know its customers' profile. 
Using the bits in the packet header simplifies the core but
wont help the edges that do the classification. I think
this issue is well understood.

2) ISPs already keep a per-customer prefix table, i.e., the 
IGP routing table. Now, what about adding an extra entries 
on badwidth allocation along the next-hop info? Not a big deal. 

Cheers
Zheng

---

From: Paul Ferguson <[email-masked]>
Date: Fri, 06 Feb 1998 22:10:59 -0500

>need to know the user profiles in order to set the bits properly.

I think is a complete red herring. Per-user profiles are not needed
at the interdomain boundary -- an aggregate service profile for all
traffic which belongs to a particular class is all that is needed.
Why on earth would an edge-to-edge transit pointy need to know per-user
profiles? This is easily accommodated by provider negotiations for
allocated resources on a per-class basis.

>on badwidth allocation along the next-hop info? Not a big deal. 

You're dancing dangerously close to QoS Routing, not simple
differential services.

- paul

---

From: Geoff Huston <[email-masked]>
Date: Sat, 07 Feb 1998 19:33:44 +1100

Again I'm sitting here thinking that my diff-serve model is different
to yours. Now my perspective is as a network manager who has to
make it work and make it run revenue positive.

Let me see - why should I _limit_ A to any pre-set limit? Gee my marketing
folk want to sell A as much as A can send with the differential
"treat me right' bits as A cares to deliver to me.  I don't want
to turn anthing back :-) Your model is different and assumes
that the network wants to limit the customer to preset chunks
with the big red stop flag being waved if you dare to over-consume!.

A framework document for Diff-Serv is really going to help nail
these things down - we really do have somewhat differnet
frameworks in mind as we discuss this.

regards,

geoff

---

From: Zheng Wang <[email-masked]>
Date: Sat, 07 Feb 1998 16:28:51 -0500

> allocated resources on a per-class basis.

Let's take premium service as an example. Suppose A 
subscribes for 1 Mbps premium class bandwidth.

A-Ra-----R1------R2-----Rb-B
     ISP1   ISP2    ISP3

For traffic endtering ISP1 from A, Ra can set the P-bit for 
traffic below 1 Mbps and send the rest as best effort.

However, for traffic to A coming from ISP2, R1 needs to know
A premium class profile in order to protect A's traffic.

Cheers
Zheng

---

From: Zheng Wang <[email-masked]>
Date: Sat, 07 Feb 1998 16:36:14 -0500

> frameworks in mind as we discuss this.
 
I totally agree with you on this:-) 

We need to be clear about the serivce models for diff-serv
before we can discuss ways of achieving the objective.

Cheers
Zheng

---

From: "Alessio Casati" <[email-masked]>
Date: Mon, 9 Feb 1998 13:59:43 +0100

> Paul Ferguson wrote:
>  
> > I think is a complete red herring. 
[...]

> A premium class profile in order to protect A's traffic.

I'm afraid complete isolation of one organization's traffic from another's
is quite a hard task to accomplish, unless some end to end signaling,
and some rather fine grain per flow treatment is available. That would not go too
far apart from the current intserv model, from my standpoint. 
I'd rather support user profiles only at the edge of the network, where
some enforcement action is appropriate. Then  per class (and class/ISP) aggregation would 
suffice in the backbone. One might also provide class/organization service to big customers,
paying million$$, but that is something the ISP has to manage when dealing with the VPN issue.
IMO we should avoid mixing up VPN and diffs issues. 

alessio

---

From: Paul Ferguson <[email-masked]>
Date: Thu, 12 Feb 1998 04:19:56 -0500

>A premium class profile in order to protect A's traffic.

I'm sure someone else has probably already said this, since I
have been trying to catch up on e-mail while traveling, so I won't
spend a lot of time trying to substantiate it.

When are start to discuss "protection" of individual data flows,
one must wonder of the intserv model & RSVP is more appropriate
than trying to muddy the waters with schemes to "protect" individual
flows. One of the scalable properties of a reasonable diffserv
approach is the ability to aggregate flows into classes, and use
a weighted probability on how each class is handled in the network.

This approach does not absolve the network administrator/engineering
staff of the responsibility of the appropriate levels of capacity
planning and traffic engineering, since no magical QoS knob will
ever be an appropriate compensation for inadequacies in the network;
people still must engineer "service quality" into a network design.

Having said that, I think any given class of traffic can be afforded
a reasonable amount of "protection" given that reasonable efforts are
made to understand that capacity planning & statistical probability of
successful delivery (or drop) will allow for different classes of
data to handled quite differently in the network. And if some
intelligence is used on ingress, rate-shaping can/will also be used on
non-adaptive flows, until such a time as we understand better how
to deal with them in the network core.

$.02,

- paul

---

From: Zheng Wang <[email-masked]>
Date: Thu, 12 Feb 1998 09:15:57 -0500

> a weighted probability on how each class is handled in the network.

I am sure I explain *several times* that I am *not* talking 
about protection of "individual flows". We are trying to protect
aggregated flows from user organization.
 
I explicitly said that I were using Van's Premium Service 
as an example where there are only two classes: premium class
and best effort class. However, you need admission control to make
the premuim service work. If you are sending traffic out to the
Internet, you can do admission at the Customer-ISP border easily
since the traffic is not yet mixed with other's traffic. However,
if you are pulling traffic in, you need to set the bit at the server
side or the ISP-ISP border. Then you need per-customer info
(disclaimer: the customer is not individual user) to do 
the admission control. 

If you think this is not a problem, please explain. If you think
we should not worry about this, well I would agree to disagree.

Cheers
Zheng

-- 
Zheng Wang                  
Bell Labs Lucent Technologies
http://www.bell-labs.com/user/zhwang

---

From: Hyogon Kim <[email-masked]>
Date: Thu, 12 Feb 1998 15:22:14 -0500 (EST)

For receiver-based marking approach, meaning positively correlating user
throughput of the _incoming_ traffic with the subscriber profile size, yes,
you do need to set the bit also at the server side or the ISP-ISP border.
Namely, if subscriber "C" below wants both her outgoing _and_ incoming (for
instance from a web server "S") traffic be treated at "Ri" based on her 
profile, the profile should be installed at Ms as well as at Mc. 

        C <-- Mc <-- R1 <-- R2 <-- ... <--Rk <-- Ms <-- S
web client   meter  routers                   meter  web server

We can't deny there is a scalability issue here (based on how may
"M"s are there along your network rim and how frequently profiles are
installed and de-installed), but more important thing than that is that 
this approach fits only with "relative" guarantee schemes like USD.
Namely, if you are a N-share holder, then you always are, no matter which
part of the market (i.e. network) you go. You will be treated 
accordingly, with weight N.

However, in Expected Capacity Framework[Clark] or the virtual leased line
service(?)[Jacobson], profile is like money. If you spend it at one
place, then you have spent it, you can't reuse it in other places. 
Namely, if "C" above is accessing (note that "C" could
be a corporate network or even a small ISP) 1,000 web sites S1, S2,
S3, ... S1000, all of which have different "M"eters, then you just can't
reuse C's profile value again and again at  different "M"s. You should 
divide the profile in some way so that "C" cannot abuse her credit. 
Currently I don't know how to do that without using some sort of signaling,
which I hate to accept. Hope there's a better way.

-- 
------------------------------------------------------------------------------
Hyogon Kim ([email-masked])	       		       |        (201) 829-4728
Research Scientist				       |           MCC 1G-231B
Internet Architecture Research Laboratory	       |      445 South Street 
Bell Communications Research			       |  Morristown, NJ 07960
------------------------------------------------------------------------------