Subject: [Tsvwg] A coment on "draft-briscoe-tsvwg-relax-fairness-00" re file-sharing

From: "Robb Topolski" <[email-masked]>
Date: Fri, 11 Apr 2008 21:30:30 -0700

Comment on 

Problem Statement: We Don't Have To Do Fairness Ourselves
draft-briscoe-tsvwg-relax-fairness-00

Dear Working Group members,

Thank you for your time working on this committee and issues such as these.

The above quoted paragraph requires a number of corrections.  This paragraph
is a basis for the problem statement that follows it -- so getting the basis
down right is key to analysing the issue and sizing up the problem.  The
corrections that I hope to effectively communicate seem to me to reduce the
stated size of the problem.  

--Quote--
Even though file-sharing generally uses TCP, it uses the
well-known trick of opening multiple connections--currently around
100 actively transferring over different paths is not uncommon. A
competing Web application might open a couple of flows at a time, but
perhaps only actively transfer data 1-10% of the time (its activity
factor). Combining 50x less flows and 10-100x lower activity factor
means the traffic intensity from the Web app can be 500-5,000x less.
However, despite being so much lighter on the network, it gets 50x
less bit rate through the bottleneck.
--EndQuote--

I am relatively certain that the draft is talking about BitTorrent above,
and it unforunately misinterprets what it does.  BitTorrent is the #1 P2P
File Sharing protocol in use today.  Furthermore, neither #2 P2P file
sharing protocol ED2K (aka eDonkey, eDonkey2000, eMule, or MFTP) nor #3
Gnutella (aka G1, G2, or Gnutella2) work like BitTorrent, nor as Briscoe's
draft described above.

In the case of BitTorrent, between 35-70 TCP connections are generated.
However, only three or four of these connections are actively transferring
data.  Therefore, any benefit to the "trick" is erased.  The reason the
remainder sit idle is described in the specification
(http://wiki.theory.org/BitTorrentSpecification#Choking_and_Optimistic_Uncho
king), chief of which is the desire that BitTorrent NOT interfere with TCP
congestion control.  

It is important to remember that P2P users are largely residential users,
most using connections that are configured with asymmetric connections where
the maximum download bandwidth allowance is several times that of their
upload allowance.  In all of the discussions about "fairness" that I've been
involved in, it has to do with sharing the tiny upload pipe.  Filling the
generously-configured download pipe is seldom a user fairness concern (it is
also very difficult to control at the receiving end).  A trend is barely
beginning toward more symmetrical residential accounts, but it is probably
too small to accurately determine changes in usage patterns created by it.

That BitTorrent uses "hundreds of connections" that are simultaneously
active is a common misperception.  While the draft likely did not mean to
inject emotionalism, the label of "trick" is both inaccurate and
inflammatory.  The reason that BitTorrent makes these idle connections is to
allow for one connection to "hunt" for the best reciprocating peer, which
allows two efficiently located peers to detect and reciprocate with each
other.  If not interfered with by network operators, this pretty much
ensures that the lowest-latency peers will eventually find one another and
engage in a long-term exchange of data.  This is not only fast for the user,
choosing the most efficient (lowest latency) pairings is friendliest to the
local ISP network and the Internet as a whole.

All three of the top P2P protocols do open multiple upload connections.
ED2K seems happiest with 3-4 KB/s upload flows, Gnutella with 10 KB/s upload
flows, and BitTorrent with anywhere from 3 KB/s to 7 KB/s flows.  Therefore,
the number of upload flows that a single peer might generate (think
different archives being sent by the same client simultaneously) is a
function of dividing the configured upload bandwidth by the amounts
decribed.  

An example: 

BitTorrent:  Upload limit set to 20 KB/s, 3-6 upload TCP connections
(remainder are idle or downloading only)
ED2K:  Upload limit set to 20 KB/s, 5-6 uploading TCP connections (remainder
of connections are downloading only or performing light-bandwidth
housekeeping, such as queue placement requests or updates)
Gnutella:  Upload limit set to 20 KB/s, 1-2 uploading TCP connections
(remainder, same as for ED2K).  

All three of the above have a UDP component, which is used for searching or
other light network housekeeping.  There is an emerging use of UDP for "NAT
hole punching" but these "connections" are still quite rare.  

I am happy to answer any questions or further clarify anything that I can.

Sincerely,

Robert M. "Robb" Topolski
[email-masked]

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 14 Apr 2008 13:05:41 +0100

Robb,

There's three types of responses to this (expanded below):
a) Could my scenario become common, even if it's not currently prevalent?
b) The discussion is over what the correct fairness /goal/ should be
c) Who is actually correct for the Internet today? (split out into 
next posting)

Anyway, TCP doesn't actually determine capacity sharing for 
residential backhaul links - operators generally use per-user fair 
queuing. This draft merely shows how bad things would be if rate 
equality was the dominant sharing regime (e.g. if, as access speeds 
increased, most congestion moved further into the network where FQ is 
too unscalable, e.g. border links).

a) Could my scenario become common, even if it's not currently prevalent?
---------------------------
The discussion below shows that (as you say) the main reason 
swarmcasting like BitTorrent doesn't open many connections is 
asymmetric access. If you look at the situation in countries like 
Japan (the [Res_p2p] reference), where symmetric 100M FTTH has become 
very prevalent over the last few years, this constraint disappears. 
Campus & enterprise networks are already symmetric.

We're designing protocols today so they will be available in a couple 
of years. We have to look ahead.

b) The discussion is over what the correct fairness /goal/ should be
---------------------------
The draft is about the /combination/ of multiple connections and high 
activity factors. It's designed to show that the /goal/ of equal flow 
rates is not the one we should be pursuing. Even if my figure of 
500-5000 times more traffic intensity is an order of magnitude wrong 
for most of today's Internet, 50-500 times more intensity (or even 
5-50) still shows the original goal of equal flow rates is inappropriate.

I don't mind altering the figures in the draft accordingly, or 
qualifying their applicability.

Bob

>Robert M. "Robb" Topolski

____________________________________________________________________________
Notice: This contribution is the personal view of the author and does 
not necessarily reflect the technical nor commercial direction of BT plc.
____________________________________________________________________________
Bob Briscoe,                           Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 14 Apr 2008 13:06:30 +0100

Robb,

I've split out this more detailed response to your mail - see my 
previous posting for these two responses:
a) Could my scenario become common, even if it's not currently prevalent?
b) The discussion is over what the correct fairness /goal/ should be

c) Who is actually correct for today's Internet?
---------------------------
The 3-4 active connections statement is just as much a fallacy as the 
hundreds of active connections statement. Which you get depends on 
the evolution of the situation over the lifetime of a swarm:
* If the swarm is immature (ie few peers have finished) your peer 
will limit open connections. How many active (unchoked) connections 
you get initially depends on configured upload slots, which is indeed 
about 3-4 per swarm (gets linearly larger for faster access lines - 
see graph in presentation below).
* But if peers stays on line after they have finished (seeders), 
other peers still downloading will activate (unthrottle) connections 
to nearly as many seeders as they can find, irrespective of the 
upload slots limit.

This is because upload capacity is the bottleneck (as you say). But 
once you're no longer downloading, BitTorrent no longer needs to 
preserve your upload capacity for the ack stream that was keeping 
your download going.

In some communities you get a lot of seeders staying online. In 
others, they're all mingey gits who disconnect once they've got what 
they want.

Also, if you're going for niche ('long-tail') content, of course you 
won't get many active connections, as it's hard to find peers.

My presentation of that draft to the Transport Area in Vancouver
<http://www.cs.ucl.ac.uk/staff/B.Briscoe/present.html#0712relax-fairness>
shows a screen shot of a colleague's Azureus client (taken randomly 
at the time I asked him for it). He had 3 swarms running (fairly 
typical) and 105-114 active TCP data transfer connections, with only 
about 10-15 choked. One of the swarms is shown. You can see ~40 
connections are active in this one swarm. This merely implies there 
were a lot of nice friendly seeders in his swarm.

Bob

---

From: "Robb Topolski" <[email-masked]>
Date: Mon, 14 Apr 2008 14:28:13 -0700

Hello Bob, thanks for the reply!

> c) Who is actually correct for the Internet today? (split out into

One problem with applications, as you are aware, is that some users often
don't follow configuration guidance or instructions.  So the answer to the
two questions above is possibly and partially.  

For the users that won't follow instructions, there will be users like your
friend (mentioned elsewhere), who was running about 40 torrent
simultaneously.  Using Azureus, presuming all 45 torrents had sufficient
"leechers," such a mis-configuration would create an instance of a user with
over a hundred active uploading TCP connections (unless he changed the
setting that also limits each task to 3-4 upload slots).  But there will
also be users that accept the reasonable installation defaults, and never go
into "Advanced mode" to break away from the parameters set at install time.

Please, though, if your statement is about some future possible condition, I
ask you to please write it up making that point clear.  Your statements
about the present condition are being relied upon in certain serious debates
in numerous circles -- including other WGs of the IETF.  

We ARE (hopefully) going to continue to transition to more symmetric
connection model.  I don't think I have any confidence in any particular
view of how things will look then.  

Personally, I've seen enough fads come and go, and I see part of the
behavior on today's P2P networks being a fad of "appropriation."  Some (or
many) of these kids are downloading media faster than they can possibly
consume it.  It's either a hoarding reaction (e.g. "someday it will go
away") or it's a bragging-rights thing (e.g. "he who dies with the biggest
DVD stack, wins").  I have no data, it's just a hunch.

> too unscalable, e.g. border links).

You're over my head at the "FQ is too unscalable" part, but since you
mentioned border links, as you consider this future issue, also consider
that there may be more borders than we all might assume.  This might offload
some of the danger, I don't know if you've included it in your
considerations.

Having never seen the topography on paper before, one thing that surprised
me was learning that the Cable MSOs cannot be modeled as a single network.
Instead, they are several smaller metro-area networks -- one in each area,
and each with its own border/gateways to upper-tier transit providers (one
of which may be a company-owned backbone to other metro-areas).  

> qualifying their applicability.

Thanks, that would definitely help.  Can I also suggest using something
other than the word, "trick?"  It's not like any of these P2P apps are
setting up redundant connections to the same host (such as "download
accelerators" do).  Knowing some of these P2P developers as I do, the last
thing they want to do is harm the network -- they just want to fill the
space available. 

Robert M. "Robb" Topolski <[email-masked]>
Hillsboro, Oregon USA

Variables won't; constants aren't.
Osborn 

> -----Original Message-----
> From: Bob Briscoe [mailto:[email-masked]]
> Sent: Monday, April 14, 2008 5:06 AM
> To: Robb Topolski
> Cc: 'IETF Transport Area working group mailing list'
> Subject: Re: [Tsvwg] A coment on "draft-briscoe-tsvwg-relax-fairness-00"
> re file-sharing
> 
> Robb,
>

---

From: "Robb Topolski" <[email-masked]>
Date: Mon, 14 Apr 2008 15:06:39 -0700

> ... How many active (unchoked) connections

Bob, this may be quite valid, and I'm not sure whether the BitTorrent
development community has put much thought into this.  As residential upload
pipes grow larger, it is possible under BitTorrent's current configuration
recommendations to have many more parallel tasks going at once.  The 3-4
upload slots could become 30-40 (assuming 512 Kbps upload becomes 5 Mbps).
It's still not the scenario of 100 or more, but 30-40 is a sizable departure
from 3-4 and BitTorrent developers might need to consider how today's TCP
stacks would treat that in a congestion situation.  Do you think this is
worthy of discussing?  

Still, I don't think the goal is "fairness."  I can't think of anyone who
wants P2P uploads to compete "fairly" with online games, telephone calls, or
active web surfing.  

What I honestly don't understand is why we can't mark these packets using
any of the Internet Standard, vetted, approved methods so that applications
give ISPs instruction and permission to handle them on a "weather
permitting" basis.  Currently, we have ISPs that ignore these flags, inspect
the packets, and pass their own brand of judgment upon them.

If the network operators enable that on their networks, I think the apps
will follow.  And yes, there will be some end-user abuse of these flags --
but that's true of almost any technical change.  The great majority won't
abuse it.

> preserve your upload capacity for the ack stream that was keeping

Your understanding has some errors.  

First, the downloaders don't activate the seeders, the downloaders only
indicate "Interest."  The seeder client is in positive control.   Before any
transfer can take place, the seeder must unchoke that peer so that peer can
make a request.  The seeder does not unchoke all of the downloaders, it only
unchokes when it has a slot available.  (Sometimes, your uploads are going
to people who can't download fast -- such as dial-up users.  Many clients
have a feature that will automatically open an additional slot if this is
detected.)

You are correct that a downloader will indicate "Interest" to all seeders,
and that many seeders might unchoke that downloader and allow it to make
requests.  But in that case, those flows are downward in direction --
currently not the big concern of residential ISP congestion.  Meanwhile the
seeding clients that are transferring to him still maintain control over the
number of uploading flows from their nodes so that they can respond to signs
of congestion.  

Since congestion is detected and responded to by the sender (the seeder),
there is nothing inappropriate with this model since the seeder is
controlling the number of outgoing flows it has.

A BitTorrent client will still respect the upload limit specified either at
install time or modified by the user, even if it's done downloading.
However, in many clients there is the option to specify both conditions --
an upload limit while downloading and upload limit while only seeding.  It's
against the users interests to turn this up higher -- for reasons similar to
the ones you mentioned.  The user still wants to surf and use other services
on the 'net.

> were a lot of nice friendly seeders in his swarm.

I think you misinterpreted that display, perhaps.  Notice that the upload
speed of all of these was only a few bytes a second.  That's overhead from
downloading.  More evidence is the fact that they're all seeders -- so they
aren't getting any data from you (they already have all of the pieces).
Client wide, it doesn't appear that he was uploading anything data all,
since he limited his upload to 11 KB/s (lower right corner) yet was only
sending out 2.2 KB/s -- that might be one upload slot or might all be
overhead chatter.

He is also misconfigured.  With a global upload limit of 11 KB/s, one might
suggest he try eMule instead, or only run one torrent at a time and let the
others queue.  In the long run, it is more efficient.

HTH

> Sent: Monday, April 14, 2008 5:07 AM
> To: Robb Topolski

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 17 Apr 2008 15:41:40 +0100

Robb,

At 22:28 14/04/2008, Robb Topolski wrote:
>simultaneously.

Er no - I said he was running 3 torrents.
I suspect you have misread "~40/torrent" for ~40 torrents.

This is a tricky area - I have tried to be as accurate as I can - so 
pls read carefully.

>Using Azureus, presuming all 45 torrents had sufficient
>ask you to please write it up making that point clear.

No, my statement was about a present case. And you will see I said 
the Azureus client and the OS was in its out-of-the-box config state.

Another colleague also did measurements with a symmetric 9Mbps 
symmetric access (6x T1 bonded together) and detected 450 
simultaneously active connections - no special effort - he just ran 
up a standard BitTorrent client and started downloading movies 
(varying between 3 & 6 over the period measured). This was well 
within the numbers expected from configuration guidelines (1200 max 
active connections at 9M upstream). However, I don't have full 
version nos of all his software for that case.

Certainly these may not be typical cases, but there was no intention 
to do an exhaustive study to find what is typical, as protocol 
engineering isn't about just typical cases. However, the fact they 
'just happened' shows they're not totally abnormal.

I believe it is mostly a function of how community spirited the 
user-base for a particular swarm is (how many seeders). My colleague 
(the 8M ADSL one) is a member of a p2p community that prides itself 
on its exclusive membership, which it limits in size, and 
collectively they police for copyright violation very fastidiously.

>Your statements
>in numerous circles

I'm aware that this is important, which is why I'm trying to be careful.

>  -- including other WGs of the IETF.

Which?

>view of how things will look then.

Who is 'We'? The IETF is an international organisation. It has to 
cater for the situation anywhere in the world. Have a look at Korea, 
Japan, Hong Kong, Sweden, Denmark, Norway.
(e.g. <http://www.bharatbook.com/detail.asp?id=25141>)

Anyway, not wishing to break the habit of a lifetime, the IETF is 
allowed to re-design protocols before the old ones cause predicted 
problems. And it's certainly allowed to try to understand problems 
that are likely to arise in the future.

>DVD stack, wins").  I have no data, it's just a hunch.

On what basis is an evolution like p2p likely to go away? 
Swarmcasting is cool. Even ancient institutions like the BBC are 
using it underneath their client to deliver their content. 
Appropriation isn't a new phenomenon. For decades now some people 
have been leaving their TVs on all day - sometimes 3 or 4 on 
different channels in different rooms. Cable networks were built to 
cope with that level of volume - it's not appropriation, but it has 
the same effect.

I'm just using swarmcasting as an example of why the "flow-rate 
equality" thinking prevalent at the IETF needs a serious update.

>some of the danger, I don't know if you've included it in your

If you mean many borders in parallel, yes certainly. But the 
scalability problem with FQ hits you for much smaller no's of users 
than you currently see at borders even today. And once you get to 
borders, you no longer have knowledge of which IP addresses should be 
weighted differently as you do on remote access servers. At a border, 
at the IP address level, a server farm looks like my gran.

>accelerators" do).

I used the word trick, because I was trying not to put a value 
judgement on it (ie I wouldn't have said 'cheat'). I guess I could 
just say 'technique'. But personally I thought 'trick' wasn't 
perjorative. In the same vein, I have written that http browsers use 
the trick of opening 2-4 connections.

>Knowing some of these P2P developers as I do, the last
>thing they want to do is harm the network

I also know p2p developers. Yes, it's not their fault personally. 
They are trapped in an arms race. That's why I'm trying to work out a 
way out of the arms race.

I also want to say explicitly that the draft is NOT intended to say 
that p2p is the root cause of the problem. It's trying to say that 
the goals of the IETF's protocols are the root cause, and we need to 
understand the problem (in the context of p2p) to find better 
protocol goals (at run-time not design-time).

>-- they just want to fill the
>space available.

The Internet was designed to take advantage of the cost efficiency of 
packet multiplexing. If everyone filled the space available, they 
might as well have a circuit-switched network.

I find nearly everyone latches on to the multiple flow issue and 
hardly anyone clocks the issue of activity factor over time.

---
Finally, does this draft at least help towards understanding that we 
have a problem in the way we are reasoning about this issue?

It was heading towards a sister requirements draft (in preparation 
with others) introducing a proposed metric to reason about fairness. 
But that's for another thread.

Cheers

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 17 Apr 2008 17:20:25 +0100

Robb,

At 23:06 14/04/2008, Robb Topolski wrote:
>recommendations to have many more parallel tasks going at once.

There is a world beyond the borders of Oregon.

>The 3-4

Need you ask? - it's my current mission in life :)

>Still, I don't think the goal is "fairness."

So what is it?

>I can't think of anyone who

An operator has to see things from all its customers' points of view. 
Those gaining and those losing out.

There is a way to consider all of these within the same reasoning 
framework so you can reason about fairness, and allow different 
communities to maintain different standards of fairness between their 
members and fairness between communities. See my paper here:
<http://www.cs.ucl.ac.uk/staff/B.Briscoe/pubs.html#rateFairDis>
(Originally an I-D in this w-g, but I let it expire)

>permitting" basis.

My own company (not me personally) is developing a Diffserv-based 
approach to this issue for our production network (no promises mind - 
it's not yet committed to as a service).

But it won't be perfect, because ultimately we would need 
inter-provider DS and a generally recognised API to DS. Many of us - 
many on this list - have been trying to sort out all the issues with 
inter-provider DS and with an API to DS.

But there's another approach that might get there first (because it's 
simpler, even tho it's less advanced through standards). It uses 
weighted congestion controls with the ISP enforcing congestion limits 
(that's the bit that requires standards). I prefer it because it can 
be fully controlled by the app/OS, and it's possible to prove the ISP 
is being 'neutral' (or not). But that's another subject.

>Currently, we have ISPs that ignore these flags, inspect

As you rightly point out, a DS-based approach gives a 
'weather-permitting' treatment, so it won't solve this problem of the 
ISP using its independent judgement either.

I have been working on this since 1997, and have finally devised a 
simple way for ISPs (and users) to be accountable for their actions 
(see I-D on re-ECN). <http://www.cs.ucl.ac.uk/staff/B.Briscoe/projects/refb/>

>but that's true of almost any technical change.

Nope - we've been designing networks and systems for years that a 
robust to abuse.

>The great majority won't abuse it.

It must be lovely to live in such a state of innocence :)

>have a feature that will automatically open an additional slot if this is

OK - yes - you're correct. But my error on which end has final say 
doesn't change the outcome - see below.

>number of uploading flows from their nodes so that they can respond to signs

You're right, except 'uploading seeders responding to signs of 
congestion' is actually one of the biggest aspects of /the/ problem. 
Most{*} currently just use normal TCP congestion control, and the 
more connections they have open the greater share of the pipe they 
get. Yes, each responds to congestion. But the set overall causes a 
much greater share of the upstream congestion precisely as described in my I-D.

{* e.g. Joost doesn't use congestion control for media streaming. DNA 
seems to be TCP-friendly but over UDP.}

And,... focusing solely on flow counts still overlooks the much 
higher activity factor, which multiplies the effect.

>controlling the number of outgoing flows it has.

Define inappropriate.

A pure uploader (seeder) uploading with 15 TCPs active 100% of the 
time causes 150 times more congestion than a user uploading with 2 
TCPs active 5% of the time.

>the ones you mentioned.  The user still wants to surf and use other services

But the user's own interest gets hurt long long after other users' 
interests have been hurt to the same degree.

>sending out 2.2 KB/s -- that might be one upload slot or might all be

None of this contradicts anything either of us have said.

It shows a downloader can open many more than 3-4 connections (114 or 
so in this case) when it can find plenty of pure uploaders (seeders).

This in itself /is/ a problem as described in the draft.

Certainly it doesn't show whether pure uploaders can have many 
connections open. But it doesn't show they can't either.

Flows have two ends. If some (and possibly many) downloaders have 
over 100 connections open, is it really likely that all uploaders of 
the same torrent have no more than 3 or 4 connections open?...

...No need to answer that - as it's not actually a water-tight 
argument. There could be only a few downloaders and many seeders for 
one torrent. I'll see if we can find some evidence on upload connection no's.

Nonetheless, I also said in the draft:
"This scenario is not meant to be an accurate model of the current 
Internet, for instance:
...
     * Upstream not downstream constrains most p2p apps on DSL (but 
not all fixed & wireless access technologies). "

The point of the draft was to use p2p to illustrate that the IETF 
needs a new way of thinking. You've shown that what I've said in the 
draft was not incorrect. I can halve the numbers and still make the 
point I was trying to make.

Bob

____________________________________________________________________________
Bob Briscoe, <[email-masked]>      Networks Research Centre, BT Research
B54/77 Adastral Park,Martlesham Heath,Ipswich,IP5 3RE,UK.    +44 1473 645196

---

From: [email-masked]
Date: Thu, 17 Apr 2008 09:52:32 -0700

Bob and committee ...please don't be discouraged from looking into this matter.

Ijust want the examination to start with an accurate understanding of
what the application really does.

It's not a trick to beat congestion, but as Bob points out, as our
upload pipes grow larger, it might turn out that way.

one question I ask Bob, how congested are the networks now? This is a
non issue when there is no Congestion. Bob, how often should the net
be congested and how long do these incidents last? (I cant find this
data anywhere.)

On 4/17/08, Bob Briscoe <[email-masked]> wrote:
> (see I-D on re-ECN).
> <http://www.cs.ucl.ac.uk/staff/B.Briscoe/projects/refb/>
>
>
> > > >generously-configured download pipe is seldom a user fairness concern
> (it
> > > is
> > > >inflammatory.  The reason that BitTorrent makes these idle connections
> is
> > > to
> > > >All three of the above have a UDP component, which is used for
> searching
> > > or

-- 
Robb Topolski ([email-masked])
Hillsboro, Oregon USA
http://www.funchords.com/