Subject: Re: [tsvwg] Naive question on multiple TCP/IP channels

From: "Eggert, Lars" <[email-masked]>
Date: Wed, 4 Feb 2015 19:49:52 +0000

Hi,

CC'ing tsvwg, which would be a better venue for this discussion.

On 2015-2-4, at 20:22, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> Today most Web browsers attempt to optimize download of images etc. by opening multiple TCP/IP streams at the same time. This is actually done for two reasons, first to reduce load times and second to allow the browser to optimize page layout by getting image sizes etc up front.
> 
> This approach first appeared round about 1994. I am not sure whether anyone actually did a study to see if multiple TCP/IP streams are faster than one but the approach has certainly stuck.

There have been many studies; for example, http://www.aqualab.cs.northwestern.edu/publications/106-modeling-and-taming-parallel-tcp-on-the-wide-area-network

> But looking at the problem from the perspective of the network it is really hard to see why setting up five TCP/IP streams between the same endpoints should provide any more bandwidth than one. If the narrow waist is observed, then the only parts of the Internet that are taking note of the TCP part of the packet are the end points. So having five streams should not provide any more bandwidth than one unless the bandwidth bottleneck was at one or other endpoint.

You don't get more bandwidth in stead state (well, with old Reno stacks, you got a little more, but not much). The real win is in getting more bandwidth during the first few RTTs of TCP slow-start, which is the crucial phase when transmitting short web objects.

> Now there are some parts of the deployed Internet that do actually perform statefull inspection. But I would expect increasing the number of channels to degrade performance at a firewall or any other middle boxen.
> 
> So we have a set of behavior that seems at odd with the theory. Has anyone done any experiments recently that would show which is right?

I haven't seen any performance study, but another concern is that middleboxes obviously need to maintain state per connection, and multiple parallel connections eat that binding space up more quickly. (And for a NAT, reduce the number of clients it can serve.)

> The reason it makes a difference is that it is becoming clear that modern applications are not best served by an application API that is limited to one bi-directional stream. There are two possible ways to fix this situation. The first is to build something on top of TCP/IP the second is to replace single stream TCP with multi-stream.

SCTP has what you call multiple streams in your second option, and is designed the same way.

> My preference and gut instinct is that the first is the proper architectural way to go regardless of the performance benefits. When Thompson and co were arguing that all files are flat sequences of bits, they were saying that was the right O/S abstraction because you could build anything you like on top.
> 
> But then I started to ask what the performance benefits to a multi-stream TCP might be and I am pretty sure there should not be any. But the actual Internet does not always behave like it appears it should.

See above.

Also, one motivation for SPDY/HTTP2.0 is to reduce the number of parallel connections, since web people have noticed that more is not always better here.

> I suspect that the preference for multiple streams probably comes from the threading strategies it permits. But that is an argument about where the boundary between the kernel and application is placed in the network stack rather than where multiplex should live in the stack. Microsoft already provides a network stack for .NET where the boundary is in the HTTP layer after all.
> 
> 
> So anyone got hard data they could share?

The TSVWG folks may have.

Lars

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 05 Feb 2015 12:54:56 +1300

> There have been many studies; for example, http://www.aqualab.cs.northwestern.edu/publications/106-modeling-and-taming-parallel-tcp-on-the-wide-area-network

GridFTP only exists because of lots of experience that several parallel FTP
streams achieve better throughput than a single stream, especially on paths
with a high bandwidth-delay product. I'm guessing that since buffer bloat
creates an artificially high BDP, that could apply pretty much anywhere.

SCTP is not the only one-acronym answer: try MPTCP. The interesting thing there
is that because there is explicit coupling between the streams, the throughput
increases sub-linearly with the number of streams.

    Brian

---

From: Piers O'Hanlon <[email-masked]>
Date: Thu, 5 Feb 2015 10:06:42 +0000

One of the snags with using multiple [slower] TCP streams is that although you might get a faster throughput you end up losing more packets as the equivalent TCP loss rate for multiple lower rate flows is higher than for one larger stream. This can be explained by looking at the TCP [simple] equation Xrate = 1.2*PacketLen/(RTT*sqrt(loss)) where it can be seen that the lower the send rate the higher the loss needs to be on a path with the same RTT.

I guess there were also other approaches like BEEP, and Google's QUIC which multiplexed multiple connections over UDP. 

Piers

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 4 Feb 2015 16:00:29 -0500

On Wed, Feb 4, 2015 at 3:47 PM, Phillip Hallam-Baker
<[email-masked]> wrote:

> I know that traditionally we have considered congestion control to be a per
> connection thing rather than per process or per user. Those don't exactly
> exist in the Internet model. But if we consider DDoS to be an extreme form
> of congestion, we have to look at the O/S implementing broader rate limiting
> anyway.

I'm not sure this scales in a useful manner...especially if you want
(for instance) to be able to read from your cloud/nas/etc at 'line
rate' but then limit access rates to other places because of 'dos'
concerns.

Policy for that is not going to be clear, or simple, or reasonable.

> If the case for multiple streams is better performance based on friendlier
> slow start parameters, maybe these should be allowed without the end run. If
> the net is surviving with people starting five streams instead of one, maybe
> the slow start parameters could start at five packets per destination host
> instead of one per connection. It would be a little more code to implement
> but speed is hardly an issue where its purpose is throttling anyway.

More connections means you may avoid the 'random early drop' problem
for some of your connections, right? Presuming somewhere between
src/dest there's a thinner pipe (or more full pipe) and RED/etc is
employed (or just queue drops in general), putting all your eggs in
one connection basket is less helpful than 5 connection baskets.

One connection (one traditional http/tcp connection) also means object
serialization gets you as well. (I think roberto alludes to this
above)

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 4 Feb 2015 17:00:04 -0500

On Wed, Feb 4, 2015 at 4:23 PM, Phillip Hallam-Baker
<[email-masked]> wrote:
>
>
> On Wed, Feb 4, 2015 at 4:00 PM, Christopher Morrow <[email-masked]>
> wrote:
>>
> No it is not simple at all!

I'm glad we agree about this.

> At a gut level, there is something wrong if a node is generating vast
> numbers of
> SYN messages and never doing anything more. But only looking for SYN floods
> at the network border means the attackers will attempt some Data as well.
>
> Working at the endpoint is a lot simpler but then we rely on the endpoint
> not
> being compromised which is a bad assumption for a machine doing DDoS
> attacks.
>

yes. there are a god number of 'not compromised' endpoints which also
are used for dos attacks. Complicit users are also a 'problem' to keep
in mind.

> The reason I am doing this work is to try to work out what controls to place
> where.Traditionally a firewall has been seen as a device that protects the
> local
> network from attacks coming in. But preventing outbound attacks is equally
> important. Nobody is going to bother attacking Comcast's customers if
> all their residential gateways have egress controls that make them
> useless as bots. Conversely nobody is going to pay for Comcast service
> if the egress controls makes it useless for browsing etc.
>

Right, this was my 'the controls are not that simple nor reasonable' comment.
ArborNetworks (for one simple example) has some decent heuristics in
place for 'anomalous traffic identification', you might ask them about
how some of that scales to 'residential gateway' sorts of locations.

Keep in mind that if you have 100,000 bots 10pps of icmp which seems
'reasonable' is 1mpps of icmp in aggregate at the far end of the
network. Individual end systems don't have to contribute much to cause
a problem.

Change icmp (becuse icmp is evil, right?) to http or https .. and
change 10pps to 1pps ... you still have a significant canon to aim
that will cause an outage for almost all websites (over 95% probably?
[note no citation] enough that 'almost all' counts). You also,
clearly, don't have to aim your http request to an actual webserver...
100kpps if syn traffic is really quite a lot for 'almost all'
endpoints on the internet to sink.

The problem of classification of traffic like this, without
coordination amongst your monitoring nodes, is ... hard.

> Hmm, again sounds like gaming the congestion parameters rather than
> an argument for a particular approach.
>

maybe? but if you look at SPDY/http2 I think the goal there is to stop
serializing, change to a binary and compressed format for data and add
channels inside the single tcp/udp stream. Fill the pipe as quickly as
possible and keep it full for the maximal amount of time.

> There are several necessary functions. One of them is MUX and another of
> them is serializing data from API calls. The question being which function
> each belongs at which layer.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Feb 2015 15:47:13 -0500

On Wed, Feb 4, 2015 at 2:49 PM, Eggert, Lars <[email-masked]> wrote:

> > Today most Web browsers attempt to optimize download of images etc. by
> opening multiple TCP/IP streams at the same time. This is actually done for
> two reasons, first to reduce load times and second to allow the browser to
> optimize page layout by getting image sizes etc up front.
> >
> > This approach first appeared round about 1994. I am not sure whether
> anyone actually did a study to see if multiple TCP/IP streams are faster
> than one but the approach has certainly stuck.
>
> There have been many studies; for example,
> http://www.aqualab.cs.northwestern.edu/publications/106-modeling-and-taming-parallel-tcp-on-the-wide-area-network

thanks, looking at it now,

>
> > But looking at the problem from the perspective of the network it is
> really hard to see why setting up five TCP/IP streams between the same
> endpoints should provide any more bandwidth than one. If the narrow waist
> is observed, then the only parts of the Internet that are taking note of
> the TCP part of the packet are the end points. So having five streams
> should not provide any more bandwidth than one unless the bandwidth
> bottleneck was at one or other endpoint.
>
> You don't get more bandwidth in stead state (well, with old Reno stacks,
> you got a little more, but not much). The real win is in getting more
> bandwidth during the first few RTTs of TCP slow-start, which is the crucial
> phase when transmitting short web objects.

Ah yes, of course, there is a built in bandwidth throttle in the stack and
it is per TCP connection...

So basically we are looking at an end run round the TCP slow start which
works because the network stack congestion algorithm throttles bandwidth by
TCP connection and not per process.

I know that traditionally we have considered congestion control to be a per
connection thing rather than per process or per user. Those don't exactly
exist in the Internet model. But if we consider DDoS to be an extreme form
of congestion, we have to look at the O/S implementing broader rate
limiting anyway.

> The reason it makes a difference is that it is becoming clear that modern
> applications are not best served by an application API that is limited to
> one bi-directional stream. There are two possible ways to fix this
> situation. The first is to build something on top of TCP/IP the second is
> to replace single stream TCP with multi-stream.
>

Yes, I know we have proposals for both approaches. I am trying to see if
there is a case for one over the other.

If the case for multiple streams is better performance based on friendlier
slow start parameters, maybe these should be allowed without the end run.
If the net is surviving with people starting five streams instead of one,
maybe the slow start parameters could start at five packets per destination
host instead of one per connection. It would be a little more code to
implement but speed is hardly an issue where its purpose is throttling
anyway.

Making use of SCTP means that I have to rely on O/S implementation. Which
is a really long wait. So a pragmatic approach is always going to mean a
TCP+Multiplex option. If I am going to need that anyway, SCTP has to
deliver a benefit other than simplifying my code.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Feb 2015 16:23:43 -0500

On Wed, Feb 4, 2015 at 4:00 PM, Christopher Morrow <[email-masked]>
wrote:

> limiting

The reason I am doing this work is to try to work out what controls to
place

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 5 Feb 2015 09:30:52 -0500

> increases sub-linearly with the number of streams.

Buffer bloat is a great example of the consequences of the co-operative
nature
of the Internet for one of the audiences I am writing for (Folk trying to
make/understand Title II regulations).

The Internet is actually a demonstration that the commons are not such a
tragedy
after all.

 If we look at the problem of buffer bloat there are several possible
solutions
and the problem is picking one:

* Persuade manufacturers to reduce buffer sizes so the old congestion
algorithms work.

* Change the congestion algorithm.

* Break with the pure end to end principle and have the middleboxen with
the
huge buffers do some reporting back when they start to fill.

The first is difficult unless you get control of the benchmarking suites
that are
going to be used to measure performance. Which would probably mean getting
the likes of nVidia or Steam or some of the gaming companies on board.

The second is certainly possible. There is no reason that we all have to
use the
same congestion algorithm. In fact a mixture might be beneficial. Instead
of looking
for packets being dropped, the sender could look at the latency and the
difference
between the rate packets are being sent and the rate at which
acknowledgements
are being received.

Whether folk are willing to consider the third depends on how serious the
buffer bloat problem is considered to be. If the stability of the Internet
really
is at stake then everything should be on the table. But my gut instinct is
that
a middlebox does not actually have any more useful information available
to it than can be derived from the acknowledgements.

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 05 Feb 2015 14:01:50 -0500

On 2/5/2015 11:51 AM, Phillip Hallam-Baker wrote:
> 
> OK yes, My Netflix download is going to kill your VOIP call.
> 
> Can't fix the queuing algorithm for just one interaction...
> 
> 
> The reason I started pushing on this is that in the wake of Title II I
> am expecting a lot of people to be asking me to explain how the Internet
> worketh and this is precisely the sort of example that shows how (1)
> things are more complex than they appear, (2) how some sort of
> coordination is needed and (3) how the coordination needs to take less
> than five years to come to a decision.
> 

The AQM working group is attempting to do helpful things here, and
could certainly use more feedback on some of the drafts:
https://datatracker.ietf.org/wg/aqm/documents/

*Please* feel welcomed to comment on the drafts, as part of the
intention is to directly deal with the sorts of problems that have
been mentioned in this thread.

-- 
Wes Eddy
MTI Systems

---

From: Jeremy Harris <[email-masked]>
Date: Fri, 06 Feb 2015 11:20:28 +0000

> huge buffers do some reporting back when they start to fill.

Data Center TCP [1], which uses ECN, seems to be doing those.
I think it misnamed; it seems applicable over the wide area.

[1] http://research.microsoft.com/pubs/121386/dctcp-public.pdf
-- 
Cheers,
  Jeremy

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 6 Feb 2015 11:49:45 +0000

> I think it misnamed; it seems applicable over the wide area.

yeah, but the rub is that you'd need to know that ECN was available along the WAN path.

Lars

---

From: John Leslie <[email-masked]>
Date: Fri, 6 Feb 2015 07:39:06 -0500

Jeremy Harris <[email-masked]> wrote:
>> the huge buffers do some reporting back when they start to fill.

   Phillip has a good point, but states it poorly. There isn't, and
can never be, a "pure end to end principle" in congestion control.

   (Alas, some of us have been acting as if there was one.)

> I think it misnamed; it seems applicable over the wide area.

   No, it is not misnamed. It defined a "walled garden" where we were
able to escape the straitjacket that ECN is "the same as loss".

   That principle -- that ECN is fundamentally different from loss --
most certainly is applicable over the wide area.

--
John Leslie <[email-masked]>

---

From: "Black, David" <[email-masked]>
Date: Fri, 6 Feb 2015 14:12:35 +0000

And there's some subtlety in getting an AQM algorithm that works well when  both DCTCP  and ordinary TCP traffic share the same queue.  I've been hearing good things about recent research progress in this area.

Thanks, --David +++Sent from Blackberry

----- Original Message -----
From: Eggert, Lars [mailto:[email-masked]]
Sent: Friday, February 06, 2015 06:49 AM
To: Jeremy Harris <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [tsvwg] Naive question on multiple TCP/IP channels

Lars

---

From: John Leslie <[email-masked]>
Date: Fri, 6 Feb 2015 09:23:21 -0500

> yeah, but the rub is that you'd need to know that ECN was available

   Perhaps... I don't quite follow what Lars means by "along the WAN
path".

   DCTCP indeed assumes ECN is implemented in any forwarder along the path
which could become congested: thus it can depend upon ECN signals being
generated and propagated before the buffers fill.

   By the nature of the DCTCP environment, ECN signals will propagate
very quickly unless the buffers fill; thus the reaction to an ECN signal
can happen well before latency grows to be noticeable to humans. This
means that the reaction to a single ECN signal need not be drastic:
another will come along soon enough if a drastic reduction is needed.

   But this benefit doesn't depend on _every_ forwarder being ECN-aware:
it depends upon the bottleneck generating ECN signals and the other
forwarders forwarding the marked packets quickly.

   Fundamentally, DCTCP requires forwarders to ECN-mark all packets
which will experience time-in-queue which exceeds a small threshhold;
and requires endpoint receivers to keep track of _every_ ECN mark. The
reduction in sending rate depends upon the proportion of packets
arriving with ECN marks (which could approach 100% in severe cases).

   (DCTCP must change the mechanism by which a receiver echos the ECN
signals to the sender: the details are not important to this summary.)

   For a number of reasons, including bottlenecks which don't implement
ECN and Round-Trip-Times which exceed human-perceptible threshholds,
DCTCP does not appear to be appropriate in the big-I Internet. But the
principle that reaction to a single ECN signal need not always be as
severe as the reaction to a packet loss clearly deserves more work.

   IMHO...