Subject: [hybi] Call for interest: multiplexing dedicated for WebSocket

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 15 May 2013 20:45:28 +0900

Hi all,

(Note: This is not WG's official process)

I'd like to survey how many people are interested in standardization and
implementation of multiplexing dedicated for WebSocket again at this point.

* Multiplexing is still important for WebSocket

HyBi WG has been working on the WebSocket multiplexing spec (
http://tools.ietf.org/html/draft-ietf-hybi-websocket-multiplexing-10).
Multiplexing functionality is really important to make WebSocket scalable
enough to be used for large scale production services. It's still true.

* Rapid adoption of HTTP/2.0

While our work is delayed and delayed again, httpbis folks started
seriously working on next generation HTTP. Now it's actively standardized
as HTTP/2.0 based on Google's SPDY experiments and input from httpbis
members. Gradually but several web service providers already started
adopting HTTP/2.0.

* WebSocket/SPDY experiment

Chrome WebSocket team has been working on both WebSocket/SPDY and dedicated
multiplexing in parallel. You can find the WebSocket/SPDY proposal
incorporated into Roberto's unofficial draft (section 4) at
https://github.com/grmocg/SPDY-Specification/blob/gh-pages/draft-mbelshe-spdy-00.txt.
Its framing is not so efficient as the dedicated one, but can coexist with
HTTP traffic in one TCP connection and developers just need to understand
and implement HTTP/2.0 framing.

* Concentrate on the best one

Developing and maintaining two different multiplexing code will be burden
for vendors. It seems that we should stop and think which approach we
should focus on.

Thanks

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 15 May 2013 20:59:16 +0900

> for vendors. It seems that we should stop and think which approach we

I'm not saying we must abandon one of them.

If it turned out both are still considered useful and supported by certain
number of people, we can continue both.

---

From: Takeshi Yoshino <[email-masked]>
Date: Sun, 19 May 2013 13:30:36 +0900

I got one comment about this.

He/she suggested that we finish the WebSocket dedicated multiplexing
quickly even by simplifying the spec much e.g. cutting some functionality
in it that would take much time to get consensus, and when HTTP/2.0 is
matured, switch to that.

---

From: Simone Bordet <[email-masked]>
Date: Sun, 19 May 2013 20:37:13 +0200

Hi,

On Sun, May 19, 2013 at 6:30 AM, Takeshi Yoshino <[email-masked]> wrote:
> He/she suggested that we finish the WebSocket dedicated multiplexing quickly
> even by simplifying the spec much e.g. cutting some functionality in it that
> would take much time to get consensus, and when HTTP/2.0 is matured, switch
> to that.

My impression is that I have not seen much request for WebSocket multiplexing.
Given the complexities related to defining and implementing ws mux
correctly, and considering that HTTP/2.0 comes with mux built-in,
building on the SPDY experience, my preference would be to
deprioritize ws mux in favour of HTTP/2.0 integration.
It is true that there may be systems that may rely on a pure-ws
communication (and therefore do not need a full HTTP/2.0
implementation) and as such would benefit from ws mux, but my guess is
that those system will be a minority or can get by without ws mux. I
may be biased on this view though, so comments welcome.

Thanks !

--
Simone Bordet
----
http://cometd.org
http://webtide.com
http://intalio.com
Developer advice, training, services and support
from the Jetty & CometD experts.
Intalio, the modern way to build business applications.

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 20 May 2013 13:01:46 +0900

Thanks for comment, Simone.

On Mon, May 20, 2013 at 3:37 AM, Simone Bordet <[email-masked]> wrote:

> quickly

Yeah, more comments from those who are using / interested in pure-ws are
appreciated.

---

From: Bruce Atherton <[email-masked]>
Date: Tue, 21 May 2013 16:02:57 -0700

I agree. During the planning stages of the protocol there was a great 
deal of interest expressed in stand-alone WebSocket servers that were 
independent of any HTTP server. That has not been the real world 
deployment behaviour, at least in my experience. So long as HTTP/2.0 
doesn't introduce so much added complexity that it becomes too costly to 
include a server for it when you are deploying WebSockets, I don't see 
much benefit to a WebSockets-only mux definition any more, much though I 
appreciate the effort that Takeshi and John have put into it.

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 22 May 2013 03:01:13 -0700

Am 15.05.2013 13:45, schrieb Takeshi Yoshino:> Hi all,
> implementation of multiplexing dedicated for WebSocket again at this point.

+1 for having WS mux and not depend on HTTP 2.0 / SPDY.

> HyBi WG has been working on the WebSocket multiplexing spec 
> (http://tools.ietf.org/html/draft-ietf-hybi-websocket-multiplexing-10). 
> Multiplexing functionality is really important to make WebSocket 
> scalable enough to be used for large scale production services. It's 
> still true.

Could you eloborate a bit on this? I'd like to better understand "the problem". As far as I can see, the current RFC text contains the following motivation for multiplexing:

"""
The WebSocket Protocol [RFC6455] requires a new transport connection
for every WebSocket connection.  This presents a scalability problem
when many clients connect to the same server, and is made worse by
having multiple clients running in different tabs of the same user
agent.
"""

What exactly is the scalability problem? Is it: scalability of a single server box is limited by the number of concurrently active TCP connections the server can handle? I.e. one is be able to handle 200k TCPs on a single box, but not 20Mio ..

If so, is the thinking that a) one needs some kind of "network concentrator" front-end servers that aggregate the many TCPs onto a few, and b) for doing that kind of concentration, WS multiplexing is one option?

However, concentration could also be done at the application level, e.g. stateless frontend server speaking WS with some app defined WS subprotocol to clients, and native database protocol to backend servers, multiplexing requests contained in the WS-based app protocol to requests on the few database protocol connections to backend servers.

That can also solve the scalability problem without requiring multiplexing at the WS level. So why do we need it at the WS level?

Sure, multiplexing at the WS level would allow to create generic concentrators, but a generic concentrator could not map incoming connections to backend boxes based on app data like UserID etc for sharding etc.

Another aspect is client-server transport efficiency.

Most efficient: exactly 1 TCP connection between given client device and server for any communication between the two.

The use of only 1 TCP connection between a given client device and a given server allows mechanisms like TCP slow start etc work efficiently.

A browser with multiple tabs opened to the same origin might be a common situation which could be handled by multiplexing in the networking code inside the browser.

Yet another option with browser could be a shared web worker opening 1 WS to the server, and have all tabs communicate only via that web worker.

However, when running _different_ apps on the same device connecting to the same origin, the multiplexing needed to happen cross-app, and hence the multiplexing needed to happen in kernel space or in a user space library used by all apps on the client device platform.

==

So my current thinking is:

- the "1 TCP for efficiency" problem can not be solved at the app layer, but needs multiplexing at WS (or within SPDY), which then solves the problem - but only partially (not across different apps, at least not without kernel-space implementation). The problem can be solved however with browsers and shared web workers also.

- the scalability problem can be solved both at the app layer or the WS/SPDY layer.

> seriously working on next generation HTTP. Now it's actively 
> standardized as HTTP/2.0 based on Google's SPDY experiments and input 
> from httpbis members. Gradually but several web service providers 
> already started adopting HTTP/2.0.
> 
> Chrome WebSocket team has been working on both WebSocket/SPDY and 
> dedicated multiplexing in parallel. You can find the WebSocket/SPDY 
> proposal incorporated into Roberto's unofficial draft (section 4) at 
> Its framing is not so efficient as the dedicated one, but can coexist 
> with HTTP traffic in one TCP connection and developers just need to 
> understand and implement HTTP/2.0 framing.

"""
4.2.1. Frame mapping

   One WebSocket frame is mapped into one proceeding SPDY HEADERS frame
   and following plural SPDY data frames. The SPDY HEADERS frame must
   contain WebSocket frame fields and following SPDY data frames contain
   payload data. These data frames are free to be reframed.

   This plan doesn't care about overheads.
"""

This seems somehow straightforward, but introduces inefficiencies: the overhead per WS message is significantly increased and an additional framing layer that allows reframing costs CPU cycles.

Btw: assuming SPDY runs over TLS exclusively, is there any need for WS frame masking anymore?

> Developing and maintaining two different multiplexing code will be 
> burden for vendors. It seems that we should stop and think which 
> approach we should focus on.

Yes, but WS over SPDY is not ready today. WS is widely implemented and works today.

As an app developer, the only thing I care about is having a message-based, reliable, ordered, full-duplex communication with browsers that is both efficient and widely implemented.

Personally I don't care a lot about improving HTTP, since with single-page web apps (and manifests), app assets would be in browser cache anyway. So I care about pure WS solutions.

Stacking framing onto framing layers or WS on SPDY in general does not seem to improve efficiency or implementation availability in browsers. It's a step backward from my POV.

So yes, I do care about WS multiplexing.

---

From: "Arman Djusupov" <[email-masked]>
Date: Wed, 22 May 2013 16:29:33 +0300

Sorry, just realized that my message was sent to Takeshi privately by
mistake a few days ago.. (wrong button). It was actually intended for the
list so I am repeating it publicly now:

Obviously continuing to develop a hybi-mux spec for another year or two
doesn't make sense. Taking into account that new specifications are coming
and that HTTP2.0 will probably provide multiplexing for both WebSocket and
HTTP traffic, at  this stage it would only be worth investing development
time in a "simple and ready to go" hybi-mux specification. Maybe if we
simplify the spec to rock bottom simplicity, even sacrificing some features
so that we can conclude it very fast, then maybe vendors might pick it up.
This way we will give to WebSocket its own mux for now, while the HTTPbis
draft might take much longer to form into the final specification. In any
case, hybi-mux must be reviewed and finalized in a very short time, plus it
should be simpler and easier to implement than both SDPY and HTTPbis based
multiplexing.

With best regards,

Arman

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Takeshi Yoshino
Sent: Sunday, May 19, 2013 7:31 AM
To: [email-masked]
Subject: Re: [hybi] Call for interest: multiplexing dedicated for WebSocket

---

From: Salvatore Loreto <[email-masked]>
Date: Wed, 22 May 2013 17:30:57 +0300

<as individual>
I tend to agree with the fact that if we want standardize mux in 
WebSocket we should speed up simplifying the spec.

/Salvatore

On 5/22/13 4:29 PM, Arman Djusupov wrote:
> mistake a few days ago.. (wrong button). It was actually intended for 
> the list so I am repeating it publicly now:
>
> Obviously continuing to develop a hybi-mux spec for another year or 
> two doesn't make sense. Taking into account that new specifications 
> are coming and that HTTP2.0 will probably provide multiplexing for 
> both WebSocket and HTTP traffic, at  this stage it would only be worth 
> investing development time in a "simple and ready to go" hybi-mux 
> specification. Maybe if we simplify the spec to rock bottom 
> simplicity, even sacrificing some features so that we can conclude it 
> very fast, then maybe vendors might pick it up. This way we will give 
> to WebSocket its own mux for now, while the HTTPbis draft might take 
> much longer to form into the final specification. In any case, 
> hybi-mux must be reviewed and finalized in a very short time, plus it 
> should be simpler and easier to implement than both SDPY and HTTPbis 
> based multiplexing.
>
> With best regards,
>
> Arman
>
> *From:[email-masked] [mailto:[email-masked]] *On Behalf 
> Of *Takeshi Yoshino
> *Sent:* Sunday, May 19, 2013 7:31 AM
> *To:* [email-masked]
> *Subject:* Re: [hybi] Call for interest: multiplexing dedicated for 
> WebSocket
>
> quickly even by simplifying the spec much e.g. cutting some 
> functionality in it that would take much time to get consensus, and 
> when HTTP/2.0 is matured, switch to that.
>
>
>
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 22 May 2013 17:12:53 +0200

> WebSocket we should speed up simplifying the spec.

+1 for the simplest possible mux that does the core job.

However, I currently think that having a per-logical channel flow 
control is required to make it useful (cannot be dropped for the sake of 
simplification). I might be wrong.

So the practical question: in what areas could we simplify the current 
mux spec?

>> should be simpler and easier to implement than both SDPY and HTTPbis

Yep, I agree.

---

From: Salvatore Loreto <[email-masked]>
Date: Wed, 22 May 2013 18:23:44 +0300

<as individual>

my concerns are just about the importance of WebSocket mux once HTTP/2.0 
will be finalized
also considering the fact that the HTTP/2.0 standardization is 
proceeding quite fast and it is supposed
to be finalized in one year and something

/Salvatore

On 5/22/13 6:12 PM, Tobias Oberstein wrote:
> control is required to make it useful (cannot be dropped for the sake 
> of simplification). I might be wrong.
>

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 22 May 2013 17:31:56 +0200

> to be finalized in one year and something

I would prefer a pure WS protocol (including mux and compression) _not_ 
layered on top of HTTP/2.0 as a long-term, widely implemented solution.

This is driven by our primary use case: have single-page web apps (with 
cached assets) that talk to backends exclusively over WS.

Layering WS on top of HTTP/2.0 does add additional overhead without 
gain, and for retrieving HTML etc assets that are persistently cached in 
browser (Web app manifests etc), HTTP/2.0 does not seem to add a lot 
over HTTP/1.1 also.

So personally I'd be fine with just WS and HTTP/1.1.

/Tobias

---

From: Joakim Erdfelt <[email-masked]>
Date: Wed, 22 May 2013 08:33:14 -0700

+1 for simplification.

I'll toss some topics onto the table now:
 * AddChannelRequest - eliminate header deltas
 * AddChannelResponse - eliminate header deltas
 * NewChannelSlot - eliminate the "channel slots" management entirely from
the spec, go back to the earlier, easier to implement, concepts from past
drafts.
 * Simplify the extension rules.
     Extensions after initial physical connection are applied only to first
logical channel.
     in AddChannelRequest, extensions provided after the mux extension are
used for logical channels.

--
Joakim Erdfelt <[email-masked]>
webtide.com <http://www.webtide.com/>
Developer advice, services and support
from the Jetty & CometD experts
eclipse.org/jetty - cometd.org

On Wed, May 22, 2013 at 6:29 AM, Arman Djusupov <[email-masked]> wrote:

> list so I am repeating it publicly now:****
>
> ** **
>
> ** **
>
> doesn’t make sense. Taking into account that new specifications are coming
> time in a “simple and ready to go” hybi-mux specification. Maybe if we
> multiplexing.****
> ** **
> With best regards,****
> Arman****
> ** **
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Takeshi Yoshino
> WebSocket****
> ** **
> I got one comment about this.****
>
> ** **
>
> matured, switch to that.****
>
>  ****
>

---

From: "John A. Tamplin" <[email-masked]>
Date: Wed, 22 May 2013 12:26:51 -0400

On Wed, May 22, 2013 at 11:12 AM, Tobias Oberstein <
[email-masked]> wrote:

> However, I currently think that having a per-logical channel flow control
> is required to make it useful (cannot be dropped for the sake of

I agree -- without flow control, there is very little benefit, and almost
nothing that an app couldn't easily do themselves by just adding a simple
framing layer inside WS messages (perhaps just the first byte is a logical
channel number).

-- 
John A. Tamplin

---

From: Simone Bordet <[email-masked]>
Date: Wed, 22 May 2013 18:48:43 +0200

Hi,

> framing layer inside WS messages (perhaps just the first byte is a logical

In SPDY, the flow control had an initial window of 64k. This proved way too
small for uploads/downloads, which resulted in a 3x or more slowdown.
So there needs to be in ws a mechanism to configure this window (which SPDY
has).
Turns out that now every client sends a huge initial window, so downloads
are now fast.
This of course impacts memory, since flow control is about memory, but
pretty much solved the flow control problem.

The next problem that arose was that now a stream/channel can take the
whole bandwidth and squat others channels.
So there is the need for a further flow control mechanism to prioritize
channels, that must be dynamic (i.e. changed during the transmissions of
frames).
SPDY is still discussing this, I am guessing it's also being discussed in
HTTP 2.0.

So "simple mux" looks like an oxymoron to me, and makes me wonder what
useful could be a ws mux solution that does not do dynamic channel
prioritization.
A typical use case Joakim presented to me was to tune video download rate
in a websocket server: a big priority initially to download faster than the
client video player plays, to make some buffer, and then reducing the
priority to a steady state with the video player's speed, or reducing
priority to almost zero if the user pauses the play.

---

From: Zhong Yu <[email-masked]>
Date: Wed, 22 May 2013 12:13:39 -0500

> framing layer inside WS messages (perhaps just the first byte is a logical
In traditional socket programming model, the application chooses the time
and the speed to perform read(). If the application is not reading fast
enough, the receive buffer is full and more incoming data are not welcome.

Contrary to that, most WebSocket API are event based - all incoming message
data are buffered without any blocking, then the complete message is sent
to the application callback. In this model, per channel flow control
doesn't seems to exist.

If per channel flow control is removed from the spec, what use cases would
suffer?

--
And I have a stupid question - what's the motivation to do mux on
WebSocket? To save local port numbers? To avoid slow start? If we drop mux
all together, what use cases would suffer?

Zhong Yu

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 22 May 2013 19:39:51 +0200

Am 22.05.2013 18:26, schrieb John A. Tamplin:
> On Wed, May 22, 2013 at 11:12 AM, Tobias Oberstein
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     control is required to make it useful (cannot be dropped for the
>     sake of simplification). I might be wrong.
>
>
> I agree -- without flow control, there is very little benefit, and
> almost nothing that an app couldn't easily do themselves by just adding
> a simple framing layer inside WS messages (perhaps just the first byte
> is a logical channel number).

Regarding implementing flow control above TCP (and below app), I found this:

"Multiplexing on top of TCP in overall fails to deliver the advantages 
it is assumed to provide."

Source: http://www.250bpm.com/multiplexing

Do you have an opinion on that?

More important: what problem exactly does multiplexing solve? The 
current mux RFC mentions "scalability", but doesn't go into detail ..

/Tobias

---

From: "John A. Tamplin" <[email-masked]>
Date: Wed, 22 May 2013 14:32:22 -0400

On Wed, May 22, 2013 at 1:39 PM, Tobias Oberstein <
[email-masked]> wrote:

> "Multiplexing on top of TCP in overall fails to deliver the advantages it
> is assumed to provide."
>
> Source: http://www.250bpm.com/**multiplexing<http://www.250bpm.com/multiplexing>
>

One thing he doesn't address are resources used to keep lots of connections
open, both at the OS level and in intermediate network nodes (such as load
balancers -- try running tons of long-lived connections through an F5 for
example).  Also, he doesn't address the interaction of TCP slow start with
lots of new connections.  I also don't believe the equivalent of a full TCP
handshake is required to allocate buffer space.

> More important: what problem exactly does multiplexing solve? The current
> mux RFC mentions "scalability", but doesn't go into detail ..
>

Here is a use-case I envision:  Imagine GMail, G+ sandbar, etc all use
WebSockets heavily.  Many users may have multiple copies of such apps
running in tabs or windows in the browser, and some of these are going to
be separate (such as GMail and the G+ sandbar probably are separate apps
running on the same page and maintain their own connections to their
backends).  Currently, that would mean you have a ton of TCP connections
open to the server.  Servers don't handle tons of long-lived connections
very well (or intermediate load balancers), so you would greatly prefer to
have one connection per browser.  WS MUX lets you do that, by having the
browser aggregate connections to the same backend.  You could do some of it
at the app level, but it is going to be complicated and in many cases these
apps won't all be from the same vendor (for example, some widget on the
page that uses the GData API) and will have separate deploy schedules.

-- 
John A. Tamplin

---

From: Zhong Yu <[email-masked]>
Date: Wed, 22 May 2013 14:00:09 -0500

On Wed, May 22, 2013 at 1:32 PM, John A. Tamplin <[email-masked]> wrote:

> One thing he doesn't address are resources used to keep lots of
> connections open, both at the OS level and in intermediate network nodes
> (such as load balancers -- try running tons of long-lived connections
> through an F5 for example).  Also, he doesn't address the interaction of
> TCP slow start with lots of new connections.  I also don't believe the
> equivalent of a full TCP handshake is required to allocate buffer space.
>
>
> open to the server.

> Servers don't handle tons of long-lived connections very well
>

Is that still true today? Recent server implementations all boast they can
handle tons of connections.

> (or intermediate load balancers), so you would greatly prefer to have one
> connection per browser.  WS MUX lets you do that, by having the browser
> aggregate connections to the same backend.  You could do some of it at the
> app level, but it is going to be complicated and in many cases these apps
> won't all be from the same vendor (for example, some widget on the page
> that uses the GData API) and will have separate deploy schedules.
>
> --
> John A. Tamplin
>

---

From: Bruce Atherton <[email-masked]>
Date: Wed, 22 May 2013 12:31:26 -0700

On 22/05/2013 12:00 PM, Zhong Yu wrote:
>
> On Wed, May 22, 2013 at 1:32 PM, John A. Tamplin <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> Is that still true today? Recent server implementations all boast they 
> can handle tons of connections.
>

Tons of short, HTTP Request/Response type connections. John is talking 
about long-lived WebSockets connections, the kind that might hang around 
for days, weeks, even months. It is a different problem.

Think of it this way. Say your server can handle 10K simultaneous 
connections to serve web pages and it takes about 330ms to serve them. 
If every one of those opens a websocket connection, you'll need 30K 
WebSocket connections just to handle a single second of traffic. If on 
average the page is kept open for only an hour, that is 108M connections.

---

From: Simone Bordet <[email-masked]>
Date: Wed, 22 May 2013 22:41:28 +0200

Hi,

>> Is that still true today? Recent server implementations all boast they can handle tons of connections.
>
>On Wed, May 22, 2013 at 9:31 PM, Bruce Atherton <[email-masked]> wrote:
> Tons of short, HTTP Request/Response type connections. John is talking about long-lived WebSockets connections, the kind that might hang around for days, weeks, even months. It is a different problem.

Well I am not sure.
I have seen the TCP Linux stack collapsing due to usage of HTTP 1.0 by
a proxy. Opening and closing connections at a high rate it's a killer.
Long lived connections, on the other hand, not so problematic, at
least in my experience, so Zhong Yu's question still stand for me.

Another point not addressed by the article linked by Tobias is the
type of usage of the protocol.
The HTTP protocol has a typical 1 request followed by 10-200 requests
in a burst, that can be performed concurrently.
In a non-mux protocol (like HTTP/1.1) browsers open 6 connections, so
to serve the burst you need multiple "passes" at using those 6
connections, which leads to the well known ziggurat profile: 6 out,
pause, 6 out, pause, etc.
In SPDY, which is muxed, the 100 requests go out at once, with a flat profile.
Roberto Peon and us (Jetty) have showed demos where this effect is
*eye visible*. SPDY is faster than HTTP just because of mux
(http://webtide.intalio.com/2012/10/spdy-push-demo-from-javaone-2012/).

Now, HTTP has a semantic which benefits clearly from multiplexing.
Does this hold true for WebSocket, which does not have semantic, but
just framing ? I think it depends.
If you use WebSocket to transport HTTP (i.e. the opposite that
HTTP/2.0 is working on) then perhaps mux would be very important.
But if you use WebSocket for chat messaging, perhaps that's not that
important to have mux.
The difference here is between a request/response model versus a
messaging model.
The model changes the need for mux.

Since WebSocket does not define a model (just framing), perhaps mux is
less important.
Application protocols may implement mux if they need to.

> Think of it this way. Say your server can handle 10K simultaneous connections to serve web pages and it takes about 330ms to serve them.
> If every one of those opens a websocket connection, you'll need 30K WebSocket connections just to handle a single second of traffic.
> If on average the page is kept open for only an hour, that is 108M connections.

That's under the assumption that WebSocket connections are closed
after a "request", which is wrong.
WebSocket connections are persistent.
If you want to serve HTTP resources via WebSocket, you better open 1
connection per client (like SPDY does), and your server will only be
able to handle 10k clients at any time.
But then you'll need mux, or you go back to HTTP/1.1, just with a
different framing (WebSocket).

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 22 May 2013 22:44:29 +0200

Am 22.05.2013 20:32, schrieb John A. Tamplin:
> On Wed, May 22, 2013 at 1:39 PM, Tobias Oberstein
>     "Multiplexing on top of TCP in overall fails to deliver the
>     advantages it is assumed to provide."
>
>     Source: http://www.250bpm.com/__multiplexing
>     <http://www.250bpm.com/multiplexing>
>

He does not address intermediaries, yes. But he does address memory 
consumption at endpoints: when running multiple TCPs between a given 
pair of endpoints, send/receive buffers reside in kernel space, whereas 
multiplexing over a single TCP requires send/receive buffers (per 
stream) in user space (to avoid TCP pushback). Hence he calls it a draw.

However, thinking about it: having send/receive buffers in kernel space 
requires one to decide how big buffers should be for the complete 
machine (TCP send / receive buffer size is usually a kernel tunable).

Having those buffers in user space (in app) allows one to tune the 
buffer size _per stream_. Thus, we can have large buffers for streams 
that carry mass data, and small buffers for chatty streams. Thats a real 
advantage.

Rgd load balancers: yep, L7 LBs will have a problem with massive numbers 
of connections. But L4 LBs that e.g. distribute connections to backend 
cluster nodes based on consistent hash of source IP not .. since they 
dont need to maintain state per connection. A L4 LB should be able to 
scale to millions of TCPs.

> equivalent of a full TCP handshake is required to allocate buffer space.

So to make the proposed advantage explicit: the per-stream handshaking 
that happens with mux consumes less resources than full TCP handshakes.

> running in tabs or windows in the browser, and some of these are going
> to be separate (such as GMail and the G+ sandbar probably are separate
> apps running on the same page and maintain their own connections to
> their backends).  Currently, that would mean you have a ton of TCP
> connections open to the server.  Servers don't handle tons of long-lived

How about having a shared Web worker in the background that handles all 
messaging for the app over a single WS?

> connections very well (or intermediate load balancers), so you would

Async servers running on top of modern APIs like epoll/kqueue/IOCP have 
no issue with massive numbers of long-lived connections. I have run 180k 
WebSocket connections on a single-threaded server running on a 2 core 
4GB Ram VM (with small TCP send/receive buffers configured).

> greatly prefer to have one connection per browser.  WS MUX lets you do
> that, by having the browser aggregate connections to the same backend.
>   You could do some of it at the app level, but it is going to be
> complicated and in many cases these apps won't all be from the same
> vendor (for example, some widget on the page that uses the GData API)
> and will have separate deploy schedules.

I agree, MUX at app level is probably less than ideal.

---

From: Tobias Oberstein <[email-masked]>
Date: Wed, 22 May 2013 22:49:02 +0200

> A typical use case Joakim presented to me was to tune video download
> rate in a websocket server: a big priority initially to download faster
> than the client video player plays, to make some buffer, and then
> reducing the priority to a steady state with the video player's speed,
> or reducing priority to almost zero if the user pauses the play.

Interesting. And yep, being able to dynamically tune the priority of a 
stream from the app seem desirable. The buffer/window size in this use 
case would probably be relatively large and fixed.

May I add another user experience centric use case where an app might 
want to tune both priority and buffer size?

Say I wanna run something like chat while also doing file transfers over 
WS. Different QoS. I don't want to increase the latency on the chatty 
stream by running a mass data stream. However, this probably requires 
both flow control _and_ message/frame priorization.

The current mux RFC reads:

" The sender MUST use a fair mechanism for selecting which logical
channel's data to send in the next WebSocket message.  Simple
implementations may choose a round-robin scheduler, while more
advanced implementations may adjust priority based on the amount
or frequency of data sent by each logical channel.
"

This seems to imply that the app has no control over QoS per stream 
(i.e. "chatty" vs "mass data").

IMHO the app should be able to control that (ultimately effecting buffer 
sizes and priorization/scheduling). If the app has no control chances 
are that people will invent something at the app level. IOW: this is 
something where MUX could really deliver added value (better UX), even 
if you are not a big G and concerned about infrastructure resource costs 
and world scale apps.

Do you think the current MUX addresses a chat+file transfer use case?

What about the proposed SPDY/HTTP2.0 multiplexing?

Are there even any plans for exposing tunables to apps?

/Tobias

---

From: Zhong Yu <[email-masked]>
Date: Wed, 22 May 2013 17:02:14 -0500

> The model changes the need for mux.

I think the main difference is that, a WebSocket channel is a meaningful
identity that cannot be reduced, so there must be an overhead associated
with each live channel; mux cannot change that fact. On the other hand,
HTTP messages have no logical relationship to HTTP connections, it doesn't
matter which connection a message is delivered on, therefore it's possible
to reduce HTTP connections to remove overhead.

Zhong Yu

> WebSocket connections just to handle a single second of traffic.

---

From: "Arman Djusupov" <[email-masked]>
Date: Thu, 23 May 2013 12:09:30 +0300

Agree on channel slot, header deltas, and simplified extension rules.

Concerning the header deltas we can simplify them by applying the
permessage-deflate compression on the entire control channel# 0. It will be
almost like SPDY header compression, only better as it will be applied on
the entire control channel payload. Since we have the permessage-deflate
spec almost ready why not put it to good use? Compressed or not compressed
the control channel could be a simple flag in mux settings.

With best regards,

Arman

From: Joakim Erdfelt [mailto:[email-masked]] 
Sent: Wednesday, May 22, 2013 6:33 PM
To: Arman Djusupov
Cc: Takeshi Yoshino; [email-masked]
Subject: Re: [hybi] Call for interest: multiplexing dedicated for WebSocket

+1 for simplification.

 

--

eclipse.org/jetty <http://eclipse.org/jetty/>  - cometd.org
<http://cometd.org/> 

 

With best regards,

Arman

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 May 2013 16:29:57 +0900

On Wed, May 22, 2013 at 7:01 PM, Tobias Oberstein <
[email-masked]> wrote:

> Sure, multiplexing at the WS level would allow to create generic
> concentrators, but a generic concentrator could not map incoming
> connections to backend boxes based on app data like UserID etc for sharding
> etc.
>

Right

> However, when running _different_ apps on the same device connecting to
> the same origin,

You meant same ws:// address?

4.2.1. Frame mapping
> This seems somehow straightforward, but introduces inefficiencies: the
> overhead per WS message is significantly increased and an additional
> framing layer that allows reframing costs CPU cycles.
>

Thanks

> Btw: assuming SPDY runs over TLS exclusively, is there any need for WS

See this post by Eric about BEAST attack. I think it's not easy to conclude
that mask is unncessary for wss.
http://www.educatedguesswork.org/2011/09/security_impact_of_the_rizzodu.html

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 May 2013 16:45:09 +0900

On Thu, May 23, 2013 at 12:33 AM, Joakim Erdfelt <[email-masked]> wrote:

>      Extensions after initial physical connection are applied only to
> first logical channel.

I'm fine with removing delta encoding though I don't think we need to stop
removing headers that don't make sense for logical channel.

      *  "Sec-WebSocket-Extensions" in an AddChannelRequest is applied to
the multiplexed connection.
      *  A client doesn't include the Upgrade and Sec-WebSocket-Accept
header as they don't make sense for multiplexed connection.

Do you prefer sending whole headers as if it's sent without mux? I agree
that we don't have to worry about any change on Upgrade and
Sec-WebSocket-Accept in the future if we send all as-is, so no strong
objection.

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 May 2013 17:25:18 +0900

On Thu, May 23, 2013 at 1:48 AM, Simone Bordet <[email-masked]> wrote:

> In SPDY, the flow control had an initial window of 64k. This proved way
> too small for uploads/downloads, which resulted in a 3x or more slowdown.
> So there needs to be in ws a mechanism to configure this window (which
> SPDY has).
>

NewChannelSlot can do this. I don't think adding SETTINGS is simpler than
NewChannelSlot.

Instead of dropping NewChannelSlot, I'd rather suggest dropping "N mux
control in 1 message" format, "Handshake size" field in AddChannelRequest,
AddChannelResponse, "Reason size" in DropChannel.

> priority to almost zero if the user pauses the play.

Isn't it in-app priority? Can't script and server address that problem? If
only the client knows current priority, the client can tell the priority to
the peer servlet using a WebSocket message. If the server know priority for
each position, it can just control transmission speed.

Even if we need help by the server, the servlet would call some special
function to tell the server about priority.

Different from HTTP/2.0's case, we don't need to provide a special field in
mux protocol to pass priority, I think.

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 May 2013 17:33:49 +0900

On Thu, May 23, 2013 at 2:13 AM, Zhong Yu <[email-masked]> wrote:

> Contrary to that, most WebSocket API are event based - all incoming
> message data are buffered without any blocking, then the complete message
> is sent to the application callback. In this model, per channel flow
> control doesn't seems to exist.
>

It's possible that one JavaScript takes longer time to process each message
than another.

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 May 2013 17:34:41 +0900

On Thu, May 23, 2013 at 5:49 AM, Tobias Oberstein <
[email-masked]> wrote:

> This seems to imply that the app has no control over QoS per stream (i.e.
> "chatty" vs "mass data").
>
> sizes and priorization/scheduling). If the app has no control chances are
> that people will invent something at the app level. IOW: this is something
> where MUX could really deliver added value (better UX), even if you are not
> a big G and concerned about infrastructure resource costs and world scale
> apps.
>

I'll fix the text. It was written considering only environments that runs
scripts without any knowledge about them such as importance, dependency,
etc.

---

From: Simone Bordet <[email-masked]>
Date: Mon, 27 May 2013 10:47:32 +0200

Hi,

> mux protocol to pass priority, I think.

Sure, but then you force every application to reinvent this mechanism
on top of WebSocket mux.
At that point, one could argue that application messages could have
their own "channel_id" in their messages, and applications can do mux
themselves, if they need it.

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 May 2013 17:51:53 +0900

On Thu, May 23, 2013 at 5:44 AM, Tobias Oberstein <
[email-masked]> wrote:

> Having those buffers in user space (in app) allows one to tune the buffer
> size _per stream_. Thus, we can have large buffers for streams that carry
> mass data, and small buffers for chatty streams. Thats a real advantage.
>

Exactly.

> messaging for the app over a single WS?

One restriction of Web Worker is that we cannot share a worker between
webapps with different origins while WebSocket can connect to a host
different from its origin.

For load balancer vendors, it's better that multiplexing is standardized
than each service provider develops its own.

Basically multiplexing should be done as close to network as possible. If
it's done in application layer, for example when compression is used, such
a load balancer need to decode the compression to see the multiplexing
information. That doesn't make them happy.

--

As Google, our motivation of mux is to make sure that total meaningful
traffic per server cost (memory, port, machine, etc.) small enough compared
to HTTP/Comet approach.

---

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 27 May 2013 18:03:19 +0900

> themselves, if they need it.

Oh, ... good point.

If there gonna be inter-app prioritization (then load balancer would want
to look at ToS), frequent priority change (performance matters. better than
JS level processing), and wide adoption of priority (standardized method is
preferred), right.

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 27 May 2013 04:59:09 -0700

>>However, when running _different_ apps on the same device connecting to the same origin,

>You meant same ws:// address?
 
Yes, sorry not origin.

I had a look into the MUX RFC again .. couldn't find anser to the following (may have missed sth):

What WS addresses would be eligible to be multiplexed over a single WS connection?

a) ws://somehost.com:999/
b) ws://somehost.com:999/foo
c) ws://somehost.com:999/bar

d) wss://somehost.com:999/
e) wss://somehost.com:999/foo
f) wss://somehost.com:999/bar

All of a) - f) are to the same target IP:port and hence could share the same TCP.

However, d) - f) use wss, and hence have a TLS handshake right after TCP establishment.

So d) - f) cannot be multiplexed over the same physical WS as a) - c)?

Or can an implementation just "silently" transport a)-c) also over wss, and hence multiplex all of a) - f) over 1 physical WS?

Lastly, a)-c) are to the same target IP:port and also WS schema (ws, not wss) - and hence can be multiplexed over 1 physical WS even though they are to different URL paths?

IOW: in the context of ws-mux, what is "same target"?

The combination of IP, port and WS schema (but not necessarily same URL path)?

>See this post by Eric about BEAST attack. I think it's not easy to conclude that mask is unncessary for wss.
 
Ok, I see. Non-trivial.

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 27 May 2013 05:51:51 -0700

>>Having those buffers in user space (in app) allows one to tune the buffer size _per stream_. Thus, we can have large buffers for streams that carry mass data, and small buffers for chatty streams. Thats a real >>advantage.

>Exactly.

In the meantime, I have read more about modern TCP/IP stacks.

E.g. beginning with FreeBSD 7, the TCP/IP stack auto-tunes the TCP send/receive buffer sizes _per socket_. This is controlled via:

net.inet.tcp.sendbuf_auto=1         # TCP send buffer size autotuning (default: on)
net.inet.tcp.sendbuf_inc=16384      # TCP send buffer autotuning step size
net.inet.tcp.recvbuf_auto=1         # TCP receive buffer size autotuning (default: on)
net.inet.tcp.recvbuf_inc=524288     # TCP receive buffer autotuning step size
net.inet.tcp.sendbuf_max=16777216   # TCP send buffer maximum size (tune up for Fat-Long-Pipes)
net.inet.tcp.recvbuf_max=16777216   # TCP receive buffer maximum size (tune up for Fat-Long-Pipes)

"""
TCP Autotuning

Beginning with Linux 2.6, Mac OSX 10.5, Windows Vista, and FreeBSD 7.0, both sender and receiver autotuning became available, eliminating the need to set the TCP send and receive buffers by hand for each path. However the maximum buffer sizes are still too small for many high-speed network path, and must be increased as described on the pages for each operating system.
"""
http://fasterdata.es.net/host-tuning/background/

Apart from autotuning, there is also the option to control send/receive buffer sizes _per socket_ using setsockopt() with SO_SNDBUF / SO_RCVBUF.

===

Given above, how would the following 2 approaches compare?

a)
- Client 1 <=> Server 1 connected with 2 TCP connections.
- TCP autotuning on both client and server OR manually set buffer sizes: small for TCP-1, and large for TCP-2
- First TCP _only_ carries "chatty" traffic, and second TCP carries "mass data" traffic.

b)
- Client 1 <=> Server 1 connected with 1 TCP connection
- TCP autotuning OR large buffers set system wide
- multiplexing done over the single TCP that carries both "chatty" and "mass data" traffic and
- the multiplexing is further optimizing buffer size (app level) and does priorization of traffic (chatty = high-prio, mass = low-prio):

"Compare": I am mostly interested in the end-user experience:

i) Is the chatty traffic still low-latency, even in the presence of concurrent mass-data traffic?
ii) Is the throughput of mass-data traffic concurrently to the chatty traffic still (nearly) as high as when only doing mass data transfer?
 
>>How about having a shared Web worker in the background that handles all messaging for the app over a single WS?
>One restriction of Web Worker is that we cannot share a worker between webapps with different origins while WebSocket can connect to a host different from its origin.

Ah, right. Same origin policy. Thus, with WS-MUX, there is _more_ potentially TCP sharing than whats possible with shared WebWorkers.

However, for a given single app (from a single origin) running in multiple browser tabs, the shared WebWorker seems not so bad. Works today. No need for WS-MUX or HTTP 2.0 MUX + WS over HTTP 2.0.

>For load balancer vendors, it's better that multiplexing is standardized than each service provider develops its own.

I don't have experience running G scale infrastructure. However, I'd like to understand why LBs need to have a look into WS at all .. why not Layer-4 LB? Doing balancing based on hash of source IP?

>Basically multiplexing should be done as close to network as possible. If it's done in application layer, for example when compression is used, such a load balancer need to decode the compression to see the >multiplexing information. That doesn't make them happy.

This seems to assume that 

- either the LB is done above layer 4 and/or
- the logical WS connections contained in the physical WS would need to be balanced to _different_ backend nodes.

>As Google, our motivation of mux is to make sure that total meaningful traffic per server cost (memory, port, machine, etc.) small enough compared to HTTP/Comet approach.
 
I see. For me, I am mostly interested in the use case: 1 app doing both chatty and mass-data coms.

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 28 May 2013 15:22:39 +0900

> following (may have missed sth):

It's not yet specified.

> All of a) - f) are to the same target IP:port and hence could share the

I think it's fine.

> However, d) - f) use wss, and hence have a TLS handshake right after TCP

Right unless TLS parameters differ (such as client cert) between paths.

See also SPDY's CREDENTIAL frame.
https://github.com/grmocg/SPDY-Specification/blob/gh-pages/draft-mbelshe-spdy-00.txt

> So d) - f) cannot be multiplexed over the same physical WS as a) - c)?

It's fine and up to the client.

> Or can an implementation just "silently" transport a)-c) also over wss,
> and hence multiplex all of a) - f) over 1 physical WS?
>
> Lastly, a)-c) are to the same target IP:port and also WS schema (ws, not
> wss) - and hence can be multiplexed over 1 physical WS even though they are
> to different URL paths?
>

Basically ip:port. We don't need to isolate them based on hostname, I
believe, unless it's required by some security/privacy policy.

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 29 May 2013 14:43:11 +0900

On Mon, May 27, 2013 at 9:51 PM, Tobias Oberstein <
[email-masked]> wrote:

> buffer size _per stream_. Thus, we can have large buffers for streams that
> carry mass data, and small buffers for chatty streams. Thats a real
> >>advantage.
>
> >Exactly.
>
> E.g. beginning with FreeBSD 7, the TCP/IP stack auto-tunes the TCP
> send/receive buffer sizes _per socket_. This is controlled via:
>
> Beginning with Linux 2.6, Mac OSX 10.5, Windows Vista, and FreeBSD 7.0,
> both sender and receiver autotuning became available, eliminating the need
> to set the TCP send and receive buffers by hand for each path. However the
> maximum buffer sizes are still too small for many high-speed network path,
> and must be increased as described on the pages for each operating system.
> """
> Apart from autotuning, there is also the option to control send/receive
> buffer sizes _per socket_ using setsockopt() with SO_SNDBUF / SO_RCVBUF.
>
> ===
>
> - the multiplexing is further optimizing buffer size (app level) and does
> priorization of traffic (chatty = high-prio, mass = low-prio):
>

a) it depends on process scheduler and network queue design of the server
and client, I think. I don't know much about per-socket buffer autotuning,
but it probably just improves memory usage and throughput but doesn't have
much effect on latency of chatty traffic?

b) the endpoints need to cut mass data into smaller chunks to reserve slots
for inserting chatty traffic. e.g. if acceptable delay for chatty traffic
is x ms, set the maximum send unit of mass traffic to (current through put)
* x.

To allow one to control the peer's message scheduling, both per-socket
buffer size and ws-mux flow control are insufficient, I think. Needs to
exchange ToS info.

> ii) Is the throughput of mass-data traffic concurrently to the chatty
> traffic still (nearly) as high as when only doing mass data transfer?
>

I think minimum delay requirement for the chatty traffic determines
fragmentation on mass-data traffic regardless it's done in OS's network
stack or ws-mux code.

> Ah, right. Same origin policy. Thus, with WS-MUX, there is _more_
> potentially TCP sharing than whats possible with shared WebWorkers.
>
> However, for a given single app (from a single origin) running in multiple
> browser tabs, the shared WebWorker seems not so bad. Works today. No need
> for WS-MUX or HTTP 2.0 MUX + WS over HTTP 2.0.
>

Yes. Thanks for reminding us about shared worker approach. We haven't
evaluated it well.

> I don't have experience running G scale infrastructure. However, I'd like
> to understand why LBs need to have a look into WS at all .. why not Layer-4
> LB? Doing balancing based on hash of source IP?
>

Traffic for various services would be multiplexed into one TCP. We wanna
demultiplex it and forward to designated backends. This means that there're
again lots of TCP connections between LB-backends, but we can make any
optimization there as we want.

> a load balancer need to decode the compression to see the >multiplexing

Yes. Sorry that we haven't told the story in our mind in detail.

> chatty and mass-data coms.

OK

---

From: Adam Rice <[email-masked]>
Date: Thu, 30 May 2013 13:12:00 +0900

> However, d) - f) use wss, and hence have a TLS handshake right after TCP

While you certainly can serve both ws: and wss: on a single port, and I can
see cases where you might want to, I would not in general consider it to be
a good idea.

> and hence multiplex all of a) - f) over 1 physical WS?

The handshake does not currently include the schema, so there would be no
way to communicate to the server that a)-c) were supposed to be ws:, not
wss:.

Even if this was amended, both client and server would have to be careful
that no ambient authority leaked from the wss: channels to the ws:
channels. For example: the client would have to be careful not to send
"secure" cookies with the ws: handshakes, and the server would have to be
careful not to apply any authority contained in a client TLS certificate to
the ws: logical channels.

For this reason, I think it would be easiest not to attempt to multiplex
ws: and wss: onto a single TCP/IP connection.

> wss) - and hence can be multiplexed over 1 physical WS even though they are

Yes. These semantics are inherited from HTTP/1.1.

Adam Rice

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 30 May 2013 16:28:14 +0900

> way to communicate to the server that a)-c) were supposed to be ws:, not

Right. I missed that point.

>  Even if this was amended, both client and server would have to be
> careful that no ambient authority leaked from the wss: channels to the ws:

Compression contexts, etc. also need to be isolated carefully to protect it
from attack like CRIME.

---

From: Tobias Oberstein <[email-masked]>
Date: Thu, 30 May 2013 10:29:45 +0200

Am 30.05.2013 09:28, schrieb Takeshi Yoshino:
> On Thu, May 30, 2013 at 1:12 PM, Adam Rice <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>         Or can an implementation just "silently" transport a)-c) also
>         over wss, and hence multiplex all of a) - f) over 1 physical WS?
>
>
>     The handshake does not currently include the schema, so there
>     would be no way to communicate to the server that a)-c) were
>     supposed to be ws:, not wss:.
>
>
>     careful that no ambient authority leaked from the wss: channels to
>     the ws: channels. For example: the client would have to be careful
>     not to send "secure" cookies with the ws: handshakes, and the
>     server would have to be careful not to apply any authority
>     contained in a client TLS certificate to the ws: logical channels.
>
>     For this reason, I think it would be easiest not to attempt to
>     multiplex ws: and wss: onto a single TCP/IP connection.
>
>
> Compression contexts, etc. also need to be isolated carefully to 
> protect it from attack like CRIME.

I also agree with Adam. To summarize, given the possible "issues" of

- client cert based auth
- "secure" cookies / ambient authority
- compression contexts / CRIME

(plus complexities like: what if an app opens ws://domain.com, and then 
_later_ opens wss://domain.com? Upgrade to TLS not possible anymore .. 
cannot TLS handshake in the middle of an already established non-TLS TCP 
where some traffic already traveled .. the WS handshake at least.)

I think it would be good to include explicit text into the MUX draft, e.g.

"""
An implementation MUST NOT multiplex multiple logical channels to 
WebSocket addresses with different schemes (ws: versus wss:) on a single 
physical WebSocket connection.

For example, two WebSocket connections to ws://domain.com and 
wss://domain.com must not be shared on a single physical WebSocket 
connection to domain.com.
"""

I also think it would be helpful to include explicit text describing 
what WebSocket addresses are eligible for sharing, e.g.

"""
An implementation MAY multiplex multiple logical channels to different 
WebSocket addresses if and only if the WebSocket addresses share the 
same WebSocket scheme, the hostname resolves to the same IP address and 
the same port.

For example, connections to all of the following WebSocket addresses may 
be multiplexed over a single physical WebSocket connection (assum 
domain.com, domain2.com and sub1.domain.com all resolve to the same IP 
address):

ws://domain.com
ws://domain.com:80
ws://domain.com/foo
ws://domain.com:80/foo
ws://domain.com/bar
ws://domain2.com
ws://sub1.domain.com
ws://sub1.domain.com:80
ws://sub1.domain.com:80/bar

Note: When a WS address does not specify an explicit port, the port is 
the default port for the given WS scheme (ws: = 80 and wss: = 443).
"""

/Tobias

---

From: Tobias Oberstein <[email-masked]>
Date: Thu, 30 May 2013 13:28:38 +0200

>     - TCP autotuning on both client and server OR manually set buffer
>     sizes: small for TCP-1, and large for TCP-2
>     - the multiplexing is further optimizing buffer size (app level) and
>     does priorization of traffic (chatty = high-prio, mass = low-prio):
>
> a) it depends on process scheduler and network queue design of the
> server and client, I think. I don't know much about per-socket buffer
> autotuning, but it probably just improves memory usage and throughput
> but doesn't have much effect on latency of chatty traffic?

Improving memory, but not necessarily latency: yep, plausible.

>
> b) the endpoints need to cut mass data into smaller chunks to reserve
> slots for inserting chatty traffic. e.g. if acceptable delay for chatty
> traffic is x ms, set the maximum send unit of mass traffic to (current
> through put) * x.
>

I now can see that flow control and priorization are somewhat orthogonal 
issues.

Priorization and ToS is not goal of WS-MUX at all?

>
>     ii) Is the throughput of mass-data traffic concurrently to the
>     chatty traffic still (nearly) as high as when only doing mass data
>     transfer?
>
>

Makes sense. The lower the latency requirement, the more fragmentation, 
and the less efficiency on mass data.

>
>      >>How about having a shared Web worker in the background that
>     handles all messaging for the app over a single WS?
>      >One restriction of Web Worker is that we cannot share a worker
>     between webapps with different origins while WebSocket can connect
>     to a host different from its origin.
>
>     However, for a given single app (from a single origin) running in
>     multiple browser tabs, the shared WebWorker seems not so bad. Works
>     today. No need for WS-MUX or HTTP 2.0 MUX + WS over HTTP 2.0.
>
>
>      >For load balancer vendors, it's better that multiplexing is
>     standardized than each service provider develops its own.
>
>     I don't have experience running G scale infrastructure. However, I'd
>     like to understand why LBs need to have a look into WS at all .. why
>     not Layer-4 LB? Doing balancing based on hash of source IP?
>
>
> demultiplex it and forward to designated backends. This means that
> there're again lots of TCP connections between LB-backends, but we can
> make any optimization there as we want.

I see. So the load-balancer needs to be a layer-7 one, and not only be 
WS-MUX enabled, but also app/service aware? Something like this:

E.g. user has 2 tabs open which (logically) connect to

wss://google.com/gmail
wss://google.com/gplus

Both would be transported over 1 physical WS (via WS-MUX).

The LB would demux traffic for logical channel

wss://google.com/gmail

to backend nodes of cluster "GMAIL", while it would demux traffic for

wss://google.com/gplus

to backend nodes of cluster "GPLUS"?

>
>      >Basically multiplexing should be done as close to network as
>     possible. If it's done in application layer, for example when
>     compression is used, such a load balancer need to decode the
>     compression to see the >multiplexing information. That doesn't make
>     them happy.
>
>     - the logical WS connections contained in the physical WS would need
>     to be balanced to _different_ backend nodes.
>
>

I think it might be helpful to add text to the MUX draft that describe 
the use cases WS-MUX addresses and how it improves thing over 
alternatives. Motivation is always good and helps by providing context 
and examples.

/Tobias

---

From: Roberto Peon <[email-masked]>
Date: Thu, 30 May 2013 11:28:51 -0700

Different schemas have nothing to do with the CRIME attack.
A pure HTTPS or WSS scheme could still be attacked with CRIME, with its
success depending on the way the compression mechanism works.
With the current HTTP/2 proposals it would be pretty difficult to attack
using the techniques in the CRIME attack.

The way CRIME works is to have a 3rd party include links to the
site-to-attack. Thus it doesn't matter if the schemes are http, https, ws,
wss, whatever.
So, a restriction against muxing ws and wss together makes no sense as it
provides no security benefit for CRIME.
-=R

On Thu, May 30, 2013 at 1:29 AM, Tobias Oberstein <
[email-masked]> wrote:

>>  The handshake does not currently include the schema, so there would be
>> no way to communicate to the server that a)-c) were supposed to be ws:, not
>> wss:.
>>
>
>  Compression contexts, etc. also need to be isolated carefully to protect
> it from attack like CRIME.
>
>
>
> I also think it would be helpful to include explicit text describing what
> WebSocket addresses are eligible for sharing, e.g.
>
> """
> WebSocket addresses if and only if the WebSocket addresses share the same
> WebSocket scheme, the hostname resolves to the same IP address and the same
> port.
>
> Note: When a WS address does not specify an explicit port, the port is the
> default port for the given WS scheme (ws: = 80 and wss: = 443).
> """
>
>
>
>
> /Tobias
>
>

---

From: Tobias Oberstein <[email-masked]>
Date: Thu, 30 May 2013 20:44:00 +0200

Am 30.05.2013 20:28, schrieb Roberto Peon:
> site-to-attack. Thus it doesn't matter if the schemes are http, https,
> ws, wss, whatever.
> So, a restriction against muxing ws and wss together makes no sense as
> it provides no security benefit for CRIME.

What about the other issues listed?

Example:

An app first connect to a ws: address. Then the app opens a 2nd 
connection over wss: to the same target (IP:port) sending some sensitive 
information (either in cookies or in payload) relying on the presumed 
privacy/encryption provided by wss:.

An implementation of MUX might run the 2nd wss: connection as a logical 
WS connection over an _unencrypted_ physical ws: that was established in 
the first place upon the app opening the 1st ws: connection.

So at least the implementation must not mux logical wss: over physical ws:.

Transporting logical ws: over physical wss: seems less problematic.

> -=R
>
>
> On Thu, May 30, 2013 at 1:29 AM, Tobias Oberstein
>>             Or can an implementation just "silently" transport a)-c)
>>             also over wss, and hence multiplex all of a) - f) over 1
>>             physical WS?
>>
>>
>>         Even if this was amended, both client and server would have to
>>         be careful that no ambient authority leaked from the wss:
>>         channels to the ws: channels. For example: the client would
>>         have to be careful not to send "secure" cookies with the ws:
>>         handshakes, and the server would have to be careful not to
>>         apply any authority contained in a client TLS certificate to
>     (plus complexities like: what if an app opens ws://domain.com, and
>     then _later_ opens wss://domain.com? Upgrade to TLS not possible
>     anymore .. cannot TLS handshake in the middle of an already
>     established non-TLS TCP where some traffic already traveled .. the
>     WS handshake at least.)
>
>     WebSocket addresses with different schemes (ws: versus wss:) on a
>     single physical WebSocket connection.
>
>     connection to domain.com <http://domain.com>.
>     """
>
>     An implementation MAY multiplex multiple logical channels to
>     different WebSocket addresses if and only if the WebSocket addresses
>     share the same WebSocket scheme, the hostname resolves to the same
>     IP address and the same port.
>
>     For example, connections to all of the following WebSocket addresses
>     may be multiplexed over a single physical WebSocket connection
>     (assum domain.com <http://domain.com>, domain2.com
>     <http://domain2.com> and sub1.domain.com <http://sub1.domain.com>
>     all resolve to the same IP address):
>
>     ws://domain.com
>     ws://domain.com:80
>     ws://domain.com/foo
>     ws://domain.com:80/foo
>     ws://domain.com/bar
>     ws://domain2.com
>     ws://sub1.domain.com
>     ws://sub1.domain.com:80
>     Note: When a WS address does not specify an explicit port, the port
>     is the default port for the given WS scheme (ws: = 80 and wss: = 443).
>     """
>
>
>
>
>     /Tobias
>
>
>     [email-masked] <mailto:[email-masked]>

---

From: Roberto Peon <[email-masked]>
Date: Thu, 30 May 2013 13:16:35 -0700

inline

On Thu, May 30, 2013 at 11:44 AM, Tobias Oberstein <
[email-masked]> wrote:

> An app first connect to a ws: address. Then the app opens a 2nd connection
> over wss: to the same target (IP:port) sending some sensitive information
> (either in cookies or in payload) relying on the presumed
> An implementation of MUX might run the 2nd wss: connection as a logical WS
> connection over an _unencrypted_ physical ws: that was established in the
> first place upon the app opening the 1st ws: connection.
>

Agreed. That would invalidate the security guarantee of wss.

> Transporting logical ws: over physical wss: seems less problematic.

Yup!
-=R

>> <[email-masked] <mailto:tobias.oberstein@**tavendo.de<[email-masked]>>>
>> wrote:
>>
>>     ______________________________**_________________
>>     hybi mailing list
>>     https://www.ietf.org/mailman/**listinfo/hybi<https://www.ietf.org/mailman/listinfo/hybi>
>>
>>
>>
>

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 31 May 2013 10:27:37 +0200

> site-to-attack. Thus it doesn't matter if the schemes are http, https,

Does above refer to the general requirement for CRIME: the attacker must 
be able to inject data into the source material before it is compressed 
and then encrypted?

If so, how would that be possible with WebSocket and the proposed 
WebSocket per-message compression?

random thoughts:

a)
permessage-deflate only compresses WS data messages, not control frames.

So the attacker would need to inject data into the app-level payload.

The only way to send app-level payload in browsers if via the JavaScript 
WebSocket API (socket.send()).

So the attacker would need to somehow modify this JS code. However, the 
origin of the JS running is (at least with browsers) sent during the WS 
opening handshake as a HTTP header, and a WS server can check and 
decline an incoming connection.

b)
With WS HTTP headers are only sent _once_: during the initial WS opening 
handshake, and by the client to the server.

CRIME requires repeated injection of attacker data.

More so: permessage-deflate does not touch the opening handshake, which 
remains uncompressed.

Side Q: Do browsers reuse an established TLS/TCP connection for multiple 
WS connections?

/Tobias