Subject: [hybi] Framing proposal with start/end demarcation

From: Douglas Otis <[email-masked]>
Date: Mon, 02 Aug 2010 07:03:39 +0200

_*Websocket framing with start/end demarcation*_

The websocket framing is changed to support binary framing and defined as:

      ws-frame    =  frame-flags
                     frame-id
                     frame-length
                     octet-data   ; binary data

     frame-flags  = %x00-FF
     frame-id     = %x00-FF
     frame-length = %x00-FF %x00-FF
     octet-data   = *( %x00-FF )
             ; the number of octets is exactly the length determined
             ; by the frame-length

The frame length is a 16 bit unsigned integer sent high octet, most
significant bit first. The maximum frame data length is 64k - 1.  When
compressed, this indicates the length of compressed data.  Larger
messages must be sent in multiple frames.

_*Proposal : Frame Flags*_

The frame flags octet  is structured as:

+--------+--------+--------+--------+--------+--------+--------+--------+
| Start  | End    |        |                                            |
| Frame  | Frame  |  zLib  |                  op-code                   |
+--------+--------+--------+--------+--------+--------+--------+--------+

The op-code values are in the range %x00-1F. Current allocated op-codes are:

%x0 - Text Data frame (utf-8 text content)
%x1 - Binary Data frame
%x2 - Ping
%x3 - %1F     TBD

When Start Frame is set, this indicates the first frame within the Frame ID.
Start Frame not set in the first frame is an error.  When End Frame is set,
this concludes the frame sequence. When Start Frame and End Frame are both
set, there is only one frame in the sequence. When neither Start or End 
Frame
is set, this is a continuation data frame. When zLib is set, data is
compressed into the zlib format.  zlib is portable across platforms and
essentially never expands the data. See http://www.zlib.net/

The frame id octet is structured as:

+--------+--------+--------+--------+--------+--------+--------+--------+
|                                                                       |
|                               Frame ID                                |
+--------+--------+--------+--------+--------+--------+--------+--------+

Frame ID is always 0.  A place holder that retains 16 bit alignment.

---
Rather than a profusion of different compression schemes being used, having
a standard compression routine available should help limit the diversity.
zLib is not expected to replace sound or video compression schemes.

-Doug

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Aug 2010 11:17:09 +1000

Douglas,

what is the advantage of a start and end bit as opposed to just a fragment
bit?

How would your treat a frame with neither bit set received immediately after
a frame that had the end bit set?

cheers

On 2 August 2010 15:03, Douglas Otis <[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Douglas Otis <[email-masked]>
Date: Mon, 02 Aug 2010 21:56:48 -0700

On 8/2/10 6:17 PM, Greg Wilkins wrote:
>
>  Douglas,
>
>  what is the advantage of a start and end bit as opposed to just a
>  fragment bit?

There are four states related to each frame passed:
1) at Start Frame.
2) at End Frame.
3) continuation of a prior frame, but neither a Start Frame or End Frame.
4) at both the Start Frame and End Frame. (A stand alone Frame)

When there are one or multiple frame sequences:
--> Start Frame state triggers resource setup operation.
--> Neither Start or End Frame triggers an append operation.
--> End Frame triggers the data exchange, and resource cleanup.

I have not seen a good definition for Frag, but it seems to mean "NOT 
Start & End Frame."
Some definitions indicate the end frame by using the multiple-bit 
op-code.  As this op-code evolves, it might muddle the ending frame 
state, and it seems there is no clear indication for the beginning 
frame.  Keep the Start and Ending state bits defined separately. These 
bits do not need to be encoded with other states.  Otherwise, this makes 
understanding these flags difficult, and fails to indicate whether the 
data transfer represents an initial frame for the identified sequence, 
assuming more than one sequence is desired.  It seems logical to 
establish provisions for having multiple streams, even if only one is 
defined in version 1. Perhaps there should be a statement that all 
frames that indicate a length not on a 32 bit boundary, should be null 
padded, where the padding is to be ignored.

RFC4960 uses the following definitions for chunk handling (a similar 
concept):

B bit: 1 bit

  The (B)eginning fragment bit, if set, indicates the first fragment
  of a user message.

E bit: 1 bit

   The (E)nding fragment bit, if set, indicates the last fragment of
   a user message.

(Start and End are shorter terms compared to beginning and ending.)

>  How would your treat a frame with neither bit set received
>  immediately after a frame that had the end bit set?

This would clearly represent an invalid state.

Legal sequences would be:

Start & End (single frame)
Start --> End (two frames)
or Start --> !start & !end *(!start & !end) --> End (three or more frames)

  RFC4960 uses the following definitions for chunk handling (a similar 
concept):

B bit: 1 bit

  The (B)eginning fragment bit, if set, indicates the first fragment
  of a user message.

E bit: 1 bit

   The (E)nding fragment bit, if set, indicates the last fragment of
   a user message.

(Start and End are shorter terms compared to beginning and ending.)
> How would your treat a frame with neither bit set received immediately 
> after a frame that had the end bit set?
This would clearly represent an invalid state.

Legal sequences would be:

Start & End (single frame)
Start --> End (two frames)
or Start --> !start & !end *(!start & !end) --> End (three or more frames)

Repeated definition:

      ws-frame    =  frame-flags
                     frame-id
                     frame-length
                     octet-data   ; binary data

     frame-flags  = %x00-FF
     frame-id     = %x00-FF

The op-code values are in the range %x00-1F. Current allocated op-codes 
are:

When Start Frame is set, this indicates the first frame within the Frame 
ID.

---

-Doug

---

From: Roberto Peon <[email-masked]>
Date: Mon, 2 Aug 2010 22:05:19 -0700

On Mon, Aug 2, 2010 at 9:56 PM, Douglas Otis <[email-masked]> wrote:

> I have not seen a good definition for Frag, but it seems to mean "NOT Start
> & End Frame."
> Some definitions indicate the end frame by using the multiple-bit op-code.
>  As this op-code evolves, it might muddle the ending frame state, and it
> seems there is no clear indication for the beginning frame.  Keep the Start
> and Ending state bits defined separately. These bits do not need to be
> encoded with other states.  Otherwise, this makes understanding these flags
> difficult, and fails to indicate whether the data transfer represents an
> initial frame for the identified sequence, assuming more than one sequence
> is desired.  It seems logical to establish provisions for having multiple
> streams, even if only one is defined in version 1. Perhaps there should be a
> statement that all frames that indicate a length not on a 32 bit boundary,
> should be null padded, where the padding is to be ignored.
>

It is a trivial difference, but I'd prefer:
B(egin) bit: 1 bit
C(ontinue) bit: 1 bit  (also known as 'frag')

Thus, one frame would be:
  0b10   B

Two frames would be:
  0b11  B | C
  0b00  0

Three frames:
  0b11  B | C
  0b01  C
  0b00  0

This makes a countdown, which just seems nice to me.
-=R

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Mon, 2 Aug 2010 22:15:45 -0700

> This makes a countdown, which just seems nice to me.

Sorry for being dense, but help me understand the advantage of this over
just using a single bit? e.g. "chunk", with the following FSM:

start state:
get a 0: interpret as single-frame message, go to start state
get a 1: interpret as multi-frame message, go to continue state

continue state:
get a 0: interpret as end of multi-frame message, go to start state
get a 1: interpret as continuation of multi-frame message, go to continue
state

you say it makes it hard to know whether a given frame is the start frame or
the end frame, but it seems to me that you either have some amount of state
you're remembering for a stream (be that cookies, path information,
whatever) in which case you can remember what state you're in, or
alternately if you have no state you can dump it all after receiving a
complete message, and then the next message you receive is interpreted as
the beginning of a message... I'm having trouble understanding why this is a
significant burden?

---

From: Roberto Peon <[email-masked]>
Date: Mon, 2 Aug 2010 22:30:11 -0700

Ian-- I think you're right. All you need is a single bit which states
whether or not you're done (one way or another).

-=R

On Mon, Aug 2, 2010 at 10:15 PM, Ian Fette (イアンフェッティ) <[email-masked]>wrote:

>> On Mon, Aug 2, 2010 at 9:56 PM, Douglas Otis <[email-masked]>wrote:
>>
>>> op-code.  As this op-code evolves, it might muddle the ending frame state,
>>> and it seems there is no clear indication for the beginning frame.  Keep the
>>> Start and Ending state bits defined separately. These bits do not need to be
> you say it makes it hard to know whether a given frame is the start frame
> or the end frame, but it seems to me that you either have some amount of
> state you're remembering for a stream (be that cookies, path information,

---

From: Douglas Otis <[email-masked]>
Date: Mon, 02 Aug 2010 22:59:34 -0700

>     On Mon, Aug 2, 2010 at 9:56 PM, Douglas Otis <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
On Mon, Aug 2, 2010 at 10:05 PM, Roberto Peon <[email-masked] 

>     This makes a countdown, which just seems nice to me.
You have inverted the End state as a C state.  Having the C bit set will 
not indicate a significant state, whereas the inverse, the End state, is 
far more significant.  Your description of this as a countdown also 
appears unclear.

-Doug

On 8/2/10 10:15 PM, Ian Fette (イアンフェッティ) wrote:

> Sorry for being dense, but help me understand the advantage of this 
> over just using a single bit? e.g. "chunk", with the following FSM:
>
> start state:
> get a 1: interpret as continuation of multi-frame message, go to 
> continue state
> you say it makes it hard to know whether a given frame is the start 
> frame or the end frame, but it seems to me that you either have some 
> amount of state you're remembering for a stream (be that cookies, path 
> information, whatever) in which case you can remember what state 
> you're in, or alternately if you have no state you can dump it all 
> after receiving a complete message, and then the next message you 
> receive is interpreted as the beginning of a message... I'm having 
> trouble understanding why this is a significant burden?
Why depend upon remembered state? Responding to each frame becomes more 
complex, especially when more than one sequence is involved.  If a 
sequence were to go astray, there would not be illegal states able to 
detect this event.  Making each state explicit in each frame keeps the 
client simpler, and more robust when errors occur.  Errors do occur. : ^(

-Doug

---

From: Roberto Peon <[email-masked]>
Date: Mon, 2 Aug 2010 23:06:34 -0700

That would argue for a decent error handling state/error signaling as
opposed to any more bits for frame signaling, no?
-=R

On Mon, Aug 2, 2010 at 10:59 PM, Douglas Otis <[email-masked]> wrote:

>>  On Mon, Aug 2, 2010 at 10:05 PM, Roberto Peon <[email-masked] <mailto:
> [email-masked]>> wrote:
>
> not indicate a significant state, whereas the inverse, the End state, is far
> more significant.  Your description of this as a countdown also appears
> unclear.
>
> -Doug
>
>
> complex, especially when more than one sequence is involved.  If a sequence
> were to go astray, there would not be illegal states able to detect this
> event.  Making each state explicit in each frame keeps the client simpler,
> and more robust when errors occur.  Errors do occur. : ^(
>
> -Doug
>
>

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Aug 2010 16:48:57 +1000

> and more robust when errors occur.  Errors do occur. : ^(

I think the 2 bit solution has to have some state because there are some
illegal transitions in the state machine. eg

Start   : 1 1 0 1 0 0 0 0 0
  End   : 1 0 1 0 0 0 1 0 1
Boundary:  |   |       |X X

So the last two combination result in the need to handle errors.

With the 1 bit solution there are no state errors because any arbitrary
ordering of the bit will result in a meaningful boundary between messages:

Frag    : 0 1 0 1 1 1 0 0 0 0
Boundary:  |   |       | | | |

So with no error states, the state machine has to be at least a little
simpler.

cheers

---

From: Douglas Otis <[email-masked]>
Date: Mon, 02 Aug 2010 23:50:48 -0700

> opposed to any more bits for frame signaling, no?
There does not appear a desire to do much in the way of detecting errors
undetected by TCP, such as bus related bit errors. Modern math
coprocessors include CRC functions. That said, using one additional bit
to explicitly expresses frame state allows the detection of errors,
whereas the proposed method of using 1 bit to express four states will
not detect. Such a one bit scheme for four states could easily find
itself in weird pathological modes that likely could have been avoided.

Error states also help with debugging. I would also suggest that any
meta-information be carried within the initial frame, rather than added
separately as more remembered state. Perhaps another TLV structure
within the initial frame would support the mime meta information.

-Doug

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Aug 2010 17:29:38 +1000

Douglas,

It is a reasonable argument to make that the error states of the 2 bit state
machine will provide some validation of the connection. However we are
trying to be complexity adverse here and I think it would be a close call
between the number of problems that bit detected vs the number of problems
implementing that bit caused.

On 3 August 2010 16:50, Douglas Otis <[email-masked]> wrote:

> I would also suggest that any
> separately as more remembered state.
>

I think that a lot of meta-data would only make sense in the initial frame
of a message.

However, there is much interest in supporting multiplexing as an extension
implemented in this meta data.
As such, Robertos proposal allows us to avoid having a more stateful
connection by allowing meta data to be associated with each and every frame.
ie the channel ID can be part of the meta data in every frame sent on that
channel.  This makes frames more self describing.

So I don' t think we want to restrict all meta-data to just the first frame.

Of course we want to avoid the situation where every frame of a message
specifies a different content-type, but essentially this is not different
from handling multiple instances of the same meta-data within the same
frame.  It's and error either way.

cheers

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 09:35:33 +0100

> zLib is not expected to replace sound or video compression schemes.

Your design implies a per-frame compression flag, and therefore (to  
me) a per-frame compression state.

I have to say I've never seen any benefit to a per-frame compression  
state in this protocol, as the messages should themselves be  
reasonably compact (and can be made very compact, given domain  
knowledge).

However, stream compression pays enormous dividends in stateful  
stream-based protocols - even IMAP, which has a very broad data  
domain, benefits from compression to the tune of (roughly) a 40%  
ratio (that is, where 100%  would be no compression).

Other protocols manage significantly better - XMPP, for example,  
compresses to around 20% typically, and much lower for the kinds of  
specialized applications we typically see deployed on the web (via  
BOSH currently).

Per message compression does work reasonably well for "classic" HTTP,  
of course, as the messages compressed are typically reasonably sized  
web pages, but this will not translate well at all onto WebSockets.

I cannot recommend strongly enough that per-frame compression is  
abandoned in favour of a stream-level compression negotiated once, at  
startup. As a side-benefit, this is also vastly simpler to code -  
which possibly explains why XEP-0138 is very widely supported, and is  
possibly still better supported than TLS-based compression. [My  
understanding is that all the common XMPP server implementations  
support XEP-0138 and TLS-based compression at this stage, as well as  
many clients].

Further, I would propose that whilst DEFLATE (not zlib, please) is an  
MTI, other compression schemes should be possible (betwixt consenting  
adults).

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 09:46:18 +0100

On Tue Aug  3 06:15:45 2010, Ian Fette (イアンフェッティ)  
wrote:
> over

If you're dense, you're in excellent company; I don't follow either.

Aside from anything else, it introduces illegal flows, which then  
need error handling.

I appreciate this may be more robust, but unless we're considering  
interleaving (which introduces a whole new set of considerations),  
then I don't think this is needed.

Dave.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 3 Aug 2010 01:59:50 -0700

Even with interleaving (I interpret that as multiplexing) it isn't
necessary. There is the start of a flow and the end. All you need in
addition to the thing that signifies the end is something that identifies
the stream.
If we're not multiplexing, there is only one stream, and the single bit is
enough to denote the end of a flow.
-=R

On Tue, Aug 3, 2010 at 1:46 AM, Dave Cridland <[email-masked]> wrote:

> On Tue Aug  3 06:15:45 2010, Ian Fette (イアンフェッティ) wrote:
>
> Aside from anything else, it introduces illegal flows, which then need
> error handling.
>
> interleaving (which introduces a whole new set of considerations), then I
> don't think this is needed.
>
> Dave.
> --
> Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]<[email-masked]>

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 10:16:36 +0100

> enough to denote the end of a flow.

Interleaving is different - it's sending multiple frames  
concurrently, such that you can have the sequence:

Chunk (id 1)
Final (id 2)
Chunk (id 1)
Chunk (id 3)
Final (id 1)
Final (id 3)

Note that I had to include an id here to make this example clear  
enough, so you can more than likely guess what would be needed in the  
protocol. FWIW, also note that this only uses a single-bit chunking  
indicator.

I don't recommend we try to support interleaving, by the way. People  
who really need to slowly deliver a large message "in the background"  
can do that protocol work themselves.

Dave.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 3 Aug 2010 02:37:07 -0700

On Tue, Aug 3, 2010 at 2:16 AM, Dave Cridland <[email-masked]> wrote:

> Interleaving is different - it's sending multiple frames concurrently, such
> that you can have the sequence:
>
> Chunk (id 1)
> Final (id 2)
> Chunk (id 1)
> Chunk (id 3)
> Final (id 1)
> Final (id 3)
>

Right, that is what I've been calling multiplexing. It isn't FIFO
multiplexing as HTTP offered, but rather "True Multiplexing".

>
> Note that I had to include an id here to make this example clear enough, so
> you can more than likely guess what would be needed in the protocol. FWIW,
> also note that this only uses a single-bit chunking indicator.
>

Yup! If you have an id and the bit that indicates scope of stream, then you
have "multiplexing" (interleaving as per your parlance)!

I think we're violently agreeing on all but terminology...

>
> I don't recommend we try to support interleaving, by the way. People who
> really need to slowly deliver a large message "in the background" can do
> that protocol work themselves.

I strongly recommend we do support it :)
The people that need it are NAT boxes (whose translation tables can get
overwhelmed by too many connections) and proxies (think loadbalancers) and
servers who will be using a lot of resources if they don't do multiplexing
(at a minimum, think of the keep-alives for all of those connections
whizzing about all the time, and then you can think of the kernel resources
tied up, etc, etc). My job is, for better and worse, ensuring that Google
scales.

The case for multiplexing/interleaving is:
  less keep-alives
    less "nonuseful" activity on network, less time that cell and mobile
device radios are powered on
  less state in NAT translation tables
    when you overflow these, say bye-bye to your connectivity.
  less state consumed by the kernel on the proxy
    loadbalancers have to terminate everything, and then have to forward
everything. If they can't share TCP state (i.e. a connection), its a
double-whammy. In all liklihood, the loadbalancer will have many fewer
servers behind it than clients, and thus could maintain many, many fewer
connections to each.
  Greater chance that your data will be sent without having to wait for the
TCP cwnd to open up-- your connection is already warm!
  Greater chance of triggering the fast rexmit TCP path when loss occurs

There are some other reasons why it is nice, but I don't think we'd get into
those for this protocol-- I don't think that level of complexity is likely
to be useful. The above, however, applies to simple
multiplexing/interleaving.

I'm perfectly happy to write a simple server with any workable multiplexing
scheme we cook up, incidentally.
I really like to play with real implementations!
-=R

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 10:46:39 +0100

> multiplexing as HTTP offered, but rather "True Multiplexing".
No, sorry, still at cross-purposes.

Interleaving needs frame identifiers, like TCP sequence numbers.

Multiplexing needs stream identifiers, like TCP address pairs.

I'm fine with the latter, but not the former.

[On multiplexing]
> I really like to play with real implementations!

All agreed 100%. I just don't want to interleave chunks of frames  
within a stream.

Dave.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 03 Aug 2010 13:27:16 +0200

On Tue, 03 Aug 2010 10:35:33 +0200, Dave Cridland <[email-masked]>  
wrote:
> Per message compression does work reasonably well for "classic" HTTP, of  
> course, as the messages compressed are typically reasonably sized web  
> pages, but this will not translate well at all onto WebSockets.
>
> startup. As a side-benefit, this is also vastly simpler to code - which  
> possibly explains why XEP-0138 is very widely supported, and is possibly  
> still better supported than TLS-based compression. [My understanding is  
> that all the common XMPP server implementations support XEP-0138 and  
> TLS-based compression at this stage, as well as many clients].
>

Another advantage of stream-level compression seems to be that we can  
leave it out for now as it does not affect framing, making it easier to  
reach the four week "milestone".

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Aug 2010 21:58:22 +1000

> I strongly recommend we do support it :)

Multiplexing without interleaving (and with out flow control), is not very
useful.

However I would qualify "we do support it", to say that I think the core
protocol should have framing that is able to well support multiplexing -
either as an extension or as a subsequent enhancement to the core protocol.

cheers

---

From: John Tamplin <[email-masked]>
Date: Tue, 3 Aug 2010 08:41:44 -0400

Two problems:
- you are compressing framing data, and flag/length bytes are going to not
compress well and will clutter up the compression dictionary.
- if you are compressing every frame equally, you must compress even things
that aren't compressible.  That means, for example, when transferring
textures or sounds for a game you will be paying the cost of expanding such
payload data, and that is going to be much larger than the compressible
frames negating any advantage of compression.

I believe forcing compression to always be on or off for the entire stream
will mean developers have to resort to more complicated schemes of passing
some resource id/url over WebSockets which is then fetched via XHR just to
avoid this problem -- workable, but awkward and more prone to error.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 05:45:12 -0700

> frames negating any advantage of compression.
even worse if you have unrelated data, e.g. you are sending 4 images.

> avoid this problem -- workable, but awkward and more prone to error.

Agreed, I do think that there should be options for compression, including
turning it on/off at a per-frame level. Perhaps we don't get compression at
the 4 week milestone, but saying that we will reserve one of the flag bits
for compression seems reasonable until we have a more concrete agreed-upon
solution.

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Tue, 3 Aug 2010 14:53:30 +0200

> Multiplexing without interleaving (and with out flow control), is not very

Indeed not.

> either as an extension or as a subsequent enhancement to the core protocol.
I can see how multiplexing would be a logical optimisation. This avoids
multiple connections to deal with different sub-protocols and probably has
the interesting aspect of providing some sort of concurrency whereby the the
client can send chunks of data as soon as it has them.

Nonetheless I'm wondering, what complexity will this bring and on whose
shoulders? What is a sensible multiplexing mechanism from the perspective of
WS libs' developers? Is it suitable to multiplex XMPP and some other
protocol for instance?  How can multiplexing not cripple on the JS API?

Moreover it seems to me this will push servers to have some sort of
dedicated component to handle the multiplexing and forward to the right
component each incoming frame. But, so far it seems most implementations on
the server side have been directly into HTTP servers (probably a side effect
of the Upgrade mechanism) and I haven't really seen stories on how WS will
be used at all in any other context. Is it expected that WS will create a
new ecosystem of servers/proxies and such?

Anyway, I agree with Greg saying that the core protocol ought to display
support for it but I'm wondering about its actual usage.

-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach

---

From: Pieter Hintjens <[email-masked]>
Date: Tue, 3 Aug 2010 15:02:17 +0200

> support for it but I'm wondering about its actual usage.

For what it's worth, we added multiplexing to AMQP from the start.
That made the protocol connection layer significantly more complex.
There was never a proven use case beyond "keep the number of TCP
connections low" and in practice, it made client-side APIs more
complex (TCP connections no longer mapped to application threads).

We observed that very few people used multiplexing in their
applications and they usually got it wrong.  Finally, we observed that
SCTP provided all this functionality already, meaning we'd duplicated
parts of the network stack that were already in place.

WebSockets are meant to remain open for longer periods, correct?  This
reduces the need for multiplexing and my advice would be to leave this
out of the design and eventually provide WebSockets over SCTP for use
cases that really benefit from multiplexing.

-Pieter

---

From: Matthew Wild <[email-masked]>
Date: Tue, 3 Aug 2010 14:04:10 +0100

> frames negating any advantage of compression.

No, I believe only the frame/chunk payload would be compressed. It
makes zero sense to compress the frame headers, and (though he can
correct me if I'm wrong) I highly doubt that's what Dave was
suggesting :)

Notably OpenSSL, when compression was originally added to it,
compressed each frame payload individually. This mistake was
eventually rectified and they received significantly better results -
not only in terms of compression ration but also processing (state can
be stored rather than requiring a setup/tear down for each frame).

Compression methods that rely on a dynamic dictionary really don't
want to rebuild the dictionary from scratch for each chunk of data.

With multiplexing ahead of us, compression could be enabled per
in-band stream (not per connection), allowing it to be turned off for
binary data streams, and enabled for text (or other
known-to-be-compressible formats).

Matthew

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Aug 2010 14:14:12 +0100

Greg Wilkins wrote:
> 
>    On 3 August 2010 19:37, Roberto Peon <[1][email-masked]> wrote:
> 
>    On Tue, Aug 3, 2010 at 2:16 AM, Dave Cridland <[2][email-masked]>
>    wrote:
> 
>      I don't recommend we try to support interleaving, by the way.
>      People who really need to slowly deliver a large message "in the
>      background" can do that protocol work themselves.
> 
>    Multiplexing without interleaving (and with out flow control), is not
>    very useful.

Multiplexing without interleaving and flow control is prone to nasty
deadlocks, so cannot be automatic, invisible to applications.

The *entire point* of multiplexing is to be invisible to apps, in the
browser at least.

So interleaving and flow control appear to be essential on technical grounds,
when multiplexing is included.

They obviously are not needed in the absence of any multiplexing.

This suggests that a multiplexing extension could use a framing layer
*outside* the basic framing layer, which is able to split and
interleave basic frames.

I don't know whether it's a good idea to have two types of framing,
one used for multiplexing, or to have the basic framing have
sufficient flexibility to accomodate the extension.

Imho,
-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Aug 2010 14:35:43 +0100

Sylvain Hellegouarch wrote:
>    Nonetheless I'm wondering, what complexity will this bring and on
>    whose shoulders? What is a sensible multiplexing mechanism from the
>    perspective of WS libs' developers? Is it suitable to multiplex XMPP
>    and some other protocol for instance?

If you do it right, it can multiplex anything that runs over TCP.

That means, if nothing else is convenient, a dedicated WS<->TCP proxy
can be used if someone wanted to hook up XMPP over it, similar to the
way people use "stunnel" to give non-TLS applications TLS capability.

However if it's missing things like flow control, then the flows
interfere more and can deadlock, which limits how it can be used safely.

>    How can multiplexing not cripple on the JS API?

That's the bizarrest question I've seen in a while.

Why would it affect the JS API at all?

I imagine it to be entirely automatic with the JS app requesting
independent WebSockets, and the underlying WS implemention converting
them to flows over fewer connections.

There may be rare uses for a JS function to request a dedicated
connection for debugging and compatibility.  That's a good idea just
in case something is unforeseen.

>    dedicated component to handle the multiplexing and forward to the
>    right component each incoming frame. But, so far it seems most
>    implementations on the server side have been directly into HTTP
>    servers (probably a side effect of the Upgrade mechanism) and I
>    haven't really seen stories on how WS will be used at all in any other
>    context. Is it expected that WS will create a new ecosystem of
>    servers/proxies and such?

If multiplexing is done transparently (as imho it should be), then it
can be a front end proxy which splits the flows into separate,
standard WebSocket connections for the back end.  That would be very
convenient as it would work with all WS servers, including trivial ones.

It if takes off, then I wouldn't be surprised to see HTTP servers and
dedicated WS servers getting the demultiplexing built in, and treating
it internally exactly as if they had received multiple connections.

>    Anyway, I agree with Greg saying that the core protocol ought to
>    display support for it but I'm wondering about its actual usage.

Imho, it's usage will be entirely dependent on whether browsers
support it automatically, below the JS API.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 3 Aug 2010 23:36:21 +1000

> complex (TCP connections no longer mapped to application threads).

I don't see why he client side APIs should be affected by multiplexing?
Multi-plexing should be transparent to applications.

> parts of the network stack that were already in place.

Which is why multiplexing should be transparent to the applications.
I think it must be implemented in the browsers and servers and the
applications should never know if they have been multiplexed or not.

> cases that really benefit from multiplexing.

There are some very good reasons why long held connections are better to be
reused by multiplexing rather than new connections being opened.
To open a new connection will take at least 2 RTT times to establish the
connection and handshake.  You then have a brand new TCP/IP connection with
it's slow start algorithm, so you are not going to get maximum through put
until you have sent a lot of data.

But even if the only reason we can find for multiplexing is "keep the number
of TCP connections low", this is a very good reason as lots of
infrastructure and servers are already connection limited.

regards

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 14:56:27 +0100

> correct me if I'm wrong) I highly doubt that's what Dave was
Oh, I do like surprising you.

I was actually, and here's why - those framing octets won't clutter  
the dictionary (because in DEFLATE, there isn't one, only a  
back-buffer), and the only time the Huffman skew will be noticable is  
when the framing data starts to dominate the stream, in which case  
compression of it is probably exactly what you *do* want to be  
doing...

I'm willing to produce data to back this assertion, if pushed.

> be stored rather than requiring a setup/tear down for each frame).
Indeed, that would be a disaster.

> known-to-be-compressible formats).

I'm not sure there's any point in practise.

When I said IMAP also has a broad domain, and compresses down to  
~40%, I meant "This is what I'm currently seeing on my client which  
supports IMAP BINARY and IMAP COMPRESS=DEFLATE". (It's currently 36%,  
BTW). Remember that IMAP mixes control, user-defined text, and binary  
data, so is a poster-child of what you're arguing against.

Now, I agree that the compression ratios could be halved if binary  
data were split out, but as for the overall octet counts, they'll  
diverge more the more incompressible data you're passing, so...

Notably:

1) DEFLATE, in common with most compression algorithms, includes a  
mechanism for sending across blocks of incompressible data as  
uncompressed aligned octets, which means in practise that the second  
of John's points seems somewhat debatable. Such incompressible blocks  
*do* get added to the back buffer, but DEFLATE will quickly recover,  
and "expanding" them is of course a single pointer arithmetic  
operation (and in zlib, unfortunately a buffer copy).

2) Typically, it's possible to make this behave better by having the  
application frame binary and text in distinct DEFLATE blocks.  
Luckily, our current proposals for framing make this advantageous for  
other reasons.

3) Although it's true that passing large objects by reference will be  
more efficient, this was always the case, and allows the browser to  
take advantage of the existing object cache inherent in REST  
architectures - this is good, and way better than compression.

I'm not saying that you won't get better compression by running  
multiple deflate codecs, one per stream, and electing to send binary  
frames - sometimes - out of band of any codec, but I don't think the  
increased compression efficiency will in any way outweight the  
increased cost in terms of complexity and resource.

For a start, each compression codec will cost 300k at each end. If  
that's what you want, I think you're going to have to stick a schwack  
more RAM in those servers.

Dave.

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Tue, 3 Aug 2010 16:03:11 +0200

> them to flows over fewer connections.

I simply assume that if you multiplex protocols that have nothing in common,
client developers will expect an easy way to dispatch and handle each
message they receive from the server in their own way. The proposed API is
so generic that you will basically need an API atop to redispatch based on
what you've received. I would not be able to handle AMQP, XMPP and binary
data in the same way directly inside onmessage...

So even though the current JS API isn't changed, it won't be really usable
as it stands *because* of the multiplexing which is what I mean by cripple
to the client's API. Either you don't do multiplexing and the client API is
fine, or you do and really it's not well suited in which case a better API
would be something that can register a callback to the content-type that
each frame carries. Unless you consider a suite of if/else like an elegant
solution.

What's so bizarre about wondering about how to actually use the API in the
context of a multiplexed WS connection?

> convenient as it would work with all WS servers, including trivial ones.

I thought the idea of multiplexing was to reduce the number of such
connections. Sure at that stage you're in your own network probably but you
still need to support the load of connections nonetheless.

Besides you can't easily close the external connection if one of the
subprotocol raises a critical error because you still have the other
subprotocols to carry. In fact, how do you safely consider the connection
can be closed with multiplexing on? At least on the server side.

> support it automatically, below the JS API.

Define "automatically" because this doesn't match with the fact you found my
earlier question so bizarre. I'm not just speaking about sending, but also
the receiving end of the client when the server sends data.

---

From: Patrick McManus <[email-masked]>
Date: Tue, 03 Aug 2010 10:09:51 -0400

On Tue, 2010-08-03 at 21:58 +1000, Greg Wilkins wrote:

> 
> However I would qualify "we do support it", to say that I think the
> core protocol should have framing that is able to well support
> multiplexing - either as an extension or as a subsequent enhancement
> to the core protocol.

I agree. 

More specifically, the framing should be able to address the flow
identifier as a first class citizen (i.e. without walking an option
list, or following an indirection). it's easy - just leave room in the
main flags in the tlv header for it. call it "reserved for future use"
for now if we're not ready to deal with it. Even 8-10 bits would be
sufficient imo.

If we get to the point of whether or not multiplexing should be defined
now I will dig up some material on the impact of connection management.
There are significant client benefits too, btw. But for now, I think it
is sufficient to reserve space for it in the basic framing.

I guess my ideal framing is:

[data len 16][flags 6][reserved 10]
[options]
[data]
[pad round frame up to 32bits]

flags
0x01 final frame of message
0x02 compressed data
0x04 options present 
0x08 UTF-8
0x10 ping req
0x20 ping resp

and an option is defined as a series of these (where the final one has
the final opt bit set to 1):

[opt data length 16 bits] [final opt 1 bit] [opt type 15 bits] 
[opt data]
[pad round option up to 32 bits]

And then we predefine an option number for meta data. 

Simple. Flexible. Very easy for a machine to both generate and consume.

-Patrick

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 00:14:55 +1000

Sylvain,

to be explicit, I would expect that a js client could do something like:

var xmpp_ws = new WebSocket("/foo/xmpp","xmpp-subprotocol");
xmpp_ws.onmessage = myXmppSupprotocolHandlerFunction.

var amqp_ws = new WebSocket("/foo/amqp","amqp-subprotocol");
amqp_ws.onmessage = myAmqpSupprotocolHandlerFunction.

and the application would work identically if they ran on a browser that
supported multiplexing or on one that did not.

There is nothing in the ws protocol that needs to be tied to a TCP/IP
connection.  The onopen and onclose callback can equally be associated with
opening virtual channels on a multiplexed TCP/IP connection.  It would be
the job of the multiplexing in the browser and server to transparently
assign a channel ID to each of the websockets opened above and to direct
only the traffic for each websocket to the correct onmessage handler.

I hope that clarifies what I mean by transparent multiplexing.

cheers

On 4 August 2010 00:03, Sylvain Hellegouarch <[email-masked]> wrote:

> I simply assume that if you multiplex protocols that have nothing in
> common, client developers will expect an easy way to dispatch and handle
> each message they receive from the server in their own way. The proposed API
> is so generic that you will basically need an API atop to redispatch based
> on what you've received. I would not be able to handle AMQP, XMPP and binary
> Define "automatically" because this doesn't match with the fact you found
> my earlier question so bizarre. I'm not just speaking about sending, but
> also the receiving end of the client when the server sends data.
>
>
> --
> - Sylvain
> http://www.defuze.org

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 15:15:31 +0100

On Tue Aug  3 13:53:30 2010, Sylvain Hellegouarch wrote:
> whose

Just to put some perspective on this, I don't think that multiplexing  
has any worth to XMPP. XMPP/WS would be a simple enough binding, and  
more or less treat WS as a more efficient BOSH.

I was assuming that the case where multiplexing becomes interesting  
is where multiple browser pages require common access, such as when a  
page is cloned, or navigation is followed, etc. These are things I  
see in XMPP/BOSH that developers want to address. I'm not sure how,  
though...

FWIW, the entire thread seems to, from where I slouch, highlight that  
many of us don't fully understand what multiplexing is all about -  
myself included.

Dave.

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Tue, 3 Aug 2010 16:19:39 +0200

> I hope that clarifies what I mean by transparent multiplexing.

It surely does. Not much to complain about now :p

Thanks,
- Sylvain

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Aug 2010 15:55:48 +0100

Sylvain Hellegouarch wrote:
>    common, client developers will expect an easy way to dispatch and
>    handle each message they receive from the server in their own way.

No, that would be "non-transparent multiplexing", because the client
can see the flow tags associated with each incoming message.

Transparent multiplexing is where the client developers don't receive
tagged messages; they receive messages on separate WebSocket objects.

(Note that simply dispatching messages to separate WebSocket objects
is *not* enough for transparent multiplexing.  That still leads to
deadlocks when one part of the client app is busy and blocks another
part, for example one browser window is processing and its event queue
is full, so prevents another from receiving messages.)

>    proposed API is so generic that you will basically need an API atop to
>    redispatch based on what you've received.

No, the client app simply opens multiple WebSocket connection objects -
exactly the same as it does without multiplexing.

>    I would not be able to
>    handle AMQP, XMPP and binary data in the same way directly inside
>    onmessage...

You can't handle binary data over the WebSocket JS API at all, yet.

If it could handle binary data, then yes it would handle those just fine.

>    So even though the current JS API isn't changed, it won't be really
>    usable as it stands *because* of the multiplexing which is what I mean
>    by cripple to the client's API. Either you don't do multiplexing and
>    the client API is fine, or you do and really it's not well suited in
>    which case a better API would be something that can register a
>    callback to the content-type that each frame carries. Unless you
>    consider a suite of if/else like an elegant solution.
>    What's so bizarre about wondering about how to actually use the API in
>    the context of a multiplexed WS connection?

I think one or both of us is really confused about what we're talking about.

In transparent multiplexing, there *isn't* any need for if/else in the
client app, *nor* any additional API.

That is why I found your question bizarre, because you seem to think
one or the other is necessary.  They aren't.

>    connections. Sure at that stage you're in your own network probably
>    but you still need to support the load of connections nonetheless.

There are several choke points being addressed.

Client's own network connection is one of them, due to mobile
power/radio usage, NAT table limits, and keepalive bandwidth on lower
bandwidth broadband; those are areas that multiplexing addresses
directly.

At the server side, medium-load services will be fine with front-end
demultiplexing, for the same reasons they use HTTP reverse proxies
now: They can just add more server hardware as their needs scale.

At risk of getting off topic...

You're right that high-load services will need to manage the load even
inside the network.  They'll tend to put the demultiplexing inside
HTTP or dedicated WS service handlers, where no TCP connection is
involved.  In fact at high load it's not enough to reduce TCP
connections: you also have to manage the amount of state per
connection, which means not having a process open for every putative
connection.  How that is done will depend on individual applications,
but publish-subscribe event patterns are likely to be common because
they're easy to understand and they scale well.

>    subprotocols to carry. In fact, how do you safely consider the
>    connection can be closed with multiplexing on? At least on the server
>    side.

Oh, that's just the difference between *bad* multiplexing and
*good* multiplexing :-)

For every connection signal (such as connection reset or abrupt
close), you need an equivalent control message in a multiplexing
protocol in order to transparently carry one in the other.

So if one of the "subprotocols" raises a critical error, you send a
flow-tagged critical error message over the shared connection.  At the
other end, it is dispatched as a critical error for that subprotocol
only.  The JS API does not see a message: it sees an error from that
WebSocket object, exactly as if it was an independent connection with
a real critical error.  The object and subprotocol are closed, but the
shared connection is not.

I presume you are thinking of protocols that run over TCP, so they
already have their own way of signalling most conditions, and the only
ones that need special consideration for multiplexing are TCP
conditions: shutdown (half-close) and connection reset.

Even without multiplexing it is desirable to have control messages for
those, because TCP conditions are not transmitted reliably over the
internet due to proxies of various kinds.

>    Define "automatically" because this doesn't match with the fact you
>    found my earlier question so bizarre. I'm not just speaking about
>    sending, but also the receiving end of the client when the server
>    sends data.

The receiving end would despatch messages inside the WebSocket
implementation (i.e. in the browser core) to the individual
WebSocket objects.

As far as the client app is concerned, it has opened several
independent WebSocket connections, by creating several independent
WebSocket objects.

The client app does not know or care that any multiplexing has
happened.  Some WS implementations won't implement multiplexing and
will actually use multiple connections; others will share connections,
without the client app knowing which.

A major goal is for multiple independent client apps to be able to
share connections without any of them knowing or caring.  This is
because of the browser security model, where multiple tabs/windows
are indepdendent.

This can only work if the multiplexing is implemented transparently
enough that it doesn't effect client behaviour, and that independent
WebSocket objects behave independently in the ways that matter.

-- Jamie

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Tue, 3 Aug 2010 17:06:09 +0200

> WebSocket objects behave independently in the ways that matter.
Yeap. Thanks for the detailed response. Once I got what you and Greg meant
by transparent multiplexing, well things fell well into place.

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Aug 2010 16:12:21 +0100

> Interleaving needs frame identifiers, like TCP sequence numbers.

No it doesn't: Frame boundaries are sufficient.
That's a single bit per frame.

Also I note that frame types, options etc. don't need to be send with
every chunk, only once per message.

> All agreed 100%. I just don't want to interleave chunks of frames  

That creates a technical problem.

If you don't allow that, multiplexing ceases to be transparent at the
point where *one* of the independent sending processes tries to send a
large frame.

Either you accept, and then all other the sending processes are
blocked for a long time, or you reject, and one of the sending process
receives an error.

Even worse, if the sending process is a proxy and it is forwarding a
message at the same time as it receives it, then this could block all
the other sending processes for an *unlimited* time.

Either way, it prevents multiplexing from being transparent enough to
use automatically.

In browsers, it would mean multiplexing could not be used because it
wouldn't be safe.

-- Jamie

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 16:32:47 +0100

Let me start again, I'm obviously not being terribly clear, sorry.

Let's assume a single WebSocket containing two streams, A and B.

I'll number each message within each stream with an independent  
ordering, such that A1 is the first frame sent to A, and B1 is the  
first frame sent to B.

If you have the following sequence:

A1 Chunk
A1 Final
B1 Final
A2 Final
B2 Chunk
B2 Final

Then there is no interleaving, but there is multiplexing. This is no  
problem, and the frame identifiers are redundant.

If you have:

A1 Chunk
B1 Final
B2 Chunk
A1 Final
B2 Final
A2 Final

Then there is multiplexing, and interleaving between streams, but no  
interleaving within the stream. Again, no need for frame identifiers,  
and I'm quite happy with this.

If you have:

A1 Chunk
A2 Final
B1 Final
B2 Chunk
A1 Final
B2 Final

Then you have multiplexing, interleaving between streams, and  
interleaving within streams. This does require a frame identifier (to  
distinguish A1 Final from A2 Final), and I'd object to this. I doubt  
anyone would seriously support it, mind.

Dave.

---

From: Jack Moffitt <[email-masked]>
Date: Tue, 3 Aug 2010 09:46:41 -0600

> Just to put some perspective on this, I don't think that multiplexing has
> any worth to XMPP. XMPP/WS would be a simple enough binding, and more or
> less treat WS as a more efficient BOSH.

Just to contradict Dave somewhat:

Being able to share the connection for multiple WS services is very
useful. One of those may be XMPP and one may be non-XMPP. I think this
will become common practice if multiplexing is available since often
developers compose whole products out of separate apps.

In my own case, we see people combining multiple widgets onto a single
page, each of which requires its own connection. Multiplexing would
make this efficient.

I also think that multiplexing would be quite useful for XMPP itself
for Jingle related things. Currently advanced XMPP applications that
must move data around have to build significant HTTP infrastructure to
proxy this or send it in-line which kills latency. Multiplexing the
connection would allow avatar transfers on a separate stream without
having to build a whole new set of HTTP applications to proxy them.
But, this is an advanced use case to be sure.

Dave's comment about multi-page applications is probably better served
with shared workers in some way, although I've just started to
experiment with them.

jack.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 16:50:55 +0100

> Just to contradict Dave somewhat:
Feel free to contradict me a lot.

> But, this is an advanced use case to be sure.

All this starts to smell like applications would need to be aware of  
multiplexing, and able to use it.

I still think we need to discuss what multiplexing is to be used for  
at a fundamental level, and be sure we're all addressing the same  
problem. I don't really get the feel that we have a tight definition  
of the problem here.

Dave.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 03 Aug 2010 10:08:49 -0700

On 8/3/10 12:29 AM, Greg Wilkins wrote:
> Douglas,
>
> It is a reasonable argument to make that the error states of the 2 bit 
> state machine will provide some validation of the connection. However 
> we are trying to be complexity adverse here and I think it would be a 
> close call between the number of problems that bit detected vs the 
> number of problems implementing that bit caused.
Take the view that network injected errors occur.  Robust protocols 
should detect errors whenever possible.   Using a single bit to 
represent four states,  means state sequence errors can not be 
detected.  This is a bad practice that may lead to security exploits.

When more than one stream is involved, by being explicit about all four 
states, state can be reported directly from bits within the frame's 
tag.  This also makes debugging less painful, as the current state is 
not derived from prior state.

Rather than tracking state to detect errors, errors can be reported 
based upon the opening and closing of streams.
Where:
a) an open stream can not be opened,
b) an unopen stream can not be closed,
c) an unopen stream can not be appended,
d) all streams should be closed at the end of the session.

Detecting these errors should always be in place, whether or not 
implicit frame state is used.  The stream processes can be driven 
directly by the asserted state held in two of the tag bits.  Surely, you 
are not suggesting open/close errors should not be detected, or that 
they are too difficult to implement?

Handling of stream open/close errors should be less painful than ending 
in an aberrant state due to undetected errors.

-Doug

---

From: Roberto Peon <[email-masked]>
Date: Tue, 3 Aug 2010 11:15:03 -0700

Being unsure of the open/closed state of a stream is far less scary than
having the stream ID changed to something that is also valid/working at the
time.

I'm not religious enough to fight over a single bit, but I'd expect a CRC or
similar if we're truly concerned with bit-level errors or other corruption.

-=R

On Tue, Aug 3, 2010 at 10:08 AM, Douglas Otis <[email-masked]> wrote:

>> state machine will provide some validation of the connection. However we are
> Take the view that network injected errors occur.  Robust protocols should
> detect errors whenever possible.   Using a single bit to represent four
> states,  means state sequence errors can not be detected.  This is a bad
> practice that may lead to security exploits.
>
> states, state can be reported directly from bits within the frame's tag.
>  This also makes debugging less painful, as the current state is not derived
> from prior state.
>
> Rather than tracking state to detect errors, errors can be reported based
> upon the opening and closing of streams.
> Where:
> Detecting these errors should always be in place, whether or not implicit
> frame state is used.  The stream processes can be driven directly by the
> asserted state held in two of the tag bits.  Surely, you are not suggesting
> open/close errors should not be detected, or that they are too difficult to
> implement?
>
> Handling of stream open/close errors should be less painful than ending in
> an aberrant state due to undetected errors.
>
> -Doug
>
>
>
>

---

From: Patrick McManus <[email-masked]>
Date: Tue, 03 Aug 2010 15:34:29 -0400

On Tue, 2010-08-03 at 11:15 -0700, Roberto Peon wrote:

> 
> I'm not religious enough to fight over a single bit, but I'd expect a
> CRC or similar if we're truly concerned with bit-level errors or other
> corruption.

+1 :: if data corruption is a real concern the 2 bit eof isn't going to
reliably address it.

if that's needed then define crc as a message level option and make sure
options can be applied to the final chunk not just the first one.

but honestly, this is something I expect to see in the application level
data stream if TCP's checksum were not strong enough. HTTP has an
optional message checksum and I don't think I've ever seen it used in
the wild. my preference would be to leave it out of the websockets
layer.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 03 Aug 2010 21:22:26 +0100

On Tue Aug  3 18:08:49 2010, Douglas Otis wrote:
>> It is a reasonable argument to make that the error states of the 2  
>> bit state machine will provide some validation of the connection.  
>> However we are trying to be complexity adverse here and I think it  
>> would be a close call between the number of problems that bit  
>> detected vs the number of problems implementing that bit caused.
Surely if the security requirements of the deployment are such that  
this becomes a risk requiring action, you'd have long-since passed  
the threshold for requiring TLS, which would surely protect against  
this?

Dave.

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 3 Aug 2010 21:50:47 +0100

> anyone would seriously support it, mind.

Thanks, that is clearer.  Imho we should use those sorts of diagrams
in the spec to illustrate ;-)

I agree, there is no point interleaving *within* an identified stream,
and actually there is a technical reason it must be avoided anyway:

It's not obvious, but since you *must* implement per-stream flow
control for the streams to be safe for independent applications, then
you cannot interleave within those streams as it would break simple
flow control.

Anyway, if someone wants to transmit multiple messages in parallel,
they can just create more streams as needed.

If done well (i.e. not like BEEP), stream creation should be
virtually free, so there's no reason to avoid using lots of them.

(If you're familiar with Git, an analogy is "branching is free" in
Git, so it changes the way you work.)

-- Jamie

---

From: Ian Hickson <[email-masked]>
Date: Tue, 3 Aug 2010 22:51:55 +0000 (UTC)

On Tue, 3 Aug 2010, Pieter Hintjens wrote:
> 
> For what it's worth, we added multiplexing to AMQP from the start. That 
> made the protocol connection layer significantly more complex. There was 
> never a proven use case beyond "keep the number of TCP connections low" 
> and in practice, it made client-side APIs more complex (TCP connections 
> no longer mapped to application threads).
> 
> We observed that very few people used multiplexing in their applications 
> and they usually got it wrong.  Finally, we observed that SCTP provided 
> all this functionality already, meaning we'd duplicated parts of the 
> network stack that were already in place.
> 

I strongly agree with this. Multiplexing, done right, is a very 
complicated problem. Given that it's possible to do this on top of Web 
Sockets using shared workers, and given that it's possible for Web Sockets 
to be layered on top of infrastructure that supports this natively, I 
really don't see any reason to implement it at the Web Sockets layer.

It's not even clear to be that it would make _sense_ to have multiplexing 
built on top of TCP -- it would interfere with TCP's own flow-control 
mechanisms. For example, say you have a video app and an IM app sharing a 
connection -- the video app might overwhelm the bandwidth, causing the TCP 
connection to be throttled down, affecting the much lower-bandwidth IM 
channel even though the IM channel is not in any way to blame.

If we find that this is not a good approach (i.e. if we find the need to 
add multiplexing), we can always add it at a loter date, using an opt-in 
option field during the handshake. This can simply add whatever 
information is needed to every frame; for example, after the option is 
selected we could say that every frame now has an additional two bytes 
before the length to specify a channel.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 03 Aug 2010 16:15:29 -0700

> before the length to specify a channel.

I agree with this approach. Multiplexing is a lot of complexity, and we should not take it as a given until we have evidence that some use cases are not feasible without it.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 09:43:52 +1000

> really don't see any reason to implement it at the Web Sockets layer.

I don't complexity as a reason we should make it an application layer
problem.

It will be no simpler in the application layer, indeed I think a shared
worker will be a lot more complex as you will have multiple frames/tabs each
having to work out who creates the shared worker, who controls it, who kills
it.    I also have some security concerns (note that I'm not fully up to
speed on the security model of shared workers), but if the security model is
good enough to stop evil.com playing with workers created by good.com, then
that model will inhibit sharing of connection.  Eg evil.com and
good.commight both want to open a connection to
neutral.com, but don't want each other to be able to eaves drop.

Shared workers are not a panacea to make this problem easier, they are just
another complication that would need to be dealt with.

On the other hand, I also think that we do not want multiplexing in the core
protocol, because there is value in having very simple clients and servers
(eg HTTP does not force all clients and servers to use persistent
connections).

I think multi-plexing true home is in an optional extension that lives below
the js API (so is transparent and secure from applications), but above the
core framing mechanism and the featureful  browsers and servers will
implement multiplexing.

What we need to support this is a simple framing protocol, but one that has
a reasonable extension mechanism.    I think the various framing threads are
converging on some reasonable solutions that will keep it simple, but also
allow such extensions to be developed.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 09:50:18 +1000

> I don't complexity as a reason we should make it an application layer

Another reason that multiplexing will not work above the API is that
optional sharing of connections in the application layer is like optional
sharing of toys at kindergarten!

There are just insufficient motivation the make application developers
ration resources that are the concern of the infrastructure developers.  If
infrastructure developers want to ration connections, then they should not
hand them out without restrictions and then expect application developers to
play nice.

cheers

---

From: Jack Moffitt <[email-masked]>
Date: Tue, 3 Aug 2010 17:53:56 -0600

> I agree with this approach. Multiplexing is a lot of complexity, and we should not take it as a given until we have evidence that some use cases are not feasible without it.

I believe it's been made clear several times that current applications
in production today on the Internet using XHR, routinely run into the
large numbers of connections problem, which multiplexing addresses.

The use case is not just one of particular features, but also of
scalability on today's Internet with today's intermediaries.

Shared workers are a possible solution for sharing a single stream
among several pages. This works fine if all the pages are from the
same application.  It doesn't really help if that's not the case.
>From the application programmers point of view, it is vastly simpler
to get good performance if the multiplexing is hidden from view.

I don't see the harm in having the multiplexing mechanics in the
bitstream even if they aren't used yet, assuming that we get consensus
on them quickly. All the complexity of multiplexing is in the
implementation; the protocol aspects of it, once defined, aren't so
hairy. That said, perhaps it's worth waiting to see what the browser
guys think of it; after all, it means very little if it's not
supported there.

jack.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 17:41:41 -0700

> really don't see any reason to implement it at the Web Sockets layer.
It's possible but it is not likely to be efficient. I really don't get this
argument. If I don't care about efficiency, I will just use the current
JavaScript libraries I'm currently using and suck it up. One of the reasons
we are interested in WebSockets is efficiency of the network protocol and
network-level characteristics. If we make WebSockets inefficient to
implement at scale, then it provides no tangible benefit over the solutions
we currently have.

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 00:44:59 +0000 (UTC)

On Tue, 3 Aug 2010, Ian Fette (ã~B¤ã~B¢ã~C³ã~C~Uã~B§ã~C~Cã~C~Fã~B£) wrote:
>
> If we make WebSockets inefficient to implement at scale, then it 
> provides no tangible benefit over the solutions we currently have.

I don't buy that multiplexing is necessary at scale, but even if it was, 
the simple benefit of having a single long-lived TCP connection, rather 
than having to track multiple HTTP connections, is huge.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 03 Aug 2010 17:48:13 -0700

On 8/3/10 1:22 PM, Dave Cridland wrote:
> this becomes a risk requiring action, you'd have long-since passed the 
> threshold for requiring TLS, which would surely protect against this?
Dave,

The normal process of moving data in and out of frames should detect 
sequence errors, don't you agree?  Such detection can utilize open, 
append, and close logic related to moving frame data.  Saving a single 
bit in a frame tag (which currently has several undefined bits) is not 
justified when it also removes an ability to detect frame sequence 
errors.  The current strategy presumes state based upon assumed (not 
explicit) prior state.  Not misidentifying an initial frame is critical, 
since this frame is also assumed to contain meta information for 
processing subsequent data.

Many applications using web sockets will not be using TLS, so unless TLS 
becomes a requirement, do not assume there is any additional protection 
afforded.  It is difficult to understand the motivation for removing one 
bit per frame that is essential in ensuring frame state remains explicit 
rather than implied.

This becomes more important when there is stream multiplexing.

Don't assume that an ability to multiplex independent frame sequences 
will require separate flow control.  TCP assures against network 
congestion.  Any application utilizing web sockets should be able to 
intermix frame sequences as desired.   Being able to intermix sequences 
helps prevent blocking issues that might occur when handling objects of 
differing sizes.  Needing to fragment just to avoid the latency induced 
by larger objects is likely to be more complex than offering a small set 
of stream handles.

The tag information as currently arranged will likely have 8 bits 
available that can serve as a Stream ID.  Version 1 can define this ID 
as always being zero for now.  The same approach could be taken to map 
out bits related to compression logic, without actually implementing 
compression in Version 1.  While there is a range of compression 
algorithms that might be better suited for sound or video, general 
purpose compression should also be available for common data objects.   
Offering a default compression option should reduce the number of bugs 
introduced by those attempting to include compression in conjunction 
with frame handling.

-Doug

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 04 Aug 2010 09:56:09 +0900

> I was actually, and here's why - those framing octets won't clutter the
> dictionary (because in DEFLATE, there isn't one, only a back-buffer),
> and the only time the Huffman skew will be noticable is when the framing
> data starts to dominate the stream, in which case compression of it is
> probably exactly what you *do* want to be doing...

Ok, but framing data is something like between 1 and 4 bytes. If that 
domainates, then you have to be very very careful to squeeze anything 
more out. Even if we are compressing by stream (which makes sense), we 
of course need to do it in a way that each frame stands on its own. That 
means that in each frame, we need at least some way to indicate how long 
it is (which doesn't necessarily have to be a length), and we need to 
round the compressed length up to the next byte. So if the framing 
octets dominate, the compression rate gets down, and assuming that each 
frame is transmitted in its own TCP packet, the TCP (and lower-level 
protocol) headers reduce the actual compression achieved even more.

> I'm willing to produce data to back this assertion, if pushed.

If you have some real data, I'd definitely be interested to see it.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 10:59:20 +1000

2010/8/4 Ian Hickson <[email-masked]>

> than having to track multiple HTTP connections, is huge.
My experience of adding websocket support to cometd was that there was no
huge saving in complexity.

Websockets being long lived does not mean they will not be dropped and have
to be reestablished.
Websocket connections may fail silently (not closed but not working) and it
is desirable to notice before network timeout.

All the activities that I have to do for HTTP ( timeouts,  keepalives,
connections that drop out and then re-establish, seeing a new connection
from a reload before noticing the old connection is closing etc), I have to
do for websocket.   There is no complexity saving.

Any reliance on the concept that TCP/IP connections are long lived and do
not fail will just make unreliable fragile applications.  Maybe that is not
a problem for tic tac toe, but it is for any reasonable application.

Websocket will give some benefits in lower max latency, higher throughput
for rapid messages and better data density, but as currently formulated
there is no complexity savings at all.
Websockets will also let some people sleep better at night because they
don't like HTTP being "abused", but that is of little technical benefit.

If we want complexity savings, then the websocket API needs to become a true
datagram API and not a connection oriented API. The infrastructure would
need to take over all responsibility for managing connections etc.  But  I
don't think that is feasible at this stage.

regards

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 18:02:38 -0700

2010/8/3 Ian Hickson <[email-masked]>

> than having to track multiple HTTP connections, is huge.

We can have a single HTTP connection using chunked encoding (for the
downstream, at least. For upstream I believe due to proxies we actually have
to close the connection to ensure delivery, but I would have to double
check). But the benefit of deploying custom code in all the servers to
handle the connections may not be worth a 50% reduction - if we could
multiplex efficiently and achieve a more significant reduction, then it's
interesting.

Again, websockets is interesting because of performance, not just the API.
We already have a reasonable API in JS via libraries. So, I am willing to
take complexity on if it gives us performance.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 11:18:55 +1000

> don't like HTTP being "abused", but that is of little technical benefit.

Oh it will also halve the number of connections (if applications are
restrained), which is a big (if not huge) saving.  My fear is that it's
current unrestricted access to connections will actually result in more
connections being held open for most non trivial applications - since create
a new connection will be far simpler than application layer multiplexing.

cheers

---

From: Roberto Peon <[email-masked]>
Date: Tue, 3 Aug 2010 21:39:47 -0700

>> than having to track multiple HTTP connections, is huge.
Ian-- I write and maintain, among other things,  the aptly named Google
Front End which is responsible for ensuring that we scale and continue to be
reliable.
You should take me at my word when I say that I have significant concern,
informed by years of experience with scaling Google scale systems, about the
scalability of one-websocket-is-one-connection websockets.
To be clear about that experience, I've been writing, maintaining and
defining the vast majority of the edge of Google's application layer's
scalability and reliability features and their implementations for years.

At the very least, counter please counter my experience with data! Where is
your data that shows that there will be negligible scalability impact? What
are your assumptions for that scenario? Have you considered the cascade
failure circumstances? Are you considering adding the appropriate
random-wait and exponential backoff to browsers so that sites which have
been hammered into the ground can reach steady state?

Here are my assumptions:
  The number and breadth of services from which products are composed will
continue to expand at a linear or super-linear pace.
  Given the required pace of changes and innovation, each service is
extremely likely to have its own websocket-- these teams will not want many
external dependencies. Large teams are extremely likely to be slow to
change.
  People will be increasingly likely to use more applications from web
service providers/web product providers (such as Yahoo, Microsoft, Google,
Amazon, etc. etc.). This means that the number of 'tabs' is likely to
increase.
  This all seemingly adds up to a multiplicative increase in the number of
connections.
  Depending on how IPv6 does, we're soon down to the last 16 bits of
addressing space for TCP connections. As NAT increases in prevalence (as it
must unless IPv6 is widely adopted), the number of connections allocated to
a single client will necessarily reduce beyond even where it is today
(constrained by the memory size of NAT tables).
  Additionally, TCP-connections use kernel resources which are very
difficult to attribute directly to a userspace application. This makes it
difficult for the server to introspect to determine how close to redline it
is running and adjust appropriately. This makes cascade failure much more
likely. This is not a joke, supposition, or a statement said for lack of
experience.

I'm not here to be unpleasant. I'm here to ensure that anything unpleasant
happens with as small an audience as possible whenever possible!

-=R

> If we want complexity savings, then the websocket API needs to become a
> true datagram API and not a connection oriented API. The infrastructure
> would need to take over all responsibility for managing connections etc.
> But  I don't think that is feasible at this stage.
>
> regards
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 06:41:18 +0000 (UTC)

On Tue, 3 Aug 2010, Roberto Peon wrote:
>
> You should take me at my word when I say that I have significant 
> concern, informed by years of experience with scaling Google scale 
> systems, about the scalability of one-websocket-is-one-connection 
> websockets.
>

That's not how standards development works. You can't just "pull rank" and 
convince everyone. You have to provide data, reasoning, and generally 
actually base your position on arguments. (This applies to me too.) In the 
absence of data, the default is to not support a feature (this is how we 
avoid rampant feature bloat).

> At the very least, counter please counter my experience with data! Where 
> is your data that shows that there will be negligible scalability 
> impact?

I'm not claiming that there is negligible scalability impact. I'm claiming 
that multiplexing can be implemented at the application layer in the 
protocol. On the server side, this would essentially look like this:

   Protocol stack:                    Implementation layer:

   |\/\/\/|\/\/\/\/\/|\/\/\/\/\|\/\/|  Applications
   | Mail | Calendar | Finance | IM | ______________
   +------+----------+---------+----+
   | Google Multiplexing Protocol   |    Google's
   +--------------------------------+   Front-end
   | WebSockets                     |    Servers
   +--------------------------------+
   | TLS                            |
   +--------------------------------+ ______________
   | TCP                            |     Kernel
   \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ 

This would let us (Google) do whatever we needed at the multiplexing layer 
without having to have every browser updated, without having to have the 
protocol revised with each new need, and so on. For all the reasons you 
list (fast growth of applications, fast change of needs, etc), this is 
IMHO far superior than relying on the core protocol to have the features.

> difficult to attribute directly to a userspace application.

Difficult because of current implementations, or difficult because of 
inherent characteristics of the problem? If the latter, what 
characteristics?

Jamie described some of the constraints here in a useful way:

   http://www.ietf.org/mail-archive/web/hybi/current/msg02850.html

It seems, though, that if we want to provide people with the equivalent of 
multiple TCP connections without using multiple TCP connections, we can't 
base it on a single TCP connection (for example because a video upload 
saturating the bandwidth and getting throttled would result in a parallel 
low-bandwidth but latency-sensitive connection from a game being throttled 
as well, which would be disastrous to the gameplay experience). So we'd 
have to use something else, like UDP, at which point we're basically 
reinventing TCP from scratch. This may be useful work, and indeed is work 
that is already happening (e.g. SCTP). But if we're going to do that we 
should just tunnel WebSocket through that work, not redesign it inside the 
WebSockets protocol.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 17:59:37 +1000

On 4 August 2010 16:41, Ian Hickson <[email-masked]> wrote:

> I'm claiming
> protocol.

Exactly - you are "claiming" and it is an entirely unsubstantiated claim.

Can you cite any examples of a scarce resource being fairly rationed and
shared in the application layer?   I think the original Mac OS had
cooperative multitasking, but that did not prove to be robust.

How can application multiplexing provide share a connection securely between
code from different origins?   Consider some auctionWatch.com site that
serves a page that uses a websocket feed from ebay.com.  AuctionWatch.com
might be a portal site and allow 3rd party gadgets to be added to the page,
all of them needing a websocket connection to ebay.com.   We don't want a
TCP/IP connection from every gadget on the page, so we need to share a
connection. But the code in those gadgets is from different sources, so we
do not want them able to incept messages to/from other gadgets.  Nor do we
want any of the gadgets to be able to look at traffic we might have on
another tab that goes directly to ebay.com.

The proponents of multiplexing below the API have given many examples and
data about how it is proven to be useful efficient and useful: TCP itself,
XHR over HTTP and SPDY are some that have been often cited here.  We have
much real world experience saying that connection limitations are real and
that connection sharing is a must have.  Yet we are all still striving to
come up with a minimal simple framing mechanism that does not mandate
multiplexing.

Throw us a bone here!!!  Would it really kill you if you at least allow for
the possibility that transparent multiplexing might be needed?  Are you
really soooooo sure that you are 100% right that you are prepared to not
even contemplate any proposals that allows the possibility of efficient
multiplexing?

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 04 Aug 2010 01:13:41 -0700

On Aug 4, 2010, at 12:59 AM, Greg Wilkins wrote:

> Can you cite any examples of a scarce resource being fairly rationed and shared in the application layer?   I think the original Mac OS had cooperative multitasking, but that did not prove to be robust.
> 
> How can application multiplexing provide share a connection securely between code from different origins?   Consider some auctionWatch.com site that serves a page that uses a websocket feed from ebay.com.  AuctionWatch.com might be a portal site and allow 3rd party gadgets to be added to the page, all of them needing a websocket connection to ebay.com.   We don't want a TCP/IP connection from every gadget on the page, so we need to share a connection. But the code in those gadgets is from different sources, so we do not want them able to incept messages to/from other gadgets.  Nor do we want any of the gadgets to be able to look at traffic we might have on another tab that goes directly to ebay.com. 
> 
> The proponents of multiplexing below the API have given many examples and data about how it is proven to be useful efficient and useful: TCP itself, XHR over HTTP and SPDY are some that have been often cited here.  We have much real world experience saying that connection limitations are real and that connection sharing is a must have.  Yet we are all still striving to come up with a minimal simple framing mechanism that does not mandate multiplexing.
> 
> Throw us a bone here!!!  Would it really kill you if you at least allow for the possibility that transparent multiplexing might be needed?  Are you really soooooo sure that you are 100% right that you are prepared to not even contemplate any proposals that allows the possibility of efficient multiplexing?

I think the burden of proof should be on those who want to add a feature, to show the benefits outweigh the costs.

Personally, I am open to the idea that multiplexing is so valuable that it should be in the core protocol, but I'd really like to see some data. So far I've mostly  seen people say "trust me, I'm an expert". Performance is an area where I have a lot more experience than networking, and one thing I've learned is that guesses without performance made without measuring are wrong more often than not. 

Thus, I think the right approach is to ensure that it's possible to sanely add multiplexing or similar features in the future, but at the same tine not bake in the assumption that it *will* be needed.

I do appreciate the people who have tried to explain the nature of the performance issues (e.g. Jamie Lokier) .

To address your auctionwatch.com hypothetical, here is one possible solution that can provide fair and secure conection sharing but does not rely on protocol-level multiplexing:

- ebay provides an html file hosted from ebay.com that can be loaded as iframe with a postMessage-based API that can be used byby any site
- the iframe uses a SharedWorker or some other mechanism to share only a single WebSocket connection
- JS code in the iframe or the SharedWorker is responsible for dispatching to various embedders; it knows their origins and can pass them to ebay.com over the WebSocket if needed

Now there is only one WebSocket to ebay, and ebay-controlled JS code mediates access.

By the way, this can save more bandwidth than using protocol-level multiplexing managed by the browser - if multiple gadgets are watching the same auction item, application-level code that knows about the protocol can register for the notification only once and multicast it on the client side.

Is this a super convenient solution? Maybe not, but it's good enough to do some live experiments to quantify the benefits.

Cheers,
Maciej

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 08:17:15 +0000 (UTC)

On Wed, 4 Aug 2010, Greg Wilkins wrote:
> shared in the application layer?

Relative to TCP, WebSockets itself is in the application layer. So surely 
that line of argumentation would suggest putting the multiplexing below 
TCP, exactly as I'm suggesting.

> I think the original Mac OS had cooperative multitasking, but that did 
> not prove to be robust.

Shared workers are isolated processes, so I don't think that analogy 
holds. Also, we're talking about multiplexing being run by code under the 
same authority as the origin server, which seems like it would be even 
more able to uphold that server's fairness policies than if the browser 
was in control.

> code from different origins?

Using a shared worker hosted by the origin of the WebSocket server.

> Consider some auctionWatch.com site that serves a page that uses a 
> websocket feed from ebay.com.  AuctionWatch.com might be a portal site 
> and allow 3rd party gadgets to be added to the page, all of them needing 
> a websocket connection to ebay.com.  We don't want a TCP/IP connection 
> from every gadget on the page, so we need to share a connection. But the 
> code in those gadgets is from different sources, so we do not want them 
> able to incept messages to/from other gadgets.  Nor do we want any of 
> the gadgets to be able to look at traffic we might have on another tab 
> that goes directly to ebay.com.

Sure. This is exactly what MessagePorts and SharedWorkers are for. In fact 
in this case you wouldn't even need a shared worker, you could do all the 
multiplexing in the main AuctionWatch.com page, with the gadgets all 
communicating with that page from their sandbxed iframes either via the 
regular postMessage() API or using MessagePorts. Alternatively if eBay 
provided this WebSocket interface, they might even provide a SharedWorker 
such that AuctionWatch.com doesn't need to do anything about it directly, 
the widgets would just use that worker and only that worker would be 
allowed to connect to the server (by blocking all origins but eBay's).

How would you do this at the browser layer? You'd end up having to add all 
kinds of additional security code, which is just asking for trouble.

> Throw us a bone here!!!  Would it really kill you if you at least allow 
> for the possibility that transparent multiplexing might be needed?

The possibility already exists.

> Are you really soooooo sure that you are 100% right that you are 
> prepared to not even contemplate any proposals that allows the 
> possibility of efficient multiplexing?

The current protocol already supports everything it needs to be extended 
to support browser-native multiplexing. I've even described a timeline in 
which I would expect us to have data showing whether this is actually 
useful or not. I'm not really sure what you're arguing against.

---

From: Rob Sayre <[email-masked]>
Date: Wed, 04 Aug 2010 01:18:13 -0700

On 8/4/10 1:13 AM, Maciej Stachowiak wrote:
>
> Thus, I think the right approach is to ensure that it's possible to 
> sanely add multiplexing or similar features in the future, but at the 
> same tine not bake in the assumption that it *will* be needed.

Yes.

You can solve any problem, if you're willing to make the problem solve 
enough.

- Rob

---

From: Roberto Peon <[email-masked]>
Date: Wed, 4 Aug 2010 01:41:41 -0700

> avoid rampant feature bloat).

Have I not been giving reasons?

It is hard to give data when we don't have implementations that implement
the idea. It is also difficult to give other data which is proprietary and
I'm fairly sure shouldn't be shared.
We *are* working on getting data for SPDY, which is attempting to solve
another problem (but at least does do multiplexing so we can talk about the
cost/benefit that we've seen), alas, we're only at the cusp of begining to
get real-world data.

The fact that we're not likely to wait for data from one of these
experiments is the source of much of my ennui and is what is causing me to
state what I do (for how long) and why I'm worried!
It is also why I'm volunteering to make/give away/open source a multiplexing
server implementation. I'd do a client, but I'm not familiar with that
codebase. Anyone up for experimenting with me on this?

>   | Mail | Calendar | Finance | IM | ______________

That does limit us to a shared-connection per domain. I can live with that,
but we really should be doing shared-connection-per-IP.
We should be doing it for HTTP, but that isn't going to happen without some
interesting signaling.
I don't think that services will want to share connections using
"cooperative multiplexing". That would be a load of complexity. Putting the
complexity on the application writers seems like a huge cop-out. That
complexity would be implemented and reimplemented over and over again. Each
time it would be different and new.

I'm curious to see what this looks like in sum-total complexity as compared
to implementing it as a base protocol feature.

> inherent characteristics of the problem? If the latter, what

We haven't been able to modify the linux kernel to do this yet. It *is*
"only software," but having the kernel track the skbufs per-application
would require lots of changes. IT hasn't happened yet, and I don't know if
the complexity in the kernel is the best thing.

>   http://www.ietf.org/mail-archive/web/hybi/current/msg02850.html

Agreed, that is very useful.

> as well, which would be disastrous to the gameplay experience).

Having parallel connections doesn't make that problem go away, but it does
change the delay characteristics.
Time to first byte for a parallel stream may be lessened (no guarantee, all
sorts of modems do queuing these days), but time to last byte of a stream
may be the same. It really depends on the network and the individual flows,
though.
Flow control over a single connections gives you more control over delays--
your queuing delay is at most the unack'd bytes plus the rest of the amount
of current frame. Both are controllable.
The fewer connections you have the more control you have since you're more
likely to be dealing with one entity instead of many. Were you dealing with
many entities on the other end, it is highly unlikely that they'll
communicate amongst themselves directly to figure out how much they as a
corpus were sending to the client.

> So we'd

SCTP has some significant deployment issues (NAT boxes). I'm not sure how
much I can say about it more than that.

-=R

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 18:55:10 +1000

> TCP, exactly as I'm suggesting.

The resource that TCP is sharing is the network interface.   The TCP
designers did not decide that multiplexing connections over the same network
interface was too complex nor did they just allocated a network interface
per application, even though it is entirely possible for network interfaces
to be shared in the application layer by using shared memory or local pipes
to distribute the packets received.

> allowed to connect to the server (by blocking all origins but eBay's).

But that is not how webservices work. Access webservices is by open
standards for which you can use a multitude of clients.
Attempts to allow access to your service only from your own software are
difficult and mostly fail as people reverse engineer the direct connection.
The likes of ebay really don't want to be writing such code - they want to
just publish standard interfaces and let others be responsible for the
clients.  Even if ebay.com did have the desire to write such a common
worker, there are going to be many establishments that do not have the
resources or skills to develop such a shared worker.

> kinds of additional security code, which is just asking for trouble.

No - you just restrict them to the websocket API.  The application will be
no more aware that the messages are sharing a TCP/IP connection than they
are aware they are sharing network interface.

> to support browser-native multiplexing.
>

If you mean that it supports it by the allocation of extra op-codes, then I
agree.  It is the first of the options that I have listed in take IV (the
hixiesque), but it has the problem of centralized control of  those
op-codes. Hence  several other proposals have been made for consideration.

So if you are happy for that capability to be there, why are you wasting so
much band width arguing against it?

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 08:55:31 +0000 (UTC)

On Wed, 4 Aug 2010, Roberto Peon wrote:
> >
> > I'm not claiming that there is negligible scalability impact. I'm 
> > claiming that multiplexing can be implemented at the application layer 
> > in the protocol. On the server side, this would essentially look like 
> > this:
> >
> > This would let us (Google) do whatever we needed at the multiplexing 
> > layer without having to have every browser updated, without having to 
> > have the protocol revised with each new need, and so on. For all the 
> > reasons you list (fast growth of applications, fast change of needs, 
> > etc), this is IMHO far superior than relying on the core protocol to 
> > have the features.
> 
> That does limit us to a shared-connection per domain. I can live with 
> that, but we really should be doing shared-connection-per-IP.

WebSockets can connect to IPs, so there's no reason we couldn't support 
that in such a "Google Multiplexing Protocol".

> "cooperative multiplexing". That would be a load of complexity.

In Google's case, if we have to implement multiplexing, it'll have to be 
implemented twice once on the server side, once on the client side. 
Whether on the client side it's implemented in Chrome or in a JS library 
doesn't seem to have any impact on the complexity seen by application 
authors targetting those protocols.

I don't think characterising JS-driven multiplexing as "cooperative 
multiplexing" is accurate, either. I see no difference in practice to 
browser-driven multiplexing in terms of whether it's "cooperative" or 
not (I'm not sure what you would call the opposite... native?). JS-driven 
multiplexing doesn't have to be any less transparent to authors on the 
client side if they target a JS library that implements such a "Google 
Multiplexing Protocol". It is certainly no different on the server side.

> Putting the complexity on the application writers seems like a huge 
> cop-out.

I prefer to see it as providing application authors with the building 
blocks they need to create whatever applications they want, rather than 
providing them with prefabricated solutions that are rather inflexible in 
terms of how they can be adapted to specific needs, but I guess one can 
spin this either way. :-)

> That complexity would be implemented and reimplemented over and over 
> again. Each time it would be different and new.

There's no reason someone can't provide a generic library for this as far 
as I can tell.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 04 Aug 2010 11:02:20 +0200

On Wed, 04 Aug 2010 10:55:10 +0200, Greg Wilkins <[email-masked]> wrote:
> If you mean that it supports it by the allocation of extra op-codes,  
> then I agree.  It is the first of the options that I have listed in take  
> IV (the
> So if you are happy for that capability to be there, why are you wasting  
> so much band width arguing against it?

Ian (and I, and maybe others) are not clear on why we need the framing to  
be consistent for multiplexing and non-multiplexing scenarios. I started a  
separate thread on the subject as none of the current thread subjects seem  
to make much sense with respect to what is being debated. I would  
appreciate if you could follow-up there.

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 04 Aug 2010 02:05:02 -0700

On Aug 4, 2010, at 1:41 AM, Roberto Peon wrote:

> We haven't been able to modify the linux kernel to do this yet. It *is* "only software," but having the kernel track the skbufs per-application would require lots of changes. IT hasn't happened yet, and I don't know if the complexity in the kernel is the best thing.

Out of curiosity, why is attributing these resources to a userspace app an issue? Is it insufficient to reconfigure the kernel to just have more of these? I'm sure there is a good reason but I think I missed that step in the explanation.

Regards,
Maciej

---

From: Roberto Peon <[email-masked]>
Date: Wed, 4 Aug 2010 02:17:16 -0700

On Wed, Aug 4, 2010 at 2:05 AM, Maciej Stachowiak <[email-masked]> wrote:

> Out of curiosity, why is attributing these resources to a userspace app an
> issue? Is it insufficient to reconfigure the kernel to just have more of
> these? I'm sure there is a good reason but I think I missed that step in the
> explanation.
>

I'll have to check to see how much I'm allowed to say on the subject before
I disclose anything more in that dimension. I can't tell you how frustrating
that is!
-=R

---

From: Pieter Hintjens <[email-masked]>
Date: Wed, 4 Aug 2010 14:09:36 +0200

On Wed, Aug 4, 2010 at 9:59 AM, Greg Wilkins <[email-masked]> wrote:

> the possibility that transparent multiplexing might be needed?

I'm going to agree with Maciej and Ian here.  Multiplexing on top of
TCP (actually on top of HTTP) is solving a problem that should be done
by the transport protocol, and is solved by transports like SCTP.
Multiplexing risks messing with TCP's own balancing algorithms.  It is
difficult to get right, you need credit based flow control mechanisms
that are delicate to get right.  It is unclear at what level you'd
authenticate a multiplexed connection.

Not to say that it's not worth doing, but without measurements that
show its value, and some testable proof of concepts, I'd be very
careful about making it a first class citizen of a new protocol.

What would make sense is to reserve space to allow multiplexing, and
to allow people who believe they really need it to provide examples of
its value.  Then, that experience can be standardized.  It's not
sensible to be experimenting and standardizing at the same time.

It would be really great to see a WebSockets protocol that already
properly covers the basic points, emerges from consensus, and fits
neatly into the HTTP mechanisms and fabric.  Such a protocol should
have space to evolve over time and I'd expect multiplexing algorithms
(plural) to show their worth and be standardized later.

While I'm on the soapbox I'd also like to plea for a pure binary
protocol (length delimited frames in the simplest possible format[1])
that can encapsulate text frames and multiplexing and other
higher-level framing semantics without these being part of the core
protocol spec.

-Pieter

[1] http://rfc.zeromq.org/spec:2

---

From: Patrick McManus <[email-masked]>
Date: Wed, 04 Aug 2010 10:19:28 -0400

On Tue, 2010-08-03 at 22:51 +0000, Ian Hickson wrote:
>  For example, say you have a video app and an IM app sharing a 

Your faith in TCP solving this problem better than multiplexed WS could
is misplaced. This is actually a severe TCP short coming.

What you describe is a classic competition between an elephant flow and
a mouse (sometimes known as thin) flow. And in TCP the elephant flow
completely dominates. Indeed if you have one flow desiring unlimited
bandwidth (i.e. it has a large document to send as fast as possible)
competing with 500 flows sending 1 unit a sec over a 200 unit/sec medium
you are going to see almost entirely the big flow being successful. It
benefits from its large measured windows and active feedback into its
loss recovery algorithms.. the mouse flows on the other hand have tiny
windows based on conservative constants and rely on crude timers to
detect losses and retransmissions. If the underlying link is truly at
capacity the mouse flows will likely have problems even connecting at
high success rates.

This is a classic TCP level problem - it is not fair between mice and
elephants. It is a significant part of the motivation for SCTP. I think
most people think that SCTP transport is not realistic for the
websockets problem, so the notion of moving this into websockets
directly where that implementation could appropriately prioritize
between flows makes a lot of sense (for this and other reasons). 

Multiplexing inside websockets gives the underlying TCP transport an
opportunity to more actively manage fairness, as well as use that
aggregation to become a more active bidirectional TCP flow. And more
active TCP flows simply perform better.

H

---

From: Patrick McManus <[email-masked]>
Date: Wed, 04 Aug 2010 12:58:02 -0400

On Wed, 2010-08-04 at 01:13 -0700, Maciej Stachowiak wrote:

> 
> Personally, I am open to the idea that multiplexing is so valuable
> that it should be in the core protocol, but I'd really like to see
> some data. 

The prime attractive properties of multiplexing are that it reduces TCP
handshakes and increases the number of messages carried over a single
TCP flow. I'll cite some advantages to that (and even some data!):

1] Save 1 RTT of latency over the parallel web sockets approach on the
client side as each three way handshake is eliminated. (a client side
benefit to multiplexing!). Trying to avoid this by piggybacking the
first data packet on the SYN or trying to pipeline the data without
waiting for the SYN ACK are not deployable schemes. Improved latency is
a goal of websockets I believe.

2] If the application serializes messages onto the same web socket
connection without multiplexing via tcp then head of line blocking
becomes a serious problem. This is property is one of the barriers to
effective deployment of pipelining in HTTP. Multiplexing lets you
control prioritization at a sub message level without introducing tcp
problems.

3] Socket accept rates are often server bottlenecks. See
http://mailinator.blogspot.com/2008/08/benchmarking-talkinator.html
where the author, who has a real talent for scaling server side
implementations, goes from 25k transactions per second to 39k
transactions when using http persistent connections to reduce his
connection rate. Multiplexing allows the connection rate to be reduced
even more as their is no need for clients to open parallel connections.
The post is actually describing a transaction with a lot of processing,
some classes of simple web sockets transactions (e.g. "mouse over") will
see even more radical ratios.

4] I cite a number of advantages regarding fair sharing of bandwidth on
multiplexed vs parallel tcp here:
http://www.ietf.org/mail-archive/web/hybi/current/msg02884.html 

5] TCP sender congestion control maps poorly to parallelized
connections. Obviously with parallel connections the chance of one being
idle is greater than if they are multiplexed onto a single tcp flow.
Idle connections are in TCP slow start - which means they are using a
cwnd based on a constant. (probably 3, but it varies.). There is nothing
particularly right about 3, its juts a relatively conservative number
totally independent of your application. If you want to send more than 3
4500 bytes it will cost you an extra (or many extra) RTT of useless
delay to do so in order to grow that window - if this was multiplexed
onto a relatively active connection the window would probably be
sufficient to send it all in one burst - to the low latency delight of
your users. The opposite problem also occurs - if you have a large
number of idle connections (e.g. 20) then they all are sitting with a
cwnd of 3. But really that is an effective cwnd of 60 if they all
decided to to send at one time - and that is a more likely scenario than
TCP anticipates. Those 20 connections are really related in a way TCP
didn't anticipate, so a certain event (e.g. page reload) might have them
all spring into action simultaneously sending 60 packets out in a burst
which causes buffer overflows and packet loss. That packet loss in turn
is severely slow to be repaired vs the multiplexed case because those
recently idle flows have useless ack clocks. Whether it is 3 or 60 they
are both wrong because they are not based on the feedback of an active
connection. Reducing the number of connections over the same path makes
it easier to get this right (its still a challenge, of course)

Some Google folks have a recent paper in sigcomm (and before the tcpm
ietf wg as well) that discusses this interaction in a very interesting
way: "An argument for increasing TCP's initial congestion window"
http://portal.acm.org/citation.cfm?id=1823844.1823848

---

From: Jack Moffitt <[email-masked]>
Date: Wed, 4 Aug 2010 10:59:23 -0600

> more able to uphold that server's fairness policies than if the browser

The cooperative multitasking comparison is completely, you are just
assuming that the shared resource is still a CPU. For example, if you
want low latency between these application multiplexed streams, then
each stream must keep messages very small.  The shared resource here
is the network pipe.

Application level multiplexing requires that some server-side
multiplexer split the message into small enough parts, and then
software in JS reconstruct those parts before handing off the message.
Not only is the software now more complicated on the client, but the
server is potentially not sending WebSocket messages that would make
sense to a non-multiplexed client app.

Multiplexing is very complicated, sure. But if it's in the app level
it's *extremely* complicated for everyone. Special multiplexing
servers that are probably app-aware are needed server side, and
special demultiplexers are needed client side. Not only that, but any
efficiency gains for this require all the developers to jump through
these hoops collectively (in the situation where many composed apps
that use the same services exist on a single page or in separate
tabs).

This sounds a lot like cooperative multitasking to me.

If multiplexing is in the protocol, client developers don't need to do
anything special. Efficiency is automatically gained whenever
possible. It does put a burden on the browser developers and the
server developers, but I believe that is exactly where such a burden
should be placed.

The latency issue is real, and I experience it all the time with XMPP
based applications on the Web. Large stanzas kill the latency. Working
around this by doing special multiplexing is so complicated that we've
always chosen to punt. It's probably way too late to add any
protocol-level multiplexing to XMPP, but XMPP does have
recommendations on stanza size to compensate for this problem somewhat
(a moderately enforceable lower limit on cooperation).

The problem with shared workers is that they work great when a single
service needs to be shared by the same developer across tabs or across
widgets in a single page. It doesn't as much (at least not without a
lot of voluntary cooperation) when every page in your browser is a
different site, but they all need the same WebSocket service. An
example of such a thing would be Google Analytics if the reporting
service were WebSocket based.  I'm sure Google would like it to be one
socket per user and not one socket per frame per tab per user.

jack.

---

From: <[email-masked]>
Date: Wed, 4 Aug 2010 18:10:15 +0100

Maciej,

I would have thought that the 'concerns about this protocol' at the top of RFC1945 due to the number of parallel TCP sessions being opened in parallel, with resource starvation on hosts, and poor performance due to multiple parallel slowstarts, and the reworking of HTTP to include (not that well used, even now...) pipelining and persistence in RFC2068, would be more than enough data to show the value of multiplexing in HTTP-like protocols, and the necessity for multiplexing when using TCP.

But then, I lived through the 1990s, and spent a lot of that decade wishing web browsers were usable.

L.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Patrick McManus
Sent: 04 August 2010 17:58
To: Server-Initiated HTTP
Subject: Re: [hybi] Framing proposal with start/end demarcation [multiplexing]

> that it should be in the core protocol, but I'd really like to see 

The prime attractive properties of multiplexing are that it reduces TCP handshakes and increases the number of messages carried over a single TCP flow. I'll cite some advantages to that (and even some data!):

1] Save 1 RTT of latency over the parallel web sockets approach on the client side as each three way handshake is eliminated. (a client side benefit to multiplexing!). Trying to avoid this by piggybacking the first data packet on the SYN or trying to pipeline the data without waiting for the SYN ACK are not deployable schemes. Improved latency is a goal of websockets I believe.

2] If the application serializes messages onto the same web socket connection without multiplexing via tcp then head of line blocking becomes a serious problem. This is property is one of the barriers to effective deployment of pipelining in HTTP. Multiplexing lets you control prioritization at a sub message level without introducing tcp problems.

3] Socket accept rates are often server bottlenecks. See http://mailinator.blogspot.com/2008/08/benchmarking-talkinator.html
where the author, who has a real talent for scaling server side implementations, goes from 25k transactions per second to 39k transactions when using http persistent connections to reduce his connection rate. Multiplexing allows the connection rate to be reduced even more as their is no need for clients to open parallel connections.
The post is actually describing a transaction with a lot of processing, some classes of simple web sockets transactions (e.g. "mouse over") will see even more radical ratios.

4] I cite a number of advantages regarding fair sharing of bandwidth on multiplexed vs parallel tcp here:
http://www.ietf.org/mail-archive/web/hybi/current/msg02884.html 

5] TCP sender congestion control maps poorly to parallelized connections. Obviously with parallel connections the chance of one being idle is greater than if they are multiplexed onto a single tcp flow.
Idle connections are in TCP slow start - which means they are using a cwnd based on a constant. (probably 3, but it varies.). There is nothing particularly right about 3, its juts a relatively conservative number totally independent of your application. If you want to send more than 3 4500 bytes it will cost you an extra (or many extra) RTT of useless delay to do so in order to grow that window - if this was multiplexed onto a relatively active connection the window would probably be sufficient to send it all in one burst - to the low latency delight of your users. The opposite problem also occurs - if you have a large number of idle connections (e.g. 20) then they all are sitting with a cwnd of 3. But really that is an effective cwnd of 60 if they all decided to to send at one time - and that is a more likely scenario than TCP anticipates. Those 20 connections are really related in a way TCP didn't anticipate, so a certain event (e.g. page reload) might have them all spring into action simultaneously sending 60 packets out in a burst which causes buffer overflows and packet loss. That packet loss in turn is severely slow to be repaired vs the multiplexed case because those recently idle flows have useless ack clocks. Whether it is 3 or 60 they are both wrong because they are not based on the feedback of an active connection. Reducing the number of connections over the same path makes it easier to get this right (its still a challenge, of course)

Some Google folks have a recent paper in sigcomm (and before the tcpm ietf wg as well) that discusses this interaction in a very interesting
way: "An argument for increasing TCP's initial congestion window"
http://portal.acm.org/citation.cfm?id=1823844.1823848

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 04 Aug 2010 19:24:49 +0200

> socket per user and not one socket per frame per tab per user.

That seems false. The script Google uses to embed Google Analytics could  
just use a SharedWorker.

---

From: Scott Ferguson <[email-masked]>
Date: Wed, 04 Aug 2010 10:40:31 -0700

> sensible to be experimenting and standardizing at the same time.
+1

How or whether to reserve that space is all we need to decide now, I think.

Here are a few ways that space could be reserved for multiplexing:

  a) Reserve 8 bits in the header for multiplexing and use them for the 
channel id.
  a') other reserve variants (variable length frame headers)
  b) Switch/flow opcodes, but keep the 3-byte frame
  c) UDP - if adding all the complexity, why not enable VOIP at the same 
time?
  d) Mime-type headers
  e) SCTP, etc.

If we can reject a few of these, we can keep the core protocol simple 
(and keep the extension discussion on track.)

a) Reserve 8 bits in the standard frame header

  +---------+-----------+-----------+------------+-------------+
  | frag(1) | unused(3) | opcode(4) | channel(8) | length (16) |
  +---------+-----------+-----------+------------+-------------+

The core spec would have unused(8) and not mention the channel. There 
would also need to be an opcode or two reserved for flow control.

+ fast and stateless channels for multiplexing
+ if not enabled, the channel bits could be used for large frames (24 bits)
 - takes up one extra byte for every frame, even when multiplexing not 
enabled
 - an extra opcode or two still need to be reserved for flow control

a') other bit-reserve variants

The channel bits could also be reserved using one of the dynamic-length 
variations that are being discussed. So the base frame could be 3-bytes, 
but the multiplexing negotiation could extend it to:

  +---------+-----------+-----------+-------------+
  | frag(1) | unused(3) | opcode(4) | length (16) |
  +---------+-----------+-----------+-------------+
  +------------+
  | channel(8) |
  +------------+

  + core frame not waste bytes multiplexing
  - multiplexing needs to reimplement frame parsing (should be trivial, 
though)
  - general extension of this idea is more complicated to implement
  - complicated specification verbiage needed to explain this

b) switch/flow opcodes

Here the frame would be the same 3-byte frame that we've been 
discussing. The switch opcode would change the current channel to a new one:

 %x6 - switch to channel (16-bit payload for the channel id)

 %x06 %x0001 %x13 - switch to channel #19

 + keeps basic frame small and simple
  - slight extra multiplexing complexity because of stateful nature
  - needs reserved opcodes

c) UDP

If you're adding multiplexing and the flow control, you're 
re-implementing a huge chunk of TCP, but keeping the minuses like the 
lost-ack penalty that makes highly-responsive applications like VOIP 
work poorly.

So, if you're adding all the complexity to add multiplexing to TCP, why 
not go all the way and enable a whole new class of applications.

  + zero impact on core
  + enables new classes of applications
  - is more complicated than tcp multiplexing
  ? better/worse memory consumption?

d) mime-type headers

The idea would be to encode channel id in a key: value header followed 
by the payload (I think)

%x04 %xNN %xNN
Multiplexing-Channel: 13 CRLF
CRLF
<data>

(This doesn't seem like a viable solution to me, but I'm adding it for 
completeness.)

 + I can't think of any
  - requires the multiplexing to parse ascii
  - payload has a variable offset

e) SCTP

Basically, decide that the kernel is better suited for multiplexing.

  + zero impact on core
  + multiplexing is the kernel's job
  - not universally available (?)
  - seems to have worse performance than TCP (?)

-- Scott

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 18:13:17 +0000 (UTC)

On Wed, 4 Aug 2010, Jack Moffitt wrote:
> each stream must keep messages very small.  The shared resource here is 
> the network pipe.

I understand what is being shared. The reason the analogy doesn't hold is 
that it presumes that the code constructing or receiving the message gets 
to decide how many WebSocket frames the message takes, which is false in 
a scenario where there is a JS library managing the connection.

> multiplexer split the message into small enough parts, and then software 
> in JS reconstruct those parts before handing off the message.

This is as opposed to browser-native-multiplexing where some server-side 
multiplexer splits the message into small enough parts, and then software 
in the browser (possibly still JS -- some browsers, e.g. Firefox, are in 
fact written in part in JS) reconstructs those parts before handing off 
the message.

> sense to a non-multiplexed client app.

Why would it matter if they made sense to a non-multiplexed client app? No 
such app is ever going to see them.

> it's *extremely* complicated for everyone.

Not at all. It's identical on the server side, and identical to client 
developsrs on the client side; the only difference is that instead of 
being complicated for browser vendors, it's complicated for JS library 
authors -- authors who typically have a better idea of what the 
subprotocol is going to be, and who understand the server's specific needs 
better, than the browser vendors could.

> Special multiplexing servers that are probably app-aware are needed 
> server side

Not at all. Just define a multiplexing subprotocol and layer everything on 
that.

> Not only that, but any efficiency gains for this require all the 
> developers to jump through these hoops collectively (in the situation 
> where many composed apps that use the same services exist on a single 
> page or in separate tabs).

Just like in browser-implemented multiplexing case requires all the 
developers to jump through the hoop of using WebSocket in the first place 
collectively. There's no difference here, it's just using a 
browser-provided API or a JS-library-provided API.

> widgets in a single page. It doesn't as much (at least not without a lot 
> of voluntary cooperation) when every page in your browser is a different 
> site, but they all need the same WebSocket service.

Sure it does. Just only provide that service via a JS library. The server 
can trivially enforce this so that only that library can connect to the 
server, so that it can't be avoided. (This is in contrast to 
browser-provided multiplexing, where the server has no way to enforce that 
the browser will actually implement the multiplexing.)

> An example of such a thing would be Google Analytics if the reporting 

Cross-origin SharedWorkers will handle this case fine. This is in fact a 
great example of where multiplexing is far more complexity on the server 
side than is warranted by the application. You don't need to establish a 
new channel and so forth; each page just needs to send a single payload. 
All you need is a SharedWorker of a few lines that queues up all the 
payloads and sends them one after the other.

---

From: <[email-masked]>
Date: Wed, 4 Aug 2010 19:18:23 +0100

"It would be really great to see a WebSockets protocol that already
properly covers the basic points, emerges from consensus, and fits
neatly into the HTTP mechanisms and fabric.
[..]
While I'm on the soapbox I'd also like to plea for a pure binary
protocol"

Inherent contradiction between those two desires, surely?

L.

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Pieter Hintjens [[email-masked]]
Sent: 04 August 2010 13:09
To: Greg Wilkins
Cc: Server-Initiated HTTP
Subject: Re: [hybi] Framing proposal with start/end demarcation

-Pieter

---

From: Pieter Hintjens <[email-masked]>
Date: Wed, 4 Aug 2010 20:28:15 +0200

> Here are a few ways that space could be reserved for multiplexing:

Very nice round up.

>  a) Reserve 8 bits in the header for multiplexing and use them for the

+1.  The extra byte will inevitably come in handy for *something* and
it rounds the header up to 4 bytes.

>  a') other reserve variants (variable length frame headers)

-1 on being more complex.

>  b) Switch/flow opcodes, but keep the 3-byte frame

-1 on being more complex for no real saving.

>  c) UDP - if adding all the complexity, why not enable VOIP at the same

+1 for pointing out the silliness of reimplementing TCP but -1 for
being too off-beat.

>  d) Mime-type headers

-1 for being provably the worst possible option.

>  e) SCTP, etc.

+1 for delegating kernel-level work to the kernel.

> If we can reject a few of these, we can keep the core protocol simple (and
> keep the extension discussion on track.)

A single reserved byte can co-exist with a recommendation to explore
SCTP and other potential alternatives to TCP.

-Pieter

---

From: Jack Moffitt <[email-masked]>
Date: Wed, 4 Aug 2010 12:29:05 -0600

> That seems false. The script Google uses to embed Google Analytics could

In one sense, it was a bad example. But plenty of other services do
not provide JavaScript embeddable gizmos to remove this issue.

It's also true that in several such cases, the service provider will
be somewhat proactive in getting developers to use their library if it
provides the service with performance benefits.

My point was that all this work revolves around the voluntary
cooperation of app developers and providers instead of the voluntary
cooperation of browser and server developers.

jack.

---

From: Pieter Hintjens <[email-masked]>
Date: Wed, 4 Aug 2010 20:34:38 +0200

> Inherent contradiction between those two desires, surely?

Is this better: "pure binary protocol following the upgrade from HTTP"?

By "pure binary" I mean framing that defines opaque blobs, giving a
protocol that acts only as a transport, neutral to whatever data it is
carrying.  This allows a pure separation between data representation
and transport, so either can be modified in the future at lowest cost.
 So no text frames, no ascii, no UTF8 delimiters, no attempt at this
level to be close to language-specific serialization.

-Pieter

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 5 Aug 2010 08:49:32 +1000

> Not at all. Just define a multiplexing subprotocol and layer everything on
The discussion in the framing threads has been about trying to nail down
more details of extension mechanism so that we can make sure that we have
solutions for issues like mixing of multiple extensions - multiplexing +
metadata + compression + myApplicationSpecificSubProtocol.    Do you have a
solution that would allow multiple subprotocols to be simultaneously
activated?

Can you please give us concrete proposals of how you would do this rather
than just hand waiving that it is possible,

regards

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 5 Aug 2010 02:05:34 +0100

> +1 for delegating kernel-level work to the kernel.

-100 for being undeployable over web client infrastructure (home routers etc.)?

Virtually everyone at home these days is behind a NAT router.

As far as I know, those can't route SCTP until they are explicitly
upgraded to handle it.  But most home routers are too old to get any
more firmware updates.

I'll be delighted if someone can tell me I'm mistaken about that.

-- Jamie

---

From: Roberto Peon <[email-masked]>
Date: Wed, 4 Aug 2010 18:27:11 -0700

That is my understanding as well.
Which is really too bad because the other SCTP features (other than the
multi-stream nature) are interesting as well.
-=R

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 05 Aug 2010 09:11:23 +0200

On Thu, 05 Aug 2010 00:49:32 +0200, Greg Wilkins <[email-masked]> wrote:
> metadata + compression + myApplicationSpecificSubProtocol.    Do you  
> have a solution that would allow multiple subprotocols to be  
> simultaneously
> activated?

You seem to be mixing subprotocols and extensions. Extensions can affect  
the protocol, subprotocols just use the protocol.

Since we want to keep the protocol simple, even though we will add  
extensions there will not be very many, and it should be relatively  
straightforward to add them when we have determined there is a need. Via  
some kind of negotiation. Either the server telling the client. Or the  
client telling the server.

I don't think we need to consider "myApplicationSpecificSubProtocol" for  
extensions by the way.

---

From: Rob Sayre <[email-masked]>
Date: Thu, 05 Aug 2010 00:16:58 -0700

On 8/5/10 12:11 AM, Anne van Kesteren wrote:
>
>
> I don't think we need to consider "myApplicationSpecificSubProtocol" 
> for extensions by the way.

I agree. This argument is basically an obfuscated variant of the the old 
"distributed extensibility" argument. Never very useful, but always very 
complex and error prone.

- Rob

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 5 Aug 2010 19:09:35 +1000

On 5 August 2010 17:11, Anne van Kesteren <[email-masked]> wrote:
>> metadata + compression + myApplicationSpecificSubProtocol.    Do you have
>> a solution that would allow multiple subprotocols to be simultaneously
>> activated?
>
> You seem to be mixing subprotocols and extensions. Extensions can affect the
> protocol, subprotocols just use the protocol.

Not me!  but there is confusion because every time people talk about
extensions, the common response is to do that in a subprotocol.

My STRONG preference is to completely separate the concept of an
extension and a subprotocol, so that is why I keep asking for
clarification from those that keep doing the "do that in a
subprotocol" hand waive.

> client telling the server.

The BIG problem I have with this is your use of "we".     I think many
will lose the will to live if in order to extend the protocol they
have to come to this group and convince everybody that "we" must add
this extension.

I do not think we want a centrally managed extension process.

> I don't think we need to consider "myApplicationSpecificSubProtocol" for

Totally agree - unless extensions are forced to use subprotocols.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 05 Aug 2010 11:31:42 +0200

On Thu, 05 Aug 2010 11:09:35 +0200, Greg Wilkins <[email-masked]> wrote:
>>> The discussion in the framing threads has been about trying to nail  
>>> down more details of extension mechanism so that we can make sure that  
>>> we have solutions for issues like mixing of multiple extensions -  
>>> multiplexing + metadata + compression +  
>>> myApplicationSpecificSubProtocol.    Do you have a solution that would  
>>> allow multiple subprotocols to be simultaneously
>>> activated?
>>
>> You seem to be mixing subprotocols and extensions. Extensions can  
>> affect the protocol, subprotocols just use the protocol.
>

The response is to first implement it in a subprotocol with added  
complexity to the application layer and if it reaches critical mass in the  
wild we can consider it as extension to the protocol. I.e. give people  
sufficient building blocks so they can implement multiplexing,  
compression, chunking, etc. using the base protocol and if developers  
actually start building these to the point where they reach critical mass  
we can add them to the base protocol taking complexity out of their hands  
in the future.

> clarification from those that keep doing the "do that in a

I thought the above was clear.

> I do not think we want a centrally managed extension process.

For the base protocol I think we do. As Rob Sayre says, distributed  
extensibility is where complexity lies for often very little gain.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 5 Aug 2010 02:41:12 -0700

On Thu, Aug 5, 2010 at 2:31 AM, Anne van Kesteren <[email-masked]> wrote:

> sufficient building blocks so they can implement multiplexing, compression,
> chunking, etc. using the base protocol and if developers actually start
> building these to the point where they reach critical mass we can add them
> to the base protocol taking complexity out of their hands in the future.
>
>
>
I really don't think we want to be forced to come back to this WG to define
a protocol extension

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 5 Aug 2010 19:48:46 +1000

Anne,

your response is not internally consistent.

You agree that extensions and subprotocols should be separate things,
but then say that all extensions should first be developed as
subprotocols?   How can transparent multiplexing be implemented if it
requires the application layer to cooperate by using a library.

You also appear to believe that the only two choices for enhancing the
protocol is either in the base protocol or in the applications
subprotocol.   I think that we need to allows a third choice of
extensions that are implemented above the base protocol, but below the
application.  In fact I would use that as the definition of an
"extension".

regards

On 5 August 2010 19:31, Anne van Kesteren <[email-masked]> wrote:
> The response is to first implement it in a subprotocol with added complexity
> to the application layer and if it reaches critical mass in the wild we can
> consider it as extension to the protocol. I.e. give people sufficient
> building blocks so they can implement multiplexing, compression, chunking,
> etc. using the base protocol and if developers actually start building these
> to the point where they reach critical mass we can add them to the base
> protocol taking complexity out of their hands in the future.
>
>

---

From: James Graham <[email-masked]>
Date: Thu, 05 Aug 2010 11:56:36 +0200

On 08/05/2010 11:09 AM, Greg Wilkins wrote:
> On 5 August 2010 17:11, Anne van Kesteren<[email-masked]>  wrote:
>> On Thu, 05 Aug 2010 00:49:32 +0200, Greg Wilkins<[email-masked]>  wrote:
>>>

What do you mean "totally separate"? Inevitably there is some overlap 
between what the subprotocol can do and what the base level protocol can 
do, all that changes is where the responsibility lies for doing that 
thing. For example let's consider a very simple hypothetical base 
protocol that has no ability to support chunking. Now an application 
developer who needs chunking for their subprotocol (which we can assume, 
for simplicity, just uses text frames) could send messages as JSON and 
use one field in the JSON object to indicate if there are more chunks of 
the message. Then the client would reassemble the message at the 
application level. Note this is not supposed to be a highly efficient 
way of adding subprotocol-level chunking, just a simple-to-follow one.

Of course if there are lots of people doing that, it makes sense to bake 
it into the core protocol where it can be done transparently. But there 
is very little that *must* be done in the core protocol. Some things 
might be more efficient, or more easilly reusable, as extensions to the 
core, but we will find out which are really necessary by studying what 
actual deployments do.

Note that given the efficiency of modern javascript, it is not a-priori 
obvious that making people add features at the application/subprotocol 
layer is a performance killer.

> have to come to this group and convince everybody that "we" must add

I don't really understand who would be adding extensions. There seem to 
be two cases:

a) Web browser clients
a) Non-browser clients

In the case that a web browser is adding an extension we certainly don't 
want it to be proprietary and want it to come through the standards 
process as it is being developed. In the non-browser case it seems like 
you can do whatever you want as long as you don't care about 
interoperability with browsers or other generic clients. Indeed if you 
don't need browsers to access your service it seems that you could just 
as well use ordinary TCP. If you do need browsers to access your 
service, and want them to use your extensions, then you should be 
bringing your extensions through the standards process.

> I do not think we want a centrally managed extension process.

I think that it is essential. I could be persuaded that there is some 
need for a scheme to indicate short-term vendor-specific extensions like 
CSS has.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 05 Aug 2010 11:57:53 +0200

> your response is not internally consistent.

I disagree :-)

> requires the application layer to cooperate by using a library.

You cannot implement transparent multiplexing in the application layer of  
course. But you can implement some form of multiplexing as explained in  
other emails by e.g. Ian. And if it turns out that will be common, we can  
make multiplexing transparent at the protocol level later.

> application.  In fact I would use that as the definition of an

I don't understand the distinction.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 05 Aug 2010 11:59:03 +0200

On Thu, 05 Aug 2010 11:41:12 +0200, Ian Fette (イアンフェッティ)  
<[email-masked]> wrote:
> I really don't think we want to be forced to come back to this WG to  
> define a protocol extension

We need to do it somewhere if we want clients to be interoperable. This WG  
seems to have the right set of people, but we could do it somewhere else.  
I don't feel too strongly about that.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 07:55:01 -0700

> to the base protocol taking complexity out of their hands in the future.

So, playing devils advocate, why are we implementing any framing at all?
It is trivial for an application developer to do framing themselves (they
can do length prefixing as a string, they can do sentinel demarcation, etc,
etc.).
Why aren't we waiting to see a strong need for this from the application
developers?
Now I don't really think that a lack of framing is a good thing, but if
we're applying this litmus test we should apply it to all features, not just
the ones that I want! :)

> extensibility is where complexity lies for often very little gain.

I don't see extensions ever actually being successful. What protocols have
had success with extensions which changed their fundamental semantics in the
past?
My fear is that multiplexing, compression, etc. will fall by the wayside and
application developers will have to do all of it over again each time (that
is what doing it as a subprotocol implies, after all!). I've proposed a
low-cost way to add metadata which would enable semantic change without
changing any framing parsers as a hedge-- I *want* to be wrong in this.

-=R

---

From: James Graham <[email-masked]>
Date: Thu, 05 Aug 2010 17:08:38 +0200

On 08/05/2010 04:55 PM, Roberto Peon wrote:
>
>
> On Thu, Aug 5, 2010 at 2:31 AM, Anne van Kesteren <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     On Thu, 05 Aug 2010 11:09:35 +0200, Greg Wilkins <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>         On 5 August 2010 17:11, Anne van Kesteren <[email-masked]
>         <mailto:[email-masked]>> wrote:
>
>             On Thu, 05 Aug 2010 00:49:32 +0200, Greg Wilkins
>             <[email-masked] <mailto:[email-masked]>> wrote:
>
>                 The discussion in the framing threads has been about
>                 trying to nail down more details of extension mechanism
>                 so that we can make sure that we have solutions for
>                 issues like mixing of multiple extensions - multiplexing
>                 + metadata + compression +
>                 myApplicationSpecificSubProtocol.    Do you have a
>             You seem to be mixing subprotocols and extensions.
>             Extensions can affect the protocol, subprotocols just use
>             the protocol.
>
>
>     complexity to the application layer and if it reaches critical mass
>     in the wild we can consider it as extension to the protocol. I.e.
>     give people sufficient building blocks so they can implement
>     multiplexing, compression, chunking, etc. using the base protocol
>     and if developers actually start building these to the point where
>     they reach critical mass we can add them to the base protocol taking
>     complexity out of their hands in the future.
>
>
> It is trivial for an application developer to do framing themselves
> (they can do length prefixing as a string, they can do sentinel
> demarcation, etc, etc.).
> we're applying this litmus test we should apply it to all features, not
> just the ones that I want! :)

Because we can't have a sensible javascript API at all without making 
the protocol message-based. In particular a typical sockets-style API 
with blocking .recv() calls doesn't play well with browsers (you don't 
want to let people block the UI thread). Since the API has to be 
asynchronous, making it messaged based and using events flows very 
naturally from the basic requirements.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 08:19:06 -0700

On Thu, Aug 5, 2010 at 8:08 AM, James Graham <[email-masked]> wrote:

> Because we can't have a sensible javascript API at all without making the
> protocol message-based. In particular a typical sockets-style API with
> blocking .recv() calls doesn't play well with browsers (you don't want to
> let people block the UI thread). Since the API has to be asynchronous,
> making it messaged based and using events flows very naturally from the
> basic requirements.

Why not do it in a Worker and provide the appropriate JS which sets this up
automatically?
Or, why use blocking recv() calls? Use nonblocking recv()!
This would result in a far more simple protocol at the cost of moving
complexity to the application writers.

(Again, this is me playing devil's advocate. I don't think a lack of framing
is a good thing).
-=R

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 16:31:37 +0100

On Thu Aug  5 16:08:38 2010, James Graham wrote:
>>                 solution that would allow multiple subprotocols to  
>> be
>>                 simultaneously
>>                 activated?
>> 
>> 
> Because we can't have a sensible javascript API at all without  
> making the protocol message-based. In particular a typical  
> sockets-style API with blocking .recv() calls doesn't play well  
> with browsers (you don't want to let people block the UI thread).  
> Since the API has to be asynchronous, making it messaged based and  
> using events flows very naturally from the basic requirements.

Well, you could do asynchronous streamed delivery just fine, but I  
readily agree that a message-based delivery makes the application  
programmer's life massively easier, not least because the numbers of  
developers who get confused by partial reads is astonishing.

And if you have messages, then you need framing.

Dave.

---

From: Jack Moffitt <[email-masked]>
Date: Thu, 5 Aug 2010 09:38:52 -0600

> making it messaged based and using events flows very naturally from the

I think that Node.js and other similar projects show very clearly that
asynchronous raw sockets APIs are possible and rather simple in
JavaScript.

However, typical client-side JavaScript programmers are probably not
used to handling things at such a low level.

I think that having the protocol be message based is prudent, and it
requires some kind of framing to demarcate those messages.

jack.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 08:40:17 -0700

> requires some kind of framing to demarcate those messages.

I agree, but what is our litmus test to show that this is the case?
-=R

---

From: Jack Moffitt <[email-masked]>
Date: Thu, 5 Aug 2010 09:57:25 -0600

> I agree, but what is our litmus test to show that this is the case?

To show that it is prudent to have a message based protocol? I think
simplicity was the reasoning. It's simpler to deal with messages than
with arbitrary portions of a stream which must then be parsed into
whatever semantic units are needed.

>From the requirements document:

    REQ. 2:  The WebSocket Protocol MUST be able to handle (send and
      receive) messages on top of a TCP data stream connection.

   Reason: transfer data as message prevents the receiver to parse/
   handle partial content.

For as long as I have known about the hybi group, this has been in the
requirements. I will note however, that many of the debates we have on
this list are debates about these requirements. Whether messages are
needed or not has not been one of the contentious issues (in my
opinion based on list traffic).

jack.

---

From: James Graham <[email-masked]>
Date: Thu, 05 Aug 2010 18:03:22 +0200

On 08/05/2010 05:19 PM, Roberto Peon wrote:
>
>
> On Thu, Aug 5, 2010 at 8:08 AM, James Graham <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     sockets-style API with blocking .recv() calls doesn't play well with
>     browsers (you don't want to let people block the UI thread). Since
>     the API has to be asynchronous, making it messaged based and using
>     events flows very naturally from the basic requirements.
>
>
> Why not do it in a Worker and provide the appropriate JS which sets this
> up automatically?

Sure. But everyone seems to agree that this makes the protocol 
gratuitously difficult to use. And that difficulty applies to everyone 
who wants to use it, which makes it relatively bad compared to something 
that only affects people working at scale.

So, for every possible feature we have to make some judgment call on 
whether we want to support it and if so how. It comes down to weighing 
up the costs of the additional complexity, against its usefulness 
against the value to authors, taking into account at every stage that we 
could be wrong about the value if we have insufficient evidence to 
support the value proposition.

I don't think there is a simple litmus test that will tell us 
unambiguously what choices to make. But I do think that if you start 
simple and build up then you are sure to end up with something that at 
least works in the simple cases, and, as you identify more pain points 
(in usability, performance, etc.), you will cover more and more 
requirements with relatively low risk of being wrong. If you try and 
bake in lots of things from the start you run a high risk of being wrong 
about how the future will pan out. You also run a higher risk of buggy 
or incomplete implementations since implementers have more to do all at 
once. There is,some risk that future additions to the core never take 
off in the same way as the basic protocol. One would imagine that as 
long as we are really adding features that people want, the only 
barriers to adoption would be technical. So it is important to avoid 
such technical barriers upfront. Otherwise making small iterations 
should make the lack of a simple litmus test for determing which 
features will be needed in the future less significant.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 09:32:46 -0700

>> complexity to the application writers.
You cut off the part where I said that I thought that the above was a bad
idea, BTW :)

>
> Sure. But everyone seems to agree that this makes the protocol gratuitously
> difficult to use. And that difficulty applies to everyone who wants to use
> it, which makes it relatively bad compared to something that only affects
> people working at scale.
>

Agreed, but with the coloring that we've been talking about the application
developers instead of the users. If users don't get to use WS to places that
need to scale, then we're penalizing a large segment of the user population.

> whether we want to support it and if so how. It comes down to weighing up
> the costs of the additional complexity, against its usefulness against the
> value to authors, taking into account at every stage that we could be wrong
> about the value if we have insufficient evidence to support the value
> proposition.
>
>
Understood.

> I don't think there is a simple litmus test that will tell us unambiguously
> what choices to make. But I do think that if you start simple and build up
> then you are sure to end up with something that at least works in the simple
> cases, and, as you identify more pain points (in usability, performance,
> etc.), you will cover more and more requirements with relatively low risk of
> being wrong. If you try and bake in lots of things from the start you run a
> high risk of being wrong about how the future will pan out. You also run a
> higher risk of buggy or incomplete implementations since implementers have
> more to do all at once. There is,some risk that future additions to the core
> never take off in the same way as the basic protocol. One would imagine that
> as long as we are really adding features that people want, the only barriers
> to adoption would be technical.

Well, unfortunately we know that isn't true. As we already know it is fairly
difficult to get all the browsers updated to something that speaks a new
protocol. Servers are a far easier chore to upgrade as there are far fewer
controlling parties. The technical challenges are the easy ones-- getting
users to upgrade or change their browsers is fairly difficult. If that
wasn't true there wouldn't be a significant population still using IE6 on
XP.

> So it is important to avoid such technical barriers upfront. Otherwise
> making small iterations should make the lack of a simple litmus test for
> determing which features will be needed in the future less significant.
>

I hope you're right. I'm afraid you won't be--  scaling is not optional. We
may not be able to use websockets if we can't get it to scale efficiently,
and I'm truly afraid that moving the complexity of multiplexing to the JS
applications will fail. And our ability to do that is limited to Google
owned properties.

So, what is the litmus test? I think that we should be thinking about the
users of the browsers in addition to the application developers, but I
haven't seen many mentioning anything like this explicitly.

-=R

---

From: James Graham <[email-masked]>
Date: Thu, 05 Aug 2010 19:05:14 +0200

On 08/05/2010 06:32 PM, Roberto Peon wrote:

>     simple and build up then you are sure to end up with something that
>     at least works in the simple cases, and, as you identify more pain
>     points (in usability, performance, etc.), you will cover more and
>     more requirements with relatively low risk of being wrong. If you
>     try and bake in lots of things from the start you run a high risk of
>     being wrong about how the future will pan out. You also run a higher
>     risk of buggy or incomplete implementations since implementers have
>     more to do all at once. There is,some risk that future additions to
>     the core never take off in the same way as the basic protocol. One
>     would imagine that as long as we are really adding features that
>     people want, the only barriers to adoption would be technical.
>
>
> Well, unfortunately we know that isn't true. As we already know it is
> fairly difficult to get all the browsers updated to something that
> speaks a new protocol. Servers are a far easier chore to upgrade as
> there are far fewer controlling parties. The technical challenges are
> the easy ones-- getting users to upgrade or change their browsers is
> fairly difficult. If that wasn't true there wouldn't be a significant
> population still using IE6 on XP.

Generally for non-IE browsers the uptake rate of new versions is rapid. 
IE upgrades have historically been slower but one may hope that IE6 is a 
special case; afaict the rate of upgrade is mainly limited by 
IE6-requiring intranet apps. Since modern versions of IE are embracing 
more standards so there should be less browser-specific code on the 
intranet systems written today. Of course enterprise upgrades will still 
be slow compared to typical end users.

Also, taken to the logical conclusion, your argument would suggest 
putting all the features we could imagine in the first version of the 
protocol. But the consequence of that would be a much longer time before 
any deployment at all, and buggy, incomplete initial implementations. 
This seems like a much worse outcome than a relatively feature-free but 
usable version now and the possibility of more features in the future.

>     So it is important to avoid such technical barriers upfront.
>     Otherwise making small iterations should make the lack of a simple
>     litmus test for determing which features will be needed in the
>     future less significant.
>
>
> I hope you're right. I'm afraid you won't be--  scaling is not optional.
> We may not be able to use websockets if we can't get it to scale
> efficiently, and I'm truly afraid that moving the complexity of
> multiplexing to the JS applications will fail. And our ability to do
> that is limited to Google owned properties.

But not everyone is Google. If Google can't use the initial releases of 
websockets that doesn't mean that it is useless or a failure. It does 
mean that there are features that we should quickly look to add after we 
first have browsers shipping the basics.

> So, what is the litmus test? I think that we should be thinking about
> the users of the browsers in addition to the application developers, but
> I haven't seen many mentioning anything like this explicitly.

Indeed we should. But users and authors and implementers are 
intertwined. If things get complex so that implementations are buggy or 
authors simply don't bother then users don't benefit from WebSockets at all.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 10:12:43 -0700

On Thu, Aug 5, 2010 at 10:05 AM, James Graham <[email-masked]> wrote:

> Generally for non-IE browsers the uptake rate of new versions is rapid. IE
> upgrades have historically been slower but one may hope that IE6 is a
> special case; afaict the rate of upgrade is mainly limited by IE6-requiring
> intranet apps. Since modern versions of IE are embracing more standards so
> there should be less browser-specific code on the intranet systems written
> today. Of course enterprise upgrades will still be slow compared to typical
> end users.
>
> Also, taken to the logical conclusion, your argument would suggest putting
> all the features we could imagine in the first version of the protocol. But
> the consequence of that would be a much longer time before any deployment at
> all, and buggy, incomplete initial implementations. This seems like a much
> worse outcome than a relatively feature-free but usable version now and the
> possibility of more features in the future.
>
>
> websockets that doesn't mean that it is useless or a failure. It does mean
> that there are features that we should quickly look to add after we first
> have browsers shipping the basics.

Google, Amazon, Rackspace, VMWare, 3Tera, whomever. I don't think this is a
solely a Google problem, I think we're just the first to start thinking
about it deeply.

> Indeed we should. But users and authors and implementers are intertwined.
> If things get complex so that implementations are buggy or authors simply
> don't bother then users don't benefit from WebSockets at all.
>

Agreed. Is the argument then that we believe that supporting scalability
induces so much complexity as to cause failed implementations?

-=R

---

From: James Graham <[email-masked]>
Date: Thu, 5 Aug 2010 19:55:50 +0200 (CEST)

On Thu, 5 Aug 2010, Roberto Peon wrote:

> Google, Amazon, Rackspace, VMWare, 3Tera, whomever. I don't think this is a solely a Google problem, I think we're just the first
> to start thinking about it deeply.

Sure, I didn't mean to imply it was only a Google problem. But for many 
potential users it is not a significant problem.

> Agreed. Is the argument then that we believe that supporting scalability induces so much complexity as to cause failed
> implementations?
>

It is certianly true that trying to introduce significant complexity right 
now, i.e. in the probably-short time before we have shipping Opera, 
Firefox, Safari and Chrome all with WebSockets support, introduces a 
serious risk of bugs or partial implementations. In practice if the 
intersection of what the aforementioned browsers support is sufficient to 
be usable then it will establish an ecosystem that depends on that 
intersection continuing to be supported in the future, effectively 
creating a de-facto base protocol.

I also believe it is true that introducing significant mandatory 
complexity to servers at any time significantly increases the barrier to 
writing a server i.e. produces failed (potential) server implementations. 
It also increases the risk of bugs in servers which one could presume may 
manifest themselves as significant problems e.g. security issues.

Although these are arguments against starting with something too complex, 
they are not the only arguments that have been made. For example it has 
been pointed out that by starting with something simple we can ensure that 
we are adding the only things that are really needed. In the "now" 
timeframe we don't have the ability to collect data about how to optimise 
for performance and scaling. If we give ourselves more time we can collect 
better data from real world use and end up with a better, more scalable, 
design in the long run.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 05 Aug 2010 12:17:48 -0700

On 8/5/10 10:55 AM, James Graham wrote:
> It is certianly true that trying to introduce significant complexity 
> right now, i.e. in the probably-short time before we have shipping 
> Opera, Firefox, Safari and Chrome all with WebSockets support, 
> introduces a serious risk of bugs or partial implementations. In 
> practice if the intersection of what the aforementioned browsers 
> support is sufficient to be usable then it will establish an ecosystem 
> that depends on that intersection continuing to be supported in the 
> future, effectively creating a de-facto base protocol.
>
> complexity to servers at any time significantly increases the barrier 
> to writing a server i.e. produces failed (potential) server 
> implementations. It also increases the risk of bugs in servers which 
> one could presume may manifest themselves as significant problems e.g. 
> security issues.
>
> Although these are arguments against starting with something too 
> complex, they are not the only arguments that have been made. For 
> example it has been pointed out that by starting with something simple 
> we can ensure that we are adding the only things that are really 
> needed. In the "now" timeframe we don't have the ability to collect 
> data about how to optimise for performance and scaling. If we give 
> ourselves more time we can collect better data from real world use and 
> end up with a better, more scalable, design in the long run.
Agreed, but there is time to establish a forward-looking foundation that 
may remain initially unimplemented in its initial form.

A websocket provides little additional utility when unable to deal with 
more than a single message stream.  Message stream multiplexing is 
indeed complex, and something that most applications should avoid, to 
minimize the number of bugs that might be otherwise generated.   As 
such, the Websocket API needs to introduce an extension that might be 
dubbed a channel or stream.   The first incarnation of this concept may 
only support a single instance.   Building upon the concept of TLV 
message framing,  this utility should strive to eventually provide 
applications the utilities currently enjoyed in other forms of sockets.

For a more secure and robust foundation, framing must use explicit, 
rather than implied state.  This provides better protection from an 
error cascading and not being detected.   In other words, deprecate the 
use of a single Frag bit to encode four states.  The use of compress 
also includes enhanced error checking.  For example, zlib socket 
extensions are able to open either a compress or non-compressed stream.  
The zlib compress algorithm will not increase the size of data being 
compressed, so there is no downside when using this extension.  The 
compression bit in the TLV tag could be hidden, and automatically handled.

When the application sends data to a socket channel, the construction of 
the outbound message should be sensitive to the network's MTU, and limit 
any frame size to fit within a typical packet.  The channels can then be 
processed in a round robin fashion, until the send bytes have been 
consumed.  The TCP transport will provide the flow control and 
congestion avoidance.  A simple round robin scheme should adequately 
support most extensions.

-Doug

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 12:22:31 -0700

On Thu, Aug 5, 2010 at 10:55 AM, James Graham <[email-masked]> wrote:

>  Google, Amazon, Rackspace, VMWare, 3Tera, whomever. I don't think this is
>> a solely a Google problem, I think we're just the first
> now, i.e. in the probably-short time before we have shipping Opera, Firefox,
> Safari and Chrome all with WebSockets support, introduces a serious risk of
> bugs or partial implementations. In practice if the intersection of what the
> aforementioned browsers support is sufficient to be usable then it will
> establish an ecosystem that depends on that intersection continuing to be
> supported in the future, effectively creating a de-facto base protocol.
>

If I had a belief (or at minimum an assurance) that if there was a
continuing worry about scalability as we go forward that we would change the
protocol and get it implemented in Opera, Firefox, Safari and Chrome, I'd
not be making noise-- I'd be cheering! We'd be on our way to an
implementation-first, then standardize process.

>
> I also believe it is true that introducing significant mandatory complexity
> to servers at any time significantly increases the barrier to writing a
> server i.e. produces failed (potential) server implementations. It also
> increases the risk of bugs in servers which one could presume may manifest
> themselves as significant problems e.g. security issues.
>
> they are not the only arguments that have been made. For example it has been
> pointed out that by starting with something simple we can ensure that we are
> adding the only things that are really needed. In the "now" timeframe we
> don't have the ability to collect data about how to optimise for performance
> and scaling. If we give ourselves more time we can collect better data from
> real world use and end up with a better, more scalable, design in the long
> run.
>

This definitely works when the set of developers is small or well
controlled. We as a set of interests are not particularly well controlled,
and certainly the set of developers implementing this won't be small.
If Mozilla, Opera, Google, and Safari were all making their own variations
of their own versions of a standard, I'd believe that we could move forward,
because in the future it would be necessary to standardize.
We're upside-down right now. We're standardizing something that doesn't meet
the needs of entities that serve a large percentage of users with little to
no incentive to change that in the future.

What is the incentive for Mozilla, Opera, and Safari to change their
implementations again in the future? When will this happen?

We're not even talking about limiting the maximum number of concurrent WS
connections that may be created to a site (something which could be
extremely simple in the protocol and something which we've already done for
HTTP).
I'm worried that we're making it so simple right now because we're assuming
that the network and the servers will not be impacted negatively. I'm
worried that we will be negatively impacting the networks, the airwaves
(bandwidth consumption on phones for many multiples more keep-alives), etc.

To me, it feels a bit like global warming! :)

-=R

---

From: Ian Hickson <[email-masked]>
Date: Thu, 5 Aug 2010 19:34:22 +0000 (UTC)

On Thu, 5 Aug 2010, Roberto Peon wrote:
> continuing worry about scalability as we go forward that we would change 
> the protocol and get it implemented in Opera, Firefox, Safari and 
> Chrome, I'd not be making noise-- I'd be cheering!

I don't really know how we can provide you with such a belief beyond what 
has been done so far, which is to say that every browser vendor who has 
commented on the topic has assured us that they intend to follow the same 
procedure with this protocol as with all other Web technologies: keep 
extending it if there's a demonstrated need.

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 5 Aug 2010 20:52:36 +0100

> make multiplexing transparent at the protocol level later.

Most applications will multiplex in the basic sense of dispatching
addressed messages.  But it's quite different because it does not
combine different tabs/windows, converge at proxies, or share among
different authors.  It's extremely complex in Javascript to implement
multi-window DOM event flow control (even with cross-origin shared
workers), and robust enough that it's safe to combine flows from
arbitrary different app pages at the same domain.  And many attempts
at that are bound to reinvent rare deadlocks, unlimited memory needed,
etc. (multiplexing is easy to do badly) - which is a strong
disincentive against its generic use at that level.

What happens if application multiplexing is not used because it's not
worth using (too complex, not robust enough, bad reputation from poor
implementations, insufficient scope to achieve network benefits), yet
transparent multiplexing would be used due to a different cost-benefit
balance?  Do we avoid implementing the latter because of the former's
faults and limitations giving a wrong measure of the benefits and
simplicity?

I think that's the wrong way to evluate the usefulness of a potential
extension.

We do need to evaluate them but that isn't a good way to do it for
things that would have very different cost-benefit in JS
vs. underneath.

> I don't understand the distinction.

The current WS handshake does not allow *browsers* to tranparently use
any protocol extensions without explicit application support.  If that
becomes the deployed standard, it will not be possible to deploy *any*
extensions, and remain compatible, except by introducing more JS API
and explicit application opt-in.

That will be a very strong barrier against adoption - even browser
makers won't be able to push anything if that happens.

The point of "extensions" is to improve network behaviour tranparently
of all applications, *without changing the JS API* for each extension.

Your suggestion - try it first in JS, then make it "transparent" -
also seems to come with another strong barrier against the latter
being adopted:

If something becomes widespread (such as a JS mediated multiplex
protocol, JS compression etc), gets widely supported on the servers
etc., to make it "transparent" it must be *replaced* with something
new and incompatible that does the same thing.  But nobody's going to
rewrite all the servers just for that.  (See also: Set-Cookie2).

The reason you can't use the *same* protocol once it has become
popular in JS form is you can't transparently use a protocol in new
browsers invisible to the API that's API-visible in old browsers -
because you'll break apps that ran on the old browsers.  It can be the
same protocol but not transparently.

Actually, I lie :-) It is technically possible to combine the two, so
that things could be tried first in JS, and if popular, deployed for
all apps under the API.  Maybe that's what you're suggesting.  I would
like that - it would be versatile and enable incremental evolution of
the infrastructure.  (I'm happy to explore how to do it if you want.)

But the current JS API is technically incapable of being used in that
way, and it would (probably) make the JS API offputtingly complex.
Basically fine-grain protocol negotiation and frame types would need
to become visible.

I think the main requirement of network performance/behaviour
extensions is to enable continuous improvement in future as we find
out what works, in a way that's tranparent to WS apps (or can iterate
towards that).  An anti-pattern would be any lock-in that blocks or
acts as a strong barrier/disincentive against improvements.
 
-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 5 Aug 2010 21:10:34 +0100

> "distributed extensibility" argument. Never very useful, but always very 

We don't want to encourage "myApp...", but we do need some way to try
out experimental extensions on the public web, take large-scale
measurements, find faults etc. and refine them before they are
standardised.

I.e. just like we have "x-safari-roundedcorners" or whatever the CSS
property is called :-) we need some place to slot in vendor WS experiments.

-- Jamie

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 13:10:50 -0700

> extending it if there's a demonstrated need.

Well, we could start with not saying things like "Thats just a Google
problem?" and by describing how we may decide that there is a need for it?

I'm already saying that we'll likely have that need, yet others are saying
that it is insignificant. I'm not even the only person that is saying that
we'll have to have multiplexing at Google.
None of the arguments about the consumption of NAT table space, nor
advantages to latency, nor the decreasing of network overhead (keep alives),
nor amount of system and userspace resources at servers seem to be having
any effect.

The complexity needed to cope with the additional DoS vectors, memory usage,
IO usage as compared to the complexity of developing multiplexing isn't
being evaluated. The assumption is that this is only a "large site" problem,
and that "large sites" are not handling a significant percentage of the
users' traffic. What is being proposed isn't going to be a net complexity
win, though it may be a temporary win in the browsers.

Thus my questions about a litmus test. I'm not even looking for a binary
answer. I'm trying to understand how we can avoid severe problems before
they occur instead of after, and whether or not that matches philosophically
with the consensus of the group.
-=R

---

From: James Graham <[email-masked]>
Date: Thu, 5 Aug 2010 22:11:56 +0200 (CEST)

On Thu, 5 Aug 2010, Roberto Peon wrote:

> What is the incentive for Mozilla, Opera, and Safari to change their implementations again in the future? When will this happen? 

Competition. If some website works better in a competitor due to some 
extra functionality, there exists a strong incentive for everyone else to 
implement that feature. The result of this is that the modern web platform 
has been stiched together from a patchwork of mutually reverse-engineered 
bugs and features. Occasionally there has even been a standards body 
involved somewhere along the way. Indeed that appears to be increasingly 
common, much to everyone's relief.

Of couse the same force that drives forward progress stops us changing the 
past; we can't take out features that are needed for existing websites to 
work even if they would have great payback over the long term. So changes 
we make to the protocol in the future have to be backwards compatible.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 08:35:06 +1000

Anne,

but if transparency is the compelling feature of the extension, then
it can't be done.

Multiplexing might spontaneously evolve in the application layer, but
it will be out competed by applications that don't bother and eat as
much TCP/IP as they can.
This is just the same as multi-tasking, which did evolve in
user-space, but was only really successful when it was genetically
engineered into kernel space.

I'll mix in some hixie for my final point on this:

On Thu, 5 Aug 2010, Ian Hixie wrote:
> every browser vendor who has

That is all fine for browser vendors to say in a world where users are
used to their OS telling them every second week that they must  update
the browser (OR ELSE ALL YOUR BANK DETAILS ARE BELONG TO US!!!).
Applications can also check for browser version and capabilities and
warn the user if an upgrade is needed.

So extending the browsers handling of HTML, DOM, JS, etc. on the fly
etc. is really really great! But you just can't do that with network
protocols.

IPv6 and SCTP are technologies that are being delay by decades because
network infrastructure does not update quickly.   We can't expect
network infrastructure to be updated with v1 of websockets only for a
year later to ask them all to upgrade again to v2.

So a little prudence in checking that our framing can be extended how
we want at this stage is justified.  It's just unacceptable to dismiss
the concerns of so many with the repeated hand waive of do it in a
subprotocol and we might update the core for it later.

regards

---

From: Ian Hickson <[email-masked]>
Date: Thu, 5 Aug 2010 23:04:44 +0000 (UTC)

On Fri, 6 Aug 2010, Greg Wilkins wrote:
> network infrastructure does not update quickly.  We can't expect network 
> infrastructure to be updated with v1 of websockets only for a year later 
> to ask them all to upgrade again to v2.
> 
> So a little prudence in checking that our framing can be extended how we 
> want at this stage is justified.  It's just unacceptable to dismiss the 
> concerns of so many with the repeated hand waive of do it in a 

Changes to WebSockets doesn't require any changes to third-party 
infrastructure. It only requires changes to the parties that are motivated 
to upgrade their software -- the same people who need to do anything to 
get WebSockets deployed in the first place.

This is distinct from IPv6 and SCTP in that these requires changes to the 
middle of the network, not just the edges, and the middle of the network 
is not as motivated to make changes.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 09:29:44 +1000

> get WebSockets deployed in the first place.

At the big end of town, the people that write the application DO NOT
control the network infrastructure.
At the small end of town, the people that write the application DO NOT
control the network infrastructure.

Somewhere inbetween there are people who write applications that
operate under the delusion that they control their network
infrastructure, but their ISP has other ideas as does the ISP of their
users, the users firewall vendor and the software firewall/filter on
the clients OS.

We might be able to tunnel ws through most infrastructure unnoticed
now, but if it is successful, then it will be "managed" by the network
infrastructure and changes to the protocol are likely to break that
management.

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 05:22:19 +0100

> is not as motivated to make changes.

There is the version skew problem, because there's always at least two
parties, and they can't upgrade at the same time.

Newer WebSocket implementations at one end will have to interoperate
with older WebSocket implementations at the other end.

For example, if a browser implements a new transparent compression
extension, but some web app opens a connection to an old server that
doesn't support transparent compression, it *must* be possible for the
connection to work without compression, rather than the old server
closing the connection because it didn't like what's offered.

Version skew can be solved by the JS web app explicitly telling the JS
API whether to use the compression feature or not, but that's a burden
and barrier to deployment.  Transparent and opportunistic is far nicer
(with, perhaps, the option to disable it available to the JS web app.)

-- Jamie

---

From: Ian Hickson <[email-masked]>
Date: Fri, 6 Aug 2010 07:07:00 +0000 (UTC)

> (with, perhaps, the option to disable it available to the JS web app.)

This is a well-understood problem in the browser technology space. The 
WebSocket handshake is designed to handle this (the client would offer a 
new feature, and then only use it if the server opts into it).

---

From: "Anne van Kesteren" <[email-masked]>
Date: Fri, 06 Aug 2010 09:47:57 +0200

On Thu, 05 Aug 2010 21:52:36 +0200, Jamie Lokier <[email-masked]>  
wrote:
> Anne van Kesteren wrote:
>> You cannot implement transparent multiplexing in the application layer  
>> of course. But you can implement some form of multiplexing as explained  
>> in
>> other emails by e.g. Ian. And if it turns out that will be common, we  
>> can make multiplexing transparent at the protocol level later.
>

I'm not at all convinced we have enough experience with SharedWorker (let  
alone cross-origin SharedWorker which is not ready yet) to draw these  
conclusions. It seems way too early to say such things.

> faults and limitations giving a wrong measure of the benefits and

Too complex I don't buy. If Facebook / Google / etc. want it they will  
write it. No problem. If there are other problems I'm pretty confident  
they will come out and we can then reevaluate the situation then.

> things that would have very different cost-benefit in JS

I'm not quite sure why the cost-benefit would be so different.

> any protocol extensions without explicit application support.

Yes it does.

> If that

This is false. The handshake allows for extensions.

> of all applications, *without changing the JS API* for each extension.

Of course.

> rewrite all the servers just for that.  (See also: Set-Cookie2).

Set-Cookie2 never got any client support (apart from Opera) and did not  
offer enough benefit. Native multiplexing would be vastly different  
situation.

> same protocol but not transparently.

Yes it can. Older browsers would simply use more connections. As they are  
declining in number rather rapidly these days that should not be of much  
issue.

> acts as a strong barrier/disincentive against improvements.

(I don't quite follow you here, but I'm not sure it matters.)

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 9 Aug 2010 11:15:38 +1000

> conclusions. It seems way too early to say such things.

Thus it is way too early to assume that SharedWorkers will be
sufficient to handle multiplexing concerns.

It is prudent to make sure that we can easily extend websocket to
support multiplexing with or without shared workers. Thus I think it
is valid to criticize framing proposals that support mutliplexing as
too complex or other technical concerns, but I do not think it is
valid to rule out multiplexing support as one of the criteria that we
can use to evaluation the various proposals.

regards

---

From: "Anne van Kesteren" <[email-masked]>
Date: Mon, 09 Aug 2010 07:41:55 +0200

> sufficient to handle multiplexing concerns.

Nobody is assuming that, I think.

> can use to evaluation the various proposals.

Agreed. The proposal in the draft (and also those in older drafts) can  
usefully be extended to support multiplexing when the time comes.

---

From: Rob Sayre <[email-masked]>
Date: Sun, 08 Aug 2010 22:48:10 -0700

> usefully be extended to support multiplexing when the time comes.

This claim definitely needs to remain true. On the other hand, I 
sincerely hope that the first WebSocket RFC leaves numerous use cases 
unaddressed. The WG needs to postpone work on valid use cases and cut 
scope whenever possible. Otherwise, it will never produce a relevant 
document.

- Rob

---

From: "Anne van Kesteren" <[email-masked]>
Date: Mon, 09 Aug 2010 07:55:37 +0200

> scope whenever possible. Otherwise, it will never produce a relevant  

I wholeheartedly agree. I'd very much like to ship to simplest thing that  
works with room for extending it -- based on experience with implementing  
features in the application layer. And as said I think the current draft  
meets that goal.

---

From: Rob Sayre <[email-masked]>
Date: Sun, 08 Aug 2010 23:15:15 -0700

On 8/8/10 10:55 PM, Anne van Kesteren wrote:
>>> Agreed. The proposal in the draft (and also those in older drafts) 
>>> can usefully be extended to support multiplexing when the time comes.
>>
> I wholeheartedly agree. I'd very much like to ship to simplest thing 
> that works with room for extending it -- based on experience with 
> implementing features in the application layer. And as said I think 
> the current draft meets that goal.

Is the current draft perfect at solving the use cases it aims for? Well, 
I don't know.

But, I do know that one great way to measure the success of a standard 
is to look at the number of later standards that build on it. Any 
standard that gets too ambitious, however initially successful, ends up 
with a number of vestigial features that become too much to bear (e.g. 
XML). Then, even the follow-on work is damaged.

One simple thing that this group should not be doing: standardizing 
features needed for unshipped and/or secret software. No hard feelings, 
but standards groups are not the right place to solve those problems. If 
you want a standard in place before you ship your product, you have 
things backwards.

This phase should be about fixing bugs. A particularly interesting type 
of bug is one that would prevent future work. An uninteresting type of 
bug would be one along the lines of "well, OF COURSE people will need 
this". Add it in a separate standard. If it is valuable, it will be 
supported.

- Rob

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 9 Aug 2010 17:00:29 +1000

this group is at a bit of an impasse with regards to the issue of extension.

We are trying to nail down the framing mechanism and there is one set
of opinions that is very keen to see a well defined extension
mechanism in the base protocol, so that we can later extend the
protocol, but without having to update every intermediary or
infrastructure that has been made websocket aware.

So the various proposals are being evaluated by asking questions like
"how would you do meta-data with this?",  "how would you do
multiplexing with this?" etc.   But unfortunately the all too frequent
response to such questions is either: "you wouldn't want to do that!"
or "that can be done in a subprotocol" or "we can just work out how to
extend later". We then get stuck into debates about why we would want
to extend in a particular way, the pros and cons of doing such
extensions in a subprotocol, the base protocol or somewhere in
between, etc. etc.

We have to break this endless loop.

I strongly believe that at this stage we need to define an extension
mechanism that works with (or is part of ) the websocket framing
mechanism, that has reasonable clear support for several types of
extensions that could reasonably be anticipated might possibly be
supported by protocol extensions in the future.  A primary requirement
of this extension mechanism should be that it will not needlessly
break intermediaries and infrastructure that has been made websocket
aware.    Ie it is not sufficient to propose a framing mechanism with
a 16 bit length field on the assumption that we can later make it 32
or 64 bits, as this will break any intermediaries/infrastructure that
are interpreting the websocket framing.   I do not think it is
sufficient to say that we will just extend the protocol for feature X
when the need for feature X is apparent.

So perhaps we should first seek consensus on the need to define such
an extension mechanism now with a no-unreasonable breakage of
intermediaries requirement.

chairs?

On 9 August 2010 16:15, Rob Sayre <[email-masked]> wrote:
>>> unaddressed. The WG needs to postpone work on valid use cases and cut scope
>>> whenever possible. Otherwise, it will never produce a relevant document.
>>
> Is the current draft perfect at solving the use cases it aims for? Well, I
> don't know.
>
> But, I do know that one great way to measure the success of a standard is to
> look at the number of later standards that build on it. Any standard that
> gets too ambitious, however initially successful, ends up with a number of
> vestigial features that become too much to bear (e.g. XML). Then, even the
> follow-on work is damaged.
>
> One simple thing that this group should not be doing: standardizing features
> needed for unshipped and/or secret software. No hard feelings, but standards
> groups are not the right place to solve those problems. If you want a
> standard in place before you ship your product, you have things backwards.
>
> This phase should be about fixing bugs. A particularly interesting type of
> bug is one that would prevent future work. An uninteresting type of bug
> would be one along the lines of "well, OF COURSE people will need this". Add
> it in a separate standard. If it is valuable, it will be supported.
>
> - Rob

---

From: Ian Hickson <[email-masked]>
Date: Mon, 9 Aug 2010 08:38:05 +0000 (UTC)

On Mon, 9 Aug 2010, Greg Wilkins wrote:
> of this extension mechanism should be that it will not needlessly break 
> intermediaries and infrastructure that has been made websocket aware.

The spec as written now does this. We can add any amount of additional 
data into the frame payload itself and the intermediaries wouldn't be 
affected.

For example, we could add a new kind of frame that is binary data by just 
minting a new frame type and saying that the payload is binary. We could 
add multiplexing by (as a simple example) saying that the first byte of 
any payload is a channel if the client and server agree to multiplex. We 
could add a new frame type that consists of a frame MIME type followed by 
a filename followed by binary data by having the MIME type and filename be 
the first few bytes of the payload, each terminated by a NULL byte or 
prefixed by the number of bytes they take within the payload. We could do 
fragmentation by using new frame types, or by having metadata regarding 
the fragments be the leading or trailing byte(s) of the payload, as agreed 
by the client and server during the handshake.

I can't think of anything we _can't_ do with the current mechanism that 
would impact intermediaries that only support the initial version of the 
protocol, other than move away from message-based communication (and back 
to streaming like TCP).

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 9 Aug 2010 19:35:55 +1000

Ian,

I agree that the current protocol can be extended with new frame types
and achieve the desired no-impact-on intermediaries.

It has been one of the options that I've continually listed, but there
is an issue of the "we" in the mechanism for how it is extended.
Currently there is no mechanism to avoid collisions between new
op-codes other than centrally managing them.

Perhaps that is OK and we will agree than centrally managed op-code
allocation is the preferred extension mechanism.

Alternately we could define an opcode negotiation mechanism, which
would have no impact on the framing, but overcome the central managed
issue.

Another option is to have a pre-allocated mime-frame opcode so that
extensions could use that frame type - again that would have no impact
on the actual framing proposed and code+length+data framing is
sufficient (give or take a debate about how length is encoded).

However, there are others that are concerned that those proposals do
not allow sufficient extended data to be associated with data frames,
as might be needed for some extension (although I think the
mime-frame does allow extended data per frame).   So there are some
proposals about how to have optional extension data in every frame.

So we need to stop bickering about if other peoples use-cases are
valid or not, and start expanding on how our preferred framing
mechanisms can actually address all (or most) of the concerns of all
(or most) that have expressed them.

regards

---

From: Ian Hickson <[email-masked]>
Date: Mon, 9 Aug 2010 17:37:52 +0000 (UTC)

On Mon, 9 Aug 2010, Greg Wilkins wrote:
> Currently there is no mechanism to avoid collisions between new op-codes 
> other than centrally managing them.

Right, this is the protocol this working group is chartered to maintain.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 09 Aug 2010 20:22:09 +0200

> Right, this is the protocol this working group is chartered to maintain.

Pointer?

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 11 Aug 2010 17:29:25 +1000

> Right, this is the protocol this working group is chartered to maintain.

So are you saying that you do think extensions should be managed
centrally by this WG?

I think that would be a real impediment to innovation and
experimentation.  For example the kind of experiments that google is
now doing with shipped versions of chromium would not be possible for
a proposed websockets extension unless this working group assigned
some opcodes to be used.   Actually, it would probably have to be IANA
that managed to opcode allocation.  This strikes me as bureaucratic
and unreasonably restrictive.

There has been several proposals to allow op-codes to be negotiated
which do not require any changes to framing (except maybe a reasonable
reserve of unallocated op-codes) and appear to be sufficient for most
of the extensions that have been discussed here.
[ I know some think that we need extended framing for extensions, but
I would argue that any extended framing mechanism can be supported
within a negotiated op-code mechanism, as you just negotiation an
op-code for the extended frame.]

What is your objection to negotiated op-codes?  A simple
implementation need not implement extensions, so it can ignore the
negotiation.

I think you will find that if we can agree on an extension mechanism,
then we will be able to quickly resolve any remaining framing issues
and move onto the important issues around the handshake.

---

From: Ian Hickson <[email-masked]>
Date: Wed, 11 Aug 2010 07:38:28 +0000 (UTC)

On Wed, 11 Aug 2010, Greg Wilkins wrote:
> 
> What is your objection to negotiated op-codes?

My objection is to building an architecture to negotiate op-codes. It's 
not needed. If someone wants to experiment, they don't need to have 
long-term stability and interoperability, so they can just make up a new 
header and say that that negotiates the experimental feature. Other than 
that, everything can go through whoever defines Web Sockets (i.e. this 
working group). No need for registries, bureaucracy, or anything.

For optimal progress, experimenters should coordinate with other 
implementers in this working group so that everyone knows what's going on.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 11 Aug 2010 09:42:48 +0200

> and unreasonably restrictive.

If the headers to negotiate support in the handshake are reasonably unique  
(i.e. prefixed or some such) Google could easily run such experiments I  
think. I do not see why it requires changes to framing.

> implementation need not implement extensions, so it can ignore the

It makes the framing more complex than it needs to be.

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 11 Aug 2010 00:43:35 -0700

> implementers in this working group so that everyone knows what's going on.

Adding to this - if any experiments are successful, I think we'll want to propagate them to all significant client implementations, even if they are not useful for servers. So it's strongly desirable for there to be central coordination before anything becomes locked in or official. If we end up with different browsers shipping support for different sets of WebSocket protocol extensions, at least beyond a short implementation/deployment period, then that would greatly suck. Servers could treat extensions as truly optional, but to make life sane for servers, we need centralized coordination on the client side.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 11 Aug 2010 18:36:41 +1000

On 11 August 2010 17:42, Anne van Kesteren <[email-masked]> wrote:
> I do not see why it requires changes to framing.
[...]
> It makes the framing more complex than it needs to be.

You are not understanding the proposals.   Negotiated op-codes do not
require any changes to framing so they cannot  make it over complex.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 11 Aug 2010 18:42:39 +1000

> working group). No need for registries, bureaucracy, or anything.

What if the experiment is successful and it would like to be widely
deployed without waiting for central approval from this WG (or an
update to the specification document?).

For example, a intermediary vendor and a server vendor might like to
cooperate on an extension for SSL offload, connection aggregation, DOS
attack protection etc. etc.   If they allocate their own op-codes or
invent their own op-code negotiation mechanism, then their systems
will be vulnerable to updates of the protocol that reallocate the same
op-codes.   This would mean that inter-operability problems would
result from any non-core extensions.

I do not think it wise to outlaw non-core extensions, because
prohibition as a policy never works.

> implementers in this working group so that everyone knows what's going on.

Some extensions may be proprietary in nature and the developers may be
unwilling to be open with details before they are ready.  I do not
think it sufficient to try to prohibit non-open extension development.

---

From: SM <[email-masked]>
Date: Wed, 11 Aug 2010 01:37:25 -0700

>implementers in this working group so that everyone knows what's going on.

Quoting Section 2.5 of RFC 5704:

   "It may be tempting for a designer to assert that the protocol
    extensions it proposes are safe even though it breaks the invariants
    of the original protocol because these protocol variants will operate
    as ships in the night.  That is, these protocol variants will never
    simultaneously exist in the same network domain and will never need
    to inter-work.  This is a fundamentally unsound assumption for a
    number of reasons.  First, it is infeasible to ensure that the two
    instances will never be interconnected under any circumstances.
    Second, even if the operators that deploy the protocols apply
    appropriate due diligence to ensure that the two instances do not
    conflict, it is infeasible to ensure that such conflicting protocols
    will not be interconnected under fault conditions."

And the Security Considerations Section:

   "The uncoordinated development of protocols poses a risk of harm to
    the Internet and must not be permitted.  The enhancement or
    modification of a protocol can increase attack surfaces considerably
    and may therefore compromise the security or stability of the
    Internet.  The IETF has a review process that combines cross-area
    review with specialist security review by experts familiar with the
    development and deployment context of the Internet protocol suite."

Regards,
-sm

---

From: Ian Hickson <[email-masked]>
Date: Wed, 11 Aug 2010 08:48:59 +0000 (UTC)

>    development and deployment context of the Internet protocol suite."

Given that both the client and the server must opt in to using any 
extension mechanism in WebSockets, how does the above apply here?

---

From: James Graham <[email-masked]>
Date: Wed, 11 Aug 2010 10:53:31 +0200

On 08/11/2010 10:36 AM, Greg Wilkins wrote:
> On 11 August 2010 17:42, Anne van Kesteren<[email-masked]>  wrote:

Perhaps you could provide a summary or a pointer to one? The list has 
been high volume recently and it is quite possible that I have missed 
something.

Given my current understanding, I see no problem with just making 
extensions trigger off fields in the handshake. Why is something else 
needed.

Also, from a client point of view, proprietary extensions are bad. We 
explicitly want everything to go through a standards process so 
providing a means to have long-lived non-standard extensions is an 
explicit non-goal. There are, however, use cases for short-lived 
extensions e.g. experimentation. These can easily be handled in the 
handshake field model e.g. we reserve fields with names like

WebSocket-vendorid-extensionname

Where vendorid is e.g. opera or mozilla or whatever. Then extensions 
that become standardized loose the -vendorid part.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 11 Aug 2010 19:08:43 +1000

On 11 August 2010 18:53, James Graham <[email-masked]> wrote:
> Given my current understanding, I see no problem with just making extensions
> trigger off fields in the handshake. Why is something else needed.
>
That is indeed what has been proposed in the various negotiated
op-code mechanisms.

> Perhaps you could provide a summary or a pointer to one? The list has been
> high volume recently and it is quite possible that I have missed something.

I will refrain from re-iterating the detail here, because invariable
the discussion get's distracted by some triviality in the proposal
that distracts from the core discussion.

At this stage it looks like we need to decide if non-centrally
approved extensions are to be supported or not before we resume debate
the actual details of how it could be done.

> explicitly want everything to go through a standards process so providing a
> means to have long-lived non-standard extensions is an explicit non-goal.
> There are, however, use cases for short-lived extensions e.g.
> experimentation. These can easily be handled in the handshake field model
> e.g. we reserve fields with names like
>
> Where vendorid is e.g. opera or mozilla or whatever. Then extensions that
> become standardized loose the -vendorid part.

The problem is that extension developers have no channel on which to
communicate other than handshake headers.  The protocol currently has
no support for meta data or any other out-of-band communication.  Thus
there are no mechanisms available to extend without rewriting every
single frame.

Provision of extensible op-codes or meta-data frames or meta-data per
message or extensible frame headers are all proposals that have been
made recently to try to achieve a standardized way of creating
extensions, so that we will have the maximum change of developing
interoperable infrastructure that will not be broken by extensions
(core or otherwise).

---

From: James Graham <[email-masked]>
Date: Wed, 11 Aug 2010 11:39:25 +0200

On 08/11/2010 11:08 AM, Greg Wilkins wrote:
> On 11 August 2010 18:53, James Graham<[email-masked]>  wrote:

I am not asking for detail; I am explicitly asking for a summary. 
Specifically, I would find something like the following helpful:

a) Examples of concrete problems you envision solving through extensions 
(I don't think "extensibility" on its own is a concrete problem. A 
concrete problem might be "transferring large text messages is slow but 
could be speeded up with per-frame compression" or something similar).

b) Why is the existing (header negotiation) mechanism insufficient to 
permit solutions

c) What other proposals have been made

d) How these address the problems in b)

I am sorry if I am being slow, but I feel like I am not following you at 
all here. I don't really understand what the real problems with the 
existing mechanism are even though you have tried to explain in general 
terms what you consider to be wrong. Maybe if you could give concrete 
examples, or a pointer to where they have been given previously, that 
would help.

---

From: SM <[email-masked]>
Date: Wed, 11 Aug 2010 07:46:33 -0700

Hi Ian,
>extension mechanism in WebSockets, how does the above apply here?

It's similar to the recommendation for coordination so that everyone 
knows what is going on.

Regards,
-sm

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 11 Aug 2010 10:17:49 -0700

On Wed, Aug 11, 2010 at 12:43 AM, Maciej Stachowiak <[email-masked]> wrote:

> Adding to this - if any experiments are successful, I think we'll want to
> propagate them to all significant client implementations, even if they are
> not useful for servers. So it's strongly desirable for there to be central
> coordination before anything becomes locked in or official. If we end up
> with different browsers shipping support for different sets of WebSocket
> protocol extensions, at least beyond a short implementation/deployment
> period, then that would greatly suck. Servers could treat extensions as
> truly optional, but to make life sane for servers, we need centralized
> coordination on the client side.
>
> Regards,
> Maciej
>
>
Maciej,

I agree that we would want to propagate successful experiments to all
significant client (and server) implementations, however at the risk of
side-tracking I'm a bit curious to why you say "even if they are not useful
for servers" as opposed to "even if they are not useful for everyone". E.g.
if an extension is beneficial for servers, but something of a wash from a
javascript developer perspective, does that mean it shouldn't see
deployment? (Sorry for playing devil's advocate here...)

Anyhow, I agree that it would be good to have coordination before we roll
something out everywhere. But, what you said above was "if any experiments
are successful" and "... before anything becomes locked in or official".
That implies that someone has to be free to experiment to figure out if
something is successful or not. In able to do a real experiment in the wild,
we need a way to conduct that experiment without intermediaries and the rest
of the world having to deal with / make changes to support the experiment.
Ideally, we also don't want un-affected intermediaries to have to re-tool
after some new extension gets added, e.g. I would like to avoid the mess
we're in with HTTP 1.0 and 1.1 where 1.0 intermediaries largely screw over
1.1.

In some sense, WebSockets is itself an experiment that has become successful
and is now being centrally coordinated. We didn't have to go to some central
place to get permission to use the Upgrade mechanism of HTTP to try out Web
Sockets, I think what Greg is suggesting is that we don't want to have
people need to come to IETF to get some assigned range of opcodes/whatever
before they can do an experiment. As such, I don't think that what Greg is
requesting is unreasonable - we need a clearly defined way for such
experiments to work such that they don't break in the presence of other
clients/servers/intermediaries/whatever that are not involved with the
experiment.

-Ian

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 12 Aug 2010 15:31:08 +1000

James,

The discussion of the extension mechanism has been mixed in with the
discussion of some proposals for changing framing (for fragmentation,
compression, etc.).

But to consider the extension mechanisms in isolation, not all of them
require changes to framing.  I've previously categorised the various
extension proposals as:

   1. hixiesque - centralized extension via op-codes (no framing
changes required)
   2. robertoesque- per frame extension via meta-data (framing changes required)
   3. martinesque - extension via dedicated mime frame (no framing
changes required other than dedicated op-code).
   4. Jamiesque - extension via op-codes negotiated in handshake (no
framing changes required)

[ Dave Cridland also made a proposal that was Jamiesque in extension
but had some other framing changes included ]

So 3 out of 4 proposed extension mechanisms don't actually need any
changes to framing - so perhaps it was a mistake to wrap them up in
the same conversations - however issues are not totally orthogonal
because a good extension mechanism may mitigate the desire to have
some features in the core protocol.  I think that less open/flexible
the extension mechanism is, then the greater the desire will be to see
features like fragmentation, mime meta-data, compression etc. in the
base protocol from day 1.

regards

On 11 August 2010 19:39, James Graham <[email-masked]> wrote:
>>> extensions
>>> been
> a) Examples of concrete problems you envision solving through extensions (I
> don't think "extensibility" on its own is a concrete problem. A concrete
> problem might be "transferring large text messages is slow but could be
> speeded up with per-frame compression" or something similar).
>
> b) Why is the existing (header negotiation) mechanism insufficient to permit
> solutions
>
> I am sorry if I am being slow, but I feel like I am not following you at all
> here. I don't really understand what the real problems with the existing
> mechanism are even though you have tried to explain in general terms what
> you consider to be wrong. Maybe if you could give concrete examples, or a
> pointer to where they have been given previously, that would help.
>

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 12 Aug 2010 15:36:13 +1000

2010/8/12 Ian Fette (イアンフェッティ) <[email-masked]>:
> I'm a bit curious to why you say "even if they are not useful
> for servers" as opposed to "even if they are not useful for everyone".

Ian,

brain fade.... I meant to say "for clients", but "for everyone" works
even better.

> As such, I don't think that what Greg is

Thanks for putting what I've been trying to say a lot clearer than I've said it.

Note the only quibble I have with that description is use of the word
"experiment". I think there is likely to be a reasonable number of
extensions that are greater than experiments (ie useful for at least
some), but less than compelling features to be in the core protocol.
It is to support deployment of these inbetweenies that would be good
to avoid breakage in the presence of other
clients/servers/intermediaries/whatever.

cheers

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 11 Aug 2010 23:51:04 -0700

Pardon the poor formatting of reply. On phone, but wanted to say I agree
with your message in full.

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 12 Aug 2010 16:43:56 -0700

> think it sufficient to try to prohibit non-open extension development.

I think non-disclosed proprietary extensions are actively harmful to the evolution of the protocol, so I think it would be better overall to discourage them. It is a negative value use case.

That being said, if two parties have a pre-arranged agreement regarding a private extension, the protocol does provide the means to negotiate opt-in, and it would be up to those parties to adjust their scheme as needed in the face of protocol changes.

Regards,
Maciej

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 12 Aug 2010 16:53:19 -0700

On Thu, Aug 12, 2010 at 4:43 PM, Maciej Stachowiak <[email-masked]> wrote:

> I think non-disclosed proprietary extensions are actively harmful to the
> evolution of the protocol, so I think it would be better overall to
> discourage them. It is a negative value use case.
>
> That being said, if two parties have a pre-arranged agreement regarding a
> private extension, the protocol does provide the means to negotiate opt-in,
> and it would be up to those parties to adjust their scheme as needed in the
> face of protocol changes.
>
> Regards,
> Maciej
>
>
Assuming I ship an intermediary at t=X and I don't want to have to update
the code (or I know it won't be updated, e.g. all the existing
proxies/intermediaries on the internet) it is largely impossible to know,
based on code at t=X, whether an extension developed at t=X+1 is a
"non-disclosed proprietary extension" or a "centrally managed and planned
extension". So, the distinction seems a bit theoretical... the protocol
needs to not fall over.

-Ian

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 12 Aug 2010 16:48:20 -0700

On Aug 11, 2010, at 10:17 AM, Ian Fette (イアンフェッティ) wrote:

> 
> 
> I agree that we would want to propagate successful experiments to all significant client (and server) implementations, however at the risk of side-tracking I'm a bit curious to why you say "even if they are not useful for servers" as opposed to "even if they are not useful for everyone". E.g. if an extension is beneficial for servers, but something of a wash from a javascript developer perspective, does that mean it shouldn't see deployment? (Sorry for playing devil's advocate here...)

The distinction I meant to draw is that some features may not be useful for all servers, only for some. So it makes sense for some features to be optional on the client side.

> 
> Anyhow, I agree that it would be good to have coordination before we roll something out everywhere. But, what you said above was "if any experiments are successful" and "... before anything becomes locked in or official". That implies that someone has to be free to experiment to figure out if something is successful or not. In able to do a real experiment in the wild, we need a way to conduct that experiment without intermediaries and the rest of the world having to deal with / make changes to support the experiment. Ideally, we also don't want un-affected intermediaries to have to re-tool after some new extension gets added, e.g. I would like to avoid the mess we're in with HTTP 1.0 and 1.1 where 1.0 intermediaries largely screw over 1.1.

I think the upgrade mechanism + defined generic framing are sufficient for people to run experiments without requiring anyone else to change.

Regards,
Maciej

---

From: "Anne van Kesteren" <[email-masked]>
Date: Fri, 13 Aug 2010 11:13:32 +0200

> extension". So, the distinction seems a bit theoretical... the protocol

Why would it fall over?

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 13 Aug 2010 21:35:03 +1000

> I think non-disclosed proprietary extensions are actively harmful to the evolution of the protocol, so I think it would be better overall to discourage them. It is a negative value use case.

There are plenty of disclosed proprietary extensions that i don't
think harmful.  For example, many existing HTTP intermediaries are
controlled by cookies, headers and special URLs.   All these are
extension mechanisms supported by HTTP which allow custom extensions
without changes to the core semantics of the protocol.

> That being said, if two parties have a pre-arranged agreement regarding a private extension, the protocol does provide the means to negotiate opt-in, and it would be up to those parties to adjust t…

That is the point of having a standardized negotiation mechanism, so
that such proprietary extensions can declare their usage. Note that
currently there is no version in the protocol, so it is impossible for
a intermediary to know if other devices they are working with have
been upgraded.  Even if they did know, it is a bit poor to say that
their only option is to fail in the face of an upgraded protocol,

regards

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 13 Aug 2010 04:49:54 -0700

On Aug 12, 2010, at 4:53 PM, Ian Fette (イアンフェッティ) wrote:

> 
> Assuming I ship an intermediary at t=X and I don't want to have to update the code (or I know it won't be updated, e.g. all the existing proxies/intermediaries on the internet) it is largely impossible to know, based on code at t=X, whether an extension developed at t=X+1 is a "non-disclosed proprietary extension" or a "centrally managed and planned extension". So, the distinction seems a bit theoretical... the protocol needs to not fall over.

I believe that is already the case, since the protocol (as currently drafted) has a generic framing that lets you identify frame boundaries. So long as extensions stick to this general form of framing, intermediaries can happily continue to forward messages. That's assuming the intermediary in question actually would care about message boundaries, which is not clear. I would expect most intermediaries can just treat the connection as a bidirectional stream.

Of course, some advanced intermediaries may actually need to understand the semantics of specific extensions, but there's nothing we can do to enable an intermediary coded at t=X to understand the specific semantics of an extension developed at t=X+1. At best, we can have a design that makes it possible to pass extensions through blind - which is already the case, so far as I can tell.

In any case, I don't see how adding more reserved fields to the frame header has any material effect.

Regards,
Maciej

---

From: Bill de hÓra <[email-masked]>
Date: Fri, 13 Aug 2010 22:35:08 +0100

> new feature, and then only use it if the server opts into it).

Interesting. This assumes the client is willing to write the state
machine to do that instead of the server - ie, complexity is on the
client side. Is that a shared assumption in this group?

Asked another way - is the fabled amateur programmer a server
programmer?

Bill