Subject: [hybi] Framing take IV

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 10:38:59 +1000

I'd like to refocus on what I believe we do have near-consensus on, so that
we don't lose progress while we debate other possible features.

I think that we have reasonable consensus on something like:

  +--------------------------------------------------+
  | frag(1) |unused(3) | opcode(4) |  Length(16)     |
  +--------------------------------------------------+
   |                      Data                        |
  +--------------------------------------------------+

    ws-frame            = frame-fragment
                          frame-unused
                          frame-opcode
                          frame-length
                          octet-data

    frame-fragment      = %x0 ; 1 bit not a fragment
                        | %x1 ; 1 bit fragment

    frame-unused        = %x0 ; 3 bits

    frame-opcode        = %x0 ; text data frame
                        | %x1 ; binary data frame
                        | %x2 ; ping
                        | %x3 ; pong
                        | %xf ; final frame

    frame-length        = %x0000-FFFF

    octet-data          = *( %x00-FF )

I think there are no strong feelings either way on padding, but it is easy
to add to any proposal, so let's sort the detail first and then consider
padding at the very end.

I do not think we have any consensus on compression, but the above framing
can be easily extended at a later date (compressed opcodes, or add
compressed bit, or just compress the stream).

We are still a bit in the wilderness with regards to meta-data and
multi-plexing.  However the above framing is at least extensible and would
allow those and other extensions to be added by defining some new
op-codes.
But the problems with this style of extensibility are:

   - The op-code space must be managed to avoid collisions, so that would
   limit experimental and/or special purpose extensions
   - Any extension data will be in it's own frame, and thus not directly
   associated with a data frame, so extensions like multiplexing will be more
   stateful and vulnerable to inserted frames.

Neither of these problems are show stoppers, although they do combine
badly.. ie multiplexing can be done with a channel switch and flow control
frames, but only if op-codes are allocated to it and that will need
centralized consensus on one true multiplexing algorithm.

We have had many proposals about how to improve the extensibility of the
framing, but so far I think only the proposal from roberto truly addresses
the issue of non-centralized extensibility and extensions per frame.  His
proposal was to optionally include meta-data with every frame - I refined
that proposal to make the meta data easily ignorable:

  +--------------------------------------------------------+
  | frag(1) |opcode(3) | meta-length (12) |  Length(16)    |
  +--------------------------------------------------------+
  |                       Meta Data                        |
  +--------------------------------------------------------+
   |                          Data                          |
  +--------------------------------------------------------+

    ws-frame            = frame-fragment
                          frame-opcode
                          frame-meta-length
                          frame-length
                          +(meta-data)
                          octet-data

    frame-fragment      = %x0 ; not a fragment
                        | %x1 ; fragment

    frame-opcode        = %x0 ; text data frame
                        | %x1 ; binary data frame
                        | %x2 ; ping
                        | %x3 ; pong
                        | %xf ; final frame

    frame-length        = %x0000-FFFF

    meta-data           = ; length framed name value pairs

    octet-data          = *( %x00-FF )

The other proposal from Martin was that we add a  MIME frame op code:

  +--------------------------------------------------+
  | frag(1) |unused(3) | opcode(4) |  Length(16)     |
  +--------------------------------------------------+
   |                      Data                        |
  +--------------------------------------------------+

    ws-frame            = frame-fragment
                          frame-unused
                          frame-opcode
                          frame-length
                          octet-data

    frame-fragment      = %x0 ; 1 bit not a fragment
                        | %x1 ; 1 bit fragment

    frame-unused        = %x0 ; 3 bits

    frame-opcode        = %x0 ; text data frame
                        | %x1 ; binary data frame
                        | %x2 ; MIME data frame
                        | %x4 ; ping
                        | %x5 ; pong
                        | %xf ; final frame

    frame-length        = %x0000-FFFF

    octet-data          = *( %x00-FF )

I think Martin proposed it as standard CRLF separated Mime, but I think an
argument could be made to put a length based encoding within that frame.
This essentially would make the meta-data of robertos proposal available
only for a single frame type.  An extension like multiplexing would be
implemented by converting all binary/text frames to mime-frames.  (I'm not
exactly sure mime is the right name for them, but let's go with it for now).

Mime frames would allow arbitrary extensions to exchange arbitrary meta-data
without centralized control.  It is a matter for either end how(if?) they
would route any such messages to their APIs.

There are many variations on these themes, but I think these are the 3 basic
approaches that have been identified, which I will call:

   1. hixiesque - centralized extension via op-codes
   2. robertoesque- per frame extension via meta-data
   3. martinesque - extension via dedicated mime frame (meta-data plus data)

Any others?

I think we really need to pick one of these approaches soon and get on with
the detail.

My own opinion is that my technical head is drawn to robertoesque solution,
but my pragmatic head feels that martinesque is sufficient, simpler and more
likely to get consensus.

cheers

PS. with a martinesque solution, who would ever use a %x1 binary data frame?

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 00:53:43 +0000 (UTC)

> I think that we have reasonable consensus on something like:

Why would we have a fixed length field with fragmentation rather than a 
variable length field?

If we can have a variable width length field, do we need to support 
fragmentation in the first version? I could see an argument for supporting 
fragmentation in the case of multiplexing, but without that it doesn't 
seem to actually gain us anything.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 17:58:20 -0700

On Tue, Aug 3, 2010 at 5:38 PM, Greg Wilkins <[email-masked]> wrote:

> Mime frames would allow arbitrary extensions to exchange arbitrary
> meta-data without centralized control.  It is a matter for either end
> how(if?) they would route any such messages to their APIs.
>
> There are many variations on these themes, but I think these are the 3
> basic approaches that have been identified, which I will call:
>
If I had to choose between Roberto's and Martin's, I would prefer Roberto's
simply because I worry that if you have metadata that changes between
messages, e.g. you are multiplexing a number of streams, you will end up
sending a control frame before each data frame, which sort of defeats the
purpose, makes it extra chatty, and opens us up to issues of frame injection
as well. So, I prefer to have whatever extension/meta-data associated with
the frame itself.

The one thing I don't like about Roberto's proposal is that I essentially
have to parse all the metadata to find the particular metadata I'm
interested in. I'm not sure if there are any ways to avoid this or make this
cheaper though if you have to support variable-length metadata, so maybe
it's just a price we pay.

I do think it would make a lot of sense to reserve a byte for stream ID
though. I know that for us (Google) to deploy, one way or another we will
need multiplexing, and I think it would be a lot more efficient if the
stream ID were at a known location than if we had to parse some metadata map
to get it out. I don't really buy the complexity arguments -- for us, the
main interest in WebSockets over JS hanging gets + posts is the ability to
pear down connections (two to one, better if we can multiplex) + more
efficiency. If we can't multiplex then there's not that much of a benefit to
the point where it becomes a lot less interesting to deploy (cost/benefit is
not so clear).

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 03 Aug 2010 17:59:14 -0700

Greg Wilkins wrote:
>
> I'd like to refocus on what I believe we do have near-consensus on, so 
> that we don't lose progress while we debate other possible features.
>
>
> badly.. ie multiplexing can be done with a channel switch and flow 
> control frames, but only if op-codes are allocated to it and that will 
> need centralized consensus on one true multiplexing algorithm.

I'd recommend opening up a multiplexing discussion using this as a 
draft/working model, and putting off the extension discussion for now, 
because the needs of a multiplexing extension would affect the basic frame.

(And also because discussing extensions can become impossible because 
everyone has a different vague idea of what kind of extensions might be 
possible/desirable, while multiplexing is concrete.)

-- Scott

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 11:07:50 +1000

> seem to actually gain us anything.
Ian,

at the f2f there was a very loud hum for a single framing mechanism.
Following that, there were loud hums for fragmentation and fixed length
frames, even without consideration of multiplexing.

I believe the feeling was that fragmentation makes sending and receiving big
messages simpler.  It also prepares the ground for multiplexing extensions.
There was very little support for variable length lengths, as that was just
seen as more complex than fragmentation.

But if you have a counter proposal that meets the single framing hum, then
please post it.

cheers

---

From: "Thomson, Martin" <[email-masked]>
Date: Wed, 4 Aug 2010 09:12:31 +0800

(BTW, email formatting makes it very hard for us with challenged MTAs to distinguish between the different parties in these emails).

Ian,

I think that perhaps you misunderstood the proposal.  The idea would be to embed the metadata in the frame.  My initial suggestion was to go full MIME and have a MIME payload, including headers in the frame body.  Header and content are separated, as they are in MIME by two end lines (\r\n\r\n).  Greg is suggesting that we could define a more efficient container for these.

I don’t think that the idea of having separate frames was ever a starter.

--Martin

If I had to choose between Roberto's and Martin's, I would prefer Roberto's simply because I worry that if you have metadata that changes between messages, e.g. you are multiplexing a number of streams, you will end up sending a control frame before each data frame, which sort of defeats the purpose, makes it extra chatty, and opens us up to issues of frame injection as well. So, I prefer to have whatever extension/meta-data associated with the frame itself.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 11:12:38 +1000

> because the needs of a multiplexing extension would affect the basic frame.
Probably a good idea.

However, I'd really like to hear feedback from those that are not that
interested in multiplexing if any of these framing approaches is attractive
and/or onerous.  If there is (for example) some acceptance of a martinesque
approach, then we can put more effort to scoping out a multiplexing solution
that works with that.

cheers

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 03 Aug 2010 18:18:29 -0700

> seem to actually gain us anything.

I agree. I can't see any benefit to fragmentation over a variable-size length field for an initial version without multiplexing. If the variable-size length field is well-designed, then in the common case where the message size is small it will only cost one extra branch to read the length. In the rare case where the message size is large, a variable-size length is easier to deal with than reassembling fragments, and easier on the sending side too.

I'm not really persuaded we need multiplexing at all until I see data proving the benefits. Useful data would be:

- Average degree of multiplexing we could likely get in real Web applications, based on measured user behavior. I'd like to see this for at least two different Web apps from two different vendors to make sure we are not overtuning to a single vendor's setup.

- Concrete performance benefits of multiplexing over a single TCP connection vs. using multiple connections. I have yet to hear an answer that involves numbers and doesn't sound handwavey.

- Performance cost of implementing multiplexing at the application level compared to doing it at the protocol level (either in terms of more connections resulting and the actual cost of that, or in terms of additional overhead on the client side from using an iframe or shared worker or whatever to share the connection.)

I'm open to trading off simplicity for performance, but only if I see data backing up the performance claims. Every performance project should start with measurement.

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 03 Aug 2010 18:20:18 -0700

On Aug 3, 2010, at 6:07 PM, Greg Wilkins wrote:

> at the f2f there was a very loud hum for a single framing mechanism. Following that, there were loud hums for fragmentation and fixed length frames, even without consideration of multiplexing.
> 
> I believe the feeling was that fragmentation makes sending and receiving big messages simpler.  It also prepares the ground for multiplexing extensions. There was very little support for variable length lengths, as that was just seen as more complex than fragmentation.
> 
> But if you have a counter proposal that meets the single framing hum, then please post it.

Hum volume is not an adequate response to a reasoned argument. If people think fragmentation is simpler than a variable size length field, they should explain why, rather than just citing the number of people who agree.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 11:29:06 +1000

On 4 August 2010 11:20, Maciej Stachowiak <[email-masked]> wrote:

> I believe the feeling was that fragmentation makes sending and receiving
> big messages simpler.  It also prepares the ground for multiplexing
> extensions. There was very little support for variable length lengths, as
> that was just seen as more complex than fragmentation.
>
> Hum volume is not an adequate response to a reasoned argument. If people
> think fragmentation is simpler than a variable size length field, they
> should explain why, rather than just citing the number of people who agree.
>
>
I did briefly explain.  I cited the hum volume not as an argument, but as an
indication that we have already had the arguments explained.

happy to explain again in a little bit more detail:

   - variable length length fields is more complex than a fixed length
   length field.
   - sending large content in a single frame is difficult if the content is
   being generated dynamically or it's length is otherwise not known.
   - a small fixed sized length allows simple implementations to have fixed
   sized buffers (eg 64k)
   - fragmentation prepares the ground for future extensions for
   multiplexing and flow control.
   - fragmentation on its own is not that complex

But if you have a counter proposal that you think might satisfy the hummers,
please make it.  The current protocol certainly does not.

cheers

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 03 Aug 2010 18:31:34 -0700

> I agree. I can't see any benefit to fragmentation over a variable-size length field for an initial version without multiplexing. If the variable-size length field is well-designed, then in the comm…

Some of us don't have infinite memory when sending messages.

-- Scott

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 01:32:33 +0000 (UTC)

On Tue, 3 Aug 2010, Scott Ferguson wrote:
> > 
> > I agree. I can't see any benefit to fragmentation over a variable-size 
> > length field for an initial version without multiplexing. If the 
> > variable-size length field is well-designed, then in the common case 
> > where the message size is small it will only cost one extra branch to 
> > read the length. In the rare case where the message size is large, a 
> > variable-size length is easier to deal with than reassembling 
> > fragments, and easier on the sending side too.
> 

Why do you need infinite memory?

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 03 Aug 2010 18:39:20 -0700

> Why do you need infinite memory?

Because we don't know the length of dynamically produced content until 
the dynamic process is complete. That's why HTTP added chunking.

-- Scott

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 01:39:44 +0000 (UTC)

On Tue, 3 Aug 2010, Ian Fette (ã~B¤ã~B¢ã~C³ã~C~Uã~B§ã~C~Cã~C~Fã~B£) wrote:
> 
> For us it's the number of open sockets on the frontend, plain and 
> simple. Each open socket == more kernel memory for socket buffers. 
> Whereas, if we have multiplexing, I can take in a frame, see what stream 
> it is, send it off to the appropriate backend and be done. Holding open 
> multiple connections for a single user kills us.

We shouldn't design a protocol for the next few decades around a 
limitation of a particular kernel implementation and how it affects one 
particular deployment. Why don't we just change the kernel to send it off 
to the appropriate backend and be done instead of using lots of kernel 
memory for socket buffers?

> At the application level we would need an <iframe> to google.com to get a
> shared worker, and then use postmessage.

On the same timeframe as multiplexing in Web Sockets, we should be able to 
get cross-origin shared workers, so this is a non-issue.

> Plus, if we wanted to multiplex sending large and small data, we would 
> probably have to break up the large data at an application level and 
> re-assemble it at an application level on the server side, and I cannot 
> imagine that being performant.

On the server side you don't have to do it at the application level. You 
have full control over the entire stack from the network fabric up, you 
can implement it however you like.

On the client side, given the right primitives, why can't it be the same 
as if the browser implemented it?

> Again, I'm much more interested in squeezing out every last drop of 
> performance. That's the whole reason for WS in my mind as opposed to 
> existing JS APIs.

It might be your reason for being interested in Web Sockets. It's not the 
whole reason Web Sockets exists, though. It's not even the main reason. :-)

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 18:33:40 -0700

On Tue, Aug 3, 2010 at 6:18 PM, Maciej Stachowiak <[email-masked]> wrote:

> variable-size length field is well-designed, then in the common case where
> the message size is small it will only cost one extra branch to read the
> length. In the rare case where the message size is large, a variable-size
> length is easier to deal with than reassembling fragments, and easier on the
> sending side too.
>
> I'm not really persuaded we need multiplexing at all until I see data
> proving the benefits. Useful data would be:
>
> - Average degree of multiplexing we could likely get in real Web
> applications, based on measured user behavior. I'd like to see this for at
> least two different Web apps from two different vendors to make sure we are
> not overtuning to a single vendor's setup.
>
>
I don't think we would be able to get any data from a Google deployment in a
reasonable time for this discussion, sadly.

> - Concrete performance benefits of multiplexing over a single TCP
> connection vs. using multiple connections. I have yet to hear an answer that
> involves numbers and doesn't sound handwavey.
>

For us it's the number of open sockets on the frontend, plain and simple.
Each open socket == more kernel memory for socket buffers. Whereas, if we
have multiplexing, I can take in a frame, see what stream it is, send it off
to the appropriate backend and be done. Holding open multiple connections
for a single user kills us.

>
> - Performance cost of implementing multiplexing at the application level
> compared to doing it at the protocol level (either in terms of more
> connections resulting and the actual cost of that, or in terms of additional
> overhead on the client side from using an iframe or shared worker or
> whatever to share the connection.)
>
>
At the application level we would need an <iframe> to google.com to get a
shared worker, and then use postmessage. The iframe itself would be enough
of a latency hit that we would consider it problematic for e.g. deployment
on the homepage. Plus, if we wanted to multiplex sending large and small
data, we would probably have to break up the large data at
an application level and re-assemble it at an application level on the
server side, and I cannot imagine that being performant. Again, I'm much
more interested in squeezing out every last drop of performance. That's the
whole reason for WS in my mind as opposed to existing JS APIs. If
Multiplexing isn't part of the base API, at the very least it needs to be
possible to do it efficiently via a protocol-level extension, and we need to
define the mechanisms for such extensions now so that we can start doing
experiments and get deployments and the data you request.

> I'm open to trading off simplicity for performance, but only if I see data
> backing up the performance claims. Every performance project should start
> with measurement.
>
> Regards,
> Maciej
>
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 4 Aug 2010 03:15:24 +0100

> the dynamic process is complete. That's why HTTP added chunking.

I agree (and favour chunking like this), however there is a
counterargument (which shouldn't be assumed to have my support):

The JS API does not permit incremental delivery of parts of a message
as they arrive, so from the JS API perspective, everything is done
with whole messages.

If you have incrementally delivered large messages, it's possible to
write a WebSocket protocol that depends on this (maybe even
bidirectionally) and is usable by non-JS clients, yet is impossible to
use from JS API browser clients.

So if you want to send large messages of unknown size that are sure to
be equally accessible to the JS API as other clients, think in terms
of "joining messages in an application-visible manner" rather than
splitting them.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 4 Aug 2010 03:27:19 +0100

>    need centralized consensus on one true multiplexing algorithm.

Will it?  If those op-codes are negotiated, then multiplexing becomes
an optional extension, and multiple competing extensions are perfectly
interoparable with each other.  It's also fine for an implementation
to offer multiple ways to do the same thing, if the negotiation is
sensible.

I think that's better than choosing the one true multiplexing
algorithm, not least because there probably isn't an ideal algorithm
for all apps, and some specialised clients and their servers may
prefer to use a different one.

>     3. martinesque - extension via dedicated mime frame (meta-data plus

4. Extension codes identified at handshake negotiation time, or a
later negotiation time in control messages.

For example: "WebSocket-Protocol: x-multiplex;control-byte=12, keepalive-aggregator;control-byte=13"

Doing it at handshake time has the advantage that it's easily
future-proof, very little needs to be defined up front, and simple
implementations can ignore them entirely.

-- Jamie

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 03 Aug 2010 19:41:06 -0700

On Aug 3, 2010, at 6:29 PM, Greg Wilkins wrote:

> I did briefly explain.  I cited the hum volume not as an argument, but as an indication that we have already had the arguments explained.

If good arguments were presented at the face-to-face, they should be presented on the mailing list, since the IETF process is that consensus cannot be declared without confirming on the mailing list.

Thanks for the explanation, here are some counterpoints:
> variable length length fields is more complex than a fixed length length field.
Is it more complex than fragment reassembly? I find that extremely unlikely based on my experience as a programmer.
> sending large content in a single frame is difficult if the content is being generated dynamically or it's length is otherwise not known.
You are correct, and that is a good reason for optional chunking. However, sending large content as chunks instead of a single frame is harder when it already exists as a data item of known length (e.g. a large file). Extremely large objects (over a gig, say) tend not to be dynamically generated. If you have to send a gig of data, chunking it into 64k frames is likely to be extremely inefficient.
> a small fixed sized length allows simple implementations to have fixed sized buffers (eg 64k)
A variable size length allows this too - you can just read up to 64k at a time. Whether with a single large frame or multiple fragments, you can choose yourself whether to process 64k at a time or reassemble at the end.
> fragmentation prepares the ground for future extensions for multiplexing and flow control.
I'm not assuming we will need such extensions until we have data to back it up.
> fragmentation on its own is not that complex
I disagree. It's more complex for senders of fixed-length content, and for receivers in nearly all cases.

On the other hand:

* If the receiver would like to read all contents into a single large memory buffer up to some fairly large size (say, a gigabyte), then it's a significant disadvantage, and likely performance hit, to get it in chunked form. The length is not known up front, therefore it is impossible to pre-allocate a buffer of the right size. To read a 256k message if chunks are 64k max, you have to perform at least 4 memory reallocations; with a single frame it could be one. Browser-hosted clients will likely have to provide a 256k message in memory to JavaScript, so it's not an option to have a 64k buffer and process the content in chunks or spool it to disk.

Thus, a single large message supports both of the following strategies:

A) Read a chunk at a time into a smallish fixed-size buffer.
B) Pre-allocate a buffer large enough for all the data and read it all in.

Mandatory fragmentation for large messages only supports strategy (A). Strategy (B) is likely preferable in the case where the browser is the receiver.

> But if you have a counter proposal that you think might satisfy the hummers, please make it.  The current protocol certainly does not.

Without knowing the rationale of any given hummer, I cannot predict what they would find satisfactory. I hope I have explained enough of my reasoning that others can guess whether a proposal would seem good me. Mandatory chunking for large messages does not sound good. 

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 12:46:29 +1000

Jamie,

Interesting idea.  I think we had assumed that negotiating extension opcodes
would be too complex, but what you have suggested looks like the start of a
simple solution.   Could you care to expand on it a bit more with exactly
how the client/server would agree on a final set of extended op-codes?

So it looks now like we have  4 possible extension approaches:

hixieasque - extension via defined opcodes
jamiesque - extension via negotiated opcodes
robertoesque - extension via per frame meta-data
martineque - extension via a dedicated mime frame

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 4 Aug 2010 03:50:42 +0100

>    I think that we have reasonable consensus on something like:

Two questions:

   1. Are intermediaries allowed to move the fragment
      boundaries to optimise their delivery?

         1a. .... "add fragment boundaries"
         1b. .... "remove fragment boundaries"

   2. Are intermediaries allowed to delay delivery of incomplete
      messages by an arbitrarily long time, including forever?

Related to 2, the JS API always delays delivery of incomplete received
messages, and in a sense the JS API is acting as an intermediary
between the server and client application.  So it should cause no harm
for other intermediaries to do it, when the receiver is a browser
providing the current JS API.  But other clients may be harmed by such
delays, or not, depending on what they expect.

Also I'd like to remind you of a past discussion where we learned that
carrying general byte stream protocols over WebSocket is slightly more
efficient if the flag meaning "tell intermediary to forward" is
independent of the flag "this boundary is adjustable".

It's tempting to specify "intermediaries shall forward all bytes they
receive eagerly, without interpretation", but that is (a) not going to
happen as firewalls will inevitably filter some WebSocket applications
so we might as well suggest what they must do right, and (b) not ideal
for TCP latency, especially where MTUs differ on either side, although
it's not too bad, and (c) a multiplexing/demultiplexing WebSocket
proxy cannot do this.

I've no issue with eager byte forwarding being the default behaviour
expected of intermediaries if nothing else is negotiated, and it makes
considerable sense to me for the questions to be deferred over to
the multiplexing-and-intermediaries discussion.

-- Jamie

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 19:53:22 -0700

As far as I can tell, what Jamie proposed is basically equivalent to what I
proposed in http://www.ietf.org/mail-archive/web/hybi/current/msg02726.html -
the problem with this approach is that it doesn't work well for extensions
that need a variable amount of space in each frame. For something like
multiplexing, compression, or fragmentation it works fine as you have a
fixed number of bits in each frame, but if you wanted to do something like a
mime extension, you would have to reserve a large number of bits in each
frame to make sure you had enough. At least, if I am understanding the
proposal correctly.

That said, it has a very nice advantage in that I don't have to parse
header-goo on each frame, and that the data I care about is at a
well-defined position, at least for all frames in a single client-server
channel (if a different client negotiates a different set of extensions,
that would be different, obviously.)

-Ian

On Tue, Aug 3, 2010 at 7:46 PM, Greg Wilkins <[email-masked]> wrote:

>
> Jamie,
>
> Interesting idea.  I think we had assumed that negotiating extension
> opcodes would be too complex, but what you have suggested looks like the
> start of a simple solution.   Could you care to expand on it a bit more with
> exactly how the client/server would agree on a final set of extended
> op-codes?
>

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 13:06:53 +1000

On 4 August 2010 12:41, Maciej Stachowiak <[email-masked]> wrote:

> But if you have a counter proposal that you think might satisfy the
> hummers, please make it.  The current protocol certainly does not.
>
>
> Without knowing the rationale of any given hummer, I cannot predict what
> they would find satisfactory. I hope I have explained enough of my reasoning
> that others can guess whether a proposal would seem good me. Mandatory
> chunking for large messages does not sound good.
>

Maciej,

These entire framing threads a have been about trying to reach a consensus
on the concerns of the hummers  while coming up with a proposal that will
seem good to you.  I for one have been really trying to keep the complexity
to the minimum and have constantly referred back to the work needed for a
minimal client and server.    I accept that you are not convinced about
fixed lengths and fragmentation, but then you've not managed to convinced
others of the contrary and are not coming up with counter proposals that
might let us move forward.

I'm not saying that your arguments for arbitrary length frames are without
merit (and the first of my frame proposals did use the variable length
frames). But if I were the chairs, I would be calling the consensus on the
side of fixed length and fragmentation.  I guess we can do nothing other
than put our proposals and wait for a consensus call from the chairs.

> I'm not assuming we will need such extensions until we have data to back it
There has already been many sources of evidence that multiplexing is a
necessary extension for many environments.  You simply don't agree with that
evidence.

However, consider the four possibilities:

0) you are right about the need for multiplexing and the framing mechanism
is not capable of well supporting it
1) you are right about the need for multiplexing and the framing mechanism
is capable of well supporting it
2) you are wrong about the need for multiplexing and the framing mechanism
is not capable of well supporting it
3) you are wrong about the need for multiplexing and the framing mechanism
is capable of well supporting it

outcomes 0,1 & 3 are good outcomes.  2 is a problematic.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 13:12:17 +1000

>         1b. .... "remove fragment boundaries"

In an earlier message I proposed that fragmentation should not change the
semantics of a message and that intermediaries should be free to fragment
and/or aggregate.

>      messages by an arbitrarily long time, including forever?

This has not been discussed, other than the discussion that we had many
months ago about a must-flush bit, that you mention below.

> the multiplexing-and-intermediaries discussion.

I think that the semantic difference between have a stream of messages and a
stream of fragments is pretty fine, so if somebody really wanted to have
partial content that was forwarded eagerly, then they can have a message
content type that is a partial content.

So I  would be happy to say that intermediaries SHOULD eagerly forward
fragment frames and MUST forward end of message frames.

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 4 Aug 2010 04:19:17 +0100

Ian Fette (イアンフェッティ) wrote:
> 
>    As far as I can tell, what Jamie proposed is basically equivalent to
>    what I proposed
>    in� [1]http://www.ietf.org/mail-archive/web/hybi/current/msg02726.html
>    � - the problem with this approach is that it doesn't work well for
>    extensions that need a variable amount of space in each frame. For
>    something like multiplexing, compression, or fragmentation it works
>    fine as you have a fixed number of bits in each frame, but if you
>    wanted to do something like a mime extension, you would have to
>    reserve a large number of bits in each frame to make sure you had
>    enough. At least, if I am understanding the proposal correctly.
> 
>    well-defined position, at least for all frames in a single
>    client-server channel (if a different client negotiates a different
>    set of extensions, that would be different, obviously.)

Actually I was suggesting control-byte = the frame type id stored in
the initial, fixed position in the frame header, to avoid the problem
of type id collisions, and the format of the frame after the command
id would be entirely specific to that type id apart from the id and
length.

Since the frame type would only be used if both sides agreed to it in
negotiation (as with all extensions), and there would be no
collisions, that would be future-proof and easy to use.

Naturally an extension frame type like "multiplex" would tend to reuse
the existing frame types *inside* the multiplex frame type, so it can
carry anything.  (It's not the same as two separate frames, because
the multiplex frame would have an ordinary length field, then the
contents would include the subsidiary frame type without a length field).

I suggest that even message splitting into chunks could be treated as
an optional extension, which simple implementations needn't accept or
care about.  It would make simple implementations simpler.  We'd still
standardise this early, but make it optional.

Everything needs to be able to parse just the command id, the length,
and any flags which bear on forwarding/splitting/aggregating behaviour
to the limited extent those are universal.  The latter don't matter,
if they can be regarded as extensions.

[It's quite desirable that everything knows hows to ignore frame types
they don't understand, by parsing the type and length, so that (a)
active intermediaries don't trash the stream (since they might not
participate in negotiation), and (b) it helps with latency reduction
at connection startup.]

-- Jamie

ps. This turns out similar to IP-style "variable-length options",
represented in a different way.  Its major difference is, if the
receiver doesn't understand the frame type, then it won't know how to
skip the "option" by itself; it'll skip the whole frame.  For some
extension frame types, that is preferable to ignoring the "option"
part, but for others (as we learn from IP options) it's good to have
options which can be ignored by themselves.  Luckily we can create
either, by encoding the option as a frame by itself, or a frame
containing the remainder to be ignored if not understood.  That is
very flexible, and at the same time very easy to implement.

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 03 Aug 2010 20:29:19 -0700

On Aug 3, 2010, at 8:06 PM, Greg Wilkins wrote:

> These entire framing threads a have been about trying to reach a consensus on the concerns of the hummers  while coming up with a proposal that will seem good to you.  I for one have been really trying to keep the complexity to the minimum and have constantly referred back to the work needed for a minimal client and server.    I accept that you are not convinced about fixed lengths and fragmentation, but then you've not managed to convinced others of the contrary and are not coming up with counter proposals that might let us move forward.

I not only am not convinced, but I produced arguments to support my position, which so far no one has replied to.

> I'm not saying that your arguments for arbitrary length frames are without merit (and the first of my frame proposals did use the variable length frames). But if I were the chairs, I would be calling the consensus on the side of fixed length and fragmentation.  I guess we can do nothing other than put our proposals and wait for a consensus call from the chairs.

Seems pretty weak to me to declare "consensus" on a proposal where the supporters have not addressed the counter-arguments. To be fair, I only posed my arguments against just now, so we should probably wait a reasonable time for anyone with a stake in the issue to reply.

> There has already been many sources of evidence that multiplexing is a necessary extension for many environments.  You simply don't agree with that evidence.

I've seen claims that it is needed, but I haven't seen evidence for those assertions. I admit with the recent high volume of mail on this list, I haven't read every message carefully, so if there's anything I missed that presents evidence, could you please give me a pointer?

Still, I think multiplexing is at best an argument for optional chunking, not for mandatory chunking of all large messages.

> 0) you are right about the need for multiplexing and the framing mechanism is not capable of well supporting it
> 1) you are right about the need for multiplexing and the framing mechanism is capable of well supporting it
> 2) you are wrong about the need for multiplexing and the framing mechanism is not capable of well supporting it
> 3) you are wrong about the need for multiplexing and the framing mechanism is capable of well supporting it
> 

I don't think these are the only relevant considerations. There is also the cost in the case where multiplexing is not used.  That's relevant both if multiplexing turns out to be unnecessary, and for services that choose not to use multiplexing even if some use it.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 13:33:10 +1000

Jamie,

so to make your proposal a little more concrete, I think that you are
advocating something like:

  +--------------------------------------------------+
  | opcode (8)            |           Length(16)     |
  +--------------------------------------------------+
   |                      Data                        |
  +--------------------------------------------------+

    ws-frame            = frame-opcode
                          frame-length
                          octet-data

    frame-opcode        = %x00 ; text data frame
                        | %x01 ; final frame
                        | %x02-ff; extension frame

A client would indicate the extensions that it wants to support with a
handshake header like:

   Websocket-Ext: fragment;mandatory=true, ping;mandatory=true, multiplex,
compression

The server could then respond with the extensions that it will be used, plus
allocations of bits/codes:

   Wesocket-Ext: fragment;bit=7, ping;code=02-03, multiplex;code=10-1F

Is that about right?

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 4 Aug 2010 04:35:20 +0100

Ian Hickson wrote:
> On Tue, 3 Aug 2010, Ian Fette (?~B??~B??~C??~C~U?~B??~C~C?~C~F?~B?) wrote:
> > 

Virtually all kernels handle sockets the same way.  Windows is a
notable exception.  WebSocket may or may not take over the world, but
it's still unlikely to matter enough to cause all major OSes to change
their kernels that much :-)

But anyway, it's a limitation of TCP, not particular kernels.

TCP requires, in practice, both ends to be able to buffer about the
same as the TCP window size, which must be large enough for reasonable
throughput.  The amount needed is getting larger as links get faster.
(Some links need gigabytes per socket, to get maximum sustained throughput!).

No kernel can avoid this, although good kernels reduce the average
memory used (allowing it to be used for caches), without reducing the
committed memory potentially used.

However let's not pretend multiplexing easily avoids this buffering.
While you can dispatch messages as they come in (with a single TCP
window's worth of buffer), you still need a comparable amount of
memory to buffer incoming *dispatched* messages in a general
implementation.

For example, browsers would buffer incoming WebSocket messages in the
DOM event queues, before they are processed by the independent
handlers of different WebSocket objects in different windows, perhaps
running in different processes (Chrome).  Those queues also use
comparable memory.

Servers will have equivalents to those queues, perhaps in pipes to
application processes or threads.

Although you can pick off one message at a time to dispatch and
process, before picking off the next one, and that *can* use much less
memory, that strategy breaks independence of flows, so it is not
usable as "transparent" form of multiplexing multiple processes over a
single socket.

Nonetheless, even though transparent multiplexing uses quite a bit of
memory too, the multiplexing way does give applications a lot more
control over buffering choices, including committing much less memory
to each flow than TCP would on a high bandwidth(-delay) link, while
maintaining efficient throughput on those links.

-- Jamie

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 21:07:45 -0700

On Tue, Aug 3, 2010 at 8:29 PM, Maciej Stachowiak <[email-masked]> wrote:

> I not only am not convinced, but I produced arguments to support my
> position, which so far no one has replied to.
>
> Seems pretty weak to me to declare "consensus" on a proposal where the
> supporters have not addressed the counter-arguments. To be fair, I only
> posed my arguments against just now, so we should probably wait a reasonable
> time for anyone with a stake in the issue to reply.
>
>
> I've seen claims that it is needed, but I haven't seen evidence for those
> assertions. I admit with the recent high volume of mail on this list, I
> haven't read every message carefully, so if there's anything I missed that
> presents evidence, could you please give me a pointer?
>
> Still, I think multiplexing is at best an argument for optional chunking,
> not for mandatory chunking of all large messages.
>

In the interest of moving towards consensus, I will say that I believe so
long as the protocol is sufficiently extensible such that things like
multiplexing and chunking can be implemented in an extension (and we are
able to actually implement such an extension in Chrome to start getting
real-world data), then I think I don't have such a strong feeling about
fixed- versus variable-length encoding. However, that depends on having some
sane extensibility mechanism, and an openness to such extensions in WebKit
;)

> I don't think these are the only relevant considerations. There is also the
> cost in the case where multiplexing is not used.  That's relevant both if
> multiplexing turns out to be unnecessary, and for services that choose not
> to use multiplexing even if some use it.
>
>
As I said on another thread, this could perhaps be re-phrased as to "what is
the cost of an extensibility mechanism that would allow for e.g.
multiplexing in an efficient manner" which might be a way to look at things
that will help us get to consensus.

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 04:13:12 +0000 (UTC)

On Tue, 3 Aug 2010, Scott Ferguson wrote:
> > > >   I agree. I can't see any benefit to fragmentation over a 
> > > > variable-size length field for an initial version without 
> > > > multiplexing. If the variable-size length field is well-designed, 
> > > > then in the common case where the message size is small it will 
> > > > only cost one extra branch to read the length. In the rare case 
> > > > where the message size is large, a variable-size length is easier 
> > > > to deal with than reassembling fragments, and easier on the 
> > > > sending side too.
> > >

Is this a common case we should expect? What are the use cases for this?

If this is a rare occurence, then it's probably simplest to support this 
at the application layer - nothing stops anyone from having a subprotocol 
that defines the first byte/character of each message as being a 
"has continuation" byte.

If this is a common occurance, then we'll be able to tell when people find 
themselves often implementing this at the application layer, and then we 
can easily add it at the WebSocket layer at that time (in a few months). 
We don't need to reserve bits specifically for it today.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 21:16:48 -0700

> We don't need to reserve bits specifically for it today.
We're adding a voice input capability, and we want to start sending the
user's voice up as soon as they start speaking, without waiting for the
recording to be finished so that we can start backend processing. Yes, we
could probably break it into multiple messages, but given that semantically
it's a single message, this is a case where we would want to start sending
data before knowing its length. Ditto for hooking up a video camera and
sending that data to some sort of web service.

---

From: "Thomson, Martin" <[email-masked]>
Date: Wed, 4 Aug 2010 12:20:27 +0800

[email-masked] writes:
> As far as I can tell, what Jamie proposed is basically equivalent to what I proposed
> in http://www.ietf.org/mail-archive/web/hybi/current/msg02726.html - 

I like this proposal.  It got lost in the email noise the first time around.  It's not exactly what Jamie proposes, but it could be made compatible with it.

+-----------------------------------------------+
|frag(1)| ???(1)|opcode(6)| Length(24)          |
+-----------------------------------------------+
|      Extensions (prearranged length)          |
+-----------------------------------------------+
|                     Data                      |
+-----------------------------------------------+

It makes new protocol features transparent to the framing layer.  All you need to do is define a negotiation for the features.

It's a shame that it doesn't work for multiplexing.  

--Martin

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 04:27:15 +0000 (UTC)

On Tue, 3 Aug 2010, Ian Fette (ã~B¤ã~B¢ã~C³ã~C~Uã~B§ã~C~Cã~C~Fã~B£) wrote:
> recording to be finished so that we can start backend processing. Yes, 
> we could probably break it into multiple messages, but given that 
> semantically it's a single message, this is a case where we would want 
> to start sending data before knowing its length. Ditto for hooking up a 
> video camera and sending that data to some sort of web service.

I would hope that audio and video data would get sent using the 
ConnectionPeer mechanism, ideally UDP-based (though that protocol hasn't 
even been sketched out yet), not over TCP-based WebSockets.

But if we do want to send live audio data over WebSockets, then we're 
going to do a bunch of work, like adding microphone support to the Web 
platform. When we do that, we can easily add a live-stream-specific frame 
code that sends fixed size blobs of audio and/or video streams. The 
protocol supports that as an extension fine, without needing to define how 
fragmentation works today.

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 04:37:19 +0000 (UTC)

On Wed, 4 Aug 2010, Greg Wilkins wrote:
> extensions. There was very little support for variable length lengths, 
> as that was just seen as more complex than fragmentation.
> 

With all due respect, I'd much rather base work on this protocol on actual 
technical arguments, and not on a self-selected group of people humming, 
as much fun as that may be.

Here's a proposal for length-based frames:

 - one byte to indicate the frame type (used by the WebSocket API to 
   decide what kind of event to fire)
 - a BER-encoded packed integer, most-significant-digits first, high bit 
   used to indicate a continuation byte, giving the payload length.
 - payload

I'm still not convinced that length-based frames are safe enough for the 
Web, but I have been convinced by the various discussions of security bugs 
related to framing that sentinel-based frames are likely just as insecure. 
I really think we should find a solution that is more secure than either, 
though; I don't think that we should go with length-based frames on the 
basis that the alternatives are equally likely to lead to security bugs.

In the absence of a better solution, though, the above (basically the same 
as what the protocol has now, except with sentinel-based frame type 
removed) seems to fit the bill fine.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 3 Aug 2010 21:38:39 -0700

On Tue, Aug 3, 2010 at 9:20 PM, Thomson, Martin
<[email-masked]>wrote:

> I like this proposal.  It got lost in the email noise the first time
> around.  It's not exactly what Jamie proposes, but it could be made
> compatible with it.
>
> It makes new protocol features transparent to the framing layer.  All you
> need to do is define a negotiation for the features.
>
Assuming multiplexing is an extension, we could make it work fine. E.g. the
multiplexing extension will reserve 8 bits in the extension section, and
we're set.

---

From: Roberto Peon <[email-masked]>
Date: Tue, 3 Aug 2010 21:56:15 -0700

On Tue, Aug 3, 2010 at 6:33 PM, Ian Fette (イアンフェッティ) <[email-masked]>wrote:

> I don't think we would be able to get any data from a Google deployment in
> a reasonable time for this discussion, sadly.
>

We do have data on at least the average number of TCP connections created
for HTTP for pages. I think we've mentioned it before, but it was a long
time ago!
I believe that the way we measured that was that we took the Alexa top 1000
and counted the concurrent connections for each page.
It is my fervent hope that websockets wouldn't be that bad, but without
multiplexing, it seems like it is at best an HTTP analogue in terms of
connection usage (with the exception that it attempts to have long lived
connections which possibly makes it far worse!).

> for a single user kills us.

Mike Belshe or I can quote numbers from experiments with SPDY, which is
attempting to solve another problem, but does show significant benefits by
using multiplexing.
We're deploying SPDY to clients and servers now, and are still in the
process of getting real-world data.

>> backing up the performance claims. Every performance project should start
I perfectly understand your concern-- I mirror that w.r.t. scalability.
Thankfully, at least with SPDY, we have data which we can speak about
publicly.

I'm happy to try (try being operative word) to find people here to do
experimentation and provide data. The trick is, of course, that it seems
that we're not wanting to wait for the amount of time it will take to get
the experiments up and analyse the data before coming to a conclusion on the
spec :(

-=R

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 05:00:39 +0000 (UTC)

>    - variable length length fields is more complex than a fixed length

I thought we weren't worried about amateur programmers?

In any case, I don't really agree that this is true. In perl, for example, 
outputting the variable-width length in the spec today is a trivial 
function call (pack('w', $length)), whereas outputting a fixed 
length field has varying levels of complexity based on whether the length 
is 32 bits, 64 bits, or 128 bits. This naturally varies based on the 
programming language, but even in the most barebone language doing bit 
manipulation is easy in practice, and not particularly prone to subtle 
hard-to-test bugs or edge cases.

>    being generated dynamically or it's length is otherwise not known.

A variable-width length does not preclude having continuation frames if 
necessary, either at the application layer today or at the WebSockets 
layer in the future.

>    - a small fixed sized length allows simple implementations to have fixed

Nothing requires a payload with a variable-width length to be read in one 
buffer, any more than a split-payload requires one buffer.

>    multiplexing and flow control.

There's nothing in the current protocol that prevents fragmentation from 
being implemented. This has nothing to do with the length field.

>    - fragmentation on its own is not that complex

It's not clear that it's less complex than variable-width lengths. That's 
not to say that we shouldn't support it, but why require it?

---

From: "Thomson, Martin" <[email-masked]>
Date: Wed, 4 Aug 2010 13:03:23 +0800

Unfortunately, that’s only true if a channel switch were prevented if a fragmented message was being transmitted.

Imagine that an intermediary wants to de-fragment fragmented messages and it sees the interleaved messages from channels A and B:

  Message (fragment), Channel A: Hello
  Message (end), Channel B: Bonjour tout le monde
  Message (end), Channel A: World

The intermediary just stomped on your messages:

  Message (end), Channel A: Hello {Channel B} Bonjour tout le monde
  Message (end), Channel A: World

Now, we could prevent this sort of thing by saying that if you don’t understand all the extensions, don’t touch, but that’s a little antisocial.

--Martin

From: Ian Fette (イアンフェッティ) [mailto:[email-masked]]
Sent: Wednesday, 4 August 2010 2:39 PM
To: Thomson, Martin
Cc: Greg Wilkins; [email-masked]
Subject: Re: [hybi] Framing take IV

Assuming multiplexing is an extension, we could make it work fine. E.g. the multiplexing extension will reserve 8 bits in the extension section, and we're set.

--Martin

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 4 Aug 2010 15:31:33 +1000

So it appears that I have over stated the near-consensus, as it looks like
that all we have is

  |  ???   (?)             |  Length(?)              |

Where there are about 4 different proposals how to handle the ???  and at
least 3 contenders for encoding the length.

I propose that we iterate here forever or until somebody stands victorious
having proved definitively beyond a  shadow of a doubt that in all
circumstances (or at least in all circumstances that the victor cares about)
that they have defined the ultimate way to encode the length of a frame that
is immune from all security attacks and is so simple that it will be
enshrined as a self evident truth for all time.

Alternately we could just pick 1 length encoding, accept that maybe it is
not exactly the ultimate encoding for all people in all circumstances, but
that it is good enough for most cases,  and then we can move on.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 04 Aug 2010 10:33:13 -0400

On Wed, 2010-08-04 at 00:53 +0000, Ian Hickson wrote:
>  I could see an argument for supporting 

In the absence of fragmentation the sender must have the entire message
available in order to prefix the total length in the header - so it
engages in store and forward instead of streaming behavior.

Benefits of streaming:

* Reduced memory consumption on the sender in cases where the message is
being "passed through". A service that does a large database query is a
good example here - the results can be passed back as they are found
even before you know how many there are.

* overall transfer time is reduced when the data to be sent takes time
to generate. Again, consider the DB query example. the transfer time for
the initial records is overlapped with the "think time" of the whole
transaction. Instead of the total send time being "processing + bulk
transfer" it is max (processing + network latency, bulk transfer)

* the receiver can overlap decompression with the receipt of streamed
chunks instead of adding decompression as a store-and-forward step onto
the end. (because without chunking the server cannot begin to send the
document until the whole thing is spooled). This will improve effective
message transfer time up to the JS layer.

* better latency properties to the recving ws stack. Other APIs than the
anticipated JS one may make use of that at no cost to the JS one.

And of course, as you say, fragmentation allows multiplexing and
effective prioritization of flows by the sender.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 04 Aug 2010 12:04:01 -0400

On Tue, 2010-08-03 at 18:20 -0700, Maciej Stachowiak wrote:

>  people think fragmentation is simpler than a variable size length
> field, they should explain why, 

My previous mail
( http://www.ietf.org/mail-archive/web/hybi/current/msg02886.html )
provided some arguments other than simplicity in favor of chunking.

But I also believe fragments with fixed lengths are simpler than a BER
because properly aligned lengths are always correct data - there are no
error cases as all the bits are defined to be some quantity regardless
of their arrangement. BER has error cases and determining the expressed
quantity requires multiple indirections and a significant algorithm to
use it. 

That contrasts with a fixed length 16 bit unsigned integer that requires
only a load instruction and perhaps a built in byte order swap to use
it. The algorithm is literally already in all hardware.

The cost of chunking vs BER is going to be 1 header (~4 bytes) every
64KB and only occurs at all on messages over 64KB (which excludes lots
of use cases). That's cheap!

I will say this though - sometimes the receiver would like to know how
big the total message is upfront in order to do non fragmented
preallocation for example. If the sender knows that information (i.e. it
is not really streaming) but is sending a > 64KB message using fragment
framing there is no way to convey that useful info to the recvr up
front. An option, unrelated to the framing, might be used to carry that
info in a variable length format (ber or maybe even ascii hex..
whatever)

---

From: Mike Belshe <[email-masked]>
Date: Wed, 4 Aug 2010 09:17:48 -0700

> > seem to actually gain us anything.

I'm going to write a bunch of stuff which makes it seem like I'm in favor of
variable length encoding.  My true position:  it's not materially different
to use a fixed length + frags vs a variable-length-encoding which supports
large sizes.

> length field for an initial version without multiplexing.

I agree with that statement.  The issue is that several folks on this list
are trying to make multiplexing work as an extension.  Of course, the frag
flag could be part of the extension rather than part of the base...

> If the variable-size length field is well-designed, then in the common case
> where the message size is small it will only cost one extra branch to read
> the length. In the rare case where the message size is large, a
> variable-size length is easier to deal with than reassembling fragments, and
> easier on the sending side too.
>

I like the variable-sized length field.  As modest "data" on this topic.
 Every place in SPDY where we used fixed, 16bit fields initially, we've
changed.  16bit fields are just on-the-cusp of being too small in some
cases.  Going to 32bits is easy to do, but leaves you with a lot of extra
zeros for most cases.  The frag-flag works around this.  Variable encoding
is also attractive.  Server folks will say "it's not scalable", but I
haven't seen any data to support that claim.

> not overtuning to a single vendor's setup.

Well, WebSockets is about letting apps write their own apps, so this is
pretty hard to do, right?  I thought from the WebSockets view, this was
about reducing the number of connections more than anything.  It's unclear
that "average degree of multiplexing" is measurable or even the right
metric?

> involves numbers and doesn't sound handwavey.

Oh - I tested this with SPDY stuff a lot.  What specifically do you want?
 But it is all app-specific.  In our case, the app was HTTP pages.  That is
probably not indicative of what people will use websockets for.

> whatever to share the connection.)

Multiplexing is relatively easy (not trivial, though).  It's the flow
control that is hard.   And unfortunately, I don't think you can seriously
claim to have multiplexing unless you address flow control (SPDY version 1
did it, it falls back to TCP flow control, which can stall out streams
inappropriately and is generally not-good).  But when you add flow control
to the multiplexed data, it's very very easy to kill performance
(underthrottle).

> backing up the performance claims. Every performance project should start

I would really love to see multiplexing as part of the websocket protocol.
 I don't see how we get there.  Roberto's hooks for "leave it in an
extension" seem about right to me.  But that does mean that the base
protocol should not be attempting to solve multiplexing issues.

One approach to getting to multiplexing it to leave it to a lower level.
 For instance, if we had SCTP, we wouldn't discuss putting multiplexing at
the WebSocket layer.  Of course, replacing lower levels is hard, so this may
sound like something that would never happen.  But SPDY intends to put
multiple streams over a TCP stream.  Maybe that would be the vehicle for
multiplexing.

Mike

---

From: Douglas Otis <[email-masked]>
Date: Wed, 04 Aug 2010 09:53:29 -0700

Please keep data handling state assertions explicit,
map out future options, and retain alignment.

     0      1      2      3          -           7
+--------------------------------------------------+
| str(1)|end(1)|cmp(1)|       Op-Code(5)           |
+--------------------------------------------------+
|                 Stream ID (8)                    |
+--------------------------------------------------+
|                 Length MS (8)                    |
+--------------------------------------------------+
|                 Length LS (8)                    |
+--------------------------------------------------+
|         Data (8) * Length + Pad to 32b           |
+--------------------------------------------------+

With this structure, the Op-Code field is independent
of data handling state.  By using "End", rather than
"Continue", null injection will not terminate handling
without error.  Detecting errors improves reliability
and security.

-Doug

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 17:55:26 +0000 (UTC)

> quantity requires multiple indirections and a significant algorithm to 

As far as I can tell there are no error cases as defined in WebSockets.

Variable-width lengths also doesn't preclude chunking at a future point 
should it prove to be needed (which we would be able to tell because we 
would see authors making their application-layer protocols implement 
chunking on top of WebSockets).

---

From: Patrick McManus <[email-masked]>
Date: Wed, 04 Aug 2010 14:51:01 -0400

> As far as I can tell there are no error cases as defined in WebSockets.

I believe the specific ber length suggestion was:

In that case, as I understand it, it would be an error if the ber type
encoding byte didn't specify a universal primitive type of 2. (or maybe
I'm wrong and it needs to be some other identifier - but that doesn't
really matter).

it is not a big deal, but it is an error case.

---

From: Ian Hickson <[email-masked]>
Date: Wed, 4 Aug 2010 19:58:17 +0000 (UTC)

On Wed, 4 Aug 2010, Patrick McManus wrote:
> > > But I also believe fragments with fixed lengths are simpler than a 
> > > BER because properly aligned lengths are always correct data - there 
> > > are no error cases as all the bits are defined to be some quantity 
> > > regardless of their arrangement. BER has error cases and determining 
> > > the expressed quantity requires multiple indirections and a 
> > > significant algorithm to use it.
> > 

The spec just uses the packed integer part, not the whole ASN.1 form. 
There are no error conditions with that as far as I can tell. There's one 
redundancy (leading 0x80 bytes), but that's not an error case, and since 
the behaviour is well-defined, doesn't even seen to be a likely attack 
vector (which redundancies can often become), though naturally we'd want 
to make sure we tested that thoroughly in our test suite.

---

From: S Moonesamy <[email-masked]>
Date: Wed, 04 Aug 2010 15:27:54 -0700

At 19:41 03-08-10, Maciej Stachowiak wrote:
>If good arguments were presented at the face-to-face, they should be 
>presented on the mailing list, since the IETF process is that 
>consensus cannot be declared without confirming on the mailing list.

Right.  And the consensus calls will be posted as soon as possible.

>as much fun as that may be.

Decisions are made based on technical arguments.  The Working Group 
Chairs do not decide what is the best technical argument; they 
determine the consensus of the Working Group.  If a person posts a 
technical argument, he/she should bear in mind that other Working 
Group participants should be convinced of the merits of the argument 
and see how consensus can be achieved.

Humming is done to get a sense of the room.  Anyone can participate 
in the HyBi Working Group and is free to provide input and weigh on 
the decision process.  The Working Group sessions at an IETF meeting 
are open.  You can walk to the microphone and state your views even 
if you do not have a badge.  There is also a Jabber room for remote 
participants and a live audio feed of the session [1].

As there hasn't been any questions about the process [2], I presume 
that all participants in this Working Group fully understand how the 
process works.

Regards,
S. Moonesamy
HyBi WG Secretary

1. http://tools.ietf.org/wg/hybi/agenda?item=agenda78.html
2. http://www.ietf.org/mail-archive/web/hybi/current/msg02534.html

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 5 Aug 2010 08:40:20 +1000

> As far as I can tell there are no error cases as defined in WebSockets.
None defined, but I'm pretty sure the following should be treated as an
error:

0x80 0xFF 0xFF  0xFF  0xFF  0xFF  0xFF  0xFF  0xFF  0xFF  0xFF  0xFF  0xFF
0xFF  0xFF ......

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 5 Aug 2010 03:03:04 +0100

>    advocating something like:

Sorry, I'm not able to decode the diagrams you are sending.  They come
out as mangled text in my editor.

But yes, the basic idea: length, opcode, data.

The length must include the opcodes as well as the data, although the
first opcode byte doesn't have to be included if we don't want to.

The order should probably be length, opcode, data, so a "wrapper"
style opcode would end up looking like these made up examples.
First the simple one:

    length, any-opcode, data  (basic simple frames, including text, binary etc.)

Extensions, to be defined by whomever wants to, no standard necessary:

    length, multiplex, flow-id, final-bit, forward-bit, any-opcode, data
    length, crc32-opcode, any-opcode, data, crc32
    length, md5-opcode, any-opcode, data, md5

Putting length after the first opcode suggests length is part of the
opcode's parameters, which it isn't, as you can see in the last three
examples, where any-opcode isn't associated with a length of it's own,
but it's the same set of opcodes.

>                                     | %x02-ff; extension

I'd rename "final frame" as "basic data frame", but it's a nit.

>           frame-length               = %x0000-FFFF

I'm not sure if this limit penalises implementations shifting large
amounts of data over jumbo frames, which are about 64kB per packet, or
over TCP segmentation accelerators, which take large packets and split
them in hardware on the way out.  I.e. a larger limit such as 24 bits
might be better, for implementations which can use "sendfile" or
"splice" style transmission methods to share memory among sockets and
avoid copying the data.  I haven't investigated or tested in
sufficient detail.

However I'm leaning towards the BER-style that Ian Hixie suggests,
with an explicit note in the specification saying what the maximum
permitted BER value is (or equivalently what the maximum number of
bytes encoding the value are), with the implication that all
implementations should count properly up to that (although they may
still reject a message on the grounds of being too large for local
policy of course).

The first byte in BER does not have to use all the bits, so we could
reserve a few bits in the first byte for universal flags _if_ we think
there are any, and the remainder of bits would be sufficient for small
lengths.  I'm not sure if this is needed - do you think there are any
universal bits needed?

>         Websocket-Ext: fragment;mandatory=true, ping;mandatory=true, multiplex, compression

>    The server could then respond with the extensions that it will be
>    used, plus allocations of bits/codes:

>    Is that about right?

Not quite.

The *client* decides opcode values for features it proposes.  The
server must ignore frames with opcodes for features it doesn't
support; this is why the length is universal (and also for
intermediaries).

If we ever use any mechanism to bypass the negotation round-trip (such
as TLS extensions, encoding initial messages in the handshake, etc.),
it is only effective if the client sets the opcodes.

Although we might not use that mechanism with the HTTP handshake, it
is a valuable and simple trap door in case we decided to avoid startup
latency in future, or with a different handshake, and it is also a
good pattern for latency avoidance in general.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 5 Aug 2010 03:11:32 +0100

Ian Fette (イアンフェッティ) wrote:
> 
>    On Tue, Aug 3, 2010 at 9:13 PM, Ian Hickson <[1][email-masked]> wrote:
> 
>    > > > > �  I agree. I can't see any benefit to fragmentation over a
>    We're adding a voice input capability, and we want to start sending
>    the user's voice up as soon as they start speaking, without waiting
>    for the recording to be finished so that we can start backend
>    processing. Yes, we could probably break it into multiple messages,
>    but given that semantically it's a single message, this is a case
>    where we would want to start sending data before knowing its length.
>    Ditto for hooking up a video camera and sending that data to some sort
>    of web service.

Ah, not really.

That one is better following Ian Hixie's method of application-visible
concatenation.  The JS API is defined to not deliver a message until
the whole message is received (even if we specify split-message chunks
in the protocol), so a JS application would not see the voice data
until a message boundary.

While other clients can use different APIs, we should try to match the
behavioural characteristics of the JS API as closely as possible for
the sake of interoperability.

There is another important reason: Intermediary behaviour isn't well
defined yet, but it would not be unreasonable for an active
intermediary to arbitrarily delay parts of a partially received
message, for example to compress the whole message (or large parts),
or forward more efficiently.

Basically, message boundaries are the only guaranteed delivery points,
and as the voice/camera application needs to guarantee regular
deliveries, it should mark those delivery points explicitly by making
them message boundaries, not chunk boundaries.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 5 Aug 2010 03:18:49 +0100

> We don't need to reserve bits specifically for it today.

Unusually, perhaps, I'm with Ian on this.

As well as the application approach, assuming we get protocol negotiation,
there is nothing to stop defining a new "continuation frame" type.

That would make trivial WS implementation even simpler: They'd just
ignore the negotiation, so wouldn't receive or need to handle
continuation frames.

(I'm really warming to the idea of a super-simple base protocol on
which just about everything we've discussed works as optional
features, ignorable by simple implementations, used by good quality
implementations automatically ("below the API"), and can be defined in
future without necessarily awaiting standardisation to try them out.)

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 5 Aug 2010 12:30:41 +1000

Jamie,

sorry the text diagrams are not working... they should be in fixed width fonts.

Anyway - I'm trying to work out the exact nature of your proposal other than
what I think are stylistic (eg opcode + length  vs BERlength + opcode )

So I think the key aspect of your  proposal I think is that the
extended op codes are negotiated in the handshake.
You say that the *client* decides opcode values for features it
proposes, but how does the client know what features the server
supports?
Surely it is best for the client to simply list the extensions it may
support and then the server picks the final list and allocates opcodes
(and/or bits)?
Could you possible do some more worked examples of how you see this
working - eg if I wanted to negotiate fragments, mime messages and
compressed frames.

Eitherway, if opcodes were able to be negotiated, then would you agree
that this would work with the binary framing more or less how it is
currently
specified;

    op-code(8) + BER length + data

I ask this because the current framing is our starting point.  I know
there are arguments for
    opcode + fixed length
    opcode + BER length
    BER opcode + BER length
    fix length + opcode
    BER length + opcode
    BER length + BER opcode

but I think they are secondary to the basic extension mechanism and
I'd really like to separate out discussion of what is the best length
encoding from what is a sufficient extension mechanism.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 5 Aug 2010 12:40:31 +1000

Ian,

the thing that stops people defining subprotocols is that the concept
of a subprotocol is entirely undefined.

Currently it is exposed in the js API as a simple string.  There is no
indication how that string can be used, how capabilities can be
discovered, declared or negotiated.   There is no mechanism to
allocate op-codes in the first byte other than to come here and try to
convince you that my extension is a good idea.

If we can clearly define an extension and/or subprotocol mechanism
then I think we would be able to answer the concerns of those that
feel the need for particular features but see no way of achieving them
in the current protocol.

If we could clearly see how a browser and server could agree on
feature X extension without breaking applications using subprotocols
or intermediaries that know about ws framing, then there would not be
pressure to support feature X in the basic framing layer.

We are beyond the point that a hand waive of do it in a subprotocol is
satisfactory (specially as many see subprotocols as part of the
application space and not something that browser/servers should be
using for transparent extensions).

regards

---

From: Scott Ferguson <[email-masked]>
Date: Wed, 04 Aug 2010 20:01:29 -0700

> Is this a common case we should expect? What are the use cases for this?

Essentially all web sockets data is dynamic. We're not sending static files.

Data from a chat application is dynamic
JSON is dynamic
SOAP/XML is dynamic
XMPP over websockets is dynamic.
> that defines the first byte/character of each message as being a 

This is incoherent.

> We don't need to reserve bits specifically for it today.

This also makes no sense.

-- Scott

---

From: Ian Hickson <[email-masked]>
Date: Thu, 5 Aug 2010 03:24:12 +0000 (UTC)

On Thu, 5 Aug 2010, Greg Wilkins wrote:
> 
> the thing that stops people defining subprotocols is that the concept of 
> a subprotocol is entirely undefined.
> 
> discovered, declared or negotiated.  There is no mechanism to allocate 
> op-codes in the first byte other than to come here and try to convince 
> you that my extension is a good idea.
> 
> If we can clearly define an extension and/or subprotocol mechanism then 
> I think we would be able to answer the concerns of those that feel the 
> need for particular features but see no way of achieving them in the 
> current protocol.

I'm not sure I follow. A subprotocol wouldn't use "opcodes".

HTTP is a subprotocol for TCP, as is SSH, FTP, or IMAP. They all just use 
TCP as two streams of bytes (one in each direction).

Similarly, you can have subprotocols on WebSockets. They all would just 
use WebSockets as two queues of events (one in each direction).

What do we need to do to indicate how this can be used? There's no 
limitation to how it can be used. Are you saying you think the WebSockets 
spec should include protocol design advice?

Is there an equivalent for Comet-style protocols? (That is, subprotocols 
that use HTTP as the transport?)

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 5 Aug 2010 14:35:39 +1000

> I'm not sure I follow. A subprotocol wouldn't use "opcodes".

I'm sure a subprotocol would not use opcodes as I'm pretty sure that a
subprotocol is an application concern and therefore nothing to do with
the infrastructure over which the connection passes.

However, things like multiplexing, compression, fragmentation, load
balancing, connection aggregation etc are all concerns of the
infrastructure that would be best implemented transparently to the
application.  That is why we have been talking about extensions as
distinct from sub protocols.

subprotocols are not sufficient for infrastructural extensions.  Let's
assume that somebody invented a multiplexed-ws subprotocol and that it
was widely implemented by browsers and servers.   It supports virtual
connections and the real application sub protocol is sent over the
virtual connections in a virtual handshake.

So the browser supporting this would always send a ws handshake to the
server with the multiplexed-ws subprotocol.   But what if the
application was deployed on a server that did not support that
subprotocol? The only choice if for the server to reject that
handshake (which at the moment also means closing the connection.   So
the browser then has to suffer 2 more RTT while it cracks open a
dedicated basic websocket connection for the application and sends the
real subprotocol this time.     So infrastructure that uses
subprotocols for extensions are heavily penalized.

There are also complications in terms of the subprotocol field is able
to be set by the user.  I want my server to be able to have some
confidence that the extensions that it is negotiating with are from
the browser and not from the user.  We could get into all sorts of
mess if the the application starts pretending to be a browser
extension.

So there is a desire to define an extension mechanism that is above
the base protocol, but below the application layer and sub protocols.
Many think that it would be entirely appropriate for such extensions
to be able to use op-codes without the need to go through a painful
standardization process first in order to get op-codes allocated that
will not clash.

> that use HTTP as the transport?)

HTTP is rich with extension points.  You can add new methods,
arbitrary headers, new content types, new URLs etc.
Comet protocols make use of  1 or more of these.

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 05 Aug 2010 06:38:58 -0700

> chunking on top of WebSockets).

No.

Chunking is a direct consequence of using a buffered stream, like fwrite 
and fread. It is not an application-level concept at all. If you do not 
add chunking, an implementation cannot implement an efficient basic 
stream for it.

If you do not understand why server-side HTTP use streams as the basic 
interface: CGI, PHP, Java, etc, you don't understand HTTP.  If you do 
not understand how streams work or understand why servers would choose 
that API, then you don't understand the basic concepts of chunking.

If you do not understand how fwrite works, or think fwrite or PHP's echo 
or CGI or Java's OutputStream are enormously-complicated concepts (or 
even new technology), then you might consider refraining on commenting 
on the usefulness of chunking until you learn these basic concepts.

-- Scott

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 15:07:17 +0100

> engages in store and forward instead of streaming behavior.
I think some of these are bad arguments.

> even before you know how many there are.
No, because the receiver cnnot process portions of messages - at  
least with our current API. And you can achieve this behaviour by  
sending multiple messages anyway.

> transfer" it is max (processing + network latency, bulk transfer)
This is true, but again, you can do this by sending multiple messages.

> message transfer time up to the JS layer.
Again, multiple messages.

> anticipated JS one may make use of that at no cost to the JS one.
Likewise.

> effective prioritization of flows by the sender.

This is true, in part.

The main reason I'd like to see fragmentation supported is to allow  
us to have a fixed-width length field, without having a maximal  
message length.

I would expect messages to be usually a single frame, and only larger  
when they couldn't be squeezed in or logically broken up.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 05 Aug 2010 08:26:23 -0700

Dave Cridland wrote:
> No, because the receiver cnnot process portions of messages - at least 
> with our current API. And you can achieve this behaviour by sending 
> multiple messages anyway.

Of course it can. Browser writers are fully capable of buffering the 
frames until the message is complete and then giving the complete string 
to the application.

The browser developers never complained about the sentinel framing and 
it has the exact same buffering requirement.

-- Scott

---

From: John Tamplin <[email-masked]>
Date: Thu, 5 Aug 2010 11:38:58 -0400

>>  I think some of these are bad arguments.

Imagine we have frame-based compression (I know this still up for debate,
though I think a good case has been made that we need to be able to compress
some payloads and not others).  Then the code is going to look something
like this:

empty output buffer
while there is input data
    read the next buffer full (if not already in a byte array/string)
    while the input buffer is not empty
      compress the input buffer into the output buffer
      if output buffer is full, send a non-final frame with the output
buffer and clear it
      advance the input buffer pointer
    end while
end while
send a final frame with the output buffer

If we don't have chunking, we must buffer the entire compressed output in
memory, which means allocating and reallocating an arbitrarily large buffer
rather than sending each output buffer's full of data.

No, because the receiver cnnot process portions of messages - at least with
> our current API. And you can achieve this behaviour by sending multiple
> messages anyway.

For per-frame compression, you certainly can decompress each chunk as you
receive it.  Yes, the JS API currently does not have a way to pass partial
messages, and perhaps that needs to get fixed when binary support is added
since it seems more likely you will want to be able to react to partial
messages as each part is received.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 16:49:38 +0100

On Thu Aug  5 16:26:23 2010, Scott Ferguson wrote:
> Of course it can. Browser writers are fully capable of buffering  
> the frames until the message is complete and then giving the  
> complete string to the application.
> 
> 
Right, giving no benefit to the application, which was my point.

> The browser developers never complained about the sentinel framing  
> and it has the exact same buffering requirement.

I'm not saying it doesn't. :-)

I am saying that this means this is not a good argument for chunking.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 16:51:33 +0100

On Thu Aug  5 16:38:58 2010, John Tamplin wrote:
> some payloads and not others).

Oh, sure. But that doesn't mean we need to have compression at the  
frame layer.

>   Then the code is going to look something
> like this:
> 
> empty output buffer
This, OTOH, is a good argument - and there's other similar ones.

> least with

I think the definition of a message is something that an application  
programmer can sensibly write code to react to, so I'd disagree.

Dave.

---

From: Yves Lafon <[email-masked]>
Date: Thu, 5 Aug 2010 12:06:01 -0400 (EDT)

> to use a fixed length + frags vs a variable-length-encoding which supports

variable-lenght-encoding + fragmentation allows to send data before having 
everything ready, it can reduce the perceived latency (like for HTTP chunk 
encoding), after that it is one of the trade-offs to think about.

> to the multiplexed data, it's very very easy to kill performance

During the past experiments on WebMux, that was the main conclusion, 
messing with TCP flow control by adding another flow control was not easy 
and could lead to some disasters, it would be far better to have mux at 
the TCP level, but I see that you already mentionned SCTP :)

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 05 Aug 2010 09:07:30 -0700

Dave Cridland wrote:
>> frames until the message is complete and then giving the complete 
>> string to the application.
>>

The bulk of the application is on the server.

The browser has infinite memory, infinite CPU, and a single user. That's 
why they could chose a string-based API instead of a streaming API.

The bulk of any websocket application will be on the server, just like 
the bulk of any HTTP application is on the server now. The bulk of 
application developers will be on the server, just like HTTP now. The 
bulk of the application complexity will be on the server.  Anything that 
makes the server application easier to develop is a benefit to the 
application.

Server code like PHP scripts are built around a solid streaming 
programming model, and that's been enormously successful, particularly 
for the "amateur" programmers. Throwing that successful programming 
model away with no good reason is absurd.

-- Scott

---

From: John Tamplin <[email-masked]>
Date: Thu, 5 Aug 2010 12:09:50 -0400

On Thu, Aug 5, 2010 at 11:51 AM, Dave Cridland <[email-masked]> wrote:

> Oh, sure. But that doesn't mean we need to have compression at the frame
> layer.

What would you suggest?  If the entire stream is compressed (which has been
proposed as an alternative to having per-frame compression [which itself
might still share a compression state across frames]), how do you send some
payload frames uncompressed?

> programmer can sensibly write code to react to, so I'd disagree.

If nothing else, updating a progress indicator for downloads of such large
content seems sensible.  It also seems reasonable that media could be
started playing before all of it has been received.

---

From: Jack Moffitt <[email-masked]>
Date: Thu, 5 Aug 2010 10:30:08 -0600

> The bulk of any websocket application will be on the server, just like the
> bulk of any HTTP application is on the server now. The bulk of application
> developers will be on the server, just like HTTP now. The bulk of the
> application complexity will be on the server.  Anything that makes the
> server application easier to develop is a benefit to the application.

I have to disagree with this. There are now many applications
springing up that have very minimal server requirements. Even more of
these applications have no 1st party server requirements (things like
Twitter clients, etc). We also have several frameworks (existing and
up and coming) that use web technology as the basis for application
development.

In my own projects, much of the work is on the client side. The server
exists only to store and forward data and to serve as an efficient
distribution system for the application itself.

Assuming WebSockets is successful, there will be frameworks akin to
Django and Rails that make doing the server side of WebSockets
extremely simple, and application developers can focus on the
frontend. I think that WebSockets has the potential to further this
trend of moving web browsers from dumb terminals into robust and
capable application platforms.

I'd like there to be some streaming API or at least an exposing of the
fragments all the way to JavaScript that could enable a more efficient
or different class of applications. Without these things, there *will*
be a lot of extra server side code to enable this in some cases. In
the world we live in now, we have XHR, which has all of these problems
and then some.

If anything, a chunk/streaming API would probably move even more
application logic to the client side, since more sophisticated things
would be possible.

jack.

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 09:47:41 -0700

The biggest reason that variable length encoding is annoying is that it
either adds complexity of buffer management or it adds overhead dealing with
the system.

A big advantage of length-prefixed data is that one need only read the
length, and then one can do a single alloc and read() to consume the rest of
the message exactly.

If any part of this is sentinel based (and that is what BER is,
effectively), you end up either doing read() one byte at a time, or you end
up doing a read for something you're guessing is larger than the max length.
If you guess wrong, you either have to have kept the encoded state from the
previous read, read more (involves another alloc), and only then do you get
to alloc and read the payload.

If you've guessed wrong and you read too much, then you have to save that
data, alloc another buffer, move it, and only then can you safely delete the
previous buffer.

This is a fair bit of complexity.
What is the gain?
-=R

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 17:53:03 +0100

On Thu Aug  5 17:09:50 2010, John Tamplin wrote:
> >> though I think a good case has been made that we need to be able  
> to
> >> compress
DEFLATE, for example, includes ways of transferring blocks of  
uncompressed data. Tweaking the compression strategy on the fly is  
perfectly reasonable.

> started playing before all of it has been received.

I'm basically happy with that concept. It does mean exposing the  
chunkiness to the Javascript layer to an extent, though.

Dave.

---

From: John Tamplin <[email-masked]>
Date: Thu, 5 Aug 2010 12:57:23 -0400

On Thu, Aug 5, 2010 at 12:53 PM, Dave Cridland <[email-masked]> wrote:

> DEFLATE, for example, includes ways of transferring blocks of uncompressed
> data. Tweaking the compression strategy on the fly is perfectly reasonable.
>

I don't know of any implementation that allows you to include an
uncompressed block during compression -- zlib certainly doesn't, not to
mention Java APIs to it.  Requiring implementors to write their own
compression library seems problematic.

---

From: John Tamplin <[email-masked]>
Date: Thu, 5 Aug 2010 13:00:27 -0400

> This is a fair bit of complexity.

I agree completely with this argument.

Since there was a post made by the chair saying that a fixed-length length
field and fragment support had strong support and would be declared the
consensus unless technical arguments for the variable-length length fields
are presented, it seems like those who don't like the fixed length field
need to present a case.

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 05 Aug 2010 10:20:29 -0700

Scott Ferguson wrote:
>
> If you do not understand how fwrite works, or think fwrite or PHP's 
> echo or CGI or Java's OutputStream are enormously-complicated concepts 
> (or even new technology), then you might consider refraining on 
> commenting on the usefulness of chunking until you learn these basic 
> concepts.
This last statement was inappropriate and I apologize for it.

Although understanding streaming is important to understand the 
importance of chunking, not being a streaming expert shouldn't prevent 
anyone from commenting on it.

-- Scott

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 08:51:50 +1000

> programmer can sensibly write code to react to, so I'd disagree.

I agree -  a message by definition is the smallest entity that should
be passed to an application.
An application is free to have their own idea of partial content and
to work on a stream of messages.

Frames are the mechanism that the transport layer uses to simplify and
optimize transport and should be entirely transparent to the
application, hence it should not be done in a sub protocol.   TCP has
it's own framing and applications may repeat this pattern for their
own efficiencies etc   (It's turtles all the way down ).

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 09:21:07 +1000

> This last statement was inappropriate and I apologize for it.

Scott,

I'd like to invite you to join the WTF WG.

The  WTF WG is an invitation only closed forum of server side and
network intermediary developers.  The focus of the WG is to privately
vent our spleens at all the pain and suffering that the browser
vendors have inflicted on the server-side over the years. All meetings
of the WTF WG will include recounting of all the stupid mistakes that
browser vendors have made over the years and the extraordinary lengths
that server implementers have had to go through to make theirs silly
web ideas actually work in the real world.  There will be much use of
the words *them* and *us*. All meetings close with lots of self
congratulations and a lament of : "WTF? if only they had listened to
*us*".

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 05:12:04 +0100

> sorry the text diagrams are not working... they should be in fixed width fonts.

Some trouble with Unicode, I think.  My poor terminal seems to be not
displaing non-ASCII characters usefully lately.

> what I think are stylistic (eg opcode + length  vs BERlength + opcode )

Agreed about style being unimportant.

> extended op codes are negotiated in the handshake.

Yup, that's the best part!

> proposes, but how does the client know what features the server

The client doesn't know, and it doesn't need to know.

The client is defining which opcodes it will use, so that it can use
them before the server has responded in negotiation.

The server ignores frames whose opcode it doesn't recognise.

The client does not know which messages the server will act on, and
which it will ignore, until the server has responded.  That's ok, it
just means only fancy clients would use this feature; simple ones
would wait for the server.

The details of handshake etc. aren't that important: This mechanism
applies whether it's a embedded in the TLS phase, in the HTTP-like
handshake, after handshake prior to server response, or even later.

The point is, if it's done this way, there's a basis whereby
latency-avoiding setup costs can be avoided, even if actually doing so
is reserved for the future and an extension.

If it's the server which decides the opcodes from the start, then even
future extensions won't have any way to encode latency-avoiding
initial messages from the client.

Note that if any feature is proposed by the server and needs to wait
for the client to confirm, the server should define the opcode in that
case.  Rule: "Proposer defines opcode(s)".

> support and then the server picks the final list and allocates opcodes

No, because that prevents sending any message until the server has
responded.  It adds an unnecessary round trip delay.  There is no cost
to the client choosing the values instead of the server, and it is
more versatile regarding latency avoidance in future.

This is a general principle of dynamic protocol design to avoid round
trips.  When defining something to use, the first sender/proposer
should do the defining, and just use it, not wait for agreement.
Multiplexing should do similar: Allow the first sender to create a
stream and use it immediately.  If creation fails, it will find out,
and the messages sent will just be ignored.  (The designer of the X
windowing protocol noted this as a design mistake in that one... it
makes X applications start slower)

> working - eg if I wanted to negotiate fragments, mime messages and

I think you're already got the basic idea.  Client sends negotiation
text which defines opcodes, and uses them after negotiation.

Note that length in all cases includes the frame's opcodes, parameters
etc. so that whole frames are skippable, as well as easily forwarded.

Negotiated fragments are simple enough: Client doesn't send fragment
opcodes until it's negotiated with the server.  It can only send whole
messages.  (Or client may send them prior to that, but on learning
that the server doesn't support them, it'll have to resend the
pre-sent content in whole messages only.  How much to pre-send is an
implementation detail, and would tend to vary over the years according
to typical server features.)  Fragments/chunks:

    length1 chunk-opcode final=0 message-opcode data1
    length2 chunk-opcode final=0 data2
    length3 chunk-opcode final=1 data3

Being equivalent to:

    (length1+length2+length3-adjustment) message-opcode data1+data2+data3

(The message-opcode is actually just part of the data, as far as the
chunks are concerned.  When concatenated together, it is interpreted
as a message).

Maybe final could always have no data; it's a stylistic detail.

Mime messages:

    length mime-opcode mime-headers data

But I see you're going to want split MIME frames, so like this:

    length1 chunk-opcode final=0 mime-opcode mime-headers data1
    length2 chunk-opcode final=0 data2
    length3 chunk-opcode final=1 data3

A split could occur inside the mime-headers, naturally.  Or even after
the opcode; as far as chunks are concerned, the whole body is an
opaque blob to reassamble and then interpret.

Compressed frames:

    length lzo-opcode lzo(data)

If you wanted a CRC and incremental LZO compression together, where
the CRC is of the compressed data:

    length crc32-opcode lzo-opcode lzo(data) crc32(lzo(data))

This begs a question: CRC and compress the message before splitting,
or each chunk?  Either way around is permissible in general, but
specific opcode definitions may mandate only one order is acceptable
use of them, and multiplexing and chunking specifically do that.

Compressed chunks looks a bit complicated.  At risk of putting people
off, this is a whole message compressed, and then the compressed
version is split:

    length1 chunk-opcode final=0 lzo-opcode lzo(data)1
    length2 chunk-opcode final=0 lzo(data)2
    length3 chunk-opcode final=1 lzo(data)3

This is a message split first, and then each chunk compressed:

    length1 lzo-opcode chunk-opcode final=0 lzo(data1)
    length2 lzo-opcode chunk-opcode final=0 lzo(data2)
    length3 lzo-opcode chunk-opcode final=1 lzo(data3)

CRCs could be combined with compression, and various flavours of
compression work.  Both incremental compression (of the whole stream)
and individual message or chunk compression, and switching off
compression for specific chunks.

It should be clear that this scheme accomodates both "chunk" opcodes
and "message" opcodes without having to define the difference.

Now multiplexing:

Multiplex encoding must be fast at identifying the stream id and
length for fast forwarding.  So the definition of multiplex-opcode
will say that it can only be used as an "outer" opcode (we might
permit it inside compression and/or checksums, maybe).

Multiplexing is extremely simple:

    length multiplex-opcode stream-id ....

Where .... is any other message- or chunk-opcode and its
parameters/data.  It really can't get much simpler.  That doesn't
cover any per-stream setup negotiation or flow control.  I'm sure this works:

    length multiplex-setup-opcode new-stream-id parameters....
    length multiplex-setup-response-opcode new-stream-id response...
    length multiplex-flow-opcode stream-id amount

With a good extension mechanism, an interesting question is what's the
_simplest_ base protocol that can be implemented and will still be
fully compatible with the most featureful future implementations.

Probably that is just this, and nothing more, in a simplest implementation:

    length text-utf8-opcode data

Or this:

    length basic-data-opcode data

Thus fragmentation, multiplexing, compression and MIME would all be
optional extensions, and all implementations of them would be expected
to work with any implementation including the most basic, if it's
meaningfully possible.

That way, we cover a wide spectrum of implementations from "amateur
programmer", to "person who can program but right now wants to write a
short Perl script", to "lead programmer at Google Wave".  We get good
scalability, it's future-extensible, and everything across the
spectrum will interoperate beautifully.  Like HTTP but better.(*)

(*) Free software comes without warranty etc.

>     op-code(8) + BER length + data

Yes.

> encoding from what is a sufficient extension mechanism.

I agree, separating detailed style/range from the fields is sensible.

The order of fields just isn't very important, although I could make a
case for "fixed fields", "length", "variable fields including
opcode(s)", and Dave Cridland makes a good case for stream id being in
an easily found place, for fast forwarding.

The range arguments affect things like: Whether parsing split messages
needs to be mandatory in the most basic protocol (for simple implementations).

For opcodes, we could, maybe, reserve opcode bytes >= 0x80, and put
off deciding if that starts a BER or not ;-) as long as the length
comes before it.  It'll just be an ignored frame either way.

-- Jamie

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 6 Aug 2010 12:23:06 +0800

Hi Jamie,

This sounds great to me!

There are a few interactions between the different concepts (the order that you apply multiplexing + chunking for example), but I think that you have described a perfectly workable protocol.

The best part is that it scales in complexity.

--Martin

...