Subject: [hybi] Framing proposals

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 Jul 2010 11:26:23 +1000

Below is a series of proposals to improve the framing of websocket with some
optional features.

[ To put these in context of Ian's plan, these are not for the 4 week
critical bug phase (which I think can probably only deal with handshake),
but for the 6 months add features phase.   I'm posting now so that we can
evaluate if the current handshake is sufficient to allow this type of future
extension ]

I have included multiple proposals in the one email because they are
somewhat interdependent and have a similar style.  However each proposal
should be considered on it's own merits and they do not need to be accepted
as a whole.

Several proposals share a common style of feature negotiation using
headers.  Firstly there are always reasonable simple defaults if the feature
header is not declared, so trivial implementations will just not specify a
header.  Then feature headers may be sent as single values that simple
declare the feature option supported and used.   Finally for full
negotiation, the headers may be sent as priority lists where the option used
must be the first option in priority order that appears on both the client
and server list.

For example, if the client sent:

   Feature: A, B, C

This says that the client supports A, B and C but that it would prefer to
use A over B, and B over C.  If the server then sent:

   Feature: C, B

This says that the server prefers to use C, but that the client may use B.
The result would be client would use B when sending messages and the server
would use option C when sending messages.

The intent of this negotiation method is to:

   -   have a trivial sub-case (not specified) for trivial implementations
   -   have a simple sub-case (single valued) for simple implementations
   -   The options can be negotiated without additional round trips or
   additional accept style headers. As soon as both ends have both priority
   lists, the chosen option can be determined.
   -   Support asymmetric negotiation, where the option used client to
   server may be different to what is used server to client.
   -   It restricts clients to be symmetric for their support of an option.
   Eg they cannot send gzip compressed frames unless they are able to accept
   gzip compressed frames.

*Proposal : Binary only framing*
The websocket framing be changed to only support binary framing as defined
by

          ws-frame            = frame-flags
                          frame-length
                          octet-data   ; binary data

    frame-flags         = %x00-7F

    frame-length        = unlimited-integer

    unlimited-integer   = *( %x80-FF ) %x00-7F
            ; concatenate 7 low order bits from each octet
            ; to form a binary integer

    octet-data          = *( %x00-FF )
            ; the number of octets is exactly the length determined
            ; by the frame-length

Unless otherwise specified, the binary content will be UTF-8 encoded text.

The support for this style of framing will be indicated by the websocket
version number in the handshake of >= 1.0 .  If the handshake does not
include a version, then the existing framing may be assumed for backwards
compatibility.

*Proposal : Content Type*

A handshake request and/or response may contain Websocket-Content-Type and
Websocket-Content-Charset headers to specify the content type for all
messages on a connection and the acceptable content types and charsets.

To specify that messages will be sent as utf16 encoded XML, the following
headers should be included in the handshake:

    Websocket-Content-Type: text/xml
    Websocket-Content-Charset: utf16

To support negotiation, the values of these headers may be priority ordered
lists.  A websocket end point must use the first content type and/or charset
in its priority list that appears in both the client and server lists. For
example if the client handshake sent:

    Websocket-Content-Type: text/plain,application/json
    Websocket-Content-Charset: utf16,utf8,iso-8859-1

and the server handshake contained:

    Websocket-Content-Type: application/json,text/plain
    Websocket-Content-Charset: utf8,iso-8859-1

Then the client would send messages as text/plain;charset=utf-8 and the
server would send messages as application/json;charset=utf-8.

If either header is not specified in the handshake, then it will be
equivalent to sending the corresponding:

    Websocket-Content-Type: text/plain
    Websocket-Content-Charset: utf8

Initially the mapping of content types other than "text/plain;charset=utf-8"
is undefined for the javascript websocket API.   The
Websocket-Content-Charset may be ignored if a char set is not applicable for
a specific content type.

*Proposal : Message chunking*

If the lowest order bit of the frame-flags is set, then the websocket frame
contains an incomplete message chunk.    A receiver should buffer and
concatinate message chunks until such time as a chunk with the lowest order
bit clear is received.

An example of a chunked message in three frames is:

    %x01 %x08 Goodbye_
    %x01 %x06 cruel_
    %x00 %x05 world

*Proposal : Frame compression*

If the 2nd lowest order bit of the frame-flags is set, then the websocket
frame contains compressed data using the algorithm negotiated in the
headers.

A websocket handshake may include a Websocket-Compression header which is a
coma separated list of compression algorithms in priorty order. If a
Websocket-Compression headers is specified, then only an algorithm that is
listed in both client and server headers may be used. The sender must used
the first common algorithm in their priority list.

For example if the client sent a handshake with

   Websocket-Compression: gzip,lzf,deflate

to which the server responded with

   Websocket-Compression: lzf,gzip

The compressed frames sent by the client would use gzip and compressed
frames sent by the server would use lzf. Compression dictionaries are shared
between all compressed messages sent in the same direction on a connection.

If no compression algorithm is specified, then it is an error to set the 2nd
lowest order bit in the frame type.  The frame must be discarded and the
connection may be closed.

*Proposal : Control frames*

If the highest order bit of the frame-flags is set, then the websocket frame
is a control frame that contains utf-8 text in the format of name and value
headers.

A websocket handshake message may advertise what meta data headers it will
accept in a control frame by including a Websocket-Control header in the
handshake with a list of headers that it will accept or can generate.

The following header in the handshake specifies that control frames may be
used to change the compression and content type during a connection:

   Websocket-Control: Websocket-Content-Type, Websocket-Content-Charset,
Websocket-Compression

If no Websocket-Control header is sent in the handshake, then control
messages will be ignored.

Control frames apply to all subsequent frames on a connection. If per
message meta data is desired, then a control frame must be sent prior to
every message. If a control frame is sent between frames of a chunked
message, than the values that existed when the first chunk was sent will
apply to the message.

Control frames may be compressed. [Note that we may wish to have separate
compression dictionaries for control frames, or at least load dictionaries
with some initially known values].

*Proposal : Keep-Alive*
An empty control frame may be sent to keep alive a websocket connection:

 %x80 %x00

It may also be desirable to define control frame headers that may be used to
adjust the period and to perhaps aggregate keep alive messages.

*Proposal : Final Frame*
If the second highest order bit of the frame-flags is set, then the
websocket frame is the final non-control frame that will be sent by the
endpoint.   Once sent, the endpoint should wait for a timeout period or
until a final frame is received from the other endpoint before closing the
connection.

A minimal final frame should also be a control from of zero length:

 %xC0 %x00

It may also be desirable to define headers that can be sent in the control
frame to give additional information about the reason for the close.

*Proposal : Multiplexing*
The control frame mechanism should be able to be used to implement
multiplexing by switching the "channel" of a connection.  However, I will
not expand on details until we see if the general approach is found to be
acceptable.

---

From: John Tamplin <[email-masked]>
Date: Tue, 27 Jul 2010 22:07:54 -0400

On Tue, Jul 27, 2010 at 9:26 PM, Greg Wilkins <[email-masked]> wrote:

> Below is a series of proposals to improve the framing of websocket with
> some optional features.
>
> This says that the server prefers to use C, but that the client may use
> B.   The result would be client would use B when sending messages and the
> server would use option C when sending messages.
>
>    -   It restricts clients to be symmetric for their support of an
>    option. Eg they cannot send gzip compressed frames unless they are able to
>    accept gzip compressed frames.
>
> I assume the server simply ignores any headers it doesn't understand, so
when the client sees no matching header in the handshake response it must
fall back to the default behavior (presumably defined for each feature)?

> include a version, then the existing framing may be assumed for backwards

+1 - I don't really see the value of having two styles of framing.

> Initially the mapping of content types other than
> "text/plain;charset=utf-8" is undefined for the javascript websocket API.
> The Websocket-Content-Charset may be ignored if a char set is not applicable
> for a specific content type.
>

+1 from me for content type.  I assume if not present it would default to
text/plain, charset=utf-8?

Is there really a value in specifying the character set on the wire rather
than just having client/server always translate to/from UTF8?

> An example of a chunked message in three frames is:
So the length of the overall message would not be known when receiving the
first frame, right?  That certainly makes things easier on the sender, such
as when compressing a long message and you can just output each buffer as it
fills, but it seems like it could complicate memory management on the
client.

> between all compressed messages sent in the same direction on a connection.

Rather than requiring that all compressed messages share compression
dictionaries, I would prefer to see that encoded in the compression
algorithm.  Ie, gzip vs gzip-frame etc.  The reason is that it may be an
unacceptable overhead to maintain compression state for either heavily
loaded servers or constrained clients, while they might still benefit from
compression.

> If no compression algorithm is specified, then it is an error to set the
> 2nd lowest order bit in the frame type.  The frame must be discarded and the

I would also like it to be acceptable to send an uncompressed frame even if
compression has been negotiated -- that would especially be useful if a
per-frame compression algorithm is being used, to avoid requiring
compression where compression increases the size.

>
> If the highest order bit of the frame-flags is set, then the websocket
> frame is a control frame that contains utf-8 text in the format of name and
> value headers.
>

It seems better to disallow control frames in the middle of a chunked
message.

> with some initially known values].

What about specific headers for e.g. compression algorithms, such as
resetting the compression dictionary, switching to a different predefined
dictionary, etc?  It seems like it would be useful to permit extension here,
and having to list them in the handshake message complicates things since
you are also negotiating the compression algorithm (or potentially other
uses) at the same time.

*Proposal : Keep-Alive*
> It may also be desirable to define control frame headers that may be used
> to adjust the period and to perhaps aggregate keep alive messages.
>

+1

> frame to give additional information about the reason for the close.

+1

> not expand on details until we see if the general approach is found to be

I'm not sure about using the control frame to switch channels.  It seems
like you would want to instead alter the framing -- for example, after
negotiating multiplexing the first n bytes of each frame's payload is a
channel ID.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 Jul 2010 13:56:21 +1000

John,

see inline.

On 28 July 2010 12:07, John Tamplin <[email-masked]> wrote:
>> [...]

Correct.

>> Proposal : Content Type
>> [...]

correct.

> than just having client/server always translate to/from UTF8?

I think we need to support charsets because there is no guarantee that
utf-8 will be favoured forever and there are already some languages
that are better sent in UTF16.    However, I split out charset so that
mostly it does not need to be specified and utf-8 will be assumed,
plus you don't need to say that each content  type  is supported in
each support charset.

>>
>> Proposal : Message chunking
>>
>> If the lowest order bit of the frame-flags is set, then the websocket
>> frame contains an incomplete message chunk.    A receiver should buffer and
>> [...]

By client do you mean  receiver?

I don't think a growing buffer is that complex. Also simple
implementations can still implement a fixed size buffer and can handle
a chunk overflow exactly as they would currently handle receiving a
single frame that is too large.  The good thing about length encoding
is that you know at the start of the frame if it is too big and you
don't need to go searching for a sentinel.

>>
>> Proposal : Frame compression
>>
>> [...]
> algorithm.  Ie, gzip vs gzip-frame etc.

Good idea.  We could even have algorithms with fixed known dictionaries.

> compression where compression increases the size.

That is supported already.  Just don't set the compression bit in the
frame-flags for that frame. The negotiated algorithm does not mean
that it has to be used, only that it will be used for frames with the
compression bit set.

>>
>> Proposal : Control frames
>> [..]

I thought about that also... but it is likely that if real
multiplexing solutions are developed, then control frames might need
to be sent for flow control and thus would need to be able to be sent
between the frames of a message.

Perhaps we could just say that changing message related headers mid
message is not supported?

> you are also negotiating the compression algorithm (or potentially other

I think that allowing arbitrary headers in control frames is good and
should be allowed for extensions.   The listing of supported headers
in a the handshake probably needs to list only those headers for which
support cannot be determined through other means.   For example if we
develop a set of control headers for compression, then perhaps we
would not need to list them as they usage can be inferred from the
negotiation of the compression.    However, I think erring on the side
of completeness and listing them all is probably not too onerous.

>> Proposal : Multiplexing

Obviously adding a channel ID to each frame is a possibility.  However
I suspect that will be a much more intrusive change than having a
control frame with a channel selector.  It also avoids the debate
about how big to make the channel ID (2 bytes? 4 bytes etc.)

Also, with compression I think a control frame that specifies a
channel number will probably compress down to approx 6 bytes
(including the frame - I'll do some experiments). If sent before every
frame, it is effectively that is just like adding those 6 bytes to the
frame header, but avoid the need for framing to be rewritten for
multiplexing.

Finally, it does allow more experimentation with different styles of
multiplexing, which would be harder to do if it was part of the core
framing.

cheers

---

From: Micheil Smith <[email-masked]>
Date: Wed, 28 Jul 2010 14:12:16 +1000

Here's some brief responses, I'll probably follow up with more detail, if wanted.

1) The singular framing method is certainly something that would simplify the protocol, however, 
     at the same time, something like this would be something that could be only implemented after 
     the binary support is added to ECMAScript / JavaScript. Probably in relation to: 
     https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/TypedArray-spec.html

2) The content type negotiation should be singular, for example, the client may send several formats, 
     and the server picks one to use. Or the server sends several and the client picks one (but how?). 
     The reason for not switching between spending and receiving is purely because I think it'd make it 
     significantly more difficult to implement correctly and get good consistency.

3) Message Chunking, Keep-alive and Final frame flag-byte proposals all look very good, and would be 
     useful to have in the protocol specification.

4) Multiplexing, Compression and Message Chunk sounds good, although, I'd definitely agree with it 
     being something to look at in say six months, when we have data as to how developers are using 
     WebSockets.

In other words, most of what you propose I do like the look of, and some of the proposals should definitely
simplify writing a frame parser. 

Yours,
Micheil Smith
--
BrandedCode.com

On 28/07/2010, at 11:26 AM, Greg Wilkins wrote:

> Below is a series of proposals to improve the framing of websocket with some optional features. 
> 
> [ To put these in context of Ian's plan, these are not for the 4 week critical bug phase (which I think can probably only deal with handshake), but for the 6 months add features phase.   I'm posting now so that we can evaluate if the current handshake is sufficient to allow this type of future extension ]
> 
> I have included multiple proposals in the one email because they are somewhat interdependent and have a similar style.  However each proposal should be considered on it's own merits and they do not need to be accepted as a whole.
> 
> Several proposals share a common style of feature negotiation using headers.  Firstly there are always reasonable simple defaults if the feature header is not declared, so trivial implementations will just not specify a header.  Then feature headers may be sent as single values that simple declare the feature option supported and used.   Finally for full negotiation, the headers may be sent as priority lists where the option used must be the first option in priority order that appears on both the client and server list.
> 
> This says that the client supports A, B and C but that it would prefer to use A over B, and B over C.  If the server then sent:
> 
>    Feature: C, B
> 
> This says that the server prefers to use C, but that the client may use B.   The result would be client would use B when sending messages and the server would use option C when sending messages.
> 
> 	•   have a trivial sub-case (not specified) for trivial implementations
> 	•   have a simple sub-case (single valued) for simple implementations
> 	•   The options can be negotiated without additional round trips or additional accept style headers. As soon as both ends have both priority lists, the chosen option can be determined.
> 	•   Support asymmetric negotiation, where the option used client to server may be different to what is used server to client.
> 	•   It restricts clients to be symmetric for their support of an option. Eg they cannot send gzip compressed frames unless they are able to accept gzip compressed frames.
> 
> 
> Proposal : Binary only framing
> The websocket framing be changed to only support binary framing as defined by
>    
>           ws-frame            = frame-flags
>                           frame-length
>                           octet-data   ; binary data
> 
>     frame-flags         = %x00-7F
> 
> The support for this style of framing will be indicated by the websocket version number in the handshake of >= 1.0 .  If the handshake does not include a version, then the existing framing may be assumed for backwards compatibility.
> 
> 
> 
> Proposal : Content Type
> 
> A handshake request and/or response may contain Websocket-Content-Type and Websocket-Content-Charset headers to specify the content type for all messages on a connection and the acceptable content types and charsets.
> 
> To specify that messages will be sent as utf16 encoded XML, the following headers should be included in the handshake:
> 
> To support negotiation, the values of these headers may be priority ordered lists.  A websocket end point must use the first content type and/or charset in its priority list that appears in both the client and server lists. For example if the client handshake sent: 
> 
> Then the client would send messages as text/plain;charset=utf-8 and the server would send messages as application/json;charset=utf-8.
> 
> If either header is not specified in the handshake, then it will be equivalent to sending the corresponding:
> 
> Initially the mapping of content types other than "text/plain;charset=utf-8" is undefined for the javascript websocket API.   The Websocket-Content-Charset may be ignored if a char set is not applicable for a specific content type.
> 
> 
> 
> If the lowest order bit of the frame-flags is set, then the websocket frame contains an incomplete message chunk.    A receiver should buffer and concatinate message chunks until such time as a chunk with the lowest order bit clear is received.
> 
> If the 2nd lowest order bit of the frame-flags is set, then the websocket frame contains compressed data using the algorithm negotiated in the headers.    
> 
> A websocket handshake may include a Websocket-Compression header which is a coma separated list of compression algorithms in priorty order. If a Websocket-Compression headers is specified, then only an algorithm that is listed in both client and server headers may be used. The sender must used the first common algorithm in their priority list.
> 
> The compressed frames sent by the client would use gzip and compressed frames sent by the server would use lzf. Compression dictionaries are shared between all compressed messages sent in the same direction on a connection.
> 
> If no compression algorithm is specified, then it is an error to set the 2nd lowest order bit in the frame type.  The frame must be discarded and the connection may be closed.
> 
> 
> If the highest order bit of the frame-flags is set, then the websocket frame is a control frame that contains utf-8 text in the format of name and value headers.
> 
> A websocket handshake message may advertise what meta data headers it will accept in a control frame by including a Websocket-Control header in the handshake with a list of headers that it will accept or can generate.
> 
> The following header in the handshake specifies that control frames may be used to change the compression and content type during a connection:
>  
>    Websocket-Control: Websocket-Content-Type, Websocket-Content-Charset, Websocket-Compression
> 
> If no Websocket-Control header is sent in the handshake, then control messages will be ignored.
> 
> Control frames apply to all subsequent frames on a connection. If per message meta data is desired, then a control frame must be sent prior to every message. If a control frame is sent between frames of a chunked message, than the values that existed when the first chunk was sent will apply to the message.
> 
> Control frames may be compressed. [Note that we may wish to have separate compression dictionaries for control frames, or at least load dictionaries with some initially known values].
> 
> 
> Proposal : Keep-Alive
> It may also be desirable to define control frame headers that may be used to adjust the period and to perhaps aggregate keep alive messages.
> 
> 
> Proposal : Final Frame
> If the second highest order bit of the frame-flags is set, then the websocket frame is the final non-control frame that will be sent by the endpoint.   Once sent, the endpoint should wait for a timeout period or until a final frame is received from the other endpoint before closing the connection.
> 
> It may also be desirable to define headers that can be sent in the control frame to give additional information about the reason for the close.
> 
> 
> Proposal : Multiplexing
> The control frame mechanism should be able to be used to implement multiplexing by switching the "channel" of a connection.  However, I will not expand on details until we see if the general approach is found to be acceptable.
> 
> 
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 28 Jul 2010 07:10:52 +0200

Hi Greg,

On Wed, Jul 28, 2010 at 11:26:23AM +1000, Greg Wilkins wrote:
(...)
>             ; the number of octets is exactly the length determined

That looks so much easier to understand and implement than the current
dual definition (text/binary) !

> adjust the period and to perhaps aggregate keep alive messages.

+1 for the keep-alive frame. The ping frame would be nice too. I see
two main uses :
  - chats : users leave their connection unused for random amounts of
    time, and silent timeouts will very frequently apply in the middle.
    Having a way to detect them and reconnect would help.

  - telephony / visio conference : the client-side application would
    be able to measure the round-trip time and report an indicator of
    quality to the user so that he can decide to switch protocol, link,
    encoding or quality.

> not expand on details until we see if the general approach is found to be

The visio-conference example above makes me realize that supporting
multiple channels may indeed be nice for such usages. For instance,
we could decide to separate audio and video channels so that audio
always has a correct quality and video is best effort, even if it's
only 0.5 fps.

+1 for your proposal.

Thanks,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 Jul 2010 16:05:11 +1000

>     https://cvs.khronos.org/svn/repos/registry/trunk/public/webgl/doc/spec/TypedArray-spec.html

I don't see that it needs to wait for binary support in javascript,
because the default is to assume that the binary content is
text/plain;charset=utf8.    So in essence the current API could be
used and the text messages would be sent simply length encoded instead
of sentinel encoded.   Only once javascript has binary support would
there be a need to send actual content-types headers etc.

>     significantly more difficult to implement correctly and get good consistency.

By singular do you mean symmetric?  ie that the content type sent has
to be the same as the content type received?     I can see the
argument for simplicity in symmetric content type, but then I also see
the use-case where asymmetric message types are being sent because
server and client environments are likely to be very different.
If multiplexing is supported, then I do think that symmetric content
types would be OK, but if it is not then I would not like to see
another connection opened just so the client could send XML to the
server and the server send thumbnail images to the client (for
example).

cheers

---

From: "Thomson, Martin" <[email-masked]>
Date: Wed, 28 Jul 2010 15:50:04 +0800

That ordering thing is not a good idea (my opinion):  you have a set of features supported by the client, which may include duplicated functions and conflicts.  The server picks the subset that it will use.  Ordering implies that you might use something else.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Greg Wilkins
Sent: Wednesday, 28 July 2010 3:26 AM
To: [email-masked]
Subject: [hybi] Framing proposals

   Feature: A, B, C

   Feature: C, B

 *     have a trivial sub-case (not specified) for trivial implementations
 *     have a simple sub-case (single valued) for simple implementations
 *     The options can be negotiated without additional round trips or additional accept style headers. As soon as both ends have both priority lists, the chosen option can be determined.
 *     Support asymmetric negotiation, where the option used client to server may be different to what is used server to client.
 *     It restricts clients to be symmetric for their support of an option. Eg they cannot send gzip compressed frames unless they are able to accept gzip compressed frames.

          ws-frame            = frame-flags
                          frame-length
                          octet-data   ; binary data

    frame-flags         = %x00-7F

Proposal : Content Type

    %x01 %x08 Goodbye_
    %x01 %x06 cruel_
    %x00 %x05 world

Proposal : Keep-Alive

 %x80 %x00

Proposal : Final Frame

 %xC0 %x00

Proposal : Multiplexing

---

From: KOMATSU Kensaku <[email-masked]>
Date: Wed, 28 Jul 2010 16:49:48 +0900

Dear Greg.

I like your proposals, especially to define Keep-Alive frame. To
develop stable websocket services, it is required to utilize
Keep-Alive frame and I think lack of this sort of definition makes it
hard to cooperate various websocket applications (I think it will be
ralistic use-case to share one websocket server by various web
applications).

BTW, why do you propose "frame-flags" as ranged-value? I know current
draft is written in same manner, but it is due to future-use
definitions. In this scenario, most programmer will choose '0x00' as
frame-frags, but some programmer will confused.

Sincerely

---

From: Dave Cridland <[email-masked]>
Date: Wed, 28 Jul 2010 09:31:25 +0100

> would use option C when sending messages.

I'm not a fan of asymmetric negotiations. I'm also not a fan of  
ordered negotiations, much, either. I think the general pattern  
should be:

--> Feature: ones, I, can, use
<-- Feature: ones, I, can, use, too
--> Feature: use

This does force a client-speak-first half-RTT, but my understanding  
was that we were doing something like this for handshake  
cross-prototocol security anyway.

>             ; the number of octets is exactly the length determined

I don't see the requirement to have unbounded integers, if you also  
have chunking. In which case you can blow 4 octets in network order  
on the framing. Or maybe only 2.

In any case, I feel very strongly that binary framing is the correct  
choice.

> Unless otherwise specified, the binary content will be UTF-8  

I wouldn't bother with this. It's payload. It might be nice if the  
client could tell the server what the payload format is, but really,  
this is defined by the subprotocol and out of scope for WebSockets.

> include a version, then the existing framing may be assumed for  

I'm okay with this, too. However, if we are to introduce this, then  
I'd also like there to be a (very) clear consensus that the time is  
right to effectively change the protocol upgrade token.

> *Proposal : Content Type*

As above, I don't care. I certainly don't see any reason to  
explicitly provide support for charsets beyond utf-8.

> concatinate message chunks until such time as a chunk with the  

Happy with this.

> frame contains compressed data using the algorithm negotiated in the

Take this one out and shoot it - you want to compress the entire  
communications stream. Compression framing itself contains  
information on whether the frame is compressed or not, and adds very  
low overhead.

> with some initially known values].
Changing features mid-flow has been discussed in XMPP-land before.  
Personally, it brings me out in goose bumps the size of Mons Olympus.  
I don't see the need, and would argue this introduces a significant  
complexity.

> adjust the period and to perhaps aggregate keep alive messages.
I would specify this differently:

A keep-alive frame which is empty is ignored by the peer. These MAY  
be used by either end to defeat NAT timers, etc. Typically, the  
client will send these in order to coordinate them, servers are  
RECOMMENDED to maintain a conservative timer.

A keep-alive frame which is not empty is a ping, and the contents  
should be returned in a response frame in a timely manner.

Incidentally, I'm beginning to wonder if frames should be marked at a  
high level whether they are fire-and-forget messages, or require a  
response? This kind of high-level definition of whether a peer should  
expect a response or not has paid dividends in XMPP, where <iq/>  
always has a response, and <message/> never does.

> until a final frame is received from the other endpoint before  

I like this.

> frame to give additional information about the reason for the close.

I like this too.

Again, very XMPPish, with </stream:stream> and an optional  
<stream:error/>.

> not expand on details until we see if the general approach is found  

Right - I don't quite follow the rationale for multiplexing, which is  
not to say there isn't one, but it feels like you need an extra octet  
in the frame header. I have a nagging feeling that if certain frames  
have a response requirement, this fits in neatly.

I also have a nagging feeling that you need frames to be marked with  
a zero multiplex id to indicate broadcast, etc.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: James Graham <[email-masked]>
Date: Wed, 28 Jul 2010 10:53:34 +0200

On 07/28/2010 03:26 AM, Greg Wilkins wrote:

> headers.  Firstly there are always reasonable simple defaults if the
> feature header is not declared, so trivial implementations will just not
> specify a header.  Then feature headers may be sent as single values
> that simple declare the feature option supported and used.   Finally for
> full negotiation, the headers may be sent as priority lists where the
> option used must be the first option in priority order that appears on
> both the client and server list.
>
> This says that the client supports A, B and C but that it would prefer
> to use A over B, and B over C.  If the server then sent:
>
>     Feature: C, B

This ordering seems overcomplex. I think we should pick the simplest 
workable solution.

> _*Proposal : Binary only framing*_
> include a version, then the existing framing may be assumed for
> backwards compatibility.

I am strongly opposed to anything that adds explicit versioning to the 
protocol. I am also opposed to pretending that the current frame type 
does not exist or trying to deprecate it somehow. I would have been 
happy with only length-prefixed frames from the start but I have not 
seen any strong enough arguments against the current framing mechanism 
(e.g. arguments that they are a security risk) to try and remove it.

> _*Proposal : Content Type*_
>
> A handshake request and/or response may contain Websocket-Content-Type
> and Websocket-Content-Charset headers to specify the content type for
> all messages on a connection and the acceptable content types and charsets.

I don't see the point of a content-type. The data sent over websockets 
need not be a resource of some particular well-known type and I see no 
reason to add protocol level support for the case where it is. I am 
extremely opposed to supporting more character sets than just utf8 since 
allowing multiple encodings allows large potential for mistakes, and for 
interoperability problems, but solves very few real problems.

In general making a laundry list of desired features without motivating 
use cases makes it hard to evaluate the proposals. Working from such 
unmotivated feature requests makes it more likely that we will end up 
designing in unnecessary complexity by adding features which are not 
really needed. So I would prefer to see such use cases of the form "I 
want to do X, but I cannot easily with the current protocol. This could 
be solved with Y." included with suggested feature additions. I know 
some use cases for some of the features you proposed have been mentioned 
on other threads, but it is useful to have them included with specific 
proposals so the specific set of problems that the proposal is trying to 
solve are readily available to readers.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 Jul 2010 19:45:55 +1000

On 28 July 2010 18:31, Dave Cridland <[email-masked]> wrote:
> I'm not a fan of asymmetric negotiations. I'm also not a fan of ordered
> negotiations, much, either. I think the general pattern should be:
>
> This does force a client-speak-first half-RTT, but my understanding was that
> we were doing something like this for handshake cross-prototocol security
> anyway.

If there is going to be an extra RTT, then we could have that style of
negotiation.

However, I think it would still be best to avoid an extra RTT, so the proposal
is for both sides to declare features they can use and both sides to know the
algorithm to pick an option, so no additional half RTT is needed to tell the
other side of the selected choice.

> I don't see the requirement to have unbounded integers, if you also have
> chunking. In which case you can blow 4 octets in network order on the
> framing. Or maybe only 2.

I'm not a huge fan of the unbounded integers... but then I don't feel strongly
enough about them to advocate a change.   If chunking is supported, then
I think a 2 byte length would be sufficient.

> I wouldn't bother with this. It's payload. It might be nice if the client
> could tell the server what the payload format is, but really, this is
> defined by the subprotocol and out of scope for WebSockets.

It is true that the protocol itself does not actually do anything with
content-type and/or character encoding.   However, I think there should
be a standardized way of exchanging content-type so that every
sub-protocol and/or application comes up with it's own mechanism.

> As above, I don't care. I certainly don't see any reason to explicitly
> provide support for charsets beyond utf-8.

Well if we just had a way to exchange meta data between application
ends, then perhaps the protocol need not have anything explicit about
content-type.

> communications stream. Compression framing itself contains information on
> whether the frame is compressed or not, and adds very low overhead.

This is also a reasonable proposal.  The negotiation could be much the
same.

> Personally, it brings me out in goose bumps the size of Mons Olympus. I
> don't see the need, and would argue this introduces a significant
> complexity.

I don't think that arbitrary changes of features should be supported, hence
the need to list the headers that can be changed.

Essentially this is a feature that can allows per message meta data without
adding any overhead to frames that do not need meta data.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 28 Jul 2010 19:49:05 +1000

On 28 July 2010 17:50, Thomson, Martin <[email-masked]> wrote:
> That ordering thing is not a good idea (my opinion):  you have a set of
> features supported by the client, which may include duplicated functions and
> conflicts.  The server picks the subset that it will use.  Ordering implies
> that you might use something else.

Martin,

The ordering is not there so that something else can be chosen.  The
intent is to avoid an extra RTT, so that both sides declare their
lists and both sides can apply the simple algorithm to know what the
selected feature is - without the need of an extra message to send a
choice.

Note that the server can avoid the ordering issue simply by sending a
single value in response.

cheers

---

From: "Thomson, Martin" <[email-masked]>
Date: Wed, 28 Jul 2010 18:33:53 +0800

> single value in response.

I'm not a big fan of leaving it to choice and that's my problem.  Your negotiation isn't really a negotiation, it's just an advertisement of capabilities.  Most similar mechanisms use a more definitive approach: client indicates features it is willing/able to support; server indicates subset of those that it is willing/able to support.  While there might be choices present in what the client provides, having a definitive list is valuable...and deterministic.

Otherwise, this sort of negotiation is excellent.  Establish a registry for labels and we've got the most important point for extensions to the protocol.

--Martin

---

From: "Thomson, Martin" <[email-masked]>
Date: Wed, 28 Jul 2010 18:44:42 +0800

I think that I generally agree with Dave, so I'm responding to him instead.

> <-- Feature: ones, I, can, use, too

To start with the exception: I disagree.  This doesn't buy anything over the simple request-response exchange.

+1

+1

+1
 
> > concatinate message chunks until such time as a chunk with the

+1 - call that bit the "more" bit and we're good.  Label chunking with a feature tag so that peers know not to do this.

+1
 
> I don't see the need, and would argue this introduces a significant

+1 - Renegotiate a new stream instead.
 
> RECOMMENDED to maintain a conservative timer.

That's a little vague, but generally OK.  The timers concern me - what does "conservative" mean?  Is that 1s, 10s, 60s, 2hr?
 
> should be returned in a response frame in a timely manner.

Neat, simple, I like it.
 
> always has a response, and <message/> never does.

Core uses acknowledged, unacknowledged, acknowledgement and forgetaboutit frames.  BEEP has 1-1 and 1-n frames with no unacknowledged frames.  I don't think we need that sort of complexity.  Let the application decide.

> > until a final frame is received from the other endpoint before

+1 - As with the above, timing becomes an issue.  You might avoid specifying timers, or building a mechanism for negotiating timers initially.  I would like to see someone write a proposal for this though.
 
> > *Proposal : Multiplexing*

Write a separate proposal, please.

Regards,
Martin

---

From: Dave Cridland <[email-masked]>
Date: Wed, 28 Jul 2010 11:58:15 +0100

> I think that I generally agree with Dave, so I'm responding to him  
Well, that's because I'm always right. Don't worry, you'll get used  
to it.

> To start with the exception: I disagree.  This doesn't buy anything  
> over the simple request-response exchange.
> 
> 
It buys us an explicit statement about what the agreement is. The  
notion that clients and servers will universally agree on the  
extension parameters in use by an algorithm fills me with some  
trepidation, not least because of the numbers of people who implement  
protocols without close examination of the specification.

> > In any case, I feel very strongly that binary framing is the  

Just to add to this, UTF-8 data can always be encoded in a binary  
framing mechanism, and if you want a security issue, then consider  
what happens when scripting interfaces can generate binary and pass  
it off as UTF-8 for WebSockets - false-framing could certainly be  
problematic.

> That's a little vague, but generally OK.  The timers concern me -  
> what does "conservative" mean?  Is that 1s, 10s, 60s, 2hr?
> 
> 
It varies, which is the real problem.

Currently, mobile gateways have a worst case, I'm told, around the 5  
minute mark. Home gateways are, shockingly, sometimes worse, at 4  
minutes. The IETF recommends 2 hours. I'm hoping these will continue  
the trend of increasing, though.

As for what to specify, I would stick with "conservative", in the  
hope that people will do what you just did, and ask at the time.

> Core uses acknowledged, unacknowledged, acknowledgement and  
> forgetaboutit frames.  BEEP has 1-1 and 1-n frames with no  
> unacknowledged frames.  I don't think we need that sort of  
> complexity.  Let the application decide.
> 
> 
I've found it provides some scope for intermediaries which don't  
understand the subprotocol to do useful things. It also provides some  
high-level building blocks for subprotocol developers. This is quite  
possibly not useful in this setting, however, but I thought the  
notion might be worth raising.

> > > websocket frame is the final non-control frame that will be  
> sent by
> > > the
> +1 - As with the above, timing becomes an issue.  You might avoid  
> specifying timers, or building a mechanism for negotiating timers  
> initially.  I would like to see someone write a proposal for this  
> though.
> 
> 
Actually, I don't think this is such a problem - the final frames are  
used to detect clean closure - it's useful to know when this doesn't  
occur, of course, but how long you're willing to wait is purely an  
application decision. I might add that if a final frame is received,  
the peer SHOULD immediately respond with a final frame.

Dave.

---

From: "Thomson, Martin" <[email-masked]>
Date: Wed, 28 Jul 2010 19:05:01 +0800

> protocols without close examination of the specification.

Disagree again.  The response from the server is that statement.

...
> It varies, which is the real problem.

Agreed.  I don't have a good solution, just saying is all.
 
...
Martin

---

From: James Graham <[email-masked]>
Date: Wed, 28 Jul 2010 13:17:09 +0200

On 07/28/2010 12:58 PM, Dave Cridland wrote:
> framing mechanism, and if you want a security issue, then consider what
> happens when scripting interfaces can generate binary and pass it off as
> UTF-8 for WebSockets - false-framing could certainly be problematic.

I don't quite follow. Presumably a scripting interface would be designed 
to send binary data in binary frames and text data in text frames. I 
don't think the API would allow you to send binary data in text frames, 
so I'm not quite sure what the problem is. I'm not quite sure what the 
exact ramifications would be if you did do this; presumably the only 
effect would be that the client thought it had sent 1 message but the 
server thought it had received 2. I suppose that could be a problem for 
some application designs.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 28 Jul 2010 12:49:46 +0100

On Wed Jul 28 12:17:09 2010, James Graham wrote:
>> what
> I don't quite follow. Presumably a scripting interface would be  
> designed to send binary data in binary frames and text data in text  
> frames. I don't think the API would allow you to send binary data  
> in text frames, so I'm not quite sure what the problem is. 

The issue is where the API is folled into thinking the binary data is  
text data.

> I'm not quite sure what the exact ramifications would be if you did  
> do this; presumably the only effect would be that the client  
> thought it had sent 1 message but the server thought it had  
> received 2. I suppose that could be a problem for some application  
> designs.

Could be. I'm thinking in some case where the client is asked to send  
some string, in which is embedded a frame which does something  
unexpected (and unwanted). I vaguely imagine this to be possible for  
a third-party attacker, who might cause an additional frame to be  
send doing something Naughty.

On the other hand, if there's binary-only framing, then the attacker  
in this case will only cause a funny string to be sent.

It's relatively easy to protect against - the fundamental trick is to  
ensure that framing is solid, and you can do this either by having  
flawless UTF-8 handling at both ends, or by using an octet-counted  
frame.

My inclination is to think people are more likely to be able to count  
than be able to handle UTF-8 flawlessly, even now.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Wed, 28 Jul 2010 13:11:18 +0100

On Wed Jul 28 12:49:46 2010, Dave Cridland wrote:
> It's relatively easy to protect against - the fundamental trick is  
> to  ensure that framing is solid, and you can do this either by  
> having  flawless UTF-8 handling at both ends, or by using an  
> octet-counted  frame.

As some input on this, RFC 3117 §3.1 has some discussion on framing  
techniques.

Neither the current Websocket method, not the XMPP method, is present  
here, but both are close in spirit to octet-stuffing - they all use  
marker data which is prohibited from occurring [unencoded] in the  
payload data.

Dave.

---

From: Arman Djusupov <[email-masked]>
Date: Wed, 28 Jul 2010 15:51:17 +0300

> with some optional features.

Excellent initiative Greg! 

> feature header is not declared, so trivial implementations will just 
> not specify a header.  Then feature headers may be sent as single values that simple
> negotiation, the headers may be sent as priority lists where the 

+1

+1

I think that this asymmetric negotiation mechanism is the best that we can have within a single handshake.

> The websocket framing be changed to only support binary framing as 
> defined by <snip> The support for this style of framing will be 
> indicated by the websocket version number in the handshake of >= 1.0 .  
> If the handshake does not include a version, then the existing framing 
> may be assumed for backwards compatibility.

+1

In the current state of things the application does not know what type of content is being sent in the binary frame. If content negotiation is introduced then the content type of each frame will be known and so differentiating between frames by using the frame header byte will become unnecessary. This will save 1 bit in the frame header byte.

+1

This should work fine. Content types should be the same as for HTTP so that we have the benefit of supporting both IANA-registered types and user-defined ones too.

> concatinate message chunks until such time as a chunk with the lowest 
> order bit clear is received.

+1

This is already implemented and tested in our own WebSocket implementation. Using this type of framing we support full duplex transfer of any type of stream.

> If the 2nd lowest order bit of the frame-flags is set, then the 
> websocket frame contains compressed data using the algorithm 
> negotiated in the headers.
> 
> A websocket handshake may include a Websocket-Compression header which 
> is a coma separated list of compression algorithms in priorty order.
> If a Websocket-Compression headers is specified, then only an 
> algorithm that is listed in both client and server headers may be 
> used. The sender must used the first common algorithm in their priority list.

Although I like the approach you suggest, why not compress the entire transport stream?

We have added a compression layer between the WebSocket protocol and the OS socket, which we activate right after the handshake is complete. This approach is simpler and more efficient than maintaining the compression state between frame contents (while not compressing frame headers). 

Stream compression can be switched ON and OFF using the control frames, if supported.

> frame is a control frame that contains utf-8 text in the format of 
> name and value headers.

Although control frames would add some flexibility to the protocol, they would also add extra complexity since some sort of service would be required on both sides in order to handle them. Currently, once WebSocket completes the handshake it does nothing more than plain message framing over TCP/IP. This really simplifies the protocol implementation and improves its performance.

I believe that support of control frames should be optional, e.g. if either of the sides does not want to receive/process/skip control messages it should be able to prohibit the remote side from sending them. Even skipping the control messages introduces a fair amount of complexity. Having the option to completely prohibit control messages would enable communication with simple implementations that don't support them.
 
Various issues related to API limitations or requirements might prohibit implementations from on-demand switching of encodings/compression or supporting control frame handling. For this reason, an implementation should not be required to negotiate everything more than once or parse and skip frames that it does not support. I would rather put it as:

"If no Websocket-Control header is sent in the handshake, then control messages are not permitted."

So those developers who wish to implement a feature-rich WebSocket based server which offers all sorts of capabilities to all connected clients would be free to do so. While those who are required to deliver resource-constrained implementation for compact devices would not be obliged to deal with control frames.

> multiplexing by switching the "channel" of a connection.  However, I 
> will not expand on details until we see if the general approach is 
> found to be acceptable.

IMHO this specification doesn't need to define how multiplexing is to be performed. The support of control frames would be sufficient to enable a sub-protocol to implement multiplexing. As it has been discussed before, if multiplexing goes into the spec than the spec will have to define many more types of control messages so that the flow control mechanism can be standardized. It is rather preferable for this specification to just enable multiplexing through the support of control messages, while a standard way to implement multiplexing can be defined in another specification.

With best regards,
Arman

---

From: James Graham <[email-masked]>
Date: Wed, 28 Jul 2010 17:45:20 +0200

On 07/28/2010 01:49 PM, Dave Cridland wrote:
>> frames. I don't think the API would allow you to send binary data in
>> text frames, so I'm not quite sure what the problem is.
>

I don't see how that would happen. Presumably the API will just have a 
single send() method that will accept either a string or a ByteArray (or 
whatever the javascript binary type is called). The right frame type 
will then be selected based on the type of the argument.

---

From: John Tamplin <[email-masked]>
Date: Wed, 28 Jul 2010 12:01:05 -0400

On Wed, Jul 28, 2010 at 11:45 AM, James Graham <[email-masked]> wrote:

> whatever the javascript binary type is called). The right frame type will
> then be selected based on the type of the argument.
>

I think the issue would be that you could supply binary data which would be
interpreted on the server as UTF8 text, possibly exploiting a bug in UTF8
decoding.

I think it would be really ugly to have the browser restrict what
Content-type could be sent to avoid that.  I imagine it would look something
like:

send(String) - encode in UTF8, set Content-Type to text/plain
send(ByteArray data, String type) - if type isn't supplied, browser sets
default based on the data or perhaps just application/octet-stream

If the JS can supply arbitrary binary data and arbitrary content type, it
can force the server to decode it as UTF8 text, and it isn't clear how you
can avoid that by restricting the Content-Type on the binary transmission
(do you block text/x-my-format?).  Maybe you can allow a content-subtype for
send(String, String), and it is always "text/" + subtype (so the browser
always controls the UTF8 encoding), and never allow the binary one to use
"text/*".

However, this general vulnerability exists with any binary type -- imagine
crafting data as image/jpeg which exploits a bug in certain server
implementations when the JPEG data is decoded (assuming it is, such as an
image compositing service running on the server).

I am inclined to say because that issue is common to any handling of
user-supplied binary data, the details of dealing with it can be deferred
until binary formats are supported in the JS API.

---

From: Scott Ferguson <[email-masked]>
Date: Wed, 28 Jul 2010 09:11:45 -0700

> with some optional features.
Excellent work, Greg. Comments below.
> defined by
> The support for this style of framing will be indicated by the 
> websocket version number in the handshake of >= 1.0 .  If the 
> handshake does not include a version, then the existing framing may be 
> assumed for backwards compatibility.
+1 for binary-only framing

+1 for the Upgrade: WebSocket/2.0 or Upgrade: BinaryWebSocket/1.0 
handshake model.

-1 for treating chunking as a separate proposal, because binary is not 
viable without chunking. (And chunking can't be an optional upgrade.  
For example, see the problems with HTTP clients not implementing 
chunking and the extra complexity of supporting both content-length and 
chunked, and of explaining to users that their HTTP/1.1 clients really 
do need to implement chunked.)

-1 for the variable-size/unlimited length. It's more complicated, which 
is particularly problematic because of the additional QA tests and 
overflow cases it introduces (and introducing 32/64 bit issues). A fixed 
size 2 or 4 byte header eliminates several potential bugs.

-1 for the fallback to the text framing inside the protocol. The spec 
will be cleaner, easier to implement and understand, and more testable 
if it has one clear purpose. If a client wants to fallback to text, it 
can retry with the Upgrade: WebSocket/1.0. If an application needs 
binary, it shouldn't be forced to implement the text protocol as part of 
the spec.

>
> _*Proposal : Message chunking*_
>
> frame contains an incomplete message chunk.    A receiver should 
> buffer and concatinate message chunks until such time as a chunk with 
> the lowest order bit clear is received.

-1 on "should buffer". There's no protocol requirement for the receiver 
to buffer the response. Since some/many receivers can handle the stream 
as it's received without the need for buffering, the spec should remain 
silent on the issue. That buffering behavior is entirely up to the 
receiver's API.

> _*Proposal : Keep-Alive*_
> _*Proposal : Final Frame*_
> websocket frame is the final non-control frame that will be sent by 
> the endpoint.   Once sent, the endpoint should wait for a timeout 
> period or until a final frame is received from the other endpoint 
> before closing the connection.
>
The control frames really do need a control opcode, instead of 
"second-highest order bit", because a ping-style keepalive needs 
separate request and response opcodes, a unidirectional keepalive needs 
a distinct opcode from ping, channel switching needs an opcode, flow 
control, "close" should just be another opcode, etc.

There are probably less than 20 opcodes we'll ever need, but we do need 
at least 3-5 to handle basic keepalives and closes and about 5-7 more 
for the multiplexing/flow control. So we should have a minimum of 6-bits 
worth (64) of opcodes to give plenty of breathing room (and 0x3f should 
probably be reserved, just in case.)

-- Scott
>
> It may also be desirable to define headers that can be sent in the 
> control frame to give additional information about the reason for the 
> close.
>
>
> _*Proposal : Multiplexing*_
> ------------------------------------------------------------------------
>

---

From: Simone Bordet <[email-masked]>
Date: Wed, 28 Jul 2010 19:20:52 +0200

Hi,

On Wed, Jul 28, 2010 at 03:26, Greg Wilkins <[email-masked]> wrote:
> every message.

I am trying to imagine a client that send both text and binary (and I
assume that by "connection" above you mean the something that maps 1-1
to the TCP connection).
In this case an implementation that wants to share/reuse connections
to the same domain must carefully key the connection with the
content-type & charset & compression of the last control frame, to
avoid that another client connecting to the same domain could mix-in
messages encoded differently, right ?

So a client doing:

ws.sendBinary(...);
ws.sendText("foo");

will send a control frame, then the binary data, then a control frame,
then the text data, right ?

I think there is value in the simplicity of stateless connections and
I like more state to be in a "channel" (as in BWTP) or "stream" (as in
SPDY), but adding these will make the protocol (and API) more complex,
etc., yet they always pop up :)

I like the idea of control frames and the power they give, but I have
not figured out yet how these can be added without carrying with them
the notion of "channel" or that of a message that carries both
metadata and data.

> If a control frame is sent between frames of a chunked

Why you want to support a control frame in the middle of a chunked message ?

Simon
-- 
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

---

From: Dave Cridland <[email-masked]>
Date: Wed, 28 Jul 2010 21:02:41 +0100

On Wed Jul 28 16:45:20 2010, James Graham wrote:
>>> in
> I don't see how that would happen. Presumably the API will just  
> have a single send() method that will accept either a string or a  
> ByteArray (or whatever the javascript binary type is called). The  
> right frame type will then be selected based on the type of the  
> argument.

Hopefully, yes. Hopefully, there is no way of passing a string object  
that happens to have bad UTF-8 in it. But my concern is that if the  
implementation is subtly flawed, then this attack becomes viable. I'm  
not saying this is a show-stopper issue, by the way, but there are  
cases *now* of people taking advantage of broken octet-stuffed  
framing in SMTP by doing:

.

(I wonder how many people will fail to get this?)

Many MTAs also have to deal with the *other* delimited framing common  
in email, which is lines beginning:

From

(The above should appear as "From" - if it appears as ">From", your  
mail system is warped...)

There's another side to this, too - we're still at a point where we  
could change this with a flag-day, so I think proponents of  
text-based delimited framing need to provide solid arguments for  
keeping it.

Dave.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 10:36:17 +1000

> etc., yet they always pop up :)

I'm tending to be convinced... and thus am thinking that a one-size-fits all
control frame of headers that effect all subsequent message is probably
not correct.

I like Scott's idea of an opcode in the control frame.  Then if we decide that
we want meta-data per frame, we can add an opcode that contains meta
data that is related to just the next message and does not change the state
of the connection.

Alternately, we can have multi-plexing and lots of channels, each with fixed
message properties.

> Why you want to support a control frame in the middle of a chunked message ?

because eventually they will be used for flow control in a multiplexed solution.
Just because I've just sent chunk 1 of 50 of a message should not mean
that I cannot
send a control message to the other end telling it that it should stop
sending on a
particular channel.

I think the solution is to make the control frames relate to
connection management
and not message state management.

cheers

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 13:45:39 +0900

I think there is some confusion about length-based and sentinel framing 
on the one hand and UTF-8 vs. binary content on the other hand. More below.

>> Unless otherwise specified, the binary content will be UTF-8 encoded

If this is a proposal for changing UTF-8 frames to use length-based 
framing, then that should clearly be said.

If this is a proposal for putting UTF-8 textual frames and binary frames 
into one bucket, without giving them different frame types, that should 
clearly be said.

If it's both of the above, then these two aspects should be very clearly 
separated.

I'm indifferent to the first aspect, but I'm very clearly against the 
second aspect.

>> websocket version number in the handshake of >= 1.0 . If the handshake
>> does not include a version, then the existing framing may be assumed
>> for backwards compatibility.

Please don't call it binary-only framing. Please call it length-based 
only framing.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 13:51:34 +0900

> reason to add protocol level support for the case where it is.

Yes. If I were to design some 'subprotocol', I'd probably use 
XMLHttpRequest or some such for uploading or downloading bulky resources 
such as images and the like. As far as the TCP metaphor is concerned, 
TCP also doesn't support content-types.

In addition, it might be good if somebody could come up with some 
examples where content-types are independent of the subprotocol used and 
therefore need to be negotiated separately. At least on the public Web, 
content negotiation is only used marginally, in that most browsers only 
send incomplete descriptions of their capabilities, and servers don't do 
too much with them on average. To me, that suggests that there may not 
be much left to be gained from content negotiation once the subprotocol 
is known.

> I am

Very, very well said.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 14:02:33 +0900

On 2010/07/28 12:56, Greg Wilkins wrote:

> On 28 July 2010 12:07, John Tamplin<[email-masked]>  wrote:
>> On Tue, Jul 27, 2010 at 9:26 PM, Greg Wilkins<[email-masked]>  wrote:

> utf-8 will be favoured forever and

Of course there is no such guarantee. But there is a clear trend on the 
Web and the Internet towards UTF-8. And the IETF overall favors UTF-8. A 
single encoding makes things much, much simpler.

Just to give you one example, your current proposal does content-type 
negotiation, but not character encoding negotiaton. How are you sure any 
arbitrary character encoding is supported at the other end? Fixing 
things to UTF-8 solves this.

> there are already some languages that are better sent in UTF16.

If needed, compression can take care of this. UTF-16 may be 1.5 times as 
efficient in some very specific cases (Indic languages, Chinese, 
Japanese, Korean,..., without any single ASCII character). The general 
experience is that there are always some ASCII characters, and in that 
case, UTF-8 quickly catches up where UTF-16 would have been better. So 
it's really not worth the trouble.

Regards,   Martin.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 15:31:45 +1000

On 29 July 2010 14:45, "Martin J. Dürst" <[email-masked]> wrote:

> I think there is some confusion about length-based and sentinel framing on
> the one hand and UTF-8 vs. binary content on the other hand. More below.
>
> framing, then that should clearly be said
>

It is.

> into one bucket, without giving them different frame types, that should

It is.

> If it's both of the above, then these two aspects should be very clearly

I think they have been.   There is a proposal for the framing and a separate
proposal for the content-type.

> I'm indifferent to the first aspect, but I'm very clearly against the

What are your reasons for being against 1 framing for both binary and text
messages?

Essentially the current protocol supports two content types:
 text/plain;charset=utf-8 and application/data.   The fact that these two
content types are carried by different framing mechanism is immaterial.
 Either frame type can actually be used to carry both content types.

My proposal is that instead of just supporting application/data, we
standardize the way the content-type of the binary messages is declared.
 If no declaration is made, then the content type is
text/plain;charset=utf-8.  Once you have this, there is no need to use
different framings to indicate if the message is text or binary.

Note that in my updated proposal, I have changed from content-type
negotiation (which is very
difficult) to content-type declaration (which is trivial).

regards

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 28 Jul 2010 23:41:04 -0700

On Jul 28, 2010, at 10:31 PM, Greg Wilkins wrote:

> 
> 
> What are your reasons for being against 1 framing for both binary and text messages?

It's not unreasonable to use the same basic framing scheme, but text and binary messages should be clearly distinguishable.

> 
> Essentially the current protocol supports two content types:  text/plain;charset=utf-8 and application/data.   The fact that these two content types are carried by different framing mechanism is immaterial.  Either frame type can actually be used to carry both content types.
> 
> My proposal is that instead of just supporting application/data, we standardize the way the content-type of the binary messages is declared.    If no declaration is made, then the content type is text/plain;charset=utf-8.  Once you have this, there is no need to use different framings to indicate if the message is text or binary.
> 
> Note that in my updated proposal, I have changed from content-type negotiation (which is very

1) I think arbitrary content types are a separate concern from text vs. binary:

1.a) We can't assume the text vs. binary distinction is handled via a content-type. The browser-based client API will deserialize text and binary frames into different JavaScript types. If it encounters a type *other* than text/plain;charset=utf-8 or application/data, how is it supposed to tell whether to deserialize the frame as text or binary? You might think this can be inferred from the top-level MIME type, but the "application" and "image" namespaces both contain a bunch of text types and a bunch of binary types. For the browser-hosted client to deserialize messages correctly, it needs a text vs binary indicator even if there is also a content-type. Thus, content-types are not sufficient.

1.b) Other than the need to distinguish text vs. data to deserialize (and serialize) correctly, it's not really necessary for the browser-hosted client API, or for that matter a server-side framework, to know the content-type. Neither the client nor the server will do anything different with text/plain vs. text/html. It can only possibly make a difference at the application level. And even at the application level, the subprotocol may be such that the type is obvious from context. This says to me that content-type is a concern best left to the application level, either by including it in-band in messages; or by using a general-purpose metadata mechanism where application-level code can specify a type or examine it, but the client API implementation and server frameworks can ignore it, and work fine if it is absent.

2) I agree with all those who have argued against supporting arbitrary encodings. Supporting many different charset encodings creates a great deal of complexity and potential for interoperability failures for very little benefit. Let's not add more complexity just because we can. Let's continue to move towards utf-8 as the standard for text sent over the wire. To me this argues against supporting the charset parameter even if we do have some notion of content type.

And finally, if we're supposed to fix the critical bugs in the next four weeks or so, it might be good to concentrate a little on critical bugs instead of proposals for later in the dev cycle. Do you think any of these ideas impact the handshake in a way that might need incompatible changes? Are they otherwise critical?

Regards,
Maciej

---

From: "Thomson, Martin" <[email-masked]>
Date: Thu, 29 Jul 2010 14:48:34 +0800

> I think there is value in the simplicity of stateless connections and

Or rather, you would like to minimize the amount of state involved with a stream in the interests of reducing the complexity of the FSM that you use (logically or otherwise) in deciding how to process requests.

> I like the idea of control frames and the power they give, but I have

I do not like the control frame idea as much.  The costs in complexity do not seem to be amply justified.

> Why you want to support a control frame in the middle of a chunked

This would be a very strong reason not to allow mid-stream state changes.  Do you continue to process the chunks with the old state?  Does this mean that we have per-message state?

Control frames seem more and more like a bad idea from a complexity standpoint.

--Martin

---

From: "Thomson, Martin" <[email-masked]>
Date: Thu, 29 Jul 2010 15:05:59 +0800

> It's not unreasonable to use the same basic framing scheme, but text
> and binary messages should be clearly distinguishable.

While this seems like a reasonable statement, it needs to be justified.  Why?

--Martin

---

From: "Thomson, Martin" <[email-masked]>
Date: Thu, 29 Jul 2010 15:06:03 +0800

HI Greg,

Please don’t make the mistake of mixing frame types.  I don’t think you actually want to do this right, but when you make statements like this, you risk some very bad things happening.  Misunderstanding is the first, and if you are confusing smart folks like Martin, then you have no idea how bad it gets.

My revised proposal would look something like this:

  Length only framing
  Op-code for a raw binary frame (the default message type).
  Additional op-code identifying a frame with a header.  This could be MIME.

If you want to send text, then it is possible to send that in the binary frame.  Each peer might have a priori knowledge that lets them know that the content is UTF-8 encoded text.  That’s OK - intermediaries don’t need this information.

If this is a proposal for putting UTF-8 textual frames and binary frames into one bucket, without giving them different frame types, that should clearly be said.

It is.

---

From: Roberto Peon <[email-masked]>
Date: Wed, 28 Jul 2010 18:22:06 -0700

> data that is related to just the next message and does not change the state

SPDY has the SETTINGS frame, which affects all the streams (it includes, for
reference, TCP-level settings for the client to persist and send to the
server, a maximum number of concurrent streams, etc.)

> send a control message to the other end telling it that it should stop

Not to mention validation of streaming data.
If you want signed, cryptographic hashes of data sent before all of the
data, you induce delay.
If you send a single hash for the entire data, you either induce a
substantial delay (all data must be downloaded before it is verified), or,
you must consume unverified data.
The only thing that makes sense is thus to send hashes in a stream
associated with the content.
One could define a separate file format for this, but why? This is a general
problem with a relatively easy solution for all content and file types.

In any case, yet another reason to support metadata at arbitrary points in
the stream.
-=R

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 17:19:34 +1000

On 29 July 2010 16:41, Maciej Stachowiak <[email-masked]> wrote:

> It's not unreasonable to use the same basic framing scheme, but text and
> binary messages should be clearly distinguishable.
>

Maciej,

currently what I'm proposing is that if no Content-Type is specified, then
the message is a text frame encoded with utf-8.     If a Content-Type is
specified, then the message is a binary message containing whatever type is
specified.   This could be done with a boolean rather than a Content-Type,
but as this is a once per connection header, I fail to see how there is any
harm in being more specific.

> 1.a) We can't assume the text vs. binary distinction is handled via a
> content-type. The browser-based client API will deserialize text and binary
> frames into different JavaScript types. If it encounters a type *other* than
> text/plain;charset=utf-8 or application/data, how is it supposed to tell
> whether to deserialize the frame as text or binary?
>

If there is no Content-Type, then it is text.
If there is a Content-Type and it is text/*   then it is text.
Otherwise it is binary.

But I think this is not really a discussion about content-type.  I think you
are really concerned about what messages will be delivered to the text based
API in the javascript.   I think that is a separate concern that how the
protocol transports them.

You could for example say that only frames without an explicit Content-Type
will be delivered to the javascript text based API.  Anything with a
content-type (including text/* types) will be delivered via a yet to be
defined "binary" API (which is might be a MIME API?) . Or you could say that
any text/* content will also be delivered to the text javascript API.
Eitherway, that is more of an issue for the HTML5 folks to discuss and less
of a protocol issue.

1.b) Other than the need to distinguish text vs. data to deserialize (and
> serialize) correctly, it's not really necessary for the browser-hosted
> client API, or for that matter a server-side framework, to know the
> content-type. Neither the client nor the server will do anything different
> with text/plain vs. text/html.
>

That is not necessarily true.  The client might parse text/xml into DOM or
handle image/jpeg by creating some holder object.   On the server side, I
can imaging using something like the java activation framework to
de-serialize arbitrary types before invoking an application.

2) I agree with all those who have argued against supporting arbitrary
> encodings.

I do not see how we can both support binary frames and not support arbitrary
encodings?
If we support binary, then other encodings may be sent.

So I don't see this as a protocol issue.  It is a Javascript API issue, and
if the JS API wants to only handle utf-8, then I think that is a reasonable
choice.   But that should not prevent those that wish to use a future binary
API from sending other encodings and declaring what encoding they are using.

> And finally, if we're supposed to fix the critical bugs in the next four
> weeks or so, it might be good to concentrate a little on critical bugs
> instead of proposals for later in the dev cycle. Do you think any of these
> ideas impact the handshake in a way that might need incompatible changes?
> Are they otherwise critical?
>

Until we have sketched out some future plans, we have no real idea if the
current handshake is suitable or not.   So I'm raising these issues now so
that we may consider the road map.    If you don't think it is important,
then there is no need to give any feedback until after the critical bugs are
fixed.

Beside, I have given plenty of feedback on the critical errors I see in the
handshake and have made little progress convincing the editor of them, nor
has there been any counter proposals to address the concerns that have been
raised.  So I don't see it as very productive to keep banging on the same
issues again and again.    If others share my concerns, then they can raise
them.  If not, then we'll go with what we've got.

---

From: James Graham <[email-masked]>
Date: Thu, 29 Jul 2010 09:27:01 +0200 (CEST)

> While this seems like a reasonable statement, it needs to be justified.  Why?

On the client side, text and binary data needs to be deserialized to 
different object types; String and ByteArray, respectively. That means 
they must be unambiguously distinguishable. Having different frame types 
is a simple, effective, mechanism for doing this. Making people specify 
content-types is neither sufficient to meet this requirement, nor 
necessary for many applications. Therefore it is not a replacement for 
having multiple frame types.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 17:28:38 +1000

On 29 July 2010 17:06, Thomson, Martin <[email-masked]> wrote:

>  HI Greg,
>
>
>
> Please don’t make the mistake of mixing frame types.  I don’t think you
> actually want to do this right, but when you make statements like this, you
> risk some very bad things happening.  Misunderstanding is the first, and if
> you are confusing smart folks like Martin, then you have no idea how bad it
> gets.
>

Have a look at my take II proposals - I think it is going in the direction
you advocate.

> My revised proposal would look something like this:
This I have proposed.

>   Op-code for a raw binary frame (the default message type).
This actually goes further than I propose.  It would allow text and binary
messages to be mixed in a stream.  I'm currently proposing that a
WebSocket-Content-Type is fixed for a connection, so that once it is a text
connection it is always a text connection.     There is no need for a frame
by frame indication that it is text or binary (unless extra meta data is
supplied).

>   Additional op-code identifying a frame with a header.  This could be

In my take II proposal, this is exactly what the meta data frame provides.
It gives MIME headers for the following message.

I consider that even if the mime-type is text, this is essentially a binary
message and need not be handled in the same way as the text frames sent to
the current javascript API.   Potentially text/* content types could be sent
to that API, but I do not think they have to be.

If you want to send text, then it is possible to send that in the binary
> frame.  Each peer might have a priori knowledge that lets them know that the
> content is UTF-8 encoded text.  That’s OK - intermediaries don’t need this
> information.
>

But surely there is no harm in declaring the content-type?  If only to
support debugging or validation that you have connected to the right service
endpoint.

Note that I'm not proposing content negotiation - only content declaration.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 17:33:25 +1000

On 29 July 2010 17:27, James Graham <[email-masked]> wrote:

> different object types; String and ByteArray, respectively. That means they
> must be unambiguously distinguishable. Having different frame types is a
> simple, effective, mechanism for doing this. Making people specify
> content-types is neither sufficient to meet this requirement, nor necessary
> for many applications. Therefore it is not a replacement for having multiple
> frame types.
>

James,

How is specifying a content-type not sufficient to unambiguously distinguish
text from binary data?

How is having two entirely separate framing mechanism simpler or more
effective that having a simple header in the handshake declaring content
type?

If a content-type is not necessary for an application that it can simply not
declare if it is using utf-8 text or it can declare application/data if it
does not wish to be specific.

There are also a whole bunch of concerns regarding sentinel framing with
regards to it's vulnerability to injection and the difficulty of handling
large frames.

regards

---

From: "Thomson, Martin" <[email-masked]>
Date: Thu, 29 Jul 2010 15:50:59 +0800

That's true, but is this not possible within the application, as opposed to the library?

If the client/server library - or an intermediary - has any reason to examine the payload, or know of its nature, then you have justification.

Otherwise, you can have the user provide this information:

WebSocket.send(data : ByteArray)
WebSocket.sendString(str : String)

On the return, it might be nice to do something like:

WebSocket.recv() : ByteArray
WebSocket.recvString() : String // throws UtfExecption

However, you probably can't do this with an upcall.  In that case, you probably have to just pass the ByteArray to the user.  If they know that it's a string, then they can call UTF8.decode.

--Martin

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> James Graham
> Sent: Thursday, 29 July 2010 9:27 AM
> To: Thomson, Martin
> Cc: [email-masked]
> Subject: Re: [hybi] Framing proposals
> 
> 
>

---

From: James Graham <[email-masked]>
Date: Thu, 29 Jul 2010 09:51:25 +0200 (CEST)

On Thu, 29 Jul 2010, Greg Wilkins wrote:

> On the client side, text and binary data needs to be deserialized to different object types; String and ByteArray,
> respectively. That means they must be unambiguously distinguishable. Having different frame types is a simple, effective,
> mechanism for doing this. Making people specify content-types is neither sufficient to meet this requirement, nor necessary
> for many applications. Therefore it is not a replacement for having multiple frame types.
> 
> 
> James,
> 
> How is specifying a content-type not sufficient to unambiguously distinguish text from binary data?

Well yes, if you use presence vs absence of some mostly-unrelated thing as 
a flag to indicate text vs binary then that will of course provide an 
unambiguous determination. I don't think it provides a protocol that is 
easy to work with though e.g I think recieving a message labelled as 
text/plain as a binary blob would be very confusing to authors.

> How is having two entirely separate framing mechanism simpler or more effective that having a simple header in the handshake
> declaring content type?

I would advocate two different frame types even in the case where both 
were length-marked. It is simple and meets all the requirements of the 
client side API. By contrast I have no idea why we would want the ability 
to declare connection-level content types (this is why I would prefer we 
worked from use cases; apologies if you provided some and I missed them, 
in that case a pointer would be appreciated). It seems to me that for many 
applications there is no sense in declaring a connection-level content 
type as the connection may be used to transmit multiple types of content. 
Even if only a single type is being used I see no point in making it part 
of the protocol because the application layer can just transmit whatever 
custom metadata it likes in the first message.

> If a content-type is not necessary for an application that it can simply not declare if it is using utf-8 text or it can declare
> application/data if it does not wish to be specific.

This seems like unneeded complexity for the common case.

> There are also a whole bunch of concerns regarding sentinel framing with regards to it's vulnerability to injection and the
> difficulty of handling large frames.

That seems orthogonal to this issue.

---

From: John Tamplin <[email-masked]>
Date: Thu, 29 Jul 2010 04:05:26 -0400

> Are they otherwise critical?

It seems like the basic framing format is something that needs to be nailed
down before version 1.0.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 18:57:46 +1000

On 29 July 2010 17:51, James Graham <[email-masked]> wrote:

> unambiguous determination.

I don't think you can characterize content-type as something
mostly-unrelated to the binary or text status of a message.     It is
certainly more related than if you use length or sentinels to frame the
data.

> I don't think it provides a protocol that is easy to work with though e.g I
> think recieving a message labelled as text/plain as a binary blob would be
> very confusing to authors.
>

How would unlabelled text/plain being delivered as a binary blob be any less
confusing than a labelled blob?  It is not the labelling that makes that
situation confusing.

Also I have no complaint if the javascipt API mapping was modified to say
that binary frames with text/* content types may be delivered via the text
API.  I only suggested that in response to those that said they did not want
to handle non utf-8 text in the js API.    Eitherway, that is an issue for
the HTML 5 folks and not a protocol issue.

I would advocate two different frame types even in the case where both were
> length-marked. It is simple and meets all the requirements of the client
> side API.

Duplication of a framing mechanism, no matter how it is deliminated cannot
be described as simple - unless you have already implemented it and don't
want to change :)

I'm  open to be persuaded that content-type can be punted to the application
(or at least a generic meta-data mechanism). But you will not convince me
that multiple framing types is better than sending a single boolean or a
string.

By contrast I have no idea why we would want the ability to declare
> connection-level content types (this is why I would prefer we worked from
> use cases; apologies if you provided some and I missed them, in that case a
> pointer would be appreciated).

I agree to some extent.  I really think that we need at least the option for
per message meta-data as content-types are best justified in a polymorphic
message environment (eg a stream of images of different formats).

The connection oriented content-type comes from the current protocol that
has such an assumption.

However, I can see use-cases for connection content-types.  If a websocket
service will only accept application/json and will only ever send text/xml,
then declaring that would allow infrastructure to validate that the service
conforms to expectation.  It would also allow infrastructure (either in
browser/server or in framework) to arrange for serialization into DOM and/or
objects.  Different clients might connect to the same service, thus

We could get by with just a per message meta data mechanism for binary.  But
I really cannot see the harm in allowing the optional declaration of a
connection content type.

 If a content-type is not necessary for an application that it can simply
> not declare if it is using utf-8 text or it can declare application/data if
> it does not wish to be specific.
>

The common case is to declare nothing.   There is no complexity at all.

> regards to it's vulnerability to injection and the

True - but I'm just indicating that the status quo is not perfect. I do not
think we have the option of changing nothing.

regards

---

From: James Graham <[email-masked]>
Date: Thu, 29 Jul 2010 11:11:42 +0200

On 07/29/2010 10:57 AM, Greg Wilkins wrote:
>
>
> On 29 July 2010 17:51, James Graham <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Well yes, if you use presence vs absence of some mostly-unrelated
>     thing as a flag to indicate text vs binary then that will of course
>     provide an unambiguous determination.
>
>

It is mostly-unrelated in the sense that something with a content type 
can "actually" be either text or binary but you are using content type 
to signal "treat this as binary".

As I said before this issue seems orthogonal to whether text frames are 
length marked or sentinel delimited. Personally I am happy with either 
design and I also have no problem with with change from sentinels to 
length markers for text frames now ("next four weeks"). I would expect 
there to be some resistance however. I think it is a less important 
issue than getting the details of the handshake right in the same time 
period. On the other hand I am not happy with only having a single frame 
type, not will I be happy if we try to deprecate an existing frame type 
after there is significant content using the current design.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 19:41:11 +1000

On 29 July 2010 19:11, James Graham <[email-masked]> wrote:

>  On the other hand I am not happy with only having a single frame type, not
> will I be happy if we try to deprecate an existing frame type after there is
> significant content using the current design.
>

James,

I think I may have misunderstood the thrust of your responses.  When you say
a "single frame type", do you mean a single frame mechanism or a single
op-code ( or similar indicator ) within a framing system.

I thought you were saying that we needed to have more than one framing
mechanism.  But I now think
you are saying something along the lines of:  we indicate a text frame with
an %x00 byte in the header and a binary frame with a %x01 byte (or similar
flag mechanism).

If so, then I'm much more accepting of that.   It's really just a discussion
then of how we allocate op-codes. While I  don't agree that binary data MUST
have a separate op-code, I don't particularly object either.

regards

---

From: James Graham <[email-masked]>
Date: Thu, 29 Jul 2010 11:43:06 +0200

On 07/29/2010 11:41 AM, Greg Wilkins wrote:
>
>
> On 29 July 2010 19:11, James Graham <[email-masked]
>       On the other hand I am not happy with only having a single frame
>     type, not will I be happy if we try to deprecate an existing frame
>     type after there is significant content using the current design.
>
>
> James,
>
> I think I may have misunderstood the thrust of your responses.  When you
> say a "single frame type", do you mean a single frame mechanism or a
> single op-code ( or similar indicator ) within a framing system.
>
> you are saying something along the lines of:  we indicate a text frame
> with an %x00 byte in the header and a binary frame with a %x01 byte (or
> similar flag mechanism).

Yes. Sorry if I was unclear.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 19:18:18 +0900

On 2010/07/29 17:57, Greg Wilkins wrote:
> On 29 July 2010 17:51, James Graham<[email-masked]>  wrote:
>

Well, there's some connection to be sure, but it's still very tentative.
What does it mean when I declare the content-type to be image/gif? Does 
this mean that every message is an image? Or that the whole 
communication in one way taken together is an image? Or that there may 
be several images, one after the other, and that the GIF format will 
help you figure these out (maybe one will end and another will start 
mid-packet)?

What does it mean when I declare the content-type to be text/plain? Does 
this mean that every message is an independent text resource (read 
file)? Or that the whole communication in one way taken together is a 
text file? Or something else?

Turning things around and looking at it from an application point of 
view, what, in terms of HTTP resources, is the back-and-forth chatter 
resulting from a chat application?

Would you expect that each subprotocol declares its own content-type, 
which applies to each message? (e.g. something like 
application/vnd.yahoo.im.websocket.response or whatever?)

This seems to clearly show that content-types don't work very well 
conceptually for WebSockets. For text/plain, we may be able to just say 
"don't bother", but for image/gif, that doesn't fly.

As for "just declare that binary means UTF-8 text if it's not labeled 
otherwise", what exactly would that mean for multibyte characters split 
up between messages? Binary as binary doesn't have any problem with 
that, but the application may want to complain heavily.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 19:24:27 +0900

> having multiple frame types.

Fully agreed. Also, having to decode each textual message by hand when 
the test cases are US-ASCII anyway is a recipe for sloppy 
implementations (US-ASCII doesn't need decode, it's only when real users 
abroad start to use it with Russian and Arabic and Chinese that it 
matters). This is clearly a point where we don't have to bring out the 
proverbial (in this WG :-) "amateur programmer", even highly 
professional programmers make this kind of mistakes.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 19:18:18 +0900

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 19:32:22 +0900

On 2010/07/29 14:31, Greg Wilkins wrote:
> On 29 July 2010 14:45, "Martin J. Dürst"<[email-masked]>  wrote:

> proposal for the content-type.

I don't mean the separation of framing and content-type, but the 
separation of "one kind of frame length identification" and "no 
top-level distinction between text and binary". To you, this may look 
like it's all the same, but to be, it isn't.

>   text/plain;charset=utf-8 and application/data.

There is no application/data at 
http://www.iana.org/assignments/media-types/application/. You may mean 
application/octet-stream. Please note that despite having 'stream' in 
its name, it has nothing to do with streaming; in the same way as all 
other media types, it denotes the type of a single resource (read file).

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 19:50:42 +0900

Hello Greg,

My thrust is quite a bit in the same direction as James's, so I'm quite 
relieved to see that you don't particularly object to having separate 
op-codes for binary data and for UTF-8 text.

Regards,   Martin.

On 2010/07/29 18:41, Greg Wilkins wrote:
> On 29 July 2010 19:11, James Graham<[email-masked]>  wrote:
>

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 29 Jul 2010 04:33:54 -0700

> If you want to send text, then it is possible to send that in the binary frame.  Each peer might have a priori knowledge that lets them know that the content is UTF-8 encoded text.  That’s OK - int…

In the case of the client API built into browsers, the browser needs to know which frames are text instead of binary and what the encoding is. It can't be based on a priori knowledge between peers because the browser has no a priori knowledge of any specific servers or their protocols.

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 29 Jul 2010 04:37:31 -0700

> While this seems like a reasonable statement, it needs to be justified.  Why?

I think I explained it in my message, but just to say it again:

When the client implementation in a Web browser gets a message from the server, it needs to know whether to hand off a string or a byte array to the JavaScript code using the client API. To do that, it needs to know which frames are text, or what the encoding is. Otherwise it is impossible for a browser to implement the draft WebSocket API: <http://dev.w3.org/html5/websockets/> (will be extended with binary support sooner or later).

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 29 Jul 2010 04:52:48 -0700

On Jul 29, 2010, at 12:19 AM, Greg Wilkins wrote:

> 1.a) We can't assume the text vs. binary distinction is handled via a content-type. The browser-based client API will deserialize text and binary frames into different JavaScript types. If it encounters a type *other* than text/plain;charset=utf-8 or application/data, how is it supposed to tell whether to deserialize the frame as text or binary?
> 

So application/xhtml+xml is binary? image/svg+xml is binary? That strikes me as a poor design.

> 
> But I think this is not really a discussion about content-type.  I think you are really concerned about what messages will be delivered to the text based API in the javascript.   I think that is a separate concern that how the protocol transports them.
> 
> You could for example say that only frames without an explicit Content-Type will be delivered to the javascript text based API.  Anything with a content-type (including text/* types) will be delivered via a yet to be defined "binary" API (which is might be a MIME API?) . Or you could say that any text/* content will also be delivered to the text javascript API.     Eitherway, that is more of an issue for the HTML5 folks to discuss and less of a protocol issue.

If things worked that way, then it would be foolish to ever use a content type for text content.

> 
> 1.b) Other than the need to distinguish text vs. data to deserialize (and serialize) correctly, it's not really necessary for the browser-hosted client API, or for that matter a server-side framework, to know the content-type. Neither the client nor the server will do anything different with text/plain vs. text/html. 
>  
> That is not necessarily true.  The client might parse text/xml into DOM or handle image/jpeg by creating some holder object.   On the server side, I can imaging using something like the java activation framework to de-serialize arbitrary types before invoking an application. 

No one has asked for features like that to be added to the API. Let's not design the protocol for imaginary use cases.

>    
> 2) I agree with all those who have argued against supporting arbitrary encodings.
> 
> I do not see how we can both support binary frames and not support arbitrary encodings?

Sure, if your binary content is secretly text in a weird encoding, then the encoding police will not break down your door and arrest you. However, this sets the expectation that other encodings will not be handled by the API - if you insist on 

> So I don't see this as a protocol issue.  It is a Javascript API issue, and if the JS API wants to only handle utf-8, then I think that is a reasonable choice. 

Supporting the JavaScript API effectively is at least one use case that the protocol must serve. Proposals that don't work reasonably with the JavaScript API are not going to fly.

>   But that should not prevent those that wish to use a future binary API from sending other encodings and declaring what encoding they are using.

If this declaration is solely for use by convention between peers that make a prior arrangement, then it doesn't need built-in support in the protocol or the API, it can just be sent in-band or use a generic metadata mechanism.

> Until we have sketched out some future plans, we have no real idea if the current handshake is suitable or not.   So I'm raising these issues now so that we may consider the road map.    If you don't think it is important, then there is no need to give any feedback until after the critical bugs are fixed.

These future ideas are dominating list traffic currently, which makes it hard to focus on the most critical and time-sensitive issues.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 22:00:09 +1000

Martin,

I think you are raising a straw man here.... but I think it is a very
valuable straw man to discuss:)

On 29 July 2010 20:18, "Martin J. Dürst" <[email-masked]> wrote:

> this mean that every message is an image? Or that the whole communication in
> one way taken together is an image? Or that there may be several images, one
> after the other, and that the GIF format will help you figure these out
> (maybe one will end and another will start mid-packet)?
>

Currently the protocol does not let you split multibyte characters over
multiple frames,
so I see no reason why an images should be split over multiple messages.

This is slightly changed when chunking is introduced, as it will allow both
multi-byte characters and images to be split over multiple frames.

But I think it VERY important to declare that a message has to be a complete
entity of what ever content-type is being sent.

If the message is text/plain;charset=utf-8 that means that all characters
have to be complete and multi-byte characters cannot be split over multiple
messages.

If the content type is application/json, then a message should contain a
fully formed json object.

If the content type is text/xml, then it should contain a valid XML
document.

So with chunking, frames become a transport mechanism for chunks of
arbitrary data.
Messages are semantic boundaries consisting of one of more frames and
represent complete entities that may be passed to an application.

Would you expect that each subprotocol declares its own content-type, which
> applies to each message? (e.g. something like

No not at all.

For me, the entire reason for having a content-type is so that my
application can be supported by infrastructure in the
browser/server/framework/libraries/debuggers that will take a lot of work
away from the application.   Ideally the infrastructure perform the de
serialization so that the application itself would never have to deal with
bytes or even unparsed json documents.

> "don't bother", but for image/gif, that doesn't fly.

Sorry I think you have just torn down your own straw man.   Yes
content-types are indeed meaningless if message boundaries are not
also semantic boundaries.

So to flip this around, how do you see binary messages working if there is
no standardized meta data? Do you really expect applications to re-invent
MIME?

Remember that this is all OPTIONAL!  if you really really think that you can
do better than MIME, then you are free to not send any standard content type
and invent your own subprotocol instead.  I just don't see that it is
reasonable to force everybody to re-invent the wheel.

> otherwise", what exactly would that mean for multibyte characters split up
> between messages? Binary as binary doesn't have any problem with that, but
> the application may want to complain heavily.
>

If you want to stream binary an not care about splitting things like
multibyte characters, then never send a frame with the chunk bit clear.
Once that final chunk is sent, you are saying that you have sent a
semantically complete unit.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 22:23:15 +1000

Maciej

all  messages are binary!    Are you saying that text/plain;charset=utf8 is
not binary?

Content type is an indication of how we should interpret that binary data as
text or other types of content.

I fail to understand why there is such an objection to being able to
OPTIONALLY label binary messages with standard MIME headers (specifically
content type), so that generic infrastructure can be built.

> No one has asked for features like that to be added to the API. Let's not
> design the protocol for imaginary use cases.
>

It is not an imaginary use case at least not the server side.  There are
entire frameworks and standard developed for handling MIME encoded messages
so that application developers do not have to deal with arrays of bytes.

Browsers have already made the mistake of not sending content type and
encoding with HTTP POSTs, so we have the stupid situation on the server side
where the application has to tell the framework what it thinks the character
encoding is.

I really do not wish to have this mistake repeated for websocket so an
infrastructure has to resort to trying to guess content types because the
client has not labelled it.

These future ideas are dominating list traffic currently, which makes it
> hard to focus on the most critical and time-sensitive issues.
>

There is no compulsion for you to read or respond to this traffic.

This thread appears to have sparked significant interest and discussion. As
a result some good progress has been made trimming down speculative ideas
and overcoming some misunderstandings.

And your contribution is to tell us to shut up!

---

From: "Alexander Philippou" <[email-masked]>
Date: Thu, 29 Jul 2010 16:26:35 +0300

> These future ideas are dominating list traffic currently, which makes it hard to focus on the most critical and time-sensitive issues.

There is another side to this coin; the very fact that these ideas are dominating list traffic is the best indication that they need to be discussed now rather than later.

Alexander

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 23:35:18 +1000

Maciej,

sorry that was out of line.

regards

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 29 Jul 2010 10:13:17 -0700

> However, you probably can't do this with an upcall.  In that case, you probably have to just pass the ByteArray to the user.  If they know that it's a string, then they can call UTF8.decode.
But wouldn't you actually want WebSocket.recvObject()?

When you're deserializing, it's generally because you want an actual 
deserialized object, not the serialized byte array.

-- Scott
>>> While this seems like a reasonable statement, it needs to be
>>>       
>> justified.  Why?
>>

---

From: James Graham <[email-masked]>
Date: Thu, 29 Jul 2010 19:27:16 +0200

On 07/29/2010 09:50 AM, Thomson, Martin wrote:
> If the client/server library - or an intermediary - has any reason to
> examine the payload, or know of its nature, then you have
> justification.
>

Is this supposed to be the javascript API? The current API looks quite 
different to this; it is event based because blocking operations in 
main-thread scripts are all kinds of bad (workers are different of 
course but overkill for many situations).

I think changing the JS API so users have to chose text vs binary 
explicitly vs it coming for free from the type system adds significant 
complexity for no obvious value.

---

From: Joshua Bell <[email-masked]>
Date: Thu, 29 Jul 2010 11:01:52 -0700

On Thu, Jul 29, 2010 at 10:27 AM, James Graham <[email-masked]> wrote:

> main-thread scripts are all kinds of bad (workers are different of course
> but overkill for many situations).
>

To summarize from
http://dev.w3.org/html5/websockets/#the-websocket-interface the JavaScript
API that an event callback will occur to a function hooked to the onmessage
event. The object passed to this callback has event type "message", and a
"data" attribute will be set to the received data. Given the current drafts
for WS, this would be a JavaScript string.

As an aside, JavaScript strings are nominally UTF-16 sequences, not UTF-8.
Using UTF-8 on the wire is a Good Thing (IMHO), but the browser will be
transcoding the UTF-8 to UTF-16 before handing the bits to JavaScript. Even
if the encoding is implicit in the protocol, there will always be
transcoding by the browser during  send/receive.

For relevant comparison, the XMLHttpRequest response objects have
responseText and responseXML attributes.
http://www.w3.org/TR/XMLHttpRequesthas all of the details, but
informally responseText is nearly always
populated as a JavaScript string (assuming content-encoding was
comprehensible), responseXML is a DOM if the response was sufficiently
XMLish (content-type, content-encoding, well formed, yadda yadda). The
http://www.w3.org/TR/XMLHttpRequest2/ draft suggests a responseBody which
contains a ByteArray of the raw data, but the binary types for JS haven't
been locked down yet.

Following either pattern (current WS API or XHR API), the API is such that
the receiving code is given the interpreted results - which may include some
metadata and/or multiple flavors.

For WS, if the frame metadata restricts it to either text/plain or
application/octet-stream then the previous suggestions of having the
message's data attribute simply be a JS string or ByteArray (final type
pending) is one possibility.

Another would be to change the API to align with XHR: dataString (JS string,
null if it wasn't textual data), dataBody (ByteArray). In that case, one
could imagine supporting a handful of other types (dataXML?) based on
content-type being present in the WS metadata. As the thread points out,
that gets a bit absurd given the variety of data types that might be getting
transmitted (image/jpeg). What's more reasonable IMHO is that the binary
data API in JS should allow re-use of the data and/or transcoding (e.g.
decoding from UTF-16, re-use as image data, etc); as is pointed out on this
thread, both ends should know what data type is being transmitted anyway.
This is the direction that the JS binary API is going anyway, so let's rely
on that rather than adding intelligence into the protocol.

If JS's binary support were mature, it might even be reasonable to suggest
that WS simply deliver the data as a ByteArray and even decoding a payload
from UTF-8 to a JS string be the responsibility of the JS code. But it's not
(in fact, it's barely scribed on soft clay at this point), so that's not a
practical option.

So I lend my support towards keeping the WS protocol simple and practical -
two frame types, UTF-8 text and binary, no extensible
content-type/content-encoding within the protocol itself. I lend my support
towards always using length-delimited frames.

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 29 Jul 2010 11:37:36 -0700

Joshua Bell wrote:
> responseText and responseXML 
> attributes. http://www.w3.org/TR/XMLHttpRequest has all of the 
> details, but informally responseText is nearly always populated as a 
> JavaScript string (assuming content-encoding was comprehensible), 
> responseXML is a DOM if the response was sufficiently XMLish 
> (content-type, content-encoding, well formed, yadda yadda). 
> The http://www.w3.org/TR/XMLHttpRequest2/ draft suggests a 
> responseBody which contains a ByteArray of the raw data, but the 
> binary types for JS haven't been locked down yet. 
>
> Following either pattern (current WS API or XHR API), the API is such 
> that the receiving code is given the interpreted results - which may 
> include some metadata and/or multiple flavors. 
>
> message's data attribute simply be a JS string or ByteArray (final 
> type pending) is one possibility. 
>
> Another would be to change the API to align with XHR: dataString (JS 
> string, null if it wasn't textual data), dataBody (ByteArray). In that 
> case, one could imagine supporting a handful of other types (dataXML?) 
> based on content-type being present in the WS metadata. As the thread 
> points out, that gets a bit absurd given the variety of data types 
> that might be getting transmitted (image/jpeg). What's more reasonable 
> IMHO is that the binary data API in JS should allow re-use of the data 
> and/or transcoding (e.g. decoding from UTF-16, re-use as image data, 
> etc); as is pointed out on this thread, both ends should know what 
> data type is being transmitted anyway. This is the direction that the 
> JS binary API is going anyway, so let's rely on that rather than 
> adding intelligence into the protocol.
>
> If JS's binary support were mature, it might even be reasonable to 
> suggest that WS simply deliver the data as a ByteArray and even 
> decoding a payload from UTF-8 to a JS string be the responsibility of 
> the JS code. But it's not (in fact, it's barely scribed on soft clay 
> at this point), so that's not a practical option.
>
> So I lend my support towards keeping the WS protocol simple and 
> practical - two frame types, UTF-8 text and binary, no extensible 
> content-type/content-encoding within the protocol itself. I lend my 
> support towards always using length-delimited frames.

Thanks for the clarification. I'm convinced now.

Supporting distinct text and binary messages will complicate the server 
APIs a bit, but it'll be manageable.

-- Scott

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 29 Jul 2010 23:23:18 +0200

> These future ideas are dominating list traffic currently, which makes it hard to focus on the most critical and time-sensitive issues.

Anyway, each time we discuss existing real issues, it looks like
once a solution seems to be found (so there is nothing more to say
about it), it finally vanishes because noone updates the draft with
it. I'll take the handshake as a good example, I got bored reminding
that it's still worded in a way that makes it incompatible with HTTP
reverse proxies. Ian himself suggested a minor change that could fix
it but this is not in the draft yet and there was no indication whether
something would be done about it. I also suggested adding some
"Connection: close" headers which Jamie agreed about to limit risks
of misinterpretations of pending data, well, no signal back...

So, discussing the future seems more efficient to continue to advance.

Regards,
Willy

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 30 Jul 2010 11:00:00 +0900

I think it would be a good think if the chairs would open tickets on 
issues such as those indicated below. Tickets also can have a target 
milestone, which would be either 4-week or 6-month.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 30 Jul 2010 14:05:04 +0900

Hello Greg,

> valuable straw man to discuss:)

Actually, it's not intended as a straw man. Please see below.

> On 29 July 2010 20:18, "Martin J. Dürst"<[email-masked]>  wrote:
>

Last point understood.

> entity of what ever content-type is being sent.

I may have missed something, but this is the first time I have seen you 
say this. It seems to have been a tacit assumption up to now. It seems 
to make sense, but if it's very important, then it should be stated very 
clearly.

> have to be complete and multi-byte characters cannot be split over multiple

I'll get to that later.

> fully formed json object.

Seems to make sense. But a json object could be sent in a text frame, 
no? RFC 4627 says UTF-8 is the default.

> If the content type is text/xml, then it should contain a valid XML

Seems to make sense, except that text/xml these days is frowned upon 
(application/xml is preferred, and most extended xml types use 
application/...+xml, not text/). That shows that the distinctions in 
MIME (mainly created for email, later used for HTTP) don't exactly match 
with the text/binary distinction

> represent complete entities that may be passed to an application.

I agree with "complete entities passed to an application" if we 
explicitly note that "entity" here may not be the same as "entity" in HTTP.

As has been noted, we are way closer here to TCP. Please imagine a 
WebSockets equivalent of SMTP, POP3, IMAP, IRC,... As an example, let's 
look at http://en.wikipedia.org/wiki/SMTP#SMTP_transport_example.
It is clear that each consecutive set of lines with S/C in the first 
column may become a message (maybe the long sequence of C..C at the end 
could be more than one message). Now in terms of MIME Media Types, are 
they text/plain? In my opinion, they are not text/plain entities (these 
would be like files if put on HD at either end), they are just protocol 
text.

> bytes or even unparsed json documents.

For json documents, this *may* make sense. But it seems inappropriate to 
think about all the messages as documents (see above SMTP,... example). 
Also, you don't know what the subprotocol wants to do with a json 
document (or an XML document, or ...).

> also semantic boundaries.

As the SMTP example shows, content-types are also not always meaningful 
if each message is a clear semantic entity (I don't have anything 
against the later).

> no standardized meta data? Do you really expect applications to re-invent

No. But look e.g. at the SMTP example. One way to convert that to 
WebSockets would be to send a C->S message for the mail headers and 
another for the mail body. The mail header would contain the necessary 
information. Another would be to have only one message. The information 
would be self-contained. The server knows that something like this comes 
after the DATA command from the client. There are all kinds of libraries 
to handle this kind of data. No WebSockets application would have to 
invent anything, at least not for this case.

> reasonable to force everybody to re-invent the wheel.

It's not about re-inventing the wheel. But while I don't deny that there 
will be cases where WebSockets messages will be MIME entities, I think 
that for many if not most of the short messages (where WebSockets is at 
its best), trying to shoehorn them into seeing them as MIME entities 
isn't appropriate (again, see the SMTP example).

Let's look at another example, HTML forms. Let's say we have a form with 
a few text fields,... and a file upload field (<input type="file" ...). 
A server-side framework would make the text fields available as strings, 
and the file would be available as an actual file in some tmp directory 
(from where it could be read or copied or relinked as appropriate). In 
WebSockets, that might work too, but how do we distinguish between a 
text field (small) and a text entity (large)?

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 30 Jul 2010 14:18:32 +0900

On 2010/07/29 21:23, Greg Wilkins wrote:
> On 29 July 2010 21:52, Maciej Stachowiak<[email-masked]>  wrote:

> all  messages are binary!    Are you saying that text/plain;charset=utf8 is

All messages are electric signals, at one point or another. That doesn't 
help this discussion. What Maciej is saying (I guess) is that once we 
have a text vs. binary distinction, in that distinction, does it make 
sense to have UTF-8 application/xhtml+xml or image/svg+xml as binary?

Thinking about this, it seems to show that there is some mismatch 
between WebSockets text vs. binary, and MIME's text/ vs. application/, 
image/,... If you could explain to us how you plan to sort that out, 
that might help.

> so that application developers do not have to deal with arrays of bytes.

Can you point to such frameworks, so that we get an idea about what you 
are talking about? Would it be impossible for a WebSockets program on 
the server side to say: "Oh, I know the next message is a png image, 
let's ask my MIME framework to handle it."?

What's the chance, in your opinion, of a subprotocol accepting arbitrary 
MIME entities at arbitrary instants in a conversation (as opposed to 
accepting single types of MIME entities at well-defined instants, or not 
dealing in MIME entities at all)?

> client has not labelled it.

This argument doesn't fly. HTTP clearly has the means to label these 
things, but it's not used. That doesn't make an argument for providing 
the possibility to label in WebSockets. (it's not an argument against, 
either)

Regards,    Martin.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 30 Jul 2010 16:01:37 +1000

On 30 July 2010 15:18, "Martin J. Dürst" <[email-masked]> wrote:

> Can you point to such frameworks, so that we get an idea about what you are
> talking about?

 The Java Activation Framework is one example.

XMLHttpRequest is another example.  It will look at the content type and use
that to decide if it should parse the content into a DOM tree or not.
Browsers also have machinery to handle all the various image/* content
types. I assume that if I was sending thumbnail images to my chat room (this
is a real use-case over a comet connection from a large mobile carrier chat
service), then I think it would be far more useful for the datagrams to
emerge as Image objects rather than as byte arrays of some unknown image
format.

Would it be impossible for a WebSockets program on the server side to say:
> "Oh, I know the next message is a png image, let's ask my MIME framework to
> handle it."?
>

The problem is in the "knowing".  Without a label it is not knowing, it is
only guessing.   Evidence from the server side of existing web applications
is that guessing content types and encodings is frequently problematic.
It's probably not so bad for English speakers, but ask I'm sure if you ask
anybody that speaks Chinese or Japanese, they will tell you how poorly
browsers and servers are at assuming the same content type/encoding.

> dealing in MIME entities at all)?

I'm not sure I understand the question.

I see two prime use-cases for labelling content-type.

1) The content type is always the same and is "known" by the application.
In this case, a content-type label is just a validation that an application
has connected to the service that it was expecting.  If it was expecting
json messages, but the service has been updated to speak xml (or the wrong
URL entered), then a content-type label will help with early and clear
detection of the problem.

2) The message stream is polymorphic.   eg Chat message might be intermixed
with thumb nails, which themselves might be jpg, png or gif.  It is not
sufficient to know if a message is text or binary, but the true content type
must be known so the image can be correctly decoded.  Ideally the
application would not have to do that decoding and a framework (or some
other infrastructure) would handle the various image formats and just pass a
bitmap for all the images types.

> things, but it's not used. That doesn't make an argument for providing the
> possibility to label in WebSockets. (it's not an argument against, either)

The argument is that not having a label forces guesses and/or assumptions of
content type/encoding and that the evidence is that such guesses are not
always correct.  I acknowledge that having support for a label will not mean
that it will be used, so this problem will not be totally avoided.
However, not have support for the label will ensure that guesses have to be
made and that this problem is more prevalent.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 30 Jul 2010 16:37:15 +1000

> Actually, it's not intended as a straw man. Please see below.

Most straw men (specially my own) are unintended :)

> I may have missed something, but this is the first time I have seen you say
> this. It seems to have been a tacit assumption up to now. It seems to make
> sense, but if it's very important, then it should be stated very clearly.
>

I think it is currently implicit due to the utf-8 sentinel based framing.
The spec says things like

    Interpret /raw data/ as a UTF-8 string, and store that
    string in /data/.

which does not allow for partial characters.

This is why I think this is a valuable discussion.  I think that this tacit
assumption needs to be clearly stated, specially if we introduce binary
and/or chunking.

> Seems to make sense. But a json object could be sent in a text frame, no?
> RFC 4627 says UTF-8 is the default.
>

I would say that json could be sent in a text *message* that could span
several frames.  Each text frame might not be a complete parseable json
string, but the entire message should be.
Of course applications are free to send partial json strings, but then they
should not declare the content type as json (because it is not, it is json
fragments).

> I agree with "complete entities passed to an application" if we explicitly
> note that "entity" here may not be the same as "entity" in HTTP.
>
> WebSockets equivalent of SMTP, POP3, IMAP, IRC,... As an example, let's look
> at http://en.wikipedia.org/wiki/SMTP#SMTP_transport_example.
> It is clear that each consecutive set of lines with S/C in the first column
> may become a message (maybe the long sequence of C..C at the end could be
> more than one message). Now in terms of MIME Media Types, are they
> text/plain? In my opinion, they are not text/plain entities (these would be
> like files if put on HD at either end), they are just protocol text.
>

When somebody maps a text based protocol to websocket, then regardless of
content type, they will have a choice of either using the semantic
boundaries provided by websocket datagrams or of using the "native"
deliminators within the protocol.  In your mime example, the mapping might
declare that every mail message has to be sent in a complete websocket
message, or they might treat websocket messages as a just a transport and
parse the data looking for CR LF '.' CR LF to terminate the message.

In either case, I don't think that labelling with a text/plain content type
gains this application very much at all (unless other charset encodings
happened to be used).

However, if the the mapping choose to send only complete mail messages in
each websocket message, then the content type should probably be
message/rfc822, which would allow infrastructure (debuggers/frameworks etc.)
to handle the text as a message, extract attachments etc. etc.

> Also, you don't know what the subprotocol wants to do with a json document
> (or an XML document, or ...).
>

Of course an application is able to not assign any meaning to the semantic
boundary of websocket message.  In such cases having a content type is not
meaningful.

But many applications will use the semantic boundaries of a websocket
message, in which case there is almost certainly a meaningful mime type that
could be assigned to assist infrastructure to validate, debug or assist and
application.

> also semantic boundaries.

As the SMTP example shows, content-types are also not always meaningful if
> each message is a clear semantic entity (I don't have anything against the
> later).
>

True. Content types are not always meaningful or useful.  I'd go as far as
to say that they are frequently neither meaningful or useful.

But it is also true that they are frequently both meaningful and useful.
Thus it seams entirely reasonable to provide a standard way to communicate
information that is frequently available, meaningful and useful.  Of course
it can be sent in non standard ways, but then that can be said for the
entire contents of any websocket message.

> WebSockets would be to send a C->S message for the mail headers and another
> for the mail body. The mail header would contain the necessary information.
> Another would be to have only one message. The information would be
> self-contained. The server knows that something like this comes after the
> DATA command from the client. There are all kinds of libraries to handle
> this kind of data. No WebSockets application would have to invent anything,
> at least not for this case.
>

Then what is the harm of labelling such a message with the content-type of
message/rfc822 so that all kinds of libraries can validate that they apply
or be applied automagicly if desired.

> will be cases where WebSockets messages will be MIME entities, I think that
> for many if not most of the short messages (where WebSockets is at its
> best), trying to shoehorn them into seeing them as MIME entities isn't
> appropriate (again, see the SMTP example).
>

There is no shoe horning going on here. There is NO PROPOSAL that message
must have a content-type.   If there is no applicable content type and/or
the application does need it, then simply do not send it.

The counter proposal (no standardized meta-data) is to shoe horn every
application into inventing it's own means of communicating content-type when
it is known and useful.

> Let's look at another example, HTML forms. Let's say we have a form with a
> few text fields,... and a file upload field (<input type="file" ...). A
> server-side framework would make the text fields available as strings, and
> the file would be available as an actual file in some tmp directory (from
> where it could be read or copied or relinked as appropriate). In WebSockets,
> that might work too, but how do we distinguish between a text field (small)
> and a text entity (large)?
>

The same way we do today - we use the appropriate content-type.  It might be
sent as application/x-www-form-urlencoded for small text fields or as
multi-part/form-data for a mix of large and small entities, each part
potentially having its own content type for further assistance with
processing.      Sure setting a content-type of text/plain would not be
useful in this example, but that is not to say that there is not a useful
type that can be used.

regards

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 30 Jul 2010 15:20:59 +0800

> When the client implementation in a Web browser gets a message from the
> server, it needs to know whether to hand off a string or a byte array
> to the JavaScript code using the client API. To do that, it needs to
> know which frames are text, or what the encoding is. Otherwise it is
> impossible for a browser to implement the draft WebSocket API:
> <http://dev.w3.org/html5/websockets/> (will be extended with binary
> support sooner or later).

With all respect, this is not a satisfactory answer.  The API could simply be changed instead to be binary only.

It is valid choice to have the API pass binary data to the higher layer.  The application can decode if it knows that it's a string.

socket.onmessage = function(data) {
  // data is binary, need to decode it
  var str = new String(data);

  // do stuff
}

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 30 Jul 2010 00:31:31 -0700

> With all respect, this is not a satisfactory answer.  The API could simply be changed instead to be binary only.

That's certainly possible, but:

(a) The proposal did not come with a suggested API change.
(b) There's no suitable type to use for binary data, so it wouldn't be practical to make that API change right now.
(c) Such an API would be much less convenient than the current API for a lot of use cases, since many useful WebSocket subprotocols are likely to be text-based, just as many useful protocols over TCP are. It's likely that binary will be the exceptional case, not the common case, and the API should not be bent around the exceptional case.
(d) If text is the common case, then having to allocate and pass off a binary data object first is suboptimal for performance.
(e) If developers have to handle text decoding themselves, it is quite likely they will make mistakes, leading to interoperability failures, internationalization problems, and security issues.
(f) No one has mentioned any technical advantage to an API that only exposes binary data and not strings. Let alone one that would outweigh the aforementioned disadvantages.

> It is valid choice to have the API pass binary data to the higher layer.  The application can decode if it knows that it's a string.

It is a valid choice, but as far as I can tell it would be a technically inferior choice.

>  var str = new String(data);

That doesn't actually work in JavaScript. Since the way of representing binary data in JS is still somewhat under construction, it's not clear what the API will be, but it will almost certainly not be that. Likely it will not involve the String constructor, and almost certainly you'll have to explicitly name the encoding to use.

Regards,
Maciej

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Fri, 30 Jul 2010 01:47:07 -0700

Apologies for the brief response, on a train from Schiphol to Maastricht and
only getting WiFi at stops at stations inbetween.

I mostly agree with Maciej here - I think much of the usage will be for text
data, and we need to make it easy to send/receive such text with a clear
mapping to a  J/S API that doesn't involve JS programmers doing all sorts of
strange things to construct a binary object in which to stuff their text.
Bad things will come from that.

That said, I do think there is some bit of separation between what we expose
as a JavaScript API in the browser and what the wire protocol is. E.g. I
don't see any reason why wanting to expose text-friendly APIs in the browser
should prohibit us from moving to a length-based framing mechanism, provided
that we have a good way to distinguish between a frame that is meant to be
interpreted as text and delivered to a JS API as such vs a frame that is
meant to be interpreted as some form of binary data that would e.g. be
delivered as a ByteArray in JS. There's some good discussion as to whether
that should be indicated as a MIME type or whether that should be indicated
by a different frame type / control bit in the frame header, which I think
is something worth discussing further.

I also see value in having a standard way to express mime types. E.g. if I
want to send a thumbnail from the server to the browser or vice versa, it
would be nice for me to have a standard way of indicating whether that's a
PNG, JPEG etc, without having to re-invent the wheel. I'm not sure I am
fully convinced on how that question should relate to the question of
indicating a frame intended to be consumed by a text- or binary- JS API.

See you all in this afternoon's session.

-Ian F (not to be confused with Ian H)

On Fri, Jul 30, 2010 at 12:31 AM, Maciej Stachowiak <[email-masked]> wrote:

> > With all respect, this is not a satisfactory answer.  The API could
> simply be changed instead to be binary only.
>
> (b) There's no suitable type to use for binary data, so it wouldn't be
> practical to make that API change right now.
> (c) Such an API would be much less convenient than the current API for a
> lot of use cases, since many useful WebSocket subprotocols are likely to be
> text-based, just as many useful protocols over TCP are. It's likely that
> binary will be the exceptional case, not the common case, and the API should
> not be bent around the exceptional case.
> (d) If text is the common case, then having to allocate and pass off a
> binary data object first is suboptimal for performance.
> (e) If developers have to handle text decoding themselves, it is quite
> likely they will make mistakes, leading to interoperability failures,
> internationalization problems, and security issues.
> (f) No one has mentioned any technical advantage to an API that only
> exposes binary data and not strings. Let alone one that would outweigh the
> aforementioned disadvantages.
>
>
> >
> > It is valid choice to have the API pass binary data to the higher layer.
>  The application can decode if it knows that it's a string.
>
> That doesn't actually work in JavaScript. Since the way of representing
> binary data in JS is still somewhat under construction, it's not clear what
> the API will be, but it will almost certainly not be that. Likely it will
> not involve the String constructor, and almost certainly you'll have to
> explicitly name the encoding to use.
>
> Regards,
> Maciej
>

---

From: Dave Cridland <[email-masked]>
Date: Fri, 30 Jul 2010 09:53:36 +0100

> that to decide if it should parse the content into a DOM tree or  

As a knee-jerk reaction to that, would the addition of a "This is XML  
data" flag be useful, alongside "This is UTF-8" and "This could be  
anything"?

And, if we really want to go along the lines of what you're thinking,  
what about a "This has a MIME header which includes the content type,  
followed by a complete MIME entity"?

My gut feeling is that this should satisfy all the use cases, and be  
efficient for the common cases of XML, text, and gloop. (Where gloop  
is uncharacterisable or incomplete binary blobbery).

Dave.

---

From: James Graham <[email-masked]>
Date: Fri, 30 Jul 2010 10:57:05 +0200

On 07/30/2010 10:47 AM, Ian Fette (イアンフェッティ) wrote:
> Apologies for the brief response, on a train from Schiphol to Maastricht
> and only getting WiFi at stops at stations inbetween.
>
> I mostly agree with Maciej here - I think much of the usage will be for
> text data, and we need to make it easy to send/receive such text with a
> clear mapping to a  J/S API that doesn't involve JS programmers doing
> all sorts of strange things to construct a binary object in which to
> stuff their text. Bad things will come from that.
>
> That said, I do think there is some bit of separation between what we
> expose as a JavaScript API in the browser and what the wire protocol is.
> E.g. I don't see any reason why wanting to expose text-friendly APIs in
> the browser should prohibit us from moving to a length-based framing
> mechanism,

I don't think anyone has argued that. Both framing mechanisms work 
equally well from the point of view of the JS API. The main argument in 
favour of sentinel-based framing for text appears to be that it is 
harder to introduce security bugs by failing to consider multibyte 
characters in server implementations. This seems like a legitimate concern.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Fri, 30 Jul 2010 02:13:58 -0700

On Fri, Jul 30, 2010 at 1:57 AM, James Graham <[email-masked]> wrote:

> I don't think anyone has argued that. Both framing mechanisms work equally
> well from the point of view of the JS API. The main argument in favour of
> sentinel-based framing for text appears to be that it is harder to introduce
> security bugs by failing to consider multibyte characters in server
> implementations. This seems like a legitimate concern.

as are the points mike belshe raised earlier about buffering, flow control,
and extensibility.

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 30 Jul 2010 02:18:19 -0700

On Jul 30, 2010, at 1:57 AM, James Graham wrote:

> I don't think anyone has argued that. Both framing mechanisms work equally well from the point of view of the JS API. The main argument in favour of sentinel-based framing for text appears to be that it is harder to introduce security bugs by failing to consider multibyte characters in server implementations. This seems like a legitimate concern.

Indeed. There are a number of separate issues on this thread, but here are my opinions on what I see as the main ones:

1) Should text and binary messages be distinguished in a clear way in the wire protocol?

Yes. This is essential to making a client API in the browser that works well and is convenient to use for common cases.

2) Should text messages use length encoding or sentinel encoding?

 No strong opinion. I think length encoding is slightly better in general, but Ian's argument about likelihood of getting length encoding wrong is a good one. That being said, I expect most WebSocket services in the long run will use a server library or framework that takes care of this issue. Sentinel-based framing schemes can also have security risks. You have to take great care to ensure you don't accidentally put the sentinel value in the message. There have been many HTTP attacks based on injecting a newline (the sentinel value used to delimit headers). There have also been attacks against multipart form uploads by guessing and replicating the multipart boundary string. 0xFF may be slightly harder to inject. So overall I don't think there is a strong advantage either way.

Note: a framing scheme that avoids both of these hazards is to length-encode text frames using a Unicode character count instead of a byte count. However, this requires streaming decoding of the message to determine when you have gotten to the end, which loses one of the main advantages of length encoding (that you can split messages without parsing).

I think MIME types and multiple text encodings are a side issue - it's plausible to see how to add these in a compatible way. However questions 1 and 2 need to be answered definitively in the shorter 4-week period in Ian's timeline, since changing on either of those fronts requires an incompatible change.

Regards,
Maciej

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 02:40:53 -0700

On Fri, Jul 30, 2010 at 2:18 AM, Maciej Stachowiak <[email-masked]> wrote:

> favour of sentinel-based framing for text appears to be that it is harder to
> introduce security bugs by failing to consider multibyte characters in
> server implementations. This seems like a legitimate concern.
>
> Indeed. There are a number of separate issues on this thread, but here are
> my opinions on what I see as the main ones:
>
> Yes. This is essential to making a client API in the browser that works
> well and is convenient to use for common cases.
>
>  No strong opinion. I think length encoding is slightly better in general,
> but Ian's argument about likelihood of getting length encoding wrong is a
> good one. That being said, I expect most WebSocket services in the long run
> will use a server library or framework that takes care of this issue.
> Sentinel-based framing schemes can also have security risks. You have to
> take great care to ensure you don't accidentally put the sentinel value in
> the message. There have been many HTTP attacks based on injecting a newline
> (the sentinel value used to delimit headers). There have also been attacks
> against multipart form uploads by guessing and replicating the multipart
> boundary string. 0xFF may be slightly harder to inject. So overall I don't
> think there is a strong advantage either way.
>
>
You've characterized the differences well, as usual :-)

If all others have no strong opinion, I'll voice a strong opinion in favor
of prefix length-encoding.  The security points are valid - we've seen those
exploits in HTTP.  I won't go so far as to say that length encodings are
immune to it, they aren't, but it is harder.  The framing also provides a
sync point and will expose most length errors quickly.  All proposals are
subject to implementor bugs.  With length-prefixed frames, if one endpoint
*does* miscalculate the size of a UTF-8 string, the protocol will detect
it.

> Note: a framing scheme that avoids both of these hazards is to
> length-encode text frames using a Unicode character count instead of a byte
> count. However, this requires streaming decoding of the message to determine
> when you have gotten to the end, which loses one of the main advantages of
> length encoding (that you can split messages without parsing).
>
> I think MIME types and multiple text encodings are a side issue - it's
> plausible to see how to add these in a compatible way. However questions 1
> and 2 need to be answered definitively in the shorter 4-week period in Ian's
> timeline, since changing on either of those fronts requires an incompatible
> change.
>

+1.  I think the mime concepts are muddled at best.   Without a really
strong use case, I think we should drop them.  (I didn't see a use case?)
 UTF-8 only.

Here is what will happen.  In practice, browsers will choose exactly one
encoding scheme.  The protocol will say that we need to be prepared for
more, but all browsers will implement one (UTF-8).  Servers will be build
which depend on this (accidentally).  Then, some smart guy will try to use a
different encoding, and realize that various clients and servers are broken.
 Why?  Because 20% of the endpoints out there never implemented features
that the browsers never utilized in practice.

We have seen this pattern before, many times, with HTTP.  Optional and
unusable generally become synonymous.

Mike

---

From: James Graham <[email-masked]>
Date: Fri, 30 Jul 2010 12:03:14 +0200

On 07/30/2010 11:40 AM, Mike Belshe wrote:

> If all others have no strong opinion, I'll voice a strong opinion in 
> favor of prefix length-encoding. The security points are valid - we've 
> seen those exploits in HTTP. I won't go so far as to say that length 
> encodings are immune to it, they aren't, but it is harder. The framing 
> also provides a sync point and will expose most length errors quickly. 
> All proposals are subject to implementor bugs. With length-prefixed 
> frames, if one endpoint *does* miscalculate the size of a UTF-8 string, 
> the protocol will detect it.

I don't follow what you mean "the protocol will detect it". Let's say
someone writes a server that does

length = len(data)
output.write(text_frame_marker)
output.write(length)
output.write(data.encode("utf8"))

In what sense will the protocol detect the error here?

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 03:15:09 -0700

2010/7/30 James Graham <[email-masked]>

> In what sense will the protocol detect the error here?

It will detect it.

In this case, say your data string was 10 characters, represented by 11
bytes in UTF8.  So in this case, we would be writing that the length was 10,
followed by 11 bytes of data.  The receiver would receive those first 10
bytes of data, and consider that a complete frame.  The receiver would then
think it has another byte of data received which is the start of the new
frame.  This byte is basically random, and unlikely to look like the start
of a new frame, thus resulting in a protocol error.  Detected.

The case which doesn't detect is the opposite case, where the frame claims
to be 11 bytes long, but only 10 bytes are written.  In this case, the
protocol "hangs" trying to get more data.  So, you can't catch everything.
 In the case of UTF-8 string related bugs, however, it's usually the
overflow, not the underflow bug.

But - compare this to the sentinel case.  With a sentinel, the protocol will
still have two modes (early sentinel or late sentinel).  The former will
deliver truncated data to the application, and keep going.  Incorrectly.
 The latter will hang until a sentinel is received.  But, it will not detect
the error in either case.

Let me know if you disagree.

Mike

---

From: Stefan Eissing <[email-masked]>
Date: Fri, 30 Jul 2010 12:18:40 +0200

Sender: "Iou $1.000.000"
length: 10 
Received Packet: "Iou $1.000"
Connection Close

Am 30.07.2010 um 12:15 schrieb Mike Belshe:

> In this case, say your data string was 10 characters, represented by 11 bytes in UTF8.  So in this case, we would be writing that the length was 10, followed by 11 bytes of data.  The receiver would receive those first 10 bytes of data, and consider that a complete frame.  The receiver would then think it has another byte of data received which is the start of the new frame.  This byte is basically random, and unlikely to look like the start of a new frame, thus resulting in a protocol error.  Detected.
> 
> The case which doesn't detect is the opposite case, where the frame claims to be 11 bytes long, but only 10 bytes are written.  In this case, the protocol "hangs" trying to get more data.  So, you can't catch everything.  In the case of UTF-8 string related bugs, however, it's usually the overflow, not the underflow bug.
> 
> But - compare this to the sentinel case.  With a sentinel, the protocol will still have two modes (early sentinel or late sentinel).  The former will deliver truncated data to the application, and keep going.  Incorrectly.  The latter will hang until a sentinel is received.  But, it will not detect the error in either case.
> 

<green/>bytes GmbH
Hafenweg 16, 48155 Münster, Germany
Phone: +49 251 2807760. Amtsgericht Münster: HRB5782

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 03:26:12 -0700

> Received Packet: "Iou $1.000"

You left off two steps before the connection close:

Receiver receives errant bytes ".000", which are an error (not a proper
start of a frame)
Receiver sends PROTOCOL_ERROR

Mike

> > The case which doesn't detect is the opposite case, where the frame
> claims to be 11 bytes long, but only 10 bytes are written.  In this case,
> the protocol "hangs" trying to get more data.  So, you can't catch
> everything.  In the case of UTF-8 string related bugs, however, it's usually
> the overflow, not the underflow bug.
> >
> > But - compare this to the sentinel case.  With a sentinel, the protocol
> will still have two modes (early sentinel or late sentinel).  The former
> will deliver truncated data to the application, and keep going.
>  Incorrectly.  The latter will hang until a sentinel is received.  But, it
> will not detect the error in either case.
> >

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 18:38:03 +0000 (UTC)

On Fri, 30 Jul 2010, Mike Belshe wrote:
> bytes in UTF8.  So in this case, we would be writing that the length was 
> 10, followed by 11 bytes of data.  The receiver would receive those 
> first 10 bytes of data, and consider that a complete frame.  The 
> receiver would then think it has another byte of data received which is 
> the start of the new frame.  This byte is basically random, and unlikely 
> to look like the start of a new frame, thus resulting in a protocol 
> error.  Detected.
> 
> [...] With a sentinel, the protocol will still have two modes (early 
> sentinel or late sentinel).  The former will deliver truncated data to 
> the application, and keep going.  Incorrectly.

But it will keep going with a byte that is basically random, and unlikely 
to look like the start of a new frame, thus resulting in a protocol error, 
no? I don't understand how this is different.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 11:45:11 -0700

> no? I don't understand how this is different.

The point is that the protocol error was detected.  We are literally in a
state where the browser or server has to throw up its hands and just tell
the user "400 PROTOCOL ERROR".

Using the sentinel approach, we simply pass partial data to the app, and are
now stuck waiting for the next read on the websocket....  No error was
detected.  The application is likely hung (waiting to read the next
sentinel?) and the user doesn't know why.

Obviously, nobody *wants* errors.  That isn't the point.  When they occur,
they are due to bugs.  I'm trying to point out that for interoperability,
the length-prefixed framing offers a level of robustness which you don't get
with a sentinel-based boundary.

Mike

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 18:53:08 +0000 (UTC)

On Fri, 30 Jul 2010, Mike Belshe wrote:
> 
> The point is that the protocol error was detected.  We are literally in 
> a state where the browser or server has to throw up its hands and just 
> tell the user "400 PROTOCOL ERROR".
> 
> Using the sentinel approach, we simply pass partial data to the app, and 
> are now stuck waiting for the next read on the websocket....  No error 
> was detected.  The application is likely hung (waiting to read the next 
> Obviously, nobody *wants* errors.  That isn't the point.  When they 
> occur, they are due to bugs.  I'm trying to point out that for 
> interoperability, the length-prefixed framing offers a level of 
> robustness which you don't get with a sentinel-based boundary.

I may be misunderstanding, so correct me if I'm wrong:

We're talking about a case where a non-malicious error has caused a frame 
to be considered too short, either because the length is given as shorter 
than it really is, or because there is a sentinel in the data.

Suppose the data is "ABC0DEF" where "0" happens to be the sentinel byte, 
and that the frame type byte is "X".

Length case (length mismeasured as "4"):
   On the wire: X 4 A B C 0 D E F
   An error is detected at "D" because "D" is not a valid frame type byte.

Sentinel case:
   On the wire: X A B C 0 D E F 0
   An error is detected at "D" because "D" is not a valid frame type byte.

What's the difference?

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 Jul 2010 11:59:01 -0700

> no? I don't understand how this is different.

It may or may not, depending on the byte in question.
It is not 100% likely that an error will be detected, however we could make
it so if we really wanted to.
It is fairly likely, however, that an error would be detected as we'd
hopefully pick a byte or word combination that wasn't typically occurring as
part of the expected frame format.

For sentinel framing, we'd never notice. Zero chance. I'd take the
90-something percent chance over the zero-percent chance every day.
-=R

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 12:08:34 -0700

>   An error is detected at "D" because "D" is not a valid frame type byte.

Sure - all bugs are possible. I specifically stated that it would not detect
all errors.  You're talking about the accidental inclusion of extra
sentinels, I was illustrating the omission of sentinels, which is more
likely, I think, but I dunno....

Previously we had discussed a different case:

  string foo;   // pretend this string is 10 characters, but 11 UTF-8
encoded bytes
  websocket.write(foo, foo.length);   // whoops - we wrote 10 bytes, but
should have written 11!

With a framing based approach, this error is caught.  With a sentinel based
approach, it hangs until a sentinel is received.  When the sentinel is
finally received, the message boundary has been lost, and combined frames
can be delivered to the application with errors undetected.

All things being equal, length-defined framing will detect more errors and
be less likely to pass errors through.

Mike

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 21:13:24 +0000 (UTC)

On Fri, 30 Jul 2010, Mike Belshe wrote:
> >
> > Suppose the data is "ABC0DEF" where "0" happens to be the sentinel 
> > byte, and that the frame type byte is "X".
> >
> Sure - all bugs are possible. I specifically stated that it would not 
> detect all errors.  You're talking about the accidental inclusion of 
> extra sentinels, I was illustrating the omission of sentinels, which is 
> more likely, I think, but I dunno....
> 
> With a [length] based approach, this error is caught.  With a sentinel 
> based approach, it hangs until a sentinel is received.  When the 
> sentinel is finally received, the message boundary has been lost, and 
> combined frames can be delivered to the application with errors 
> undetected.

That is a far preferable failure condition than the situation we would get 
with length framing, which is that an attacker could trick the 
implementation into sending a length shorter than the text it actually 
sent, which could include a fake frame to perform an XSS attack.

> All things being equal, length-defined framing will detect more errors 
> and be less likely to pass errors through.

Yes, but things aren't equal. Some bugs lead to vulnerabilities. Others 
lead to failure. Failure is fine. Vulnerabilities are not.

(Sentinel-based framing can also lead to vulnerabilities, but more things 
have to go wrong, and they have to go wrong in components unrelated to 
WebSockets, which means it's a better tradeoff.)

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 14:31:28 -0700

> sent, which could include a fake frame to perform an XSS attack.

Ian -

This is just untrue.  Sentinel based framing DOES NOT offer additional
protection against bugs than does fixed length encoding.

Here is an example where sentinel encoding fails, but length-encoding does
not.  It's bitten us in HTTP many times, as you  know.
http://www.securiteam.com/securityreviews/5GP0220G0U.html

Lets be clear:
  * You're correct that, IFF the browser has a bug where it writes the wrong
length, the attacker can do bad stuff to fixed-length framing.
  * I am also correct that, IFF the browser allows the content to embed
sentinels (or skip sentinels) the attacker can do bad stuff.
  * Neither approach is more "secure" from random security bugs than the
other.

But, more importantly, fixed-length framing IS MORE ROBUST against
accidental attacks.

So let's review:
   a) On security, the two are the same.  Both approaches are vulnerable if
implemented incorrectly.
   b) On catching errors, fixed-length framing is better.

Mike

> lead to failure. Failure is fine. Vulnerabilities are not.

Stop.  There is no vulnerability.  You're simply making that up.

Sentinel based framing is far more susceptible to this type of attack:
http://www.securiteam.com/securityreviews/5GP0220G0U.html

<http://www.securiteam.com/securityreviews/5GP0220G0U.html>

> WebSockets, which means it's a better tradeoff.)

Ian, you're ignoring all the arguments you don't like.

length-based framing *is* less error prone and *more* secure.

Sentinel based framing is what leads to this:
http://www.securiteam.com/securityreviews/5GP0220G0U.html

It would be wrong of me to say that length-based framing is a panacea.  It
is not.  But, you're

<http://www.securiteam.com/securityreviews/5GP0220G0U.html>Mike

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 22:19:25 +0000 (UTC)

> protection against bugs than does fixed length encoding.

Length based framing means that if they output the length as UTF-8 rather 
than as bytes, something that won't be caught if they don't use anything 
other than ASCII, which itself is likely to be a common situation, and if 
their protocol has any aspect which involves echoing text back, then if 
you can trick the page into sending specially-constructed packets, you can 
perform an XSS attack.

Sentinel based framing means that if they output the data without checking 
it for markers, something that generally won't be caught in testing, and 
if they allow content to come from another source than Web Sockets, then 
if you can trick the other source into accepting specially-constructed 
data, and then trick the user into seeing that data, you can perform an 
XSS attack.

Length-based framing requires these stars to line up:
 - they mismeasure text
 - they have a protocol that echoes content
 - you can trick them into sending constructed data

Sentinel-based framing requires these stars to line up:
 - they don't check output
 - they get input from another source
 - you can get data into that other source
 - that other source doesn't check the data's validity
 - you can trick the user into seeing that data

IMHO the former is more likely. The latter requires far more things to go 
wrong.

That isn't to say that sentinel-based framing is a panacea. It's 
vulnerable to poor implementations too. It's just that to take advantage 
of such vulnerabilities, far more things have to go wrong.

> length, the attacker can do bad stuff to [pre]fixed-length framing.

This is about servers, not browsers. Browsers will get plenty of testing 
and won't have these bugs for long enough to matter.

> But, more importantly, [pre]fixed-length framing IS MORE ROBUST against 
> accidental attacks.

I don't buy that.

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 15:50:52 -0700

>  - you can trick the user into seeing that data

You're self-constructing the argument how you like it.   The steps you've
outlined are arbitrary and subjective.

I think we agree that
   - implementations will have bugs
   - any protocol, regardless of framing, is susceptible to attack if
implemented improperly.

You assert that your approach is more secure, but the data doesn't prove
that at all.  Just a lot of opinion.

I think what is really happening is that you simply prefer UTF-8 frames.
 Previously, you said this was about solving the "amateur programmers"
requirement, because that took the argument away from the objective and into
the subjective.  But when the group finally (and painstakingly!) killed that
requirement, you shifted to the security argument.  You're pretending this
is a security argument when it is not, because security would be a good
trump card.  We've proven (and you agree) that bugs make UTF-8 framing
vulnerable to attack just as any other framing.  So, stripped of the
objective argument that it might be more secure, you're now appealing to a
subjective cause again, that although it's not immune to bad
implementations, it is "better" (in Ian's mind). Here, we just disagree, but
nobody can be right, because it's subjective...

Let's call an apple an apple.

You prefer UTF-8 framing.  That's fine.  You should vote that way.

I'm in favor of length delineated frames.

Either choice can be used to build equivalent security.

Mike

---

From: Patrick McManus <[email-masked]>
Date: Fri, 30 Jul 2010 19:23:45 -0400

On Fri, 2010-07-30 at 14:31 -0700, Mike Belshe wrote:

> So let's review:
>    a) On security, the two are the same.  Both approaches are
> vulnerable if implemented incorrectly.

I agree. I would also add

  c) fixed-length framing is considerably more efficient to consume.

More efficient in terms of buffer management, cpu cycles, and cpu cache
friendliness. The latter has been mentioned in this forum (e.g.
zero-copy friendliness is similar, and Willy just made reference to
layered implementations today) but not nearly often enough imo given its
high value to a busy multiplexing server or intermediary. 

I say that with implementation experience on high volume http
intermediaries.

I find the concern that an implementor will mistake the length of data
she writes to the network much less compelling than the concern of
sentinel-conflicting data being written to the network due to the
machinations of a clever attacker.

I do support typing the messages, and I don't have a strong preference
about how that is done (i.e. through a protocol byte and registry,
through a mime type, etc..) but all types should use the same fixed
length and chunked framing.

I would support the insertion of 1-4 "pad" characters of fixed known
value to be placed in between chunks. We can accomplish two things with
this:

1] align the next chunk on a 32 bit boundary, which will create a
processor efficiency for interpreting the frame/chunk header as a number
which can be easily decomposed into bit fields and lengths. I always
appreciate protocols that start objects well aligned.

2] serve as a sanity check on the "I miscalculated my length" concern.
This is basically the equivalent of the poison patterns some access
checkers lay down to detect buffer overruns. 

If you think that #1 is a worthwhile thing, then the cost of #2 is
basically an average of 1 byte a chunk. (because 1/4 of the time you
would need to add 4 pad bytes instead of 0 to an already aligned
buffer).

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 31 Jul 2010 03:49:11 +0200

>Sentinel-based framing requires these stars to line up:

I have three comments on that. The first is that these two are not mutu-
ally exclusive, you can have a prefixed length and a sentinel to verify
that length. Secondly, either we have experience with a comparable pro-
tocol or protocols where it has been found that one is better than the
other, or we would be doing an experiment here with an unclear outcome.
Advocates of passing lengths have made a good case there, pointing at C
strings (even though that is not entirely related); I've yet to hear of
sentinel success stories. If you can demonstrate such experience, that
would put a lot a weight behind your argument.

Thirdly, even if one agrees that for one approach to lead to bugs, more
things have to go wrong than for the other, that does not mean the com-
bination of fewer elements is also more likely in practise and will lead
to greater damages. Passing a character count where a byte count is ex-
pected is a well-known problem, while it is easy to make that mistake,
it is also easy to recognize, even automatically, and avoid; code with
that problem is unlikely to pass a review in an environment where that
is a concern.

The goal is to minimize harm, not to minimize the number of buggy imple-
mentations. I don't care about my name being mangled in the "who is on-
line" widget of some web forum because they passed a character count in-
stead of a byte count, but I do care a lot about bugs on my bank's web
site because they have an off-by-one bug in their buffering search-for-
sentinel algorithm.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 Jul 2010 20:08:35 -0700

> I say that with implementation experience on high volume http

Yes. Same (high transaction rate intermediary/multiplexing server)
experience here, same observation about efficiency.
 -=R

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 31 Jul 2010 07:25:08 +0200

On Sat, Jul 31, 2010 at 03:49:11AM +0200, Bjoern Hoehrmann wrote:
> strings (even though that is not entirely related);

I took C strings as an example of bad use of sentinel-framing. C strings
don't have a length prefix and it's now well known that many C programs
which process user data have buffer overflows, because you can only
detect the end while processing the data, there's usually no easy way to
detect it before. So we're already aware of a large known class of
problems caused by sentinel framing on a receiver side : failing to
properly detect when the data is too large and will finally not fit.
There is nothing in the proposals here to indicate why this would
suddenly disappear with websocket while this problem is fairly common.
And BTW the first example code pointed to by Ian was an example of
program which will get that wrong : it already did not check for the
end of the handshake while looking for strings in the buffer, I don't
see why the same guy would care about the received frames to fit into
buffers while processing them.

> I've yet to hear of

In my opinion, the sentinel can be a good check when the length is
known. But that's all. A length prefix ensures you get the framing
right. After all, length-prefix works like datagrams. As long as the
API makes it easy to send the advertised number of bytes, it is not
even possible to get it wrong. And if the user has to care himself
about encoding, that's where the misplaced ending sentinel will
make wrong messages immediately detectable by the receiver. So I
really don't see the problem with length prefix here.

Willy