Subject: [hybi] Framing Proposals - take II

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 12:20:05 +1000

All,

I've updated the proposals with much of the feedback received so far and
also tried a few simplifications plus a few new ideas (eg sub protocol
op-codes).

*Feature Negotiation*
For feature negotiation, the general pattern to apply will be that the
client will send a list of options for a feature in priority order.  The
server will then pick a supported option and reply with the single value to
be used.   If the feature supports asymmetric values, then the server can
respond with 2 values, the first will be the one the server will use when
sending and the second will be the for the client to use.

For example, if the client sent:

  Feature: A, B, C

This says that the client supports A, B and C but that it would prefer to
use A over B, and B over C.  The server could response with

  Feature: B

Which says that feature B will be used by both client and server.  IF
asymmetric values are supported, the server could respond with 2 values:

  Feature: C, B

This says that the server prefers will use C, but that the client will use
B.   The result would be client would use B when sending messages and the
server would use option C when sending messages.

*Proposal : Binary only framing*

The websocket framing be changed to only support binary framing as defined
by

          ws-frame            = frame-flags
                          frame-length
                          octet-data   ; binary data

    frame-flags         = %x00-FF

    frame-length        = %x00-7F
                        | %x80-7F %x00-7F

    octet-data          = *( %x00-FF )
            ; the number of octets is exactly the length determined
            ; by the frame-length

The maximum frame data size is 32k and the minimum frame size is 2 bytes.

[ This frame length is an attempted compromise between those that like the
variable length field because it will often be 1 byte, and those that want a
fixed length.  Maybe this is the best of both, or maybe the worst of both.
Either way, this proposal does not hinge on the exact length encoding picked
]

Unless otherwise specified, the binary content can be assumed to be  UTF-8
encoded text.

The support for this style of framing will be indicated by a websocket
version number (if available) in the handshake of >= 1.0 or some other
distinguishing feature of the handshake (to be decided).

*Proposal : Content Type*

A handshake request and/or response may declare the content type and charset
that will be sent on a connection.  This is NOT a negotiated field and
simply an informational header to allow the application layer to confirm
they have correctly connected to the expected service and for debugging
tools to interpret the frames.  The is no proposed mechanism to advertise
acceptable content-types.

For example, the client handshake could contain

    Websocket-Content-Type: text/xml;charset=utf16

Which declares that the client will be sending UTF-16 encoded XML on the
channel.  The content type can be asymmetric, so the server may declare a
different content type in it's handshake response.

If a websocket endpoint receives a connection with a content type that it is
unable to handle, then it may reject the connection.

*Proposal : Frame Flags*

The frame flags byte will be structured as

+--------+--------+--------+--------+--------+--------+--------+--------+
|chunk   |compress| sub    | op-code                                    |

+--------+--------+--------+--------+--------+--------+--------+--------+

The op-code values are in the range %x00-1F and will be discussed in detail
below. Current allocated op-codes are:

%x00 - Data frame
%x01 - Keep Alive
%x02 - Meta Data
%x1E - Message rejected
%x1F - Final Frame

If the sub (sub protocol bit is set), then the frame is a control frame for
the sub protocol and the allocation and interpretation of the op-code is
dependent on the sub protocol.

*Proposal : Message chunking*

If the highest  order bit of the frame-flags is set, then the websocket
frame contains an incomplete message chunk.    A receiver should should not
consider the message complete until a frame is received with the chunk bit
clear.

An example of a chunked message in three frames is:

    %x80 %x08 Goodbye_
    %x80 %x06 cruel_
    %x00 %x05 world

*Proposal : Frame compression*

If the 2nd highest order bit of the frame-flags is set, then the websocket
frame contains compressed data using the algorithm negotiated in the
headers.

A websocket handshake may include a Websocket-Compression header to
negotiate the algorithm used and asymmetric compression is allowed.

For example if the client sent a handshake with

   Websocket-Compression: lzf,gzip,deflate

to which the server responded with

   Websocket-Compression: gzip

The compressed frames sent  use gzip.

By default, compression dictionaries are shared between all compressed
messages sent in the same direction on a connection, unless otherwise
defined by the particular algorithm specified. If no compression algorithm
is specified, then it is an error to set the 2nd lowest order bit in the
frame type.  The frame must be discarded and the connection may be closed.

[ There is also a reasonable proposal to compress the entire stream, which
may be simpler than frame by frame compression, but would not allow
uncompressed frames to be sent .  If this proposal is preferred, the I
suggest the same negotiation is used, but then there is no need for a
compress bit in the flags ]

*Proposal : Keep-Alive

* A control frame with the op-code of %x01 is a keep alive frame, for
example:

 %x01 %x00

Keep-Alive frames are sent to ensure a connection does not idle out. If a
keep alive frame has data content, then it is assumed to be utf-8 encoded
name/value headers. The following headers are defined:

Period    - The period in seconds before the next keep-alive will be sent
Sequence  - A sequence number that identifies a specific keep-alive message
(used with Ping this can act as a flush/ack of a connection).
Ping      - A boolean, which if true request that a copy of the keep alive
is returned by the other endpoint
Timeout   - The time in seconds that the endpoint will wait for a message
before closing the connection.

*Proposal : Message Meta-Data

* A control frame with the op-code of %x02 is a frame containing meta data
for the next message sent on the connection. The content of the frame will
be utf-8 encoded name/value headers taken from the mime specification.

For example, if a connection is sending images thumbnails, then each image
may need meta data to indicate exactly what image format is being sent, so a
content type may be sent before a data frame as follows

%x02 %x1A Content-Type: image/jpeg CR LF
%x00 %2B %A3 <image data>

Meta-Data headers are not acted on by the protocol and are provided to allow
application layers to communicate additional meta data. If a meta-data frame
contains a content type, there is no requirement that it comply with the
connection content-type previously declared. However it is recommended that
a connection content type is a super type of the individual message types.
For example if a channel is sending image/jpeg and image/gif messages, then
the WebSocket-Content-Type should be image/*

The content of the meta data message relates entirely to the application
layer handling of the next message. Nothing within a message meta data frame
should affect the behaviour of the protocol.

*Proposal: Message Rejected*

A control frame with the op-code of %x1E is an notification that a message
has been rejected by the other end (eg too large, unsupported content type
etc.).  The content of this frame will provide the reason for the rejection
in a format to be determined.  It is the sender of the message that can then
decide if rejection indicates that the connection should be terminated or if
it can continue.   Currently there is no mechanism to indicate which
specific message that was rejected, only that a message was rejected.

*Proposal : Final Frame*

A control frame with the op-code of %x1F indicates the final frame to be
sent on the connection. A minimal final frame should also be a control from
of zero length:

 %x1F %x00

It may also be desirable to define headers that can be sent in the control
frame to give additional information about the reason for the close.

An endpoint that has sent a final frame should close the connection only
once it has received a final frame. There is no time limit for how long a
connection may be half closed (other than normal idle timeouts).

*Proposal : Multiplexing*

The control frame mechanism can be used to implement multiplexing
potentially using several op codes for channel creation and flow control.
However as there are a wide range of multiplexing solutions possible, it is
proposed that these initially be implemented as sub-protocols using the sub
protocol control frames.  If a single multiplexing solution emerges, it may
eventually be allocated unused op-codes in the protocol op-code space.

---

From: "Thomson, Martin" <[email-masked]>
Date: Thu, 29 Jul 2010 16:01:56 +0800

This take II is better, but I still don’t like the three way negotiation.  A server should be unequivocal about what is being used.  It’s easier, faster and it works.

_As of right now_, don’t see the need for a content-type header in the negotiation, or the meta data frame.  Control frames I don’t like either.  I think that means that the reject frame goes away too.

What does that leave?

Binary framing = great.
Keep alive and final frame = great.
Feature negotiation = good, modulo my reservations above

These are things that are important for _right now_.  These are the sorts of things that you can’t shoehorn in later.

However, limit scope.  Otherwise, we have no hope of progressing to Ian’s plan.

Add to this a solid suggestion for mutual authentication of client and server both being websocket-capable and I think we have all that we need.  We need to look at expanding on Adam’s suggestion, which I think has promise.

--Martin

From: [email-masked] [mailto:[email-masked]] On Behalf Of Greg Wilkins
Sent: Thursday, 29 July 2010 4:20 AM
To: [email-masked]
Subject: [hybi] Framing Proposals - take II

All,

I've updated the proposals with much of the feedback received so far and also tried a few simplifications plus a few new ideas (eg sub protocol op-codes).

Feature Negotiation
For feature negotiation, the general pattern to apply will be that the client will send a list of options for a feature in priority order.  The server will then pick a supported option and reply with the single value to be used.   If the feature supports asymmetric values, then the server can respond with 2 values, the first will be the one the server will use when sending and the second will be the for the client to use.

  Feature: A, B, C

This says that the client supports A, B and C but that it would prefer to use A over B, and B over C.  The server could response with

  Feature: B

Which says that feature B will be used by both client and server.  IF asymmetric values are supported, the server could respond with 2 values:

  Feature: C, B

This says that the server prefers will use C, but that the client will use B.   The result would be client would use B when sending messages and the server would use option C when sending messages.

Proposal : Binary only framing

The websocket framing be changed to only support binary framing as defined by

          ws-frame            = frame-flags
                          frame-length
                          octet-data   ; binary data

    frame-flags         = %x00-FF

    frame-length        = %x00-7F
                        | %x80-7F %x00-7F

[ This frame length is an attempted compromise between those that like the variable length field because it will often be 1 byte, and those that want a fixed length.  Maybe this is the best of both, or maybe the worst of both.  Either way, this proposal does not hinge on the exact length encoding picked ]

Unless otherwise specified, the binary content can be assumed to be  UTF-8 encoded text.

The support for this style of framing will be indicated by a websocket version number (if available) in the handshake of >= 1.0 or some other distinguishing feature of the handshake (to be decided).

Proposal : Content Type

A handshake request and/or response may declare the content type and charset that will be sent on a connection.  This is NOT a negotiated field and simply an informational header to allow the application layer to confirm they have correctly connected to the expected service and for debugging tools to interpret the frames.  The is no proposed mechanism to advertise acceptable content-types.

Which declares that the client will be sending UTF-16 encoded XML on the channel.  The content type can be asymmetric, so the server may declare a different content type in it's handshake response.

If a websocket endpoint receives a connection with a content type that it is unable to handle, then it may reject the connection.

Proposal : Frame Flags

The op-code values are in the range %x00-1F and will be discussed in detail below. Current allocated op-codes are:
%x00 - Data frame
%x01 - Keep Alive
%x02 - Meta Data
%x1E - Message rejected
%x1F - Final Frame
If the sub (sub protocol bit is set), then the frame is a control frame for the sub protocol and the allocation and interpretation of the op-code is dependent on the sub protocol.

Proposal : Message chunking

If the highest  order bit of the frame-flags is set, then the websocket frame contains an incomplete message chunk.    A receiver should should not consider the message complete until a frame is received with the chunk bit clear.

    %x80 %x08 Goodbye_
    %x80 %x06 cruel_
    %x00 %x05 world

Proposal : Frame compression

If the 2nd highest order bit of the frame-flags is set, then the websocket frame contains compressed data using the algorithm negotiated in the headers.

A websocket handshake may include a Websocket-Compression header to negotiate the algorithm used and asymmetric compression is allowed.

By default, compression dictionaries are shared between all compressed messages sent in the same direction on a connection, unless otherwise defined by the particular algorithm specified. If no compression algorithm is specified, then it is an error to set the 2nd lowest order bit in the frame type.  The frame must be discarded and the connection may be closed.

[ There is also a reasonable proposal to compress the entire stream, which may be simpler than frame by frame compression, but would not allow uncompressed frames to be sent .  If this proposal is preferred, the I suggest the same negotiation is used, but then there is no need for a compress bit in the flags ]

Proposal : Keep-Alive

A control frame with the op-code of %x01 is a keep alive frame, for example:

 %x01 %x00

Keep-Alive frames are sent to ensure a connection does not idle out. If a keep alive frame has data content, then it is assumed to be utf-8 encoded name/value headers. The following headers are defined:

Period    - The period in seconds before the next keep-alive will be sent
Sequence  - A sequence number that identifies a specific keep-alive message (used with Ping this can act as a flush/ack of a connection).
Ping      - A boolean, which if true request that a copy of the keep alive is returned by the other endpoint
Timeout   - The time in seconds that the endpoint will wait for a message before closing the connection.

Proposal : Message Meta-Data

A control frame with the op-code of %x02 is a frame containing meta data for the next message sent on the connection. The content of the frame will be utf-8 encoded name/value headers taken from the mime specification.

For example, if a connection is sending images thumbnails, then each image may need meta data to indicate exactly what image format is being sent, so a content type may be sent before a data frame as follows
%x02 %x1A Content-Type: image/jpeg CR LF
%x00 %2B %A3 <image data>

Meta-Data headers are not acted on by the protocol and are provided to allow application layers to communicate additional meta data. If a meta-data frame contains a content type, there is no requirement that it comply with the connection content-type previously declared. However it is recommended that a connection content type is a super type of the individual message types.  For example if a channel is sending image/jpeg and image/gif messages, then the WebSocket-Content-Type should be image/*

The content of the meta data message relates entirely to the application layer handling of the next message. Nothing within a message meta data frame should affect the behaviour of the protocol.

Proposal: Message Rejected

A control frame with the op-code of %x1E is an notification that a message has been rejected by the other end (eg too large, unsupported content type etc.).  The content of this frame will provide the reason for the rejection in a format to be determined.  It is the sender of the message that can then decide if rejection indicates that the connection should be terminated or if it can continue.   Currently there is no mechanism to indicate which specific message that was rejected, only that a message was rejected.

Proposal : Final Frame

A control frame with the op-code of %x1F indicates the final frame to be sent on the connection. A minimal final frame should also be a control from of zero length:

 %x1F %x00

It may also be desirable to define headers that can be sent in the control frame to give additional information about the reason for the close.

An endpoint that has sent a final frame should close the connection only once it has received a final frame. There is no time limit for how long a connection may be half closed (other than normal idle timeouts).
Proposal : Multiplexing

The control frame mechanism can be used to implement multiplexing potentially using several op codes for channel creation and flow control.  However as there are a wide range of multiplexing solutions possible, it is proposed that these initially be implemented as sub-protocols using the sub protocol control frames.  If a single multiplexing solution emerges, it may eventually be allocated unused op-codes in the protocol op-code space.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 18:16:15 +1000

Martin,

On 29 July 2010 18:01, Thomson, Martin <[email-masked]> wrote:

>  This take II is better, but I still don’t like the three way
> negotiation.  A server should be unequivocal about what is being used.  It’s
> easier, faster and it works.
>
that is what I proposed.    Client provides a list of options and the server
picks one (or two if asymmetric options are to be supported).   There is no
three way negotiate.

> _*As of right now*_, don’t see the need for a content-type header in the
> negotiation, or the meta data frame.  Control frames I don’t like either.  I
> think that means that the reject frame goes away too.
>

I'm not intending any of these proposals for _right_now_.  In terms of Ians
road map, I think it is impossible to change the framing in a few weeks even
if we all wanted to.

These are strictly for consideration in the 6 month time frame.

Perhaps the semantics of keep-alive and final-frame need to be resolved for
the 4 week goal, but the syntax as I've proposed is not suitable for the
current framing mechanism.

cheers

---

From: John Tamplin <[email-masked]>
Date: Thu, 29 Jul 2010 04:44:36 -0400

> road map, I think it is impossible to change the framing in a few weeks even

How can we extend it later to support the different framing?  If that is
done with a optional header like "WebSocket-Framing: v2" or whatever,
wouldn't that be painful to maintain forever more rather than making the
framing more extensible to start?

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 19:04:40 +1000

> framing more extensible to start?

John,

I'm a soft advocate for a protocol version for this very reason.   Until we
have broad agreement that we have a framing mechanism that will stand the
test of time and will be extensible, then we have to prepare for a breaking
change in the protocol.  A protocol version would allow that - even if we
went from no version to a 1.0 version and then stuck with 1.0 for all
time.   In fact, if we ever had to go to a 1.1, it would mean that we got
the framing/extension mechanisms wrong.

But I'm unclear in Ian's plan what would be called 1.0?  Is that the 4 week
target or the 6 month target?

However, as much as I think we need to change the framing, it is unrealistic
to think that it can be done in less than 6 months, nor should we give up
the opportunity to deploy the current implementations to a wider audience in
the short term, as there will be other issues unrelated to framing that may
be discovered.

cheers

---

From: Dave Cridland <[email-masked]>
Date: Thu, 29 Jul 2010 10:23:53 +0100

> framing more extensible to start?

I agree - I think we need to get the handshake and the framing sorted  
out now, while we have involvement and energy.

I just think of Cookie2.

Doing this inside of four weeks is not actually impossible.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Dave Cridland <[email-masked]>
Date: Thu, 29 Jul 2010 10:23:59 +0100

> also tried a few simplifications plus a few new ideas (eg sub  
This is looking good, thanks. I'm already more comfortable with  
what's being described in this thread compared to the current  
specification.

I've sketched out a few counter-proposals in this message; I've also  
gone back over them as I've caught up on the thread. As such, there  
may be some disjointedness in them...

> server would use option C when sending messages.

I'm still not keen on assymetry, but this is looking better. If your  
proposal hasn't changed, then my reading of it has. You're saying  
that:

client-feature-field-val = option *(sep option)
server-feature-field-val = option [sep option]
sep = "," WSP

Such that servers can signal either one bidirectional feature option,  
or two unidirectional ones.

My only concern is that if implementors typically only see the single  
option value, then they may behave badly - in addition, there's the  
potential for a substantial increase in the numbers of combinations  
that need QA testing.

The first concern could be mitigated by changing to:

server-feature-field-val = option sep option

ie, always having two options explicitly for asymmetric features.

> Either way, this proposal does not hinge on the exact length  

I think this is the worst of both. I've come to the conclusion that  
even in the most drastically resource-tight situations (and mobile  
FACH thresholds would count here), the overhead of a fixed-length  
fixed-length field would be okay.

But I do have an alternative, with which I hope to get a consensus.

> acceptable content-types.

Phrased this way I have no strong objection.

> unable to handle, then it may reject the connection.

I'd actually rather that this were a SHOULD NOT - if you're  
tunnelling through, say, IMAP, this becomes less than useful.

> The frame flags byte will be structured as

I still don't follow the need for per-frame compression.

I'm also, I have to admit, somewhat sceptical on the need for a  
subprotocol non-message, as well as a non-subprotocol data frame -  
I'd be inclined to say that messages are either subprotocol messages  
(ie, destined for the application) or control messages (ie, destined  
for the WebSocket stack).

With that in mind, and in addition thinking about variable length  
encodings:

Bits:
0	-	Fragment [Frames with this flag are non-terminal fragments]
1	-	Sub	[Frames with this flag are for the subprotocol]
2/3	-	2-bit length length, 0-4 octets.
4-7	-	4-bit opcode.

For "control" opcodes, I'd specify:

0xF	-	Extended opcode
0xE	-	(End) Final Frame
0x1	-	Ping Response
0x0	-	Keep Alive

For "subprotocol opcodes", I'd go for:

0x0	-	Text frame (MUST be UTF-8 payload)
0x1	-	Binary frame (MAY not be UTF-8 payload)
0xF	-	Extended opcode

All unextended "short" opcodes are reserved. Subprotocols may use  
extended opcodes for any purpose, however.

Note that I've magicked the length-length, to avoid 3-octet lengths  
in favour of 4-octet lengths:

0	-	No length, no payload.
1	-	1 octet length, 0-255 payload.	(as-is)
2	-	2 octet length, 0-16k payload.	(ntohs())
3	-	4 octet length, 0-64k payload.	(ntohl())

Extended opcodes would mean pulling another opcode in off the header  
for an additional 255 possible extended opcodes. The intent would be  
that most opcodes would be extended unless common use - there's  
actually an argument for making the Final Frame opcode an extended  
one.

Note that this changes the overall ABNF for a frame:

frame = flags [extended-opcode] [length payload]

flags = OCTET
; As above

extended-opcode = %x00-%xFE / %xFF extended-opcode
; Infinite. Possibly silly, I suspect that 270 opcodes is actually  
enough.

length = OCTET / OCTET OCTET / OCTET OCTET OCTET OCTET
; Number of octets depends on length flags.
; Held in network-byte-order.

payload = *OCTET
; Number decoded from length above.

Also, frames now measure from a single octet up to a maximum of 65541  
octets (1 flags, 1 extended-opcode, 4 length, 65535 payload).

> An example of a chunked message in three frames is:
+1

> compress bit in the flags ]
I prefer simple stream compression.

Also:

*Proposal: Inline TLS*

A websocket handshake MAY include a Websocket-Encrypt header to  
negotiate TLS. THe only feature values defined at this stage are  
"TLS" and "None". Inline TLS is a symmetric option; servers MUST NOT  
include more than one option in their response. Clients SHOULD  
attempt to negotiate TLS.

For example, should a client send:

  Websocket-Encrypt: tls,none

And a server respond:

  Websocket-Encrypt: tls

[[ EITHER: ]]

Then a TLS session will be negotiated on the TCP socket immediately  
following the handshake, and frames will be transferred within this  
session.

[[ OR: ]]

Then the initial frames of the WebSocket session MUST be control  
frames of opcode TLSNEGO, containing TLS handshake record to  
negotiate a session. Subsequence frames will consistently contain TLS  
records as payload.

[[ END ]]

I do infinitely prefer the former, I'm just not clear on whether the  
latter would be better for intermediaries.

My rationale for this is that deploying TLS as easily as possible  
seems like a generally useful thing - this method would mean that the  
URI to the WebSocket service would remain unchanged whether TLS is  
deployed or not.

> before closing the connection.
I don't think that these headers are needed. In my proposal above,  
you'll note I've a single-octet Keepalive, and I'm sticking with my  
original proposal of "and if you include a payload it'll be echoed  
back to you".

I'd also say that a ping MUST be responded to prior to processing any  
further frames.

Incidentally, it is critical, as I follow things, to have the  
keepalive frames weigh in at less than 127 octets including TCP and  
TLS overhead, in order to avoid triggering DCH mode on 3G devices.  
TCP+TLS is (if memory serves) 57 octets. This is really my driver for  
minimizing the size of KA, and allowing pings to be arbitrarily small  
(minimum 3 octets) at the sender's control.

(I wrote up some details I'd gleaned from cleverer people on mobile  
stuff as it relates to protocol work for the XSF, currently at  
http://xmpp.org/extensions/inbox/mobile.html - this may help).

> should affect the behaviour of the protocol.
I'm fully ambivalent to this. I would argue that application data is  
entirely at the whim of the application, but I can follow that such a  
structure may be useful. That said, I've included the  
counter-proposal of marking UTF-8 vs binary frame distinctly.

> specific message that was rejected, only that a message was  
I think this is useless without including a message identifier.  
Application errors can be signalled by the application, which might  
have such identifiers.

I do think a more general fatal error signalling frame would be  
interesting, though.

> connection may be half closed (other than normal idle timeouts).

You'll note I've readily included provision in my header  
counter-proposal (in a single octet, although I recognise that's  
possibly a waste).

> eventually be allocated unused op-codes in the protocol op-code  

I did wonder if there was any mileage in attempting this yet.

My inclination would be to allocate out opcodes in the subprotocol to  
indicate that the frame was addressed to a specific endpoint, and  
have the presence of this include an extra octet to describe the  
endpoint (limiting us to only 255 tabs open).

But I disliked the idea of an opcode causing an additional qualifier  
in the frame header, and in addition, I wanted to allow subprotocols  
to allocate their own opcodes, and finally, what if you want to send  
an opcode to an endpoint?

I do note that it's possible to repurpose the fragment bit if we use  
another method there. Two that spring to mind are either use of an  
opcode for this (which breaks my design rule above), or use of a  
maximal length. The latter breaks things twice - you'd need a  
zero-length frame following a real maximal length, and in addition it  
supposes that the sole purpose in chunking is to break the length  
boundary.

The final option, therefore, is a repurposing of an opcode bit,  
halving the number of "short" op-codes. This is physically worse that  
allocating a special opcode, of course, but allows for a bit more  
flexibility. Note that this only lowers the number of opcodes by 8,  
and I expect that we can identify the opcodes needed to be "short"  
very quickly.

This essentially means that the frame would have an optional,  
single-octet, endpoint identifier - in a browser, I'm assuming that's  
effectively a document object (tab, iframe, whatever). Who knows what  
this means in the server-side. :-)

I imagine that we'd also need an opcode or two to signal the  
availability of an endpoint.

Dave.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 18:47:33 +0900

> target or the 6 month target?

Good point. Even if version numbers are a bad idea for extensibility and 
such, the press, the publishers, and so on will very much ask for them 
(or some equivalent textual labels). So we should think about what 
labels to give Ian's 4-week and 6-month 'versions'.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 18:54:22 +0900

Overall, I like the proposal below quite a bit, in particular the
 > 0x1 - Binary frame (MAY not be UTF-8 payload)
part. But while I see some good reasons for every part, I still get a 
feeling that some of this should be even simpler.

Regards,    Martin.

On 2010/07/29 18:23, Dave Cridland wrote:

> Note that I've magicked the length-length, to avoid 3-octet lengths in
> favour of 4-octet lengths:
>
> Extended opcodes would mean pulling another opcode in off the header for
> an additional 255 possible extended opcodes. The intent would be that
> most opcodes would be extended unless common use - there's actually an
> argument for making the Final Frame opcode an extended one.
>
> ; Infinite. Possibly silly, I suspect that 270 opcodes is actually enough.
>

---

From: Julian Reschke <[email-masked]>
Date: Thu, 29 Jul 2010 11:55:48 +0200

On 29.07.2010 11:47, "Martin J. Dürst" wrote:
> ...
> ...

WSv4 and WSv6?

Julian (ducks)

---

From: Arman Djusupov <[email-masked]>
Date: Thu, 29 Jul 2010 13:18:08 +0300

> be utf-8 encoded name/value headers taken from the mime specification.

Wouldn't it be more effective if the meta-data frame applied to all subsequent messages, up until the next meta-data frame?

With best regards,
Arman

---

From: Dave Cridland <[email-masked]>
Date: Thu, 29 Jul 2010 11:32:28 +0100

On Thu Jul 29 10:54:22 2010, "Martin J. Dürst" wrote:
> part. But while I see some good reasons for every part, I still get  
> a feeling that some of this should be even simpler.
> 
> 
Feel free - indeed encouraged - to take the proposal and bend it into  
a shape more to your liking.

That's all I've done with Greg's, after all.

I'm hoping nobody's wedded to anything at this stage.

Dave.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 21:14:22 +1000

On 29 July 2010 19:23, Dave Cridland <[email-masked]> wrote:

>>  If the feature supports asymmetric values, then the server can
>> I'm still not keen on assymetry, but this is looking better.

I'm not that keen on asymmetry either.  If we have multiplexing, then we
definitely don't need it. So let's drop asymmetry for now  - but we have a
place holder for it, if it proves necessary in future.

> If your proposal hasn't changed, then my reading of it has. You're saying
> that:
>
> Such that servers can signal either one bidirectional feature option, or
> two unidirectional ones.
>

Correct - I did change the proposal. The server just declares what will be
used and does not advertise what it could do.

> I still don't follow the need for per-frame compression.

My only concern with connection compression is that it may make the life of
intermediaries more difficult if they are trying to do things like
aggregation and/or load balancing.  But I don't think that is a huge hurdle.
If others prefer connection compression to frame compression, then I'm happy
either way.

> I'm also, I have to admit, somewhat sceptical on the need for a subprotocol
> non-message, as well as a non-subprotocol data frame - I'd be inclined to
> say that messages are either subprotocol messages (ie, destined for the
> application) or control messages (ie, destined for the WebSocket stack).
>

I see three destinations for a message:
 1) a data message intended for the primary application AP
 2) a control message for the protocol stack
 3) a sub protocol message

> 2/3     -       2-bit length length, 0-4 octets.

ARGH!!! another length encoding proposal :-)

> 0x1     -       Binary frame (MAY not be UTF-8 payload)

I can live with this (while not thinking it strictly necessary) and it
appears to be to the liking of thers

> 0xF     -       Extended opcode
>
> All unextended "short" opcodes are reserved. Subprotocols may use extended
> opcodes for any purpose, however.
>

I can live with any of the alternatives, so I'll follow the herd on this
one... but if I had to pick I'd probably go for yours because it gives a
minimal frame size of 1 byte!

> Extended opcodes would mean pulling another opcode in off the header for an
> additional 255 possible extended opcodes. The intent would be that most
> opcodes would be extended unless common use - there's actually an argument
> for making the Final Frame opcode an extended one
>

That a good get-out-of-goal-free card for future extension.

> *Proposal: Inline TLS*

Maybe it is just a connection encoding that could be gzip or TLS?

>  *Proposal : Keep-Alive
>>
>>  I don't think that these headers are needed. In my proposal above, you'll
> note I've a single-octet Keepalive, and I'm sticking with my original
> proposal of "and if you include a payload it'll be echoed back to you".
>

I'd be fine with a 1 byte keep-alive.

> structure may be useful. That said, I've included the counter-proposal of
> marking UTF-8 vs binary frame distinctly.
>

If binary data is to be useful, then we need as much infrastructure and
frameworks to help with it, plus appending MIME headers to data is such a
common thing to do and is widely understood.

> I did wonder if there was any mileage in attempting this yet.

almost certainly.  I'm torn between my thoughts that it is necessary vs my
thoughts that it will be more complex than expected.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 21:17:03 +1000

On 29 July 2010 20:18, Arman Djusupov <[email-masked]> wrote:

> Wouldn't it be more effective if the meta-data frame applied to all
> subsequent messages, up until the next meta-data frame?
>

I originally proposed it like that - ie it altered the default state of the
connection.  But there were objections to a stateful connection.   So I've
now proposed a per message meta data mechanism.

I guess there might be scope to have op codes for both, but as there is
still some resistance to any form of meta-data I don't want to advocate an
over complex solution.

regards

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 29 Jul 2010 14:10:51 +0200

I think what we should focus on now is coming to agreement for the  
handshake and framing. Most other details can be added later. I tried to  
evaluate your proposals from this perspective. In the future it would help  
if instead of just feature proposals you would cast them in terms of use  
cases. Going from use cases to solutions is much more effective than the  
other way around.

> *Feature Negotiation*

If it turns out this is needed this can be added later as fields can be  
added by the client and/or server that will be ignored by existing  
clients/servers.

> *Proposal : Binary only framing*

This is something we should settle on within the next four weeks, imo.  
Making a backwards incompatible change to the framing is just not worth it.

> The websocket framing be changed to only support binary framing [...]

I kind of like the sentinel-delimited frames so you do not have to  
calculate lengths when sending from the server and can just write to a  
buffer, but I do not feel very strongly about it.

> The maximum frame data size is 32k and the minimum frame size is 2 bytes.

I think constraining the size is a bad idea. It is very likely we'll be  
sending way more traffic in the future and if that then would require lots  
of frames that would be a shame and not very efficient.

> *Proposal : Content Type*

I don't see why this is needed at all. Frame types handle this and more  
fine-grained control can be done on the application layer. We are trying  
to stay close to the TCP layer.

> *Proposal : Frame compression*

Others have commented on these.

> * A control frame with the op-code of %x01 is a keep alive frame, for

Since unknown frame types are supposed to be ignored this can easily be  
added later.

> *Proposal: Message Rejected*

I don't see why these cannot be handled at the application level.

> sent on the connection. A minimal final frame should also be a control  

How is this different from the closing frame currently in the  
specification?

> *Proposal : Multiplexing*

This can also be added later, if it turns out it is needed.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Dave Cridland <[email-masked]>
Date: Thu, 29 Jul 2010 13:43:06 +0100

On Thu Jul 29 13:10:51 2010, Anne van Kesteren wrote:
> I think constraining the size is a bad idea. It is very likely  
> we'll be  sending way more traffic in the future and if that then  
> would require lots  of frames that would be a shame and not very  
> efficient.
> 
> 
The message size isn't constrained, as I think you follow, due to  
chunking. With Greg's proposal as-is, this means that with huge  
messages, you're looking at a 3/32767 overhead, or 0.009%. With my  
alternative, it's 5 or 6 per 65535 bytes of payload, which is  
approximately the same (0.008% - 0.009%). The worst cases are at 32k  
and 64k respectively, where overhead becomes 0.0122% in both cases  
(assuming, in my case, that there is an extended opcode in play as  
well), due to a 1-octet frame. The overheads are actually higher with  
smaller data, where they reach this level at 24576 and 49152 octets  
respectively.

Compare this to TCP+TLS, where the typical packet size is around  
1500, with (as I recall) a 57 octets overhead, giving a 3.8% overhead  
in the best case. This overhead will dominate in terms of bandwidth.

> Others have commented on these.

It'd still be useful to have your input - it's not clear from this  
whether you're in favour, violently against, etc.

> Since unknown frame types are supposed to be ignored this can  
> easily be  added later.

I'm not so sure. Long-running protocols over mobile (and through home  
NAT gateways) need such things, unfortunately. Very easy to specify  
now.

> This can also be added later, if it turns out it is needed.

Possibly.

I'd really appreciate a discussion of requirements, since that would  
allow me some idea of whether it's practical to bolt on afterward, or  
whether we need to tackle this now.

Dave.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 29 Jul 2010 14:56:49 +0200

On Thu, 29 Jul 2010 14:43:06 +0200, Dave Cridland <[email-masked]>  
wrote:
>> I think constraining the size is a bad idea. It is very likely we'll  
>> be  sending way more traffic in the future and if that then would  
>> require lots  of frames that would be a shame and not very efficient.
>
> approximately the same (0.008% - 0.009%). The worst cases are at 32k and  
> 64k respectively, where overhead becomes 0.0122% in both cases  
> Compare this to TCP+TLS, where the typical packet size is around 1500,  
> with (as I recall) a 57 octets overhead, giving a 3.8% overhead in the  
> best case. This overhead will dominate in terms of bandwidth.

Well yes, but having to use chunking is not exactly elegant, imo, and  
makes writing away very large sets of data more complicated. I rather have  
something that can handle the transfer of petabytes in the initial version  
and add chunked encoding later based on needs of applications written on  
top of the initial version.

> whether you're in favour, violently against, etc.

Fair enough. I do not think frame flags are needed other than frame types  
the current proposal has and compression of any kind can wait until later.

>> Since unknown frame types are supposed to be ignored this can easily  
>> be  added later.
>
> NAT gateways) need such things, unfortunately. Very easy to specify now.

It's also very easy to add later, no? To me that seems better. As our  
solution will be even better informed and the initial version stays  
simpler.

> whether we need to tackle this now.

This can be easily done in JavaScript using SharedWorker. Again it seems  
better to figure out if this is an actual roadblock in practice and  
address it based on experience than making the initial version more  
complicated than it has to be.

---

From: John Tamplin <[email-masked]>
Date: Thu, 29 Jul 2010 09:13:36 -0400

On Thu, Jul 29, 2010 at 8:56 AM, Anne van Kesteren <[email-masked]> wrote:

>> chunking. With Greg's proposal as-is, this means that with huge messages,
>> you're looking at a 3/32767 overhead, or 0.009%. With my alternative, it's 5
>> or 6 per 65535 bytes of payload, which is approximately the same (0.008% -
>> 0.009%). The worst cases are at 32k and 64k respectively, where overhead
>> becomes 0.0122% in both cases (assuming, in my case, that there is an
>> extended opcode in play as well), due to a 1-octet frame. The overheads are
>> actually higher with smaller data, where they reach this level at 24576 and
>> 49152 octets respectively.
>>
>> with (as I recall) a 57 octets overhead, giving a 3.8% overhead in the best
>> case. This overhead will dominate in terms of bandwidth.
>>
>
> Well yes, but having to use chunking is not exactly elegant, imo, and makes
> writing away very large sets of data more complicated. I rather have
> and add chunked encoding later based on needs of applications written on top
> of the initial version.

Having to count everything up front is rather expensive for the sender.
 Imagine compressing this petabyte frame -- I would have to complete
compressing it before I could send the first compressed byte.  With chunked
encoding, I could send each buffer as it fills up.

---

From: Arman Djusupov <[email-masked]>
Date: Thu, 29 Jul 2010 16:19:09 +0300

> best case. This overhead will dominate in terms of bandwidth.

The overhead is not a big problem. But in the case of a single frame messages it is quite comfortable to know the complete message size prior to reading it so that memory for the message can be allocated right way (or the message refused). Otherwise even relatively small messages are going to be framed and will require pulling frames one by one and growing the buffer.

Arman

---

From: Arman Djusupov <[email-masked]>
Date: Thu, 29 Jul 2010 16:38:04 +0300

> acceptable content-types.

I suggest that sides advertise the content-types they accept and the content-types they send in two separate headers, in a manner similar to the one that has worked out for HTTP.

For example, an application sends JSON or text/xml request messages and is ready to accept image/*,application/json,text/xml responses. In such a case the client advertises the supported formats like this:

Websocket-Content-Type: application/fastinfoset,application/json,text/xml
Websocket-Accept: application/fastinfoset,application/json,text/xml,image/jpeg

Then the server evaluates the client handshake and defines what shall be used by the client, say:

Websocket-Content-Type: image/jpeg
Websocket-Accept: application/json, text/xml

The offer sent by the server is final. If the client does not accept it must drop the connection.

When the header specifies more than one content types, the content-type must be explicitly specified in the message meta-data. When the header specifies a single content type in one of those headers, the given content-type is considered as being the default for each message that does not have meta-data.

This approach would clearly define which content types are expected to be exchanged and when the content type is expected to be specified in the message metadata. It also gives each side the ability to impose a single-content-type symmetric scenario.

Arman

---

From: Arman Djusupov <[email-masked]>
Date: Thu, 29 Jul 2010 16:38:25 +0300

Arman

---

From: Dave Cridland <[email-masked]>
Date: Thu, 29 Jul 2010 15:21:56 +0100

On Thu Jul 29 14:38:25 2010, Arman Djusupov wrote:
> I suggest that sides advertise the content-types they accept and  
> the content-types they send in two separate headers, in a manner  
> similar to the one that has worked out for HTTP.

There's really no need for this - both ends know what content type  
should be transferred anyway. It's the lower levels and third parties  
which may need this information.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Thu, 29 Jul 2010 16:38:20 +0100

> place holder for it, if it proves necessary in future.

That works for me.

> used and does not advertise what it could do.

That's good.

> If others prefer connection compression to frame compression, then  

If compression (and TLS) are negotiated at stream (ie,  
higher-than-frame) level, then intermediaries are the endpoint of the  
compression (and/or TLS), and can still do this, I think.

> subprotocol
Oh. My understanding of "subprotocol" was essentially (1). I may have  
misunderstood something. I was mostly using the opcode facility to  
provide "interesting" things for applications to use. That and I  
thought that 2-octet frames for various weird application-layer  
signalling tasks might be a handy tool in the box.

> ARGH!!! another length encoding proposal :-)
Yeah, sorry... Well, it seems to cover all the requirements short of  
infinite length frames, which I think chunking addresses much more  
effectively.

> appears to be to the liking of thers

That's more or less my take on it too. I don't feel strongly against  
this, and others feel strongly for.

> minimal frame size of 1 byte!

Right, and it also uses a stock encoding for which there are library  
calls to decode - which may be no-op - in every sane language and on  
every platform.

> That a good get-out-of-goal-free card for future extension.

I'll claim it's my idea, and not in any way a direct copy of RFC 861;  
it's actually older - NIC 16239 proposed the idea, and that was  
around spring 1973 I think, making it difficult for me to claim to  
have written it...

> Maybe it is just a connection encoding that could be gzip or TLS?

In principle, TLS can negotiate compression, yes. I think the  
situation may be that most TLS stacks can, now, do compression. The  
problem is there certainly used to be a marked tendency for the  
devices most in need of it to be missing it.

> counter-proposal of

Yes, but attaching the MIME headers directly to the data packets  
isn't - there's a lot of MIME-based protocols where the MIME data is  
transferred at a different point to the data it refers to. IMAP  
springs to mind as an obvious case in point - the MIME data is part  
of the BODYSTRUCTURE, and you're unlikely to fetch that at the same  
time as the parts (since it tell you about the existence of those,  
too).

In any case, I'm not strongly against this, I just note that the  
payload data will be mostly interpreted by the client, and the client  
will (presumably) be downloaded from the same source as the service  
itself.

> >> However as there are a wide range of multiplexing solutions  
> possible, it
> >> is
> >> proposed that these initially be implemented as sub-protocols  
> using the
> >> sub
> >> protocol control frames.  If a single multiplexing solution  
> emerges, it
> >> may

If it's more complex than expected, that suggests that support at the  
framing level will be required, and we need to figure this out now.

Dave.

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 29 Jul 2010 13:38:11 -0700

Dave Cridland wrote:
> subprotocol non-message, as well as a non-subprotocol data frame - I'd 
> be inclined to say that messages are either subprotocol messages (ie, 
> destined for the application) or control messages (ie, destined for 
> the WebSocket stack).

+1

> 2/3    -    2-bit length length, 0-4 octets.

+1  This is fine.  A single fixed length would be fine, too.

> 0x1    -    Binary frame (MAY not be UTF-8 payload)

+1 based on the browser requirement clarifications

> actually an argument for making the Final Frame opcode an extended one.

+1 for the extension

I'm not sure it would be a good idea to expose those subprotocol opcodes 
to an application, though.

> original proposal of "and if you include a payload it'll be echoed 

+1 The payload proposal avoids the need for separate ping vs keepalive 
opcodes which is nice.

> indicate that the frame was addressed to a specific endpoint, and have 
> the presence of this include an extra octet to describe the endpoint 
> (limiting us to only 255 tabs open).
>

I was thinking a switch-channel opcode would avoid the need to modify 
the subprotocol framing by keeping the channel-id outside of the 
subprotocol frame itself, but switch-channel wouldn't address the 
send-an-opcode-to-an-endpoint issue. For example, I think a 
switch-channel opcode would need to add extra opcodes to resolve 
channel-close vs connection-close.

-- Scott

---

From: "Alexander Philippou" <[email-masked]>
Date: Fri, 30 Jul 2010 11:05:56 +0300

> There's really no need for this - both ends know what content type should be transferred anyway. It's the lower levels and third parties which may need this information.

No. The whole point of content negotiation is that the two sides do not know the content type beforehand. Content negotiation offers the flexibility to support multiple encodings and to dynamically select the most appropriate/efficient one depending on runtime conditions so that different clients may use different encodings to access the same service. This is extremely useful when some clients support more encodings than others (more capable clients can be supported without excluding lesser capable clients), or when clients are communicating over different connections (a highly compact but slow encoding might not make sense in a lan but might be great for gprs), or in order to have the ability to introduce a new encoding on the server without breaking interop with existing clients. As for using asymmetric content types, this is useful when the connection is asymmetric (think adsl, you want minimum message size when sending in the low bandwidth direction) or when you have different processing resources available at each side (some encodings are symmetric, some are asymmetric: slow to encode, fast to decode). Greg's proposal includes metadata that can specify the content type on a per-message basis. It is good to have a mechanism that specifies when the default encoding applies (in which case no metadata need to be present) and when the message is to include the content-type into its metadata.

Alexander