Subject: [hybi] Changed framing

From: Ian Hickson <[email-masked]>
Date: Fri, 6 Aug 2010 00:41:41 +0000 (UTC)

I've changed the WebSocket protocol's framing to use a 64 bit length 
prefix instead of the combination of sentinel-based framing and 
variable-width lengths that we had before.

   Text version
   http://www.whatwg.org/specs/web-socket-protocol/

   HTML version - as part of the multipage version of Web Apps 1.0
   http://www.whatwg.org/specs/web-apps/current-work/complete/network.html#websocket-protocol

   HTML version - as part of the one-page version of Web Apps 1.0
   http://www.whatwg.org/specs/web-apps/current-work/complete.html#network

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 11:21:13 +1000

> variable-width lengths that we had before.

Ha! just as I was almost convinced of  variable length!

But unfortunately I have a quibble.  While I appreciate the sentiment
of making to so big that it should not be a problem, I think 64 bits
is perhaps a little too big.  The language I use (java) doesn't have
an unsigned 64 bit integer and I suspect there are other environments
which will also have difficulties with such a value.

Perhaps it should be 63 (or 31) bits, and the high bit is used for the
fragmentation bit.  This would help avoid errors associated with
signed types being used for the signed value.

regards

---

From: Douglas Otis <[email-masked]>
Date: Thu, 05 Aug 2010 19:08:14 -0700

On 8/5/10 6:21 PM, Greg Wilkins wrote:
> On 6 August 2010 10:41, Ian Hickson<[email-masked]>  wrote:
>    
Greg,

The advantage of WS multiple streams or channels is to avoid slow-start, 
connection-setup, and being directed to the correct server which often 
involves other connection specific processes.  These goals would be in 
conflict with attempts to continuously send large files.  The overhead 
related to large file transfers will not benefit much from connection 
optimization.  Even so, optimization from an Internet server perspective 
is likely limited to handling objects roughly equal to that of the 
network MTU which will never exceed 64kB.   Keeping connections active 
by combining message streams also realizes the highest bandwidth 
possible per connection.  The servers that I have been using are not CPU 
bound, where the overhead related to chaining together a multiplexed 
data stream is likely insignificant.

For transfers that need to exchange aggregate data size ahead of 
incremental transfers, could be done within meta-information preceding 
the payload, which could be identified as representing the initial frame.

<state()><type()><channel()><length()> (32 bits)
The initial frame would be a set of tlv structures which could include 
the overall length when needed.

Concatenation as a state could help in dealing with the bundling of 
smaller framed objects.

-Doug

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 05 Aug 2010 21:18:21 -0700

> signed types being used for the signed value.

I think 63 bits is a reasonable choice. However, if we do that, it would be a bad idea to reuse the sign bit for another purpose, particularly if it is initially always zero. That makes it really easy to make a mistake, and write code that gets a negative length, if it is initially only ever tested when the magic bit is 0.

It is kind of sad that Java does not have unsigned integer types. But then again, JavaScript has no integer types at all, and may be limited in practice to 53 bits for some time.

Regards,
Maciej

---

From: Ian Hickson <[email-masked]>
Date: Fri, 6 Aug 2010 07:01:38 +0000 (UTC)

> which will also have difficulties with such a value.

While waiting for Java to get an unsigned 64 bit integer type, you can 
just send a 32 bit zero and a 32 bit unsigned integer. This is only a 
problem if you're sending messages bigger than 4GB, which I assure you no 
browser today is going to support anyway. Similarly when receiving, for 
now you can quite safely just read it as two unsigned 32 bit numbers and 
bail if the first one is non-zero.

(I'd be fine with using variable-width lengths instead. I just went with a 
fixed-width length because people hummed, which apparently is what passes 
for technical arguments here.)

---

From: <[email-masked]>
Date: Fri, 6 Aug 2010 08:50:34 +0100

Well, so much for java. Interesting history at the end of:
http://www.darksleep.com/player/JavaAndUnsignedTypes.html
-- seems implementing any kind of unsigned was just too hard? That would make the suggestion below (32-bit zero and 32-bit unsigned int) unworkable, since you'd lose a bit.

4GB is no longer large enough to cover all possible messages, really; DVD iso's can already be bigger. But taking the overheard of a large length field when you're spending most of the time delivering small frames is not ideal either.

A while back we designed a protocol to deliver very large files using 128-bit integers describing sizes and pieces (while using 64-bit or 32-bit or 16 bit integers for describing smaller files - the choice is indicated by a pair of binary flags up front), and the same suggested 'bail if high bytes is non-zero' has been used in implementation (which later went on to handle 128 bits as two unsigned 64-bit integers and do all the necessary arithmetic). So, yes, variable-width length indicators for varying lengths can be made to work nicely.
http://tools.ietf.org/html/draft-wood-tsvwg-saratoga
On the other hand, that particular example didn't have to worry about browser constraints or amateur programmer requirements. And no-one is writing an implementation in java.

On humming - my understanding from Ian Fette's message is that 
> Sigh, perils of editing. Should read "The hum _was_ agnostic as to whether we use fixed or variable length encoding."

The hum is a summary of responses to technical arguments previously made on the mailing list and in the meeting. Surprising a WG editor doesn't know that...

L.

> for technical arguments here.)

Lloyd Wood
[email-masked]
http://sat-net.com/L.Wood

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Fri, 6 Aug 2010 11:42:21 -0700

On Fri, Aug 6, 2010 at 12:50 AM, <[email-masked]> wrote:

> -- seems implementing any kind of unsigned was just too hard? That would
> make the suggestion below (32-bit zero and 32-bit unsigned int) unworkable,
> since you'd lose a bit.
>
> 4GB is no longer large enough to cover all possible messages, really; DVD
> iso's can already be bigger. But taking the overheard of a large length
> field when you're spending most of the time delivering small frames is not
> ideal either.
>
> A while back we designed a protocol to deliver very large files using
> 128-bit integers describing sizes and pieces (while using 64-bit or 32-bit
> or 16 bit integers for describing smaller files - the choice is indicated by
> a pair of binary flags up front), and the same suggested 'bail if high bytes
> is non-zero' has been used in implementation (which later went on to handle
> 128 bits as two unsigned 64-bit integers and do all the necessary
> arithmetic). So, yes, variable-width length indicators for varying lengths
> can be made to work nicely.
> On the other hand, that particular example didn't have to worry about
> browser constraints or amateur programmer requirements. And no-one is
> writing an implementation in java.
>
> > Sigh, perils of editing. Should read "The hum _was_ agnostic as to
> whether we use fixed or variable length encoding."
>
> The hum is a summary of responses to technical arguments previously made on
> the mailing list and in the meeting. Surprising a WG editor doesn't know
> that...
>
>
Just so that I'm not mis-interpreted here, what I was saying is that the
phrasing of the particular hum put to the mailing list at the top of the
email message you excised my quote from did not mention variable or
fixed-length framing, and instead would work with either.

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>