Subject: [hybi] 64-bit maximum message size

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 03:53:58 +0100

Greg Wilkins wrote:
> On 6 August 2010 11:28, Jamie Lokier <[email-masked]> wrote:
> > I.e. it just means "don't mandate that chunks must be small".
> > It's not asking much.
> 
> So is 16 exebytes enough for you (see Ian's email).
> Not sure how I'm going to write a unit test for that one:)

Ian's email pleases me.

I'm inclined to variable-length encoding to keep short messages short,
but it is a trivial style issue.

For split chunks, 24-bit works well enough for chunks from a
sendfile-ish perspective and current hardware, but I like the idea of
splitting being _optional_ for the sender, and that's not enough as a
message size.

Any arbitrary limit on message size, that is actually reachable, will
bite someone.  Even if it's just HTML fragment updates to a page,
someone will write one that's just a bit bigger than expected, and the
whole software stack should not fail just because of this spec.

32-bit is reachable when throwing around arbitrary blobs of data these
days (and especially over the next 10 years); 64-bit is not.

Ian's suggestion, settling at 64/63-bit sounds good to me, since the
whole world has settled on that as maximum file size, and it's
starting to get sensible for string implementations to use a 64-bit
length, so it's a natural match for "write file length, write file
contents" type of ultra-simple code.

It could bite in the browser receiving a 2^36 byte message, though.
Will the browser run out of memory and crash? :-)  Seriously, browsers
will have to take a bit of care.

Unit testing >32-bit sizes is realistic these days.  It doesn't take
too long to shift that much data over a local connection.  If >32-bit
works, chances are the implemention got large sizes right, except
maybe close to the limits which aren't going to be of practical
concern.

Some languages can't process >32 bits.  But then, some can't process
>24 bits (Emacs Lisp), and some get inaccurate around 40 bits (Perl
when not compiled with 64-bit integer support, so switches to IEEE
doubles).  Core infrastructure should be absolutely fine with it,
especially if 64-bit is mentioned in the spec.

One added bonus of 64-bit size, as opposed to say 32-bit or 48-bit:

People wanting a "pure byte stream" from their WebSocket could,
legitimately, just send a 2^63-1 sized chunk in both directions, and
then stream their bytes as much as they like.  (Actually I think we
should define 2^63-1 == Infinity, which would make no difference in
practice, except for very long lived connections on extremely fast
networks of the future.)

Naturally, such "pure byte streams" will multiplex, compress
etc. perfectly well through any endpoint/proxy that happens to do so,
just like any other message :-)

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 14:10:31 +1000

> but it is a trivial style issue.

I agree.

I happy with either:
 + variable length.
 + a very long size so that it is never ever a practicable problem

I'd accept
 + a very short size so that chunking is almost always used and thus
is well tested

No so keen on
 + something in between (eg 24 bits).

> days (and especially over the next 10 years); 64-bit is not.

agreed.

But 63 bits is also not reachable and does not suffer sign issues.
56 bits is also beyond immediate reach and would result in a 64bit
total header and no sign issues.

cheers

---

From: Pieter Hintjens <[email-masked]>
Date: Fri, 6 Aug 2010 11:24:22 +0200

>  + a very long size so that it is never ever a practicable problem

I'd point you to http://rfc.zeromq.org/spec:2

Short frames have a 1-byte size, longer frames (255 octets or more)
have a 64-bit size.  Trivial to read and write.

-Pieter