Subject: [hybi] frame length encoding

From: John Tamplin <[email-masked]>
Date: Fri, 20 Aug 2010 15:08:08 -0400

>From the discussions related to my previous proposal, it seems like how we
encode the length is the remaining contentious issue (at least there were no
other comments on the rest of the 4 issues I outlined).  It is hard to get a
sense of who prefers what to what extent.  Also, in a comment not intended
as a serious proposal I mentioned a 1/2/8 option, I had a couple of people
reply to me privately that they liked the idea.  So, I thought I would ask
for a "show of hands" about three possibilities.  If you feel strongly for
or against a particular option, please reply and say so.  If you are fine
with any of the three, then there is no need to reply.

A sample code snippet (not suggesting this is optimal for any of the cases)
is included to make sure what is intended is clear.

Option 1
========
Use 7 bits in the frame header, and if the value is 0-126, that is the
length.  If the value is 127, the next 8 bytes are a 63-bit length (the high
bit must be zero).

long long len = readByte();
if (len == 127) {
  len = readLongLong();
}

Option 2
========
Use 7 bits in the frame header, and if the value is 0-125, that is the
length.  If the value is 126, then the next 2 bytes are a 16-bit length.  If
the value is 127, the next 8 bytes are a 63-bit length (the high bit must be
zero).

long long len = readByte();
switch (len) {
  case 126:
    len = readUnsignedShort();
    break;
  case 127:
    len = readLongLong();
    break;
}

Option 3
========
Use 8 bits in the frame header (assigning RSV2 in the most recent proposal
to be part of the length field).  If the high bit is set in that byte, shift
the current length seven bits and read the next byte.  Leading 0x80 bytes
are disallowed, and at most 9 length bytes may be included

int byte = readByte();
if (byte == 0x80) // fail
long long len = byte;
for (int i = 1; (byte & 0x80) && i < 9; ++i) {
  byte = readByte();
  len = (len << 7) | (byte & 0x7f);
}
if (byte & 0x80) // fail

I would be willing to live with any of the three personally.  My preference
is roughly even for #1 or #2 over #3 because #3 uses one of our reserved
bits (reducing expansion options) and has error states which must be
checked.  Also, most of the lengths don't buy you much in terms of reducing
overhead, as the overall frame is much longer than the header at that point
anyway.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 20 Aug 2010 15:20:44 -0400

>     len = readUnsignedShort();

+1

Much superior than Option 1.

Reason is because with Option 1 overhead for small packets 127+ in size,
is significant (6% not including the rest of header).  And this Option 2,
only slightly increases the overhead for only one size value, 126.

---

From: gustav trede <[email-masked]>
Date: Fri, 20 Aug 2010 21:23:01 +0200

> long long len = readByte();
Strong negative feeling against option 1

>     len = readUnsignedShort();
Strong positive feeling for option 2.

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 20 Aug 2010 21:29:27 +0200

Hi John,

>     len = readUnsignedShort();

This looks really nice and optimal to me !

+1 !

Willy

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 20 Aug 2010 12:48:21 -0700

+1

Looks good to me.

-- Scott

---

From: Roderick Baier <[email-masked]>
Date: Fri, 20 Aug 2010 22:49:25 +0200

+1

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 21 Aug 2010 16:31:17 +0200

Option 1 is simplest and sufficient for all use cases. There is no advantage
in a more complex algorithm and non-trivial costs in complexity, and
inefficiency for frames of 125-254 bytes if we use option 2.

Option 1 gets my vote.

-Pieter Hintjens

Sent from my Android mobile phone.

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 10:56:22 -0400

> inefficiency for frames of 125-254 bytes if we use option 2.

At least 267% incorrect.

The Option 2 inefficiency is only 3/125 to 3/254, which is 2.4% to 1.2%.

Whereas, Option 1 is 8/127 to 8/254, which is 6.3% to 3.1%.

Also Option 1 increases inefficiency by 267% from 3/255 to 8/255 and all
values between to 2/65535 to 8/65535.

Also Option 1 causes a big discontinuity in inefficiency at 126, and that
could be a source of random performance patterns, which is hard to
quantify.

Also I want to suggest that we change Option 2 to re-use the 7 bits from
the first byte instead of adding 2 bytes, then we only add 1 byte.  The
only disadvantage is the values from 32768 to 65335 which incur a 0.002%
inefficiency.  But then our efficiency for 125 to 32767 will increase by
50%!!

So I think it is a "no brainer" that we adopt my modification for Option 2:

long long len = readByte();
switch (len) {
  case 126:
    len = ((len & 0x7f) << 8) + readByte();
    break;
  case 127:
    len = readLongLong();
    break;
}

>>>> length. If the value is 126, then the next 2 bytes are a 16-bit
>>>> length.
> If

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 11:14:03 -0400

Correction:

if (len >= 128)
else if (len == 127)
    len = readLongLong();

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 12:15:42 -0400

>     len = ((len & 0x7f) << 8) + readByte();

Ok, so you are suggesting using RSV2 to differentiate 7/63-bit from 15-bit
lengths, and then within the 7/63-bit option using a value of 127 to do so,
right?

I still dislike using up one of the reserved bits, as it reduced our
expansion options.  Also, as posted previously, I think once compression is
supported, an overwhelming majority of real-world WebSocket messages will
fit in ~126 bytes.  That is why I am mostly indifferent between option 1 and
2 -- I don't think the 2-byte length value will get used very often (I
expect mostly small frames with some very large frames where an entire file
is sent in one message), but the cost of adding it over option 1 is
relatively small (packets of exactly 126 bytes now take 2 bytes longer,
everything else is the same).

I do prefer your proposal over Option #3, which also uses one reserved bit,
but otherwise I still prefer either #1 or #2 because I think removing 1/3 of
our expansion capability is excessive for what is gained.

The reason I feel strongly about keeping reserved bits available for
expansion, is that we have not yet agreed on how extensions should work.  If
we decide to go the nested opcode approach (an extension opcode uses the
first part of the payload data for its extension data, then the next opcode
etc, until you get to a "data" opcode that uses what is left), then I think
we are going to need to assign both RSV1 bits to the opcode to be able
to accommodate extensions there (remember that we don't want to have to
change the base framing protocol at all and this protocol will hopefully be
used for a long time).  If we decide to have per-frame compression
(certainly still up for debate), then we will likely need a single bit per
frame to indicate compression efficiently (there are certainly other ways to
do it as well).  If we decide to have separate extension metadata, we will
need a bit to indicate an extension length and extension data is present.
 And those are just the extensions we have been talking about the past few
months -- who knows what things we may come up with in the future that might
be more efficiently implemented if we assign one of the reserved bits rather
than an extension data block or an opcode?

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 21 Aug 2010 18:37:58 +0200

On Sat, Aug 21, 2010 at 4:31 PM, Pieter Hintjens <[email-masked]> wrote:

Sorry, John.  Here was the original 8/64 proposal that for me remains
the simplest model:

Option 0
========
Use 8 bits in the frame header, and if the value is 0-254, that is the
length.  If the value is 255, the next 8 bytes are a 64-bit length.

if (len == 255) {
  len = readLongLong();
}

-Pieter

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 13:31:52 -0400

> lengths, and then within the 7/63-bit option using a value of 127 to do

Yes.

I assume that reserved bit is in the size field because all the other
8-bit groups of bit fields were allocated? That wastes 7-bits. We could
recover the 7-bits and keep the reserved bit, if we are willing to have a
header which is not byte aligned (1 extra bit in length), but the code
gets more convoluted:

long long read = readUnsignedshort();
long long len = read >> 7;
read &= 0x7f;
if (len >= 128)
    len = ((len & 0x7f) << 8) + (read << 8) + ((read = readByte()) >> 7);
else if (len == 127)
   len = (read << 8) + ((read = readLongLong()) >> 7);
read &= 0x7f;

> expansion options.

Why did you decide to put a reserved bit in middle of a size field if that
bit is not already allocated to specific use?

Apologies I have not yet read the entire frame header design discussion.

>  Also, as posted previously, I think once compression
> is

And just in case there is ever a use case for the frames larger than 126
but less than 65535 in size.

Agreed it is only the 126 frame length that has any penalty.

And if we don't need the extra reserved bit (or if we can use a 1 bit
longer header), then there is no penalty at all.

But I guess the logic is that it is better throw away 7 bits for the 126
to 65535 frame byte sizes, than to add 1 bit to frame smaller than 126
bytes in size?

7 bits / (126 x 8 bits) = 0.7%

The break even point is 126 / 7 = 18 byte frame sizes.  We would need to
know the distribution of frame sizes we expect to make a decision of what
is more cost effective.  But it is minor in either case, and the bit
twiddling for non-byte aligned headers has some cost mostly in terms of
software error.

> I do prefer your proposal over Option #3, which also uses one reserved

Agreed, gets rid of the error case.

> our expansion capability is excessive for what is gained.

Okay but we could get it back if we want to have non-byte aligned headers.

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 13:34:58 -0400

Okay now I understand and retract my "incorrect" word..  There was no way
I could know what you meant (well perhaps I could have deduced you might
have been talking about a different option 0).

John, maybe you need to restart the voting with all 5 options, and
reference your points about the reserved bit value?

---

From: Daniel Stenberg <[email-masked]>
Date: Sat, 21 Aug 2010 20:39:20 +0200 (CEST)

On Fri, 20 Aug 2010, John Tamplin wrote:

> I thought I would ask for a "show of hands" about three possibilities.  If 
> you feel strongly for or against a particular option, please reply and say 
> so.  If you are fine with any of the three, then there is no need to reply.

I think I would prefer a plain 32 bit size, with no special treatment for 
smaller fragments.

-- 

  / daniel.haxx.se

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 14:55:52 -0400

> I think I would prefer a plain 32 bit size, with no special treatment for

See previous discussions -- many people thought it was unacceptable to have
to fragment a message which was already entirely in memory or a disk file
which could be sent with sendfile.  My conclusion from that discussion was
that there was no way we were going to achieve consensus with anything less
than a very large maximum frame size.

Aside from that, your suggestion more than doubles the frame size for small
frames.  As previously indicated, I think that is important because traces
of real-world WebSocket apps show that once compression is added, 98-99.9%
of frames will fit in approximately 126 bytes.  Paying three extra bytes for
the most common frames in exchange for a small gain in simplicity seems a
poor trade-off.

Combining these requirements seems to indicate that some form of encoding
the length in a variable number of bytes is required -- we are debating the
best way to do so.

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 15:07:30 -0400

> long long len = readByte();

So, this is exactly the same as option #1, except it allocates RSV2 to the
short length field.  As such, it doesn't address the primary complaint
people had about option #1, which is that an implementation that flushes a
buffer a WebSocket frame is likely to use something in the 4-64k buffer
size, and would therefore have to use the 8-byte version.

Comparing it to option #1, the question is the additional gain of being able
to encode values 127-254 more efficiently worth using one of the 3 reserved
bits?  I think if you believe that there is value in short frames (ie, that
98%+ of WebSocket frames will fit in ~126 bytes after compression), then
there is little value in being able to save additional bytes for the small
portion of that <2% that is between 127 and 254 bytes.

Thus, personally I would rate this behind all the other proposals -- if we
are going to give up the reserved bit, at least get some benefit from it.

---

From: gustav <[email-masked]>
Date: Sat, 21 Aug 2010 21:15:40 +0200

On 21 August 2010 20:39, Daniel Stenberg <[email-masked]> wrote:

> Do you care to elaborate why ?.
Its not the lazy coders who pay others net bills.

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 21 Aug 2010 21:24:01 +0200

> size, and would therefore have to use the 8-byte version.

John, thanks for taking the time to collect all the knowledge we've
gathered so far and steer this group towards consensus.

The advantage of #0 is simplicity.  Frames can be read and written
with the utter minimum of logic: no special cases, no ceilings, no bit
mangling.

Simplicity of the framing is also an essential part of reducing CPU
cycles needed to process frames, which matters in low-latency
scenarios.

Our main use case for high performance messaging is market data and
the generally-accepted average size of a FIX message is 157 bytes.  We
also have Twitter messages at 140 bytes and SMS text messages at 160
bytes.  This argues (gently) that a "short" message is under 255 bytes
rather than 127.

As for 64-bit long frames, it's again a matter of simplicity.  With
32-bit frames some use cases MUST switch to fragmentation.

We've been using the 8/64 framing model for several years in ZeroMQ
and it has proven to be robust and optimally fast, see
http://www.zeromq.org/whitepapers:design-v01#toc5.

At the same time if the group's consensus is towards a more complex
length encoding we'd look at adopting that in ZeroMQ.

-
Pieter Hintjens
iMatix - www.imatix.com

---

From: Daniel Stenberg <[email-masked]>
Date: Sat, 21 Aug 2010 21:30:19 +0200 (CEST)

> Do you care to elaborate why ?.

I think most of the arguments around the sizes already have been mentioned and 
I'm not sitting on any new secrets.

I think sizes larger than 32bit will be very rare for Websockets, even for a 
long time to come into the future. Thus a vast vast majority of all frames 
will be sent with at least 4 bytes set to zero (with a fair amount of them not 
even implementing >32 bit support). In my mind, I think the few users that 
acually want to send more than 32bits of data can send it in multiple packets 
to get around the limitation. It's not like having a 32bit limit here will 
prevent anyone from sending more than 4GB of data.

I don't think 4 bytes size is any particular size "loss" for small packets to 
feel bad about, so I prefer the simplicity of always just getting the size in 
a single specified way.

> Its not the lazy coders who pay others net bills.

Uh?

-- 

  / daniel.haxx.se

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 15:34:36 -0400

> with the utter minimum of logic: no special cases, no ceilings, no bit

It has the same special case that option #1 does.

> cycles needed to process frames, which matters in low-latency

I think the difference between option #0 and option #1 will be exactly a
single AND instruction, which hardly seems significant.

> bytes.  This argues (gently) that a "short" message is under 255 bytes

But those are going to be highly compressible, and will easily compress to
under 126 bytes, even with some application level framing around them.

The main difference between #0 and #1 is that #0 uses one of three reserved
bits.  They are easy to spend, but then when we find out we need them it
becomes much more expensive to get them back.  We already know, today,
several potential uses for the reserved bits.  They are reserved simply
because we haven't agreed on how to use them and we are trying to separate
the base framing from how we will implement extensions so we can make
progress, but I think it is very likely that the extension mechanism we
define, starting shortly after the base framing is finished, will
immediately use at least 2 of the bits one way or another.  That doesn't
even include future extensibility requirements we haven't even thought of
yet.

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 15:40:41 -0400

> header which is not byte aligned (1 extra bit in length), but the code

It isn't in the size field, it is a separate reserved bit.  It is where it
is because there was a request that the opcode be aligned on a nibble
boundary for easily reading a hex dump (admittedly minor benefit) -- any of
the fields could be rearranged however you desire without impacting anything
functionally.  The base header has 16 bits total, none of them wasted
(though 3 reserved, plus additional opcode values reserved).

> bit is not already allocated to specific use?

The reserved bits are reserved because we haven't decided how to use them.
 Some discussions we have had about frame-by-frame compression or extension
metadata would need a single reserved bit in the header, the opcode field
could be extended, and there are likely things we haven't thought of yet
that could use it.

> The break even point is 126 / 7 = 18 byte frame sizes.

Depending on the app, more than 98% of real-world WebSocket frames should
fit into 126 bytes after we add compression support.  Paying one bit for
those 98% seems a poor trade to save 7 bits on the <<2% frames which would
be between 127 and 64k.  There is also a threshold effect here -- if we
never need that reserved bit, then it was free to use it.  However, if we
turn out to have needed it, it will cost us a minimum of 1 byte to get it
back by adding an extension opcode.

> We would need to know the distribution of frame sizes we expect to make a
> decision of what is more cost effective.  But it is minor in either case,
> and the bit twiddling for non-byte aligned headers has some cost mostly in
> terms of software error.

I published such data for GWT Quake and Wave in April -
http://www.ietf.org/mail-archive/web/hybi/current/msg01810.html - since
then, I have add percentiles of frames <=126 bytes, assuming stream-based
compression with a 12-bit window: 98.88%/99.03% for GWT Quake S->C (2
trials), 99.98%/99.97% GWT Quake C->S, 98.22% Wave C->S, 98.31% Wave S->C.

No, I think that is a non-starter, as it adds a great deal of complexity for
very little gain.

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 15:54:48 -0400

> prevent anyone from sending more than 4GB of data.

No because for any file on disk larger than 4GB, we will have to move the
entire file through the CPU, when we wouldn't had to otherwise:

http://www.ietf.org/mail-archive/web/hybi/current/msg03402.html

More abstract explanation of the above link:

http://www.ietf.org/mail-archive/web/hybi/current/msg03410.html
http://www.ietf.org/mail-archive/web/hybi/current/msg03417.html

>> Its not the lazy coders who pay others net bills.

Imagine a WebSocket used to push wakeup events to real-time application
(game perhaps), so that is only perhaps 1 byte of payload. You are
proposing to double the network traffic.

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 21 Aug 2010 21:59:26 +0200

> under 126 bytes, even with some application level framing around them.

Compression is not an option for low-latency cases (where volumes are
in the millions of messages /second).  We're already finding that
zero-copy techniques halve the cost of processing.  So the sweet spot
for these really high volume messages remains above 127 bytes IMO.
However, websockets is not intended for the financial markets, and
probably won't ever be used for such volumes.

> becomes much more expensive to get them back.

Indeed.  Adding another 8 reserved bits in the header does not seem wasteful.

Again, I plead for simplicity.  Mixing a length with other bits just
makes it more complex to process WebSocket frames and the benefits
seem marginal.

Best regards
-
Pieter Hintjens
iMatix - www.imatix.com

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 21 Aug 2010 22:04:02 +0200

On Sat, Aug 21, 2010 at 9:54 PM, Shelby Moore <[email-masked]> wrote:

> entire file through the CPU, when we wouldn't had to otherwise...

More pragmatically it means every single WebSocket implementation,
both readers and writers, have to either impose a 4GB limit on frames,
or else have extra functionality to detect large frames and fragment
and defragment them.  It creates significant additional cost and risk
and interoperability concerns, and the benefits are to win 4 bytes
with negligible benefit.

It's not just disk files, my box has 8GB and it's more than a year
old.  Loading, processing, and sending 8GB chunks of memory will be
banal in a few years' time.  That really should remain as simple as
sending a 8KB chunk of memory.

-- 
-
Pieter Hintjens
iMatix - www.imatix.com

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 16:11:46 -0400

> probably won't ever be used for such volumes.

Obviously, I don't know the details of your environment, but I suspect CPU
speed will continue to increase faster than network speeds.  Already, it is
nearly even between the time to send the extra bits and the time to compress
them, so over time I suspect that we will see compression used even more.

> makes it more complex to process WebSocket frames and the benefits

I am a bit confused at your position -- in one sentence you suggest we
should add another byte just in case we might use it, yet you are still
worried about a single AND instruction?  Surely the cost of sending that
byte is higher than one instruction?

Regardless of the differences between #0 and #1, it already seemed clear
that #1 was losing out to #2 or yet more complicated schemes, so I don't
think option #0, which removes the one advantage compared to #3 or Shelby's
proposal of preserving a reserved bit, will get any more backers than #1
did.

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 16:15:24 -0400

[snip, because understood]

> be between 127 and 64k.

Agreed if that turns out to be the distribution of the frame sizes, as I
had mentioned (see below).

> There is also a threshold effect here -- if we

That is the case regardless of whether we make the header 17 bits or 16
bits, so isn't relevant to what I was writing about.  I was not arguing to
get rid of the reserved bit in this portion of the prior reply, when I
proposed adding the extra bit and was comparing the tradeoffs of 17 and 16
bits, with the 2 different RLE encoding methods.

> trials), 99.98%/99.97% GWT Quake C->S, 98.22% Wave C->S, 98.31% Wave S->C.

Okay but we don't know what sort of applications may be built in future. 
But I am not saying this to argue for the 17 bits, rather to argue for
Option 2 versus versus Option 1.  I don't like the discontuinity (sharp
increase) in header size at 126+ bytes in Option 1.  Why assume 2% packets
in that size, when it only costs next to nothing to use Option 2 as
compared to Option 1?

> No, I think that is a non-starter, as it adds a great deal of complexity

I wasn't arguing for it (note how I wrote above that too much bit
twiddling and no strong advantage).  I am just being complete, so that
later no one can say we didn't analyze all possible options.

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 16:20:29 -0400

> in that size, when it only costs next to nothing to use Option 2 as

I have no problem with Option #2 -- it is slightly more complex (which some
people have been complaining about), but it will be better for some
percentage of frames so I am fine with it, especially if it removes the
remaining objections people had with #1.

Mostly I just want us to agree on something and go with it.

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 21 Aug 2010 22:27:48 +0200

On Sat, Aug 21, 2010 at 10:11 PM, John Tamplin <[email-masked]> wrote:

> speed will continue to increase faster than network speeds.

CPU core count increases but I think it's fair to say CPU speeds have
hit the top of the curve.  Thus we're constrained by the capacity of
one core to process the data off a network, while network capacity
continues to double every 18-24 months.

> byte is higher than one instruction?

The single AND instruction is not a problem.  Treating messages above
126 bytes as "long" means we miss the sweet spot.

> proposal of preserving a reserved bit, will get any more backers than #1

Indeed, there are probably very few people in the open source world
concerned with efficiency at the level we are.  I've explained our
experience, and will leave it at that.

Thanks again for steering this issue, consensus is more important here
than the actual design, whatever that may be.

-
Pieter Hintjens
iMatix - www.imatix.com

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 16:35:59 -0400

> Mostly I just want us to agree on something and go with it.

Peter am I correct that option 2 addresses your concerns about
uncompressed packets possibly being larger than 126 bytes? Your option 0
eliminates the extra reserved bit. Bring back the reserved bit and it is
no different than option 2, except for one frame size value we add 2 bytes
in order to get the 3 bytes instead of 8 for medium size frames.

Peter do you still object to the CPU overhead of option 2 as compared to
option 0 or 1?  John I think argued fairly convincingly that the CPU load
is minimal difference and declining over time relative to network speed.

If we need more reserved bits, we could go for a 24-bit header?? Then we
could use more bits for the base case size.  This would cost us very
little even for very small payloads, and it would eliminate 1 byte from
the medium size packets, and would give us more reserved bits if we want.
Should we play it safe instead of trying to squeeze the last drop of
performance for the very small packets?

---

From: Patrick McManus <[email-masked]>
Date: Sat, 21 Aug 2010 16:41:51 -0400

> entire file through the CPU, when we wouldn't had to otherwise:

there seems to be a misunderstanding about ZC techniques here. To be
efficient they certainly require a large-ish chunk size, but there is no
all-or-nothing requirement like you assert that would require copying
through the CPU to only send a subset of the data.

32 bits with chunking would be fine to accommodate zero copy.

there are some folks that want to be able to send any message in one
chunk for unrelated reasons.

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 16:42:27 -0400

> than the actual design, whatever that may be.

I bet in this case, the optimum design and the concensus will converge,
once everyone understands all the issues.  And I think it is clearly
Option 2.  It addresses your sweet spot concern I think?

The more I think about it, it does no good to use 24 bit header, because
if we need an extra reserved bit, we can just use one of the reserved bits
to indicate there is an extra byte on the header.

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 16:57:19 -0400

> 32 bits with chunking would be fine to accommodate zero copy.

My understanding is there would be copying, if you read my link again
carefully:

The reason is because I understand from what I read at John's links that
the entire file is contiguous in memory (because it is a memory mapped or
memory cached copy of the file on the disk), and thus to insert the frame
headers at the 32bit chunks, you would have to move the entire file
through the CPU.  Is my understanding incorrect?  Can you simply rewrite
one sector on disk and the memory manager remaps?

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 16:59:49 -0400

> one sector on disk and the memory manager remaps?

I haven't tried it personally, but I would suspect you could use TCP_CORK,
write a header, use sendfile to send a chunk of the file to finish the
frame, uncork, repeat until done.  I don't think there would be any extra
copying involved here, but you can't just issue the sendfile and go on to
the next message, you need some thread that will queue up the next frame
when it is ready.

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 17:25:02 -0400

[snip]

> chunk for unrelated reasons.

How can they send in one chunk if their file size is greater than 32 bit
length, if you limit the size to 32bit?

If you had a typo above and meant that some folks want to send in smaller
chunks than the proposed 64bit, they would still be able to send in any
size chunks regardless how large (and up to the size) we make the maximum
size.

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 21 Aug 2010 23:27:51 +0200

On Sat, Aug 21, 2010 at 10:35 PM, Shelby Moore <[email-masked]> wrote:

> uncompressed packets possibly being larger than 126 bytes?

Option 2 is efficient enough for shorter messages but it still seems
unnecessarily complex.

The same frame can be encoded in different ways.  The magic numbers
are not obvious, and people will get them wrong.  The rules for
writing frames will be broken and things will work for a while and
then break.  It feels like a brittle design, bound to break in real
use.

We're balancing different needs but the most important requirement of
all must surely be simplicity so that implementors take less risk of
getting it wrong.

-
Pieter Hintjens
iMatix - www.imatix.com

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 17:32:58 -0400

> all must surely be simplicity so that implementors take less risk of

That sounds dangerously close to the discarded "amateur programmer"
requirement.

Personally, I think having a base framing that won't have to be changed
breaking existing implementations is the most critical thing we need to
accomplish -- surely anyone implementing a protocol can figure out how to
write the same code I gave in the first message on this thread.  Look at
IPv6 as just one example of how long it takes to replace a working protocol
with a new version because there weren't enough options for extensibility in
the first one.

---

From: John Tamplin <[email-masked]>
Date: Sat, 21 Aug 2010 17:33:20 -0400

> continues to double every 18-24 months.

I'm not sure I would agree with you on the network bandwidth.  Many places I
know of still use 100baseT to the desktop, 17 years after its introduction
and are only slowly migrating to GigE (itself 11 years old) as
infrastructure costs have come down to about equal.  I don't know of anyone
running higher than GigE to the desktop.  Beyond that, anecdotally the
typical size of the Internet connection at businesses doesn't seem to have
increased since I was in the ISP business 10 years ago (home and mobile
definitely has, though I still don't think at your suggested rate of
doubling).

I will grant your doubling pattern in Infiniband and Myrinet offerings, but
I doubt any significant portion of WebSocket traffic will be running on such
networks.

> experience, and will leave it at that.

It may also be that WebSocket, which is primarily designed to give TCP-like
capabilities to web browsers, is not appropriate for your backend network --
perhaps a frontend would speak WebSocket to end-users, and then a protocol
more optimized for your needs going back to your servers.

Anyway, I agree with you that getting agreement on the framing, as long as
it is reasonable, is more important than what it actually is so we can get
on to the next problem.

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 21 Aug 2010 23:55:47 +0200

> That sounds dangerously close to the discarded "amateur programmer"

Perhaps it sounds like it, but it's not.  Simplicity is about shaving
off friction costs and the more a design is used the more those
savings add up.

> accomplish.

That is easily accomplished if we stop aiming for ultra-efficiency.
Make no assumptions about limits, assume a 50-year lifespan during
which data sizes will double every 24 months.  Allow sufficient space
at every junction for 20x more variability than we think we will ever
need.  Do not try to save bits.  There will be waste but we won't hit
limits.

Dropping the assumption that we have to save every byte possible will
IMO make it easier to build a solid, future-proof framing.

This makes sense, WebSocket is not meant for supercomputing but to
connect browsers to servers easily and interoperably.

-
Pieter Hintjens
iMatix - www.imatix.com

---

From: gustav <[email-masked]>
Date: Sat, 21 Aug 2010 23:56:23 +0200

On 21 August 2010 23:27, Pieter Hintjens <[email-masked]> wrote:

> Personality driven by fear and incompetence has nothing to do in a
programmer, that attitude is for the users ,
its for them that people who are driven by inspiration and innovation are
supposed to develop good stuff =).
 It only goes bad when users come and dictate how programmers should code...
 it uhm must be so simple.. so there is no need to think at all..
Its only by pushing limits, your own limits that you ever can become better,
it natural to fail sometimes - now i come better !.

regards
 gustav trede

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 18:00:55 -0400

>> continues to double every 18-24 months.

Why can't you utilize more than one core?

I realize we need a new computer language for multi-core and I started
working on that.  It is all about minimizing function side-effects, which
I call purity.  Copute is the name, but just in early idea stage.  I
realize such discussion is out-of-scope of this thread, and you may be
referring to real-time or other issues that I am not aware of.

> running higher than GigE to the desktop.

My observations may not apply to more wealthy countries, but they probably
do apply to India and other developing countries in Latin America.

Last time I checked in 2009, afaiu that most of the low-end PC
motherboards (what sells here in Philippines in huge volumes), have buggy
wake-up on LAN if the GigeE is turned on, so many revert to the old
Ethernet.  I don't know if it is driver or OS issue.

In terms of mobile, here in Philippines where I am, which is perhaps #1
texting county in world, the 3G has very minimal coverage and in the large
population centers where there is coverage, it is overloaded and the
networks drop you back to EDGE/GPRS level of throughput after you hit
certain limits.  And during peak hours everything slows down here on every
consumer network.  There are no high end choices, except I think the call
centers somehow get the wide pipes they need.

>  Beyond that, anecdotally the

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 18:13:09 -0400

> off friction costs and the more a design is used the more those

I love abstract points (I make many myself and I guess sometimes people
don't understand me), but I can't find any meaning in what you wrote
there.  We are trying to shave friction where it is quantifiable. You are
asking us to quantify the incompetence of network protocol programmers
between some very minor difference.  If they can't get a RLE header
correct, I doubt they can implement this protocol successfully. RLE has
been around since beginning of computing.

RLE = run length encoding

I wrote one of the first printer drivers back in mid-1980s that took
advantage of RLE encoding in printers to make printing faster over serial
connections we had before USB.  Was named TurboJet since it was targetted
to the laser jet printers and the slow print times of sending 300 DPI over
serial.

> need.  Do not try to save bits.  There will be waste but we won't hit

But we only need to reserve one bit for that, then we can always extend
the protocol without breaking the prior versions.  And then the extended
header can reserve one bit, and so on.

---

From: Pieter Hintjens <[email-masked]>
Date: Sun, 22 Aug 2010 00:14:16 +0200

> Why can't you utilize more than one core?

There is no effective way to turn "process the frames on this
connection" into a concurrent process.

Yes, there are many very slow networks around, just like there are
many devices with limited memory.  However even those slow networks
get consistently faster.  Just because GPRS is not 10GigE does not
mean mobile users won't get LTE upgrades.

Shelby, if you are interested in discussing parallel architectures and
high performance networking, we can talk off-list.  This is not
relevant to HyBi any longer.

Regards
-
Pieter Hintjens
iMatix - www.imatix.com

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 18:32:13 -0400

> connection" into a concurrent process.

Actually just remembered I had this public discussion last year in design
of Copute, and the conclusion is you can't backup the iteration and still
be pure (or something like that, I will go look at my notes again).

The channel is serial and forks (side-effects), so you can't parallize it
unless you are willing to do predictive branches and throw away CPU time
for incorrect guesses.

Okay I think understand.

> mean mobile users won't get LTE upgrades.

But it seems subjectively that the rate speed is introduced at the
bleeding edge, is running away from the rate that speed is improving at
the dull edge.  But perhaps I am wrong about that.

I am highly frustrated that afaik, I can not buy a faster pipe here for
any price.  My solution was to buy multiple crappy pipes and load balance
them :D

I think it is economics over here, so I tend to think you will be proven
correct.  Once the dollar and the western nations collapse by 2011 or
2012, we will see massive boost in relative wealth in the developing
world, and then a massive increase in the bandwidth in the consumer
markets here.  Time to get positioned now for the big shift (are you
listening Google?).

> relevant to HyBi any longer.

Is that an interest of yours?  Okay I may hit you up later.  Thanks.

---

From: Patrick McManus <[email-masked]>
Date: Sat, 21 Aug 2010 19:58:10 -0400

>          Can you simply rewrite
> I haven't tried it personally, but I would suspect you could use
> TCP_CORK, write a header, use sendfile to send a chunk of the file to
> finish the frame, uncork, repeat until done.  I don't think there
> would be any extra copying involved here, but you can't just issue the
> sendfile and go on to the next message, you need some thread that will
> queue up the next frame when it is ready.

That is correct (and I have done this personally), you need multiple
calls of the ZC API interleaved with the write of the chunk header.
There is no copying of the data through the CPU just because it requires
multiple calls. And there is no reason you couldn't implement some kind
of writev() like API that lets you interleave copied and zero-copied
data.

And yes, multiple calls are more overhead than 1 call. but at 1 per 4GB,
it really is inconsequential.

---

From: Patrick McManus <[email-masked]>
Date: Sat, 21 Aug 2010 20:00:14 -0400

> length, if you limit the size to 32bit?

I did not say they could fit 33 bits of stuff in a 32 bit bucket.

I said that some folks want a >32 bit chunk size for reasons unrelated
to zero copying performance.

---

From: "Shelby Moore" <[email-masked]>
Date: Sat, 21 Aug 2010 23:23:07 -0400

> it really is inconsequential.

That is an extremely technical solution applicable to experienced sysadmins.

But WebSockets is supposed to be used also by web programming amateurs. 
It is much easier for them to issue a single sendFile() call from PHP,
Perl, Ruby, etc., without all the fuss and low-level network stack API
synchronization.

---

From: Brian <[email-masked]>
Date: Sun, 22 Aug 2010 00:57:50 -0700

Hey all!  I've been watching this list for two or three weeks now, and
decided to finally chime in.

As much as I feel that 63-bits is ridiculously large for the frame length,
I'd be happy to accept it and just move on to the next item of discussion --
though the WS server I'm writing right now will consider any frame with a
non-zero bit in the high 31-bits of the long long length field to be an
error condition, and immediately drop the rogue/attacking/misbehaving
client.  I don't want to be bound to treat absurdly large frames as
potentially legitimate by an overzealous specification.  Furthermore, I'm
writing my server with Node.js, which, being a JavaScript-based environment
itself, doesn't even have a 64-bit integer type of any sort.  I *can't*
correctly handle a 63-bit frame.  Ah, the irony of a protocol meant for
JavaScript applications that's impossible to fully implemented in JavaScript
(at least efficiently -- no bignum libraries allowed.)  And who knows what
would happen with the actual frame data in, say, a JavaScript String, with a
frame of a few terabytes?  Assuming anyone had that much RAM?   We're
talking about JavaScript clients here, so no streaming to disk, etc....

In my not-so-humble opinion, trying to accommodate sending large files
through WebSockets is a complete and utter misdirection.  The protocol
exists to provide real-time interactivity support to browsers.  That's what
it's meant for.  If someone wants to send large chunks of data (>4GB) in one
frame, they should find another protocol, because they're clearly not
planning to process it in JavaScript in a web-browser client.  And if
they're not running in the browser, there's no reason they have to use WS.
 Making abusers of the protocol chunk at 4GB boundaries is more than
reasonable when you consider that the vast majority of users won't begin to
consider sending fragments anywhere near even a fraction of that size.  This
is not and should not be an all-things to all-people protocol.  We shouldn't
be designing for these edge cases just because the protocol could
theoretically enable any number of diverse applications.

Lets get back to basics!  WebSockets was conceived to provide super-simple
bi-directional persistent communications on a platform -- the browser --
that had no such pre-existing canonical ability.  It's meant exclusively for
communicating with JavaScript-based applications running in a web browser.
 WebSockets may end up being used or abused for other things, but that is
not the intended purpose and, in my opinion, such uses shouldn't be the
concern of this working group.

Pedantic discussion of details and attempting to figure out how to
accomodate theoretical (ab)uses of this protocol has become a very real
distraction to the group.  The volume of mail generated by this list has
skyrocketed in the last week or so and most of the content can really be
summed up as quibbling over minutia.  Let's pick a reasonable solution and
move forward, even if a tiny subset of people who want to abuse WS for
something outside the browser don't like it.

For the record, my vote for frame size encoding goes to John's
original option #2.  It's simple, clean, easy, and efficient.  The only
change I'd make is to go to a 32-bit maximum, but I'm happy to accept a
63-bit field in the spec if it means we can move on.  Also, thank you, John,
for taking the time to attempt to steer this issue to consensus!  It's hard
to keep up with the current theoretical consensus of where the spec is going
with all the pedantry on the list at the moment.

As for the concern about keeping a sufficient number of reserved bits...
throw in a whole reserved byte and put a bow on it. :-)

Cheers,
Brian McKelvey

---

From: Roberto Peon <[email-masked]>
Date: Sun, 22 Aug 2010 01:16:44 -0700

> Perl, Ruby, etc., without all the fuss and low-level network stack API

No, that (amateur programmers) is not a requirement currently as I
understand things.
The expectation is that we develop the minimal featureset necessary to
ensure a robust and hopefully efficient transport.
There are different perspectives about what that featureset need be assuming
competent professional programmers, and that is how you should be looking it
it!
-=R

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 04:36:27 -0400

Hi Brian,

Although I do appreciate your sentiments and also appreciate John and
others for trying to move towards concensus, I have some feedback to you.

> I'd be happy to accept it and just move on to the next item of discussion

Noted.

> client.

64-bit computing is rogue? I thought it was a desired advancement.

>  I don't want to be bound to treat absurdly large frames as
> potentially legitimate by an overzealous specification.

You do not want to ever be bound to moving to 64-bit computing?

And if you do one day accept the need to move to 64-bit computing, how
difficult will it be to come back and redo WS and get the new version
adopted in reasonable time?

I say the push to 64-bit is now and we are derelict in our duties if
release a new protocol that is looking backwards about 2 decades.  I
relented on my 128-bit suggestion.

>  Furthermore, I'm
> itself, doesn't even have a 64-bit integer type of any sort.

Javascript will never support 64-bit?

In meantime, WS browser code can do the re-fragmentation, you will never
see larger than 32-bit frame in Javascript, until that browser's
Javascript is 64-bit.

>  I *can't*
> correctly handle a 63-bit frame.

Yes you can. See above.

>  Ah, the irony of a protocol meant for

Again see above, you will never have to store something that Javascript
can't store.

> consider sending fragments anywhere near even a fraction of that size.

I deduce/conjecture that WS will also be used to replace Flash in HTML5,
so that users can get video streaming without a plugin or Flash server
license. And I think we also need to consider the amateur webserver
programmers who simply want to dump a sendFile() of their DVD to client
who can display it.  We don't want to limit the simple optimization (DMA
-> NIC, no CPU load) that capability to sysadmins who understand low-level
TCP tricks.  It doesn't cost you anything on the intermediary or client
side, as we have to support re-fragmentation (and/or negotiation on
handshake) any way (because no matter what maximum size we choose, not all
intermediaries will necessarily support it).

> This
> is not and should not be an all-things to all-people protocol.

Why not?

>  We
> shouldn't

Web video is going to be edge case?  I don't think so!  It may end up
being the majority of the traffic (if measured in bytes, not frames) for
WS.

> that had no such pre-existing canonical ability.

What is not simple that could be designed away from our frame header?

>  It's meant exclusively
> for

Disagree as noted above.

> concern of this working group.

Web video is an abuse of WS?

> summed up as quibbling over minutia.

Disagree with that "quibbling over minutia" characterization.  I showed
that for the bi-direction events you are batting for, that "minutia" could
cause a doubling of network traffic in the extreme case.

Good design is a lot of work.

>  Let's pick a reasonable solution and

Agreed!  But I think the abusers are the ones who argue they can't check
one bit in the header and switch to 64-bit processing.

> 63-bit field in the spec if it means we can move on.

Noted.  I am obviously arguing for the 63-bit field.

>  Also, thank you,
> John,

That might not be a bad idea.  But I think we will know that more as we
move forward on designed the extensions?

1 byte is not that ridiculous, because we already have 3 bytes minimum
including a 1 byte payload.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 04:42:16 -0400

> competent professional programmers, and that is how you should be looking

My understanding is that the "amateur programmers" requirement was only
removed for those who process the protocol directly, not for those who use
WS in scripting applications such as Javascript in browser, PHP, Perl, etc
on server. Am I incorrect?

I would strongly disagree if we go against "amateur programming" in
scripting uses of WS.  I do agree with the elimination of "amateur
programming" requirement for protocol implementation programmers.

---

From: Roberto Peon <[email-masked]>
Date: Sun, 22 Aug 2010 01:49:15 -0700

> on server. Am I incorrect?

The API should be easily used and there should be trivial applications which
may use it.
I don't believe there is any requirement for this which has been explicitly
stated, but there have been concerns voiced about the feasibility of
representing various data in JS. I believe that consensus was reached on
binary/utf representation of data as a result of these concerns.

Which is a long-winded way of saying that you're correct that protocol
implementors (servers, browsers, intermediaries) had the "amateur
programmer" requirement lifted.

> programming" requirement for protocol implementation programmers.

Excellent, then I think that matches what I perceived as the stated
consensus.

-=R

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 05:34:22 -0400

Hi Brian,

I think you reply is really important and I need to make it clear to the
list that I am not trying to cause bad feelings.  My sincere apologies.

> I never said "64-bit is rogue."  You frequently use hyperbole on this list
> and reframe other peoples' comments in order to try to put more weight
> behind your arguments and subtly vilify your debate opponent.

I apologize.  I will try to be more conscious of the considerate tone in
the way I word my feedback. I was just being frank as to what is my
reaction. I am historically accustomed to working in small groups that
speaking frankly about design considerations. A common joke of my former
boss when I was much younger, was "NFG" (no f-ing good). Given he could
memorize a whole page of a phone book by looking at it for a few seconds,
I guess I learned to respect his method (shocked me when I was first
indoctrinated into the culture).

below...

>  That said,
> I'm going to take a minimal approach in my response:
>
> Regarding all your 64-bit comments.. the frame size has absolutely nothing
> to do with "64-bit computing."  A discussion for "64-bit computing" should
> have no bearing whatsoever on this protocol's length field in the header.

But you argued that Javascripts inability to do 64-bit computing was a
justification for not doing a 64-bit frame size. So I thought you started
that discussion. Apologies for that.

>  The only thing that truly matters are what kinds of messages we want to
> handle with the protocol.  My point is that there's no reason for a
> websocket client or server to handle a single message >4GB.

You won't have to.  You will tell the other end your maximum frame size on
the connection handshake, then the other end will re-fragment if
necessary.

>  For my chat
> server that I'm building, any incoming message larger than 64KB is likely
> to
> be ill-intended, and that's a very generous threshold.  Clearly the
> threshold would be different for someone else's application.

Apologies I forgot to make the above point in my prior reply. I could have
been more concensus driven with that.

> More hyperbole.  See above.  128-bit is way beyond absurd for a WebSocket
> message length, whether today, tomorrow, or ever.

I said I relented on the 128-bit.  I wasn't still arguing for it.  I had
presented it in order to be complete in our design analysis, so that
anyone in future wants to know why we did not support 128-bit, then they
will see our logic written down.

I do agree that I should not have made the point that you need to process
64-bit frames.  That would not be a free market. Rather we must have
re-fragmentation, so that no WS server has to be forced into any maximum
size.

> <snip>
> It may well be helpful in replacing flash for things like games, but not
> video.
>
> There are already competing methodologies for streaming HTML5 video and
> even
> advanced functionality like dynamic ad insertion.  For just one example
> see
> http://www.mdialog.com/
>
> A standardized way to do HTML5 Video streaming is something that should be
> undertaken by another working group in relation to the <video> tag.  So
> yes,
> I feel rather strongly that video streaming would be an abuse of
> WebSockets.

Thank you for clarifying that.

So then why are people mentioning a DVD in conjunction with sendFile()
issue?  Is that all hyperbole too?

And why couldn't video be streamed on top of WS? There can be any protocol
on top of WS that can do extra features such as ads.  How do we know the
standardized way won't end up being on top of WS?

> Because it wasn't originally intended to be.  It solves a specific problem
> rather well -- the ability of JavaScript-based applications in the browser
> to participate in bi-directional communications with a server.  That's
> what
> it was meant for.  Opening up the scope of the protocol is a distraction
> from getting it optimally correct for its originally intended purpose.

What is sub-optimal about the Option 2 with 63-bit capability with respect
to Javascript?

> Yes it is.  And there's a lot of pressure to do good quality work rather
> quickly to get it finalized, since the browsers are already implementing
> drafts and people are starting to build real-world applications on it.

We could get this frame header decision done this week I expect. The more
we talk the better. The faster we can hash out the understandings.

>  Arbitrarily widening the scope of the protocol and then quibbling over
> those edge cases that aren't applicable to the browser takes time away
> from
> the work of truly getting it right.

But why kill the 63-bit if it does not cost us anything?  I would prefer
we not try to predict the future, when it doesn't cost us anything to
allow more capability.

Even if we did 32-bit maximum, we still need re-fragmentation correct?

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 05:53:14 -0400

> including a 1 byte payload.

I forgot that I had already mentioned in this thread that we only need to
keep 1 reserved bit free, so that we can always add another bit as needed,
by allocating that 1 bit to signal there is an extra byte.  Repeat in next
byte, and so on...

So we don't need the extra byte now, unless we see an extension now that
needs that many bits.

Any one disagree?

---

From: gustav <[email-masked]>
Date: Sun, 22 Aug 2010 11:53:29 +0200

On 22 August 2010 11:34, Shelby Moore <[email-masked]> wrote:

> Nothing.
What do we know about the future java script capabilities ?.
There is no reason to ask for limitations based on the current inabilities
of java script,
neither regarding performance or function, those metrics are indeed evolving
extremely fast.
Jscript clients can like any other use what they currently can, the rest is
optional and gives room to grow into.

regards
  gustav trede

---

From: gustav <[email-masked]>
Date: Sun, 22 Aug 2010 11:53:29 +0200

regards
  gustav trede

---

From: Brian <[email-masked]>
Date: Sun, 22 Aug 2010 02:59:22 -0700

Apologies, the first time I sent this I forgot to hit "Reply All" and sent
the response directly to Shelby:
----

> that discussion. Apologies for that.
Then I apologize for not being more clear -- that comment was an aside.  I
just thought it was ironic.  Regarding JavaScript handling 64-bit
datatypes... JavaScript's number format is specified as a 64-bit float.  So
it is indeed a 64-bit datatype, technically, but it can only handle 53 bits
of integer precision.

> the connection handshake, then the other end will re-fragment if
Has there been discussion of implementing maximum accepted frame size
declaration in the handshake?  This is news to me.  Admittedly, I've only
been able to keep up with about 85% of the list messages lately.  There's a
lot to read.

If the handshake has some accepted way to declare your maximum accepted
frame size, then that might solve my immediate complaint for the chat
server. Then we're on to discussions about the mechanics fragmentation,
total message length, etc...  I think that discussion still has more to go.

> issue?  Is that all hyperbole too?
Yes, I believe it's a complete misdirection.  The discussion of such things
is one of the main reasons I felt compelled to write to the list to entreat
people to stay on track.  Sending a DVD over a WS link is not a useful thing
to do with a protocol whose client is meant to be a JavaScript-based browser
application.

> standardized way won't end up being on top of WS?
Because WS isn't really meant to be an efficient video streaming protocol,
and designing it to be able to handle that application would add all kinds
of unnecessary complexity.  Sure you *could* implement video streaming on
top of WS, but I suggest that it wouldn't be the best idea for anyone.
 There are already perfectly good well-supported video streaming protocols
that would be better suited to that particular task.

> What is sub-optimal about the Option 2 with 63-bit capability with respect

Only that JavaScript is incapable of interpreting a full 63-bit length field
with full accuracy (it only has 64-bit floating point numbers.)  If an
individual implementation is free to punt on the larger sizes in this length
field and still be considered to be fully compliant, then my only complaint
is the rather minimal waste of space in the header.  I'm happy to accept the
63-bit length if we can move on to something else.

> we not try to predict the future, when it doesn't cost us anything to
Keep it then.  It's not a huge deal to me.  Just being able to nail down
framing is what I want.

> Even if we did 32-bit maximum, we still need re-fragmentation correct?

Yes.  It was accepted as a requirement by consensus and I see no need to
debate that.

Brian

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 06:16:23 -0400

[snip]

> re-fragmentation, so that no WS server has to be forced into any maximum

Given the above point, those who are arguing against the CPU load to test
for value 126 and 127 to fork to 16-bit and 63-bit frame size of Option 2,
we instead could consider adding a byte to the header and give 1 to 32766
sizes without any test.  The end could set its maximum frame size to 32766
on connection handshake, and then it never needs to perform that logic in
CPU.  They just load 2 byte size field and use it.

That eliminates the 16-bit size field of Option 2.  It is extends the size
field of Option 1.

That would cost us an extra byte on every frame, but the minimum frame
size is 3 already, and those who were arguing that point, said that
network speed will increase faster than CPU because there is no way to
parallelize (use multi-cores) for a serial channel that has no rewind
feature (unless you buffer and use pipelining perhaps?).

Also what are we using the 0 size value for? Couldn't we add 1 to the size
field to get the size?

Also for the special case where the maximum frame size is set on
connection handshake to 32767 (instead of 32766), no need to escape that
into an 8 byte size field.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 07:31:44 -0400

Just got back from 2-mile jog, nice way to process and focus thoughts...
(due the fact that the brain solves the N-dimension space via annealing,
so doing something entirely different can jolt your Newton gradient over a
hill into a deeper N-space global minima)

below for the key point...

> feature (unless you buffer and use pipelining perhaps?).

This could be referred to as Option 1/15-bit.

Note that although the re-fragmentation will still incur CPU overhead for
large frames, the point was that this new option eliminates the CPU
overhead when there wasn't going to be re-fragmentation (most frames' byte
size smaller than 15-bit).  And we must have the re-fragmentation any way,
because some servers simply won't be able to handle large frame sizes for
any number of possible reasons.

So if we place important priority on the possibility that network speeds
will outpace CPU speeds (due to multi-core point), then our choice boils
down to either Option 1/15-bit or Option 1/7-bit.  But 7-bits seems aweful
small as a useful maximum size.  So I think Option 1/15-bit is our only
choice that will bring concensus?

And the Option 1/15-bit is actually more efficient than Option 2, for
frame size between 126 to 32767.  It is equally efficient for > 32767. 
And it only costs 1 extra byte below 126 in size.

And what happens if we don't do Option 1/15-bit and indeed network speed
has a radical burst upward relative to single-core CPU speed?  See my
point about a possible radical shift in the realized market speed soon due
to global financial rebalancing via hyper-inflation (networks here may
have fast enough CPUs but overloaded pipes due to fact that pipes are not
regulatory subsidized here like they are in the socialized "rich"
countries):

http://www.ietf.org/mail-archive/web/hybi/current/msg03454.html

Btw, the above is an example of why insurance (surety about future) is a
fraud. Statistics can not see long-tail events:

http://goldwetrust.up-with.com/knowledge-f9/book-ultimate-truth-chapter-1-science-is-never-certain-t148.htm#3159

Some of that abstract discussion I did before this thread was relevant:

http://www.ietf.org/mail-archive/web/hybi/current/msg03348.html

---

From: gustav <[email-masked]>
Date: Sun, 22 Aug 2010 13:53:06 +0200

On 22 August 2010 13:31, Shelby Moore <[email-masked]> wrote:

> Where is the _proof_ of network speed is outgrowing cpu speed on the long
term ?.
Its imo more about some people having a hard time to design efficient
scalable code and hence even modern multi core CPUs becomes a rock wall  for
their throughput.

Should doomsday theories, or the presumed but unquantified incompetence of
the general programmer be the basis for a new protocol ?. I don't think so,
but its just me.

regards
 gustav trede

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 08:22:31 -0400

>> >>>  The only thing that truly matters are what kinds of messages we
>> want
>> >>> to
>> >> You won't have to.  You will tell the other end your maximum frame
>> size
>> >> on
>> > for value 126 and 127 to fork to 16-bit and 63-bit frame size of
>> Option
>> 2,

My understanding now that someone else mentioned it:

is that you can't do multi-core on a serial process unless it is
reversible or you can load the entire thing in memory (which you can't if
you want your CPU to run in parallel to your NIC).  That is a inviolable
mathematical truth I think?

> Should doomsday theories,

I appreciate the point that we shouldn't design for something that happens
very rarely (perhaps not even in a lifetime). But when we don't design for
them and they do happen, we can possibly get catastrophic failure. If it
costs us nothing to design for them, then why not?

The only thing this costs us is 1 byte for frame size < 126 size. And we
get 1 byte savings (increased efficiency) for frame size 126 - 32767.

And we gain that it is maximally efficient for CPU if maximum frame size
is hard-coded by one end at 32767 (and actual frames are less than that
size at the source, otherwise it is not maximally efficient because
re-fragmentation is happening at other end).

> or the presumed but unquantified incompetence of

I tend that due to Coase's Theorem, artifical barriers will be destroyed
by the market.  So to introduce artificially frame size boundary if there
is a way around it, will not stand test of time.

However, if only a single-core can be used for mathematical reasons (?),
then it is not incompetence, but a fact of nature.  And we should pay
attention to the expert feedback, because it may affect routers in ways we
(if we are non-experts in routers as is case for me) can't really
understand.

It is not just the issue of how fast the connections are increasing but
also how many connections your multi-plex in one CPU, etc.  I don't really
understand their issues well.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 08:45:08 -0400

Sorry I had to get that one reply before on list, so I could publicly
apologize and make that very important point about maximum frame size at
handshake.

>> But you argued that Javascripts inability to do 64-bit computing was a
justification for not doing a 64-bit frame size. So I thought you
started
> Then I apologize for not being more clear -- that comment was an aside.

Me too apologize again.  It is amazing how increased knowledge improves
concensus and good feelings.  Once I realized that the re-fragmentations
solves your issue, then there was a way for me to re-phrase my reply that
wasn't so contentious.

>  I
datatypes... JavaScript's number format is specified as a 64-bit float.
So
> it is indeed a 64-bit datatype, technically, but it can only handle 53 bits

Oh great, I didn't look at that yet.

necessary.
> Has there been discussion of implementing maximum accepted frame size
declaration in the handshake?  This is news to me.

I saw it only briefly mentioned this week (or maybe it was late last
week...the days and nights are running together for me because I work from
home with no set hours).

>  Admittedly, I've only

Apologies (:guilty:)

frame size, then that might solve my immediate complaint for the chat
server. Then we're on to discussions about the mechanics fragmentation,
total message length, etc...  I think that discussion still has more to
go.

Yes that was the epipheny I just realized too.  I think that could
possibly move many of us closer to concensus?

> to do with a protocol whose client is meant to be a JavaScript-based
browser

Okay but I replied to that on list, so we can get the others to justify
that.  I also too have been keeping my eye out to make sure we all don't
push something that we don't all need.  We all seem to have good design
principles overall.  I am happy with the list folks, seems like a great
bunch of guys and 1 gal from opera.

>> on top of WS that can do extra features such as ads.  How do we know
the standardized way won't end up being on top of WS?
>>
>>
> Because WS isn't really meant to be an efficient video streaming
protocol, and designing it to be able to handle that application would
add all kinds of unnecessary complexity.  Sure you *could* implement
video streaming on top of WS, but I suggest that it wouldn't be the best
idea for anyone.

Understood, but sometimes the market chooses something that is 80% as
good, because it is 80% more compatible or simpler or other reason.

So lets not be shocked if it happens, especially if it does not cost us
anything that we didn't have to do any way.

> with full accuracy (it only has 64-bit floating point numbers.)  If an
individual implementation is free to punt on the larger sizes in this
length
> field and still be considered to be fully compliant,

Yes I think it can, but the beauty is the browser will not be punting.  It
will take the full frame and then fragment it client side for the
Javascript.  Well at least that is my understanding.

> then my only
> complaint

There is no waste if we adopt my Option 1/15-bit.  We won't get any extra
header bytes except for frames > 32767.

>
>
>
>>
>> But why kill the 63-bit if it does not cost us anything?  I would
prefer we not try to predict the future, when it doesn't cost us
anything to allow more capability.
framing is what I want.

Great.  I want everyone to be happy. I think there should be an optimum
balance.

debate that.

Great.

Thanks for the reply.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 09:53:59 -0400

>>> so doing something entirely different can jolt your Newton gradient
>>> over
>>> a
>>> >> Apologies I forgot to make the above point in my prior reply. I
>>> could
>>> >> have
>>> > on connection handshake, and then it never needs to perform that
>>> logic
>>> in
> is hard-coded by one end at 32767

> (and actual frames are less than that

My point above was that the efficiency bottle neck with respect to framing
CPU overhead, is conceptually the ratio of the state-machine logic at the
frame boundaries to the frame size. Thus many small frames is where it
matters most. And for very large frames that can't be sent with a
straightforward sendFile() because the maximum frame size was set too
small by the other end, and thus extra re-fragmentation overhead and
potentially unoptimized sendFile(), well this was going to happen any way
because the other end was going to set the maximum frame size that
eliminates any branching logic.  So there really isn't any loss of
efficiency that wasn't going to be lost any way.  Coase Theorem can not go
around this.

The reason for this was mentioned in private just a few minutes ago. The
multi-core overhead is the signalling between cores, whose complexity and
cost grows dramatically (probably geometrically), until adding more cores
actually reduces performance. I had actually predicted this at
Lambda-the-Ultimate.com last year (my focus at that time was in gaming
events in a scene), and I think others too have.

My high level abstract mathematical understanding of this (which btw is
what is driving my design of the Copute language), is that the lambda
calculus tells you that pure functions with no side-effects, will scale to
infinite cores.  But once you introduce side-effects (i.e. a
state-machine, Turing complete), then the parallelism fitting degrades and
quite dramatically.

So as I understand it from a high level of abstraction, the problem is the
real-time conditional tests for different maximum frame sizes causes a
dramatic rise in side-effects between multi-cores which have to
coordinated, especially the smaller the frame sizes.  I hope my
restatement of the phenomenon is abstractly correct.

The more I think about it, I don't think very many routers are going to
support an open-ended maximum frame size (i.e. 2^63), so the ability to
use the simplified sendFile() is going to be luck??  But there is no
reason to not support 63-bit, because there is no one maximum size that is
going to fit every instance.  The maximum has to be negotiated on
connection handshake?

Hope I haven't conflated issues.  I rushed this.  I broad stroked it, so
feel free to correct me.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 11:03:38 -0400

> going to fit every instance.  The maximum has to be negotiated on

Most routers aren't going to inspect above the IP layer - they just forward
network-layer packets.  TCP streams essentially have infinite frame length -
does that mean a router can't forward them?

Where it will come into play are the endpoints that must generate/process
the messages and proxies that browsers use to connect to the outside world,
since they will have to store and forward the frames.  That is one of the
reasons that fragmentation is required, so that such a proxy may choose to
dedicate a fixed buffer for each connection coming into it, and send a
fragment as it reaches its buffer size.

Regarding negotiating a maximum frame size, as soon as we finalize the
framing we will start trying to get consensus on the handshake.  I am
certain we can include a maximum frame size in the handshake if consensus is
that it is needed.

---

From: gustav <[email-masked]>
Date: Sun, 22 Aug 2010 17:29:09 +0200

On 22 August 2010 17:03, John Tamplin <[email-masked]> wrote:

> yes. lets move forward =)

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 11:32:34 -0400

> feature (unless you buffer and use pipelining perhaps?).

I posted a while back and keep referring to packet-traces of real-world
apps.  Nobody else has posted any, so all I can go with is the data I
collected.  That data shows that after compression, not just a lot of frames
will fit in ~126 bytes, but *an overwhelming majority*.  Even if we assume
that as new applications are developed the frame size will go up, it still
seems clear that more frames will fit in the small frame size than will
require larger frames.  Thus, it seems a foolish trade to add one byte to
the most frequent group of frames to save a byte or two on the infrequent
group of frames.

Does anyone dispute this?  The only rationales I have heard against are:

   - In some extremely high-volume cases the CPU time for compression can't
   be spared, which may push a number of frames above that threshold.
   I don't think that will apply to most cases, and even without
   compression, 78% of Wave S->C, ~99.8% of GWT Quake C->S, ~33% of GWT Quake
   S->C frames would fit (only 1.2% of Wave C->S though), so it still seems
   worth optimizing for.
   - Implementations that don't special-case a 7-bit length can be simpler.
   Sure, but we aren't trying to design the simplest possible protocol, but
   rather one that will meet a certain class of needs -- clearly it is a
   slippery slope and we shouldn't add complexity without clear benefit, but I
   don't think that applies here.

> field to get the size?

It is quite likely that some control messages will have a 0-byte payload.
 In particular, close/ping/pong seem likely to be frequently sent with no
payload.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 11:44:12 -0400

> So we don't need the extra byte now, unless we see an extension now that

Yes, I agree that as long as we don't use our last reserved bit for
something besides "more reserved bits", we can arbitrarily extend the
protocol.  However, we have several things which we know we want to
implement as extensions in short order - multiplexing and compression among
them, and perhaps additional metadata.  If we only have one spare bit in the
base protocol, then that means we must define it as the "more reserved bits"
flag, and we will then be paying a cost for what are expected to be common
extensions.  Also, note that we would have to negotiate adding these
reserved bytes in the handshake for interoperability with base protocol
implementations.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 12:22:00 -0400

> will fit in ~126 bytes, but *an overwhelming majority*.

In that case we can go with Option 2 as optimum.

But I think you are missing my point as follows?

Any ends that have a CPU load problem due to multi-core synchronization
overhead, may set their maximum size to 125, if eliminating the
conditional test will improve throughput. The downside is that is a lot of
fragmentation that would not be necessary for the large source frames. 
The CPU load problem improves the larger the maximum frame can be, but
apparently not if there is conditionals in the size. That was the only
reason I proposed Option 1/15-bit, which gives a maximum frame size of
32767 without any conditionals, and the cost is only adds 1 byte to header
for frame sizes smaller than 126, and removes 1 byte for the 2% of frame
sizes between 126 - 32767.

I think that is the most concise I can be on that matter.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 12:29:54 -0400

> fragmentation that would not be necessary for the large source frames.

I think it is ridiculous to suggest an implementation would avoid a couple
of conditional tests and instead would fragment a large frame into 125-byte
frames to save CPU - the cost of the doing the fragmentation will dwarf the
few instructions.

> 32767 without any conditionals,

There is still a check of whether the value is the flag to indicate an
8-byte length.  Having 2 compares to byte constants and associated branches
instead of 1 compare to a short constant and a branch seems hardly a
significant additional processing requirement.

> and the cost is only adds 1 byte to header

Yes, and my point above is that you add a byte in the 98% case and save a
byte for the <2% case, which on average costs at least 0.96 bytes more per
frame.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 12:39:12 -0400

> theoretically enable any number of diverse applications.

Just because the JS API today passes the messages totally in RAM, there is
no reason that it can be extended to support a stream-like API if processing
larger messages becomes justified.  However, at the framing level, we have
to have something in place such that v1.0 implementations written against
this spec can interoperate with later extensions, so it seems prudent to
make sure the protocol is sufficient for any reasonably forseeable needs.
 There were several strong arguments that 32-bits isn't enough, and the next
logical step up is 63/64-bits.

Personally, I would be perfectly happy with a 32-bit maximum frame length,
and let senders fragment larger messages.  However, there were significant
objections (see the discussions link from
http://www.ietf.org/mail-archive/web/hybi/current/msg03399.html for details)
about being required to fragment a message when its size was known, and the
fact that it made "fire and forget" of sending the message contents with a
single sendfile() call impossible (when larger than 4G).  Thus, it seemed
likely the only way to gain consensus was to support a maximum frame length
much larger than 32 bits.

I agree in the spec we need to clarify behavior regarding frame size.  I
suspect we will wind up adding some way of declaring maximum acceptable
frame size during the handshake, but we have yet to start working on trying
to nail that down.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 22 Aug 2010 18:50:37 +0200

On 22.08.2010 18:39, John Tamplin wrote:
> ...
> Just because the JS API today passes the messages totally in RAM, there
> is no reason that it can be extended to support a stream-like API if
> processing larger messages becomes justified.  However, at the framing
> level, we have to have something in place such that v1.0 implementations
> written against this spec can interoperate with later extensions, so it
> seems prudent to make sure the protocol is sufficient for any reasonably
> forseeable needs.  There were several strong arguments that 32-bits
> isn't enough, and the next logical step up is 63/64-bits.
>
> Personally, I would be perfectly happy with a 32-bit maximum frame
> length, and let senders fragment larger messages.  However, there were
> significant objections (see the discussions link from
> http://www.ietf.org/mail-archive/web/hybi/current/msg03399.html for
> details) about being required to fragment a message when its size was
> known, and the fact that it made "fire and forget" of sending the
> message contents with a single sendfile() call impossible (when larger
> than 4G).  Thus, it seemed likely the only way to gain consensus was to
> support a maximum frame length much larger than 32 bits.
> ...

Well, we need only rough consensus.

32 bits has the advantage that it's big enough for almost all use case 
we can currently think of.

32 bits has the drawback that people who cross the 2/4GB barrier might 
encounter interop problems because of insufficient testing.

One way to address this problem are BIG WARNINGs in the spec, and a test 
suite.

Best regards, Julian

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 12:57:29 -0400

> Well, we need only rough consensus.

True, but as one of the potential implementors (Maciej of WebKit) was
strongly against it (
http://www.ietf.org/mail-archive/web/hybi/current/msg02977.html) and various
others were also strongly against requiring fragmentation for messages over
4G, it seems like we would not achieve consensus with a 32-bit maximum frame
length.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 12:59:50 -0400

> frames to save CPU - the cost of the doing the fragmentation will dwarf

I sensibly assumed that the end that has the load issue (thus is not the
end doing the fragmenting) will choose the 125 frame size, because it has
no way to know that the sender wants to send large frames at the time the
maximum size handshake is negotiated. Even if it did know, I think it may
act selfishly due to economics.

> significant additional processing requirement.

I do not understand. With Option 1/15-bit, the 16-bits is read, the top
bit is cleared, and that is the size to read (plus extra byte for rest of
frame header).  No conditionals (notwithstanding extensions which is an
orthogonal discussion).

> byte for the <2% case, which on average costs at least 0.96 bytes more per

Say the average size is 50 bytes, then that is only a neglible 2% network
overhead cost.  If we fragment large messages to 125 instead of 32767 byte
frames, not only do we increase the network overhead to 1.6% but we
drastically increase CPU overhead.

I am arguing that maybe it is not better to squeeze the last 2% of blood
of performance and have it pop out as aliasing error in our design with a
huge bottleneck in another scenario.  That is the way nature (entropy,
Murphy's Law) works if we aren't careful in our analysis.  I am also
trying to propose concensus that makes everyone happy.

---

From: gustav <[email-masked]>
Date: Sun, 22 Aug 2010 19:46:11 +0200

On 22 August 2010 18:50, Julian Reschke <[email-masked]> wrote:

> 32 bits has the advantage that it's big enough for almost all use case we
> can currently think of.
>
"Rough consensus" as you and one more person runs the rest over or exactly
how do you mean ?.

"Interop problems". so your trying to quantifify programmers assumed
incompetence in this very specific detail and use it as an  argument ? =)
Even if your right - so what ? , that's how it always is with new stuff, and
how people tend to be,  should that be a valid argument to stop innovation
?. it can be used against everything !

I was against 63/64 bit frames too, but now when i heard peoples arguments,
i agree that its good to have,
especially since we can still get optimal bandwidth encoded  short frames,
option2 and some later proposal offers that.

regards
  gustav trede

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 13:54:58 -0400

>> significant additional processing requirement.

CLARIFICATION of the context:

I meant in the case where maximum size had been negotiated to 15-bits.

Of course Option 1/15-bit would still support 8 byte size when the maximum
size was negotiated to larger than 32767.

========
I hope everyone understands my point.  I am saying that if you are going
to assume that some routers will decided to negotiate a fixed maximum size
that has no conditional tests for size header, then it is more efficient
overall to use Option 1/15-bit, than Option 1 or Option 2.

It is a bit convoluted to understand, but I think my logic is correct. I
hope others convey that they understand or they correct my logic.

---

From: gustav <[email-masked]>
Date: Sun, 22 Aug 2010 20:05:38 +0200

> trying to propose concensus that makes everyone happy.

There is focus on extremely short frames and insanely large frames.
There applications that will have a lot of frames that are inbetween, and
hence wants it efficient as in option2 (1/2/8 header lengths) or similar
compact concept.

My game server with webgl clients needs to send state changes for what many
other clients do do each of them.
The client sends does indeed get away with 7bit frame lengths, compressed
chat text fits good !.
However it can be several 100 clients that must get their discrete state
machine changes from the server.
For increase of overall efficiency i use rather long server ticks per logic
loop,  the adapted game mechanic design makes it transparent to the user
(client side prediction helps).
The 'long' server ticks leads to more logical state changes being sent per
data frame,
Compression etc works better and i need fewer expensive socket writes, that
each carry a _relative_ heavy penalty.
So it will from the server side exist frames with with more then 7bit
length, and i want them to be optimally encoded regarding both bandwidth and
cpu where bandwidth takes somewhat precedence.

regards
 gustav trede

---

From: Salvatore Loreto <[email-masked]>
Date: Sun, 22 Aug 2010 20:45:28 +0200

On 8/21/10 9:59 PM, Pieter Hintjens wrote:
> On Sat, Aug 21, 2010 at 9:34 PM, John Tamplin<[email-masked]>  wrote:
>
>    
(as individual)

I agree on the fact that with the fact that adding an extra byte to the 
WebSocket header would
cost very little also for very small payload.

I don't understand because 4bytes basic header (+length field +payload) 
as defined in http://tools.ietf.org/html/draft-ietf-core-coap-01#section-3.1
is good for constrained devices and constrained networks: btw 
constrained devices are the one usually just exchanging 1byte payload

and we can not define a 2byte basic header (+length field +payload) for 
WebSocket, reducing the complexity present in the previous proposals
using 1 or 2 bits to indicate the length header,
and also providing more room (bits) for future extensions (actually we 
could also think to use 2bits for version the protocol)

cheers
/Sal

-- 
Salvatore Loreto
www.sloreto.com

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 15:01:03 -0400

Resending because I forgot to quote what I had written, which was critical
to the point!

Angry at myself.  Apologies.

Also clarified the point more, because I am still wondering if anyone
understands what I mean?

========

The point above is that some agents may choose to always set their maximum
frame size to the size of the _SMALLEST_ size field, so that they don't
have to do comparison tests.

It was reported that on this list (and further discussed in private):

http://www.ietf.org/mail-archive/web/hybi/current/msg03453.html

that multi-core coordination overhead may not scale proportionally with
network bandwidth, and thus making the protocol as stateless as possible
is very important in that case. I am not arguing that is the case. I am
trusting the expertise of concensus. I am also saying why should we risk
it for 2% savings in network bandwidth overhead?

---

From: Brian <[email-masked]>
Date: Sun, 22 Aug 2010 12:24:31 -0700

Then that one bit isn't reserved, it's defined as a bit that adds an
optional "flags" field that contains 7 undefined bits and another "add
another flags field" bit.

Seems simpler to just throw in the one-byte "reserved" field that can be
used for whatever down the line.  If we later need more fields in the
header, then one of those bits could be designated to indicate such at that
time.

I guess we need to decide how many reserved bits we care to keep in the
initial header and let that drive the solution we come up with.  I'm happy
to punt on the issue and keep plenty by adding an extra byte.

Brian

---

From: Scott Ferguson <[email-masked]>
Date: Sun, 22 Aug 2010 12:31:52 -0700

Salvatore Loreto wrote:
> I agree on the fact that with the fact that adding an extra byte to 
> the WebSocket header would
> I don't understand because 4bytes basic header (+length field 
> +payload) as defined in 
> http://tools.ietf.org/html/draft-ietf-core-coap-01#section-3.1
> and we can not define a 2byte basic header (+length field +payload) 
> for WebSocket, reducing the complexity present in the previous proposals

Something like the following?

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+---+-------+---------------+
|F|T|LEN|OPCODE | reserved(8)   |
|I|M|   |       |               |
|N|L|   |       |               |
+-+-+---+-------+---------------+

LEN: 00 = 1 byte length
     01 = 2 byte length
     10 = reserved (or 4 byte)
     11 = 8 byte length

It might also be possible to define 00 to use the reserved byte for an 
ultra-compact header, or 00 could be ultra-compact, and 01 could be the 
normal 1 byte after the reserved and 10 be 2 byte.

LEN: 00 = 0 byte length (uses reserved byte)
     01 = 1 byte length
     10 = 2 byte length
     11 = 8 byte length

(For versioning, a hello opcode might be better than taking up frame 
bits, and a hello opcode would also allow digest-authentication 
credentials to piggyback on the handshake replacing the 8 random bytes, 
which is an idea I'd like to bring up when we get to discussing the 
handshake because I'd love to save a roundtrip when authenticating 
non-browser clients.)

-- Scott

---

From: Salvatore Loreto <[email-masked]>
Date: Sun, 22 Aug 2010 21:44:26 +0200

> Something like the following?
(again as individual)

Hi Scott

almost, what I had in mind was

  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+---+-+-------+--------------+
|F|T|LEN|R|OPCODE | reserved(7)  |
|I|M|   |E|       |              |
|N|L|   |S|       |              |
+-+-+---+-+-------+--------------+

in this way we have the possibility to add 1 extra bit to OPCODE in the 
case we will run out of it

+1
> handshake because I'd love to save a roundtrip when authenticating

For versioning I don't have a strong opinion, it was just an example of 
the fact that we need enough room (bits) for extansions

-- 
Salvatore Loreto
www.sloreto.com

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 15:48:45 -0400

> size was negotiated to larger than 32767.

I don't think it would be a good idea to have the interpretation of the
framing itself vary depending on the handshake.  I realize it could work,
but I would prefer that if the field is a 15-bit length then it always means
that, not with a special case of 7FFF meaning an extra 8-byte length field
is present, but only when the negotiated maximum frame size was greater than
32766.

If people are complaining about the complexity of the existing proposals, I
would hate to hear them after this.

Also, it effectively ties defining the framing to defining the handshake,
since you cannot properly interpret a frame without knowing the handshake.

You could define the 32767 value as reserved and treat the 8-byte length
field as an extension, though I suspect those who object to having to
fragment a message would not like punting that to an extension.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 15:55:14 -0400

> have to do comparison tests.

I don't believe that is what was being suggested, and if you account for the
processing going on on the client I am sure it will be much more to process
n fragments rather than one condition in looking at the length field.

> it for 2% savings in network bandwidth overhead?

I don't understand "stateless" in this context -- we are not talking about
maintaining state across different frames, but rather you look at one byte
and if it is a given value (or two given values in option 2) then you read
following bits for the length instead.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 16:05:53 -0400

> almost, what I had in mind was

Why not just move the opcode all the way to the right, and then we can
allocate reserved bits from the left as separate flags or from the right to
extend the opcode field?

>> LEN: 00 = 0 byte length (uses reserved byte)

I still think it is a mistake to increase the size of the overwhelming
majority of frames to make things easier for the less-frequent (and larger,
so that the extra overhead is negligible) frames, but I am tired of arguing
about it.  If this is what people want then let's go with it.

---

From: Brian <[email-masked]>
Date: Sun, 22 Aug 2010 13:06:30 -0700

> logical step up is 63/64-bits.

Yes.  What is "reasonable" is what I'm getting at.  Compression and
fragmentation seem reasonable... multiplexing seems like something that
should be handled by the subprotocol to me, as most of them won't need it
and how to multiplex seems more likely to depend on the subprotocol itself.

As for the arguments for 63/64 bits, I don't think they were that strong.  I
think it's more that fewer people felt like expending the energy to debate
them and keep the discussion on track in terms of what this protocol is for,
so the edge case people won.

If JavaScript-based browser clients are truly thought to be doing something
useful with large frames >4GB, then fine, though I doubt it.  Even an abuse
of WS like video streaming would use a series of shorter frames in practice.
 If that length requirement was envisioned for usage outside of JS and the
browser, then it's not appropriate.

> details) about being required to fragment a message when its size was known,
> and the fact that it made "fire and forget" of sending the message contents
> with a single sendfile() call impossible (when larger than 4G).  Thus, it
> seemed likely the only way to gain consensus was to support a maximum frame
> length much larger than 32 bits.
>
>
Yes.  That whole sendfile() debate, from where I'm sitting, was completely
irrelevant to the discussion of a protocol meant to enable realtime
bi-directional communications with browser clients.

The purpose of this protocol is basically to enable the kinds of
communications that Comet attempts to solve by abusing the HTTP protocol,
and to open new frontiers to things we haven't considered doing in the
browser yet (binary frames etc).  Some things are useful and easy to throw
in.  Some things are just a distraction from the true purpose of the
protocol.  64-bit lengths is "easy to throw in" until a discussion of the
details consumes a full week+ of the working group's extremely valuable
time.  At this point, we need to pick something and put a bow on it so it
doesn't cost any more than it already has.

Brian

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 17:14:22 -0400

> since you cannot properly interpret a frame without knowing the handshake.

So then the maximum size from the handshake is not a maximum size and the
other end can violate it.

I can really hear the complaining from that.  "What the maximum size is
not a maximum size, it is just a mild suggestion?!?!"

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 17:22:08 -0400

> not a maximum size, it is just a mild suggestion?!?!"

I would think that the spec will have to say an implementation is free to
drop a frame larger than an implementation-defined limit.  The value in the
handshake, then, is just a promise of "I won't drop your frame for excessive
length if it is below this size".

---

From: Scott Ferguson <[email-masked]>
Date: Sun, 22 Aug 2010 14:27:28 -0700

John Tamplin wrote:
> On Sun, Aug 22, 2010 at 3:44 PM, Salvatore Loreto 
> <[email-masked] <mailto:[email-masked]>> 
> wrote:
>
>     (again as individual)
>
>     Hi Scott
>
>     in this way we have the possibility to add 1 extra bit to OPCODE
>     in the case we will run out of it
>
>
> allocate reserved bits from the left as separate flags or from the 
> right to extend the opcode field?

Those are good points. Moving it to the right would also keep the opcode 
aligned, which is nice for debugging.

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+---+-+-+---------------+-------+
|LEN|T|F| reserved(8)   |OPCODE |
|   |M|I|               |       |
|   |L|N|               |       |
+---+-+-+---------------+-------+

> majority of frames to make things easier for the less-frequent (and 
> larger, so that the extra overhead is negligible) frames, but I am 
> tired of arguing about it.  If this is what people want then let's go 
> with it.

I think the compact frame (LEN 00 = 0 byte length) would give you that 
efficiency because the compact frame only uses 2 bytes total for the 
header and length:

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+---+-+-+-------------+-+-------+
|00 |T|F| length(7)   |R|OPCODE |
|   |M|I|             |E|       |
|   |L|N|             |S|       |
+---+-+-+-------------+-+-------+

Where length(7) is the payload length and the payload immediately 
follows the header.

If we were willing to trade some extra complexity for extension bits, we 
could even reuse the TML and FIN bits for the compact header, because 
those bits aren't needed for small frames. So the compact (2 byte) 
header might be:

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+---+---------------+---+-------+
|00 | length(8)     |RES|OPCODE |
|   |               |   |       |
|   |               |   |       |
+---+---------------+---+-------+

I'm not sure if it's a good idea because it does make the compact frame 
a special case, but it frees up some extra reserved bits.

I think that would give you the compact frame with the minimal overhead 
that you want, while still reserving 8 bits for the standard frames.

-- Scott

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 17:32:19 -0400

> n fragments rather than one condition in looking at the length field.

I think you may not understand the issue that was stated by Pieter.  I get
the feeling the no one but Pieter understands this issue?

below...

> following bits for the length instead.

Pieter refused to explain it further on list:

only in private and now I am beginning to understand why.  I have tried 3
times to explain it.  Let me try again.  I hope I am not misstating the
issue that concerned Pieter.  This was not my issue.  I adopted it,
because I think I understand it, and seems no else did?  It sounds like it
could be very important in about 5 years...

Network bandwidth is increasing faster than single-core speeds are.
Multi-cores are being used to keep up with Moore's Law.

But the problem is that stateful (Turing complete) algorithms don't scale
well to multi-core.  Stateless (lambda pure) algorithms do scale well to
multi-core. Maybe you are aware that pure Haskell can scale by compiler to
any number of multi-core, whereas stateful languages are brittle mashups
of mutex and semaphore heuristics (not mathematically proveable).

The problem with stateful code is that there is always point where adding
more cores actually starts to reduce the speed (because of the
non-mathematical fit of the algorithm and the communication/coordination
overhead between cores growing faster than the speed added by the
additional cores).  Basically you end up in mutex/semaphore deadlock hell
of inefficiency.

If I am not mistaken, the comparisons in the size header field logic,
create a state-machine that has to be communicated/coordinated with the
multiple cores.  The comparisons change the algorithm from stateless to
stateful.  Without the comparisons (and other opcodes and state changes to
be added), more cores can be used before the inefficiency overwhelms.

Okay maybe I have misstated the issue, but that is my understanding.

And it is worse the smaller the frames are.

And also I just think the 15-bits versus 7-bits as the maximum size that
can go through without comparisons is safer.  We can not see every
possible issue from this WG.  We don't lose much at all, maybe 2 - 5% in
network bandwidth, but we gain a lot of comfort zone against the
unexpected.

That is my 2 cents.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 17:48:18 -0400

On Sun, Aug 22, 2010 at 5:27 PM, Scott Ferguson <[email-masked]> wrote:

> efficiency because the compact frame only uses 2 bytes total for the header
> and length:
>
>
> Where length(7) is the payload length and the payload immediately follows
> the header.
>

The problem with this is we only have a single reserved bit, which is highly
unlikely to be sufficient when we get to extensions.  That means if those
extensions are commonly used, we will be paying the extra byte anyway and be
doing it in a more awkward way.

> could even reuse the TML and FIN bits for the compact header, because those
> bits aren't needed for small frames. So the compact (2 byte) header might
> be:

If you are concerned about latency, you might well want to flush the buffer
with a small frame, with more fragments to follow when the dynamic content
is generated.

> I'm not sure if it's a good idea because it does make the compact frame a
> special case, but it frees up some extra reserved bits.
>

I think it is a false economy, because you only have 3 reserved bits in this
form, while 8 in the others.  If you assign a 4th reserved bit, you now have
to come up with a new way of representing it in the short frame, so those
extra reserved bits are less useful than the others.  Also, given the
complaints about complexity for simpler things, I can't imagine having
MORE/CMLP bits be dependent on the length-length bits would go over well.

I still think option #2 is the best compromise, and a number of people
agreed (and a number disagreed).  If that option isn't deemed acceptable and
having 4 length steps is preferred, I would rather just eat the extra byte
for small frames and go with the suggestion above, having headers of 3, 4,
6, and 10 bytes with 8 reserved bits between the defined flags and the
opcode, as in:

   - More - 1 bit
   - CMLP - 1 bit
   - LenLen - 2 bits
   - Reserved - 8 bits
   - Opcode - 4 bits
   - Length - 2^LenLen bytes
   - CML - 8 bytes - present only if CMLP=1
   - Payload - Length bytes

Otherwise, as before.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 17:57:30 -0400

> Multi-cores are being used to keep up with Moore's Law.

I disagreed with that point as did others, but it is irrelevant to the
discussion at hand.  He brought that point up in the discussion about
compressing frames to get the benefit of a small frame.

> of mutex and semaphore heuristics (not mathematically proveable).

Yes, I am aware.  You aren't going to be parallelizing decoding this frame
anyway.  You can't start processing the payload until you have understood
the opcode (which indirectly gets you the start of the payload data), the
CMLP flag, and parsed the length (which gets you the end of the payload
data).  Even if the length is a fixed 8-byte field, there is still an
ordering requirement.

> be added), more cores can be used before the inefficiency overwhelms.

You aren't going to have mutliple cores trying to decode a single frame
anyway -- the synchronization overhead will be far more than the cost to
process even the most complex framing we have discussed here.  You might try
and farm off (de)compression to another core, but again that is different
than trying to decide how many bytes there are in the frame.

Note that Pieter initially voted for Option #1, then introduced Option #0
which was the same but assigned a reserved bit to the length field.  It
still had a comparison for the length, so I don't think you are accurately
representing his opinion.

> network bandwidth, but we gain a lot of comfort zone against the

Please explain how it can go through without comparison.  Even if you
allocate a bit to define which length to use, you still have to test that
bit.  If you negotiate in the handshake a maximum, you have to test some
flag you set during negotiation.

---

From: Salvatore Loreto <[email-masked]>
Date: Mon, 23 Aug 2010 00:12:44 +0200

On 8/22/10 11:48 PM, John Tamplin wrote:
> On Sun, Aug 22, 2010 at 5:27 PM, Scott Ferguson <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>         I still think it is a mistake to increase the size of the
>         overwhelming majority of frames to make things easier for the
>         less-frequent (and larger, so that the extra overhead is
>         negligible) frames, but I am tired of arguing about it.  If
>         this is what people want then let's go with it.
>
>
>     I think the compact frame (LEN 00 = 0 byte length) would give you
>     that efficiency because the compact frame only uses 2 bytes total
>     for the header and length:
>
>
> The problem with this is we only have a single reserved bit, which is 
> highly unlikely to be sufficient when we get to extensions.  That 
> means if those extensions are commonly used, we will be paying the 
> extra byte anyway and be doing it in a more awkward way.
>
>     If we were willing to trade some extra complexity for extension
>     bits, we could even reuse the TML and FIN bits for the compact
>     header, because those bits aren't needed for small frames. So the
>     compact (2 byte) header might be:
>
>
> If you are concerned about latency, you might well want to flush the 
> buffer with a small frame, with more fragments to follow when the 
> dynamic content is generated.
>
>     I'm not sure if it's a good idea because it does make the compact
>     frame a special case, but it frees up some extra reserved bits.
>
>
> I think it is a false economy, because you only have 3 reserved bits 
> in this form, while 8 in the others.  If you assign a 4th reserved 
> bit, you now have to come up with a new way of representing it in the 
> short frame, so those extra reserved bits are less useful than the 
> others.  Also, given the complaints about complexity for simpler 
> things, I can't imagine having MORE/CMLP bits be dependent on the 
> length-length bits would go over well.
>
> agreed (and a number disagreed).
(as individual)
Hi John,

I know your preferences... but this is just an alternative/compromise if 
we can find a larger consensus
> If that option isn't deemed acceptable and having 4 length steps is 
> preferred,
(as individual)

it could be just a possible compromise among who wants 32 and who 64.
Then we do not need to use all 4 length steps, but also keep 1 step for 
future definition...
> I would rather just eat the extra byte for small frames and go with 
> the suggestion above, having headers of 3, 4, 6, and 10 bytes with 8 
> reserved bits between the defined flags and the opcode, as in:
>
>     * More - 1 bit
>     * CMLP - 1 bit
>     * LenLen - 2 bits
>     * Reserved - 8 bits
>     * Opcode - 4 bits
>     * Length - 2^LenLen bytes
>     * CML - 8 bytes - present only if CMLP=1
>     * Payload - Length bytes
>
(as individual)

+1
I like this proposed order.

cheers
/Sal

-- 
Salvatore Loreto
www.sloreto.com

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 23 Aug 2010 00:39:03 +0200

> 32 bits has the advantage that it's big enough for almost all use case we

At least one person has cited a case where a 31-bit length caused a
failure.  It's also naive to design for today's use cases and ignore
the basic rule that data volumes and sizes double every 24 months.
640K is enough for anyone, right?

The only cases that really matter here are:

- very short frames, where every byte is significant
- very long frames, where any ceiling is significant

Medium length frames can be treated as long frames with zero
significant cost.  You could stick a 100-byte header on a 100K frame
and it would not be significant.

There is no point optimizing any case except the two: very short (trim
space), very long (trim complexity).

It's also naive to claim that WebSockets is only for JavaScript, only
for dumb browsers, only for medium size data.  We're designing "TCP
over HTTP" in a real sense and there are generations of applications
that will use this, way beyond today's use cases.  Please, think of
the children!

-
Pieter Hintjens
iMatix - www.imatix.com

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 18:51:46 -0400

Exactly processing the header is a stateful and ordered process.

Thus, the time that the CPU spends processing the header, is time that the
network might be idled. The bottleneck is processing the header and then
shunting the frame processing to the other cores. So as the frames get
smaller, the header is larger % of network bandwidth. If the CPU needs
more time processing the header than it takes to move the header over the
network, then at the asymptote of 0 size payload, then network bandwith is
more than 50% wasted.  I nearly certain that as Pieter's point.

Thus eliminating the comparisons is crucial for very small packets.

So it is likely that massively multi-core implementations will choose to
set the maximum size to 125 if you go with Option 2, or 126 with Option 1,
because that is where their economy is maximized.

That is why I proposed you go with Option 1/15-bit, so they can set the
maximum size to 32767 instead.

The comparisons can be eliminated if you set maximum at handshake and
expect the sender to honor it.  You can have a separate version of your
code for that hardcoded maximum.  Even self-writing code if necessary.

Do you finally get my point?

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 18:58:03 -0400

> length if it is below this size".

That forces the receiver to check for the larger size.  See my prior post
on the multi-core issue, that it would be more efficient in that case if
the sender honored the maximum.

Why can't the sender honor the agreed maximum size?

I think many people here requested a simple 16bit or 32bit size field with
no escape to 8 bytes.  By honoring the maximum size and using Option
1/15-bit, you will give them that, and still keep the 8 byte escape for
those receivers and senders who agree to use it on handshake.

Any way, you all know best, I was just trying to help process the requests
of others.  I listened to what they said.  I didn't try to force my Option
2 preference on them.

---

From: John Tamplin <[email-masked]>
Date: Sun, 22 Aug 2010 19:08:00 -0400

> Why can't the sender honor the agreed maximum size?

The receiver will still have to check, because the sender might be an
attacker hoping to take advantage of a receiver that only checks the first 2
bytes of the length while firewalls/etc would have filtered it had it been
correct.  Getting different entities to make different decisions about some
framing of data is a fundamental attack tactic.

---

From: Scott Ferguson <[email-masked]>
Date: Sun, 22 Aug 2010 16:23:29 -0700

> space), very long (trim complexity).

That's only true when you know the length before serializing the 
payload. For dynamic content like JSON or a Wave stanza, which are 
frequently short, the medium length of the serialization buffer is 
better, which is why the 1/2/8 variations like option #2 are good 
compromises.

1 is best for hand-optimized protocols like Quake.
2 is best for dynamic content that is frequently short
8 is best for sendfile

-- Scott

---

From: Brian <[email-masked]>
Date: Sun, 22 Aug 2010 16:26:17 -0700

+1

Brian McKelvey

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 23 Aug 2010 01:33:12 +0200

Ack.

-Pieter

---

From: Brian <[email-masked]>
Date: Sun, 22 Aug 2010 16:41:56 -0700

> that will use this, way beyond today's use cases.  Please, think of
Indeed... But not entirely.  What we're attempting to build is more like a
single TCP connection, rather than the whole of TCP.  We're not concerning
ourselves with things like src/dest port numbers, etc.  We can leverage TCP
itself for a lot of things.

Brian

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 19:53:29 -0400

> framing of data is a fundamental attack tactic.

Isn't that incorrect logic?

If the spec says that the maximum is the maximum, then firewall should
respect the spec.

Any sender can send any thing any time.  That is not relevant here.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 20:17:52 -0400

+1

This fulfills what I was trying to achieve with Option 1/15-bit, but uses
the extra byte more efficiently.  I predict this will be the winner.  It
hits all the needs.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 22 Aug 2010 21:01:18 -0400

Correction. Pieter are you sure you want to vote for this?

http://www.ietf.org/mail-archive/web/hybi/current/msg03510.html

Seems to me that Option 1/15-bit is superior, because then you won't have
to check the LenLen every header and you get larger maximum size that can
be used with no comparisons?

But I guess reading the LenLen and shifting by it and reading that many
bytes is an efficient operation, still no comparisons.  So I will keep my
vote for 2^LenLen option, but I will also vote for Option 1/15-bit.

I will no longer vote for Option 2, because of the above link.

>>>> agreed (and a number disagreed). If that option isn't deemed
>>>> acceptable
>> and
>>>> for small frames and go with the suggestion above, having headers of
>>>> 3,
>> 4,

---

From: gustav <[email-masked]>
Date: Mon, 23 Aug 2010 10:32:39 +0200

On 23 August 2010 03:01, Shelby Moore <[email-masked]> wrote:

> That argument is focusing only on one of of many ways to do handle IO in an
optimal way.
Its not a universal truth that its optimal do only parse the header before
off loading to other threads.
It all depends on your exact needs regarding latency,overall throughput etc,
and how the different alternatives perform on your OS and configuration.
For some situations it is optimal to have one IO thread per core and do a
lot of work in them.
For my specific application its the best strategy i have found this far
using dev poll on solaris.

regards
 gustav trede

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 23 Aug 2010 06:58:15 -0400

My reply is here:

http://www.ietf.org/mail-archive/web/hybi/current/msg03545.html

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 23 Aug 2010 07:05:06 -0400

> does that mean a router can't forward them?

Fragmentation is one example of why intermediaries may need to read
WebSockets protocol:

I understand that if an intermediary does not want to inspect the
WebSocket data, they can just operate at the TCP layer.  Others have
pointed out that enterprise firewalls will whitelist any protocol that
they can't filter the data (e.g. TLS).

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 23 Aug 2010 07:36:47 -0400

> space), very long (trim complexity).

On complexity of large frames:

"Tangentially, the limit can be if avg is too high, requiring intra-frame
(splitting) parallelization, which depends on the intra-core signalling
cost due to complexity of the state-machine for the frame data processing
(extension protocols, any application on top, etc)."

Btw, I appreciated Pieter's point very much.  WebSockets is at least TCP
over HTTP, not just Javascript push.

And I also think WebSockets should be TCP for WebApps, not just HTTP,
since WebApps was the main point of the original WhatG.

And that is why I proposed that we should not be ignoring the opportunity
to do P2P:

http://www.ietf.org/mail-archive/web/hybi/current/msg03231.html
(whole thread)

I had a private discussion yesterday that went like this slam-dunk:

======================================
> You can't be the economy-of-scale of server farms (client-server)

What is the cost barrier that makes uneconomic the harvesting billions of
wasted CPU and network cycles?

> Centralized enables things that can't be done without, e.g. improved
anti-spam

There are distributed models to deal with spam.  The social network is one
model, which could be distributed.

But more importantly, there are probably very important apps that can only
be done distributed.  For example video chat.  It makes no sense for my
packets to go back to your server farm, when they can travel directly to
the person who is in my same island.  And downloading is another one.  No
sense that my packets can't come from someone else who is downloading at
same time.

But nobody can do P2P in a browser now.  Our hands are tied, so we don't
know what apps would be created.

> Windoze users won't switch to Linux, because they are too lazy to
> appreciate the freedom of rolling your own configuation.

Linsucks will never succeed in mass-market because it violates the
economic law of economy-of-scale, otherwise known as the maximum
division-of-labor.

People prefer a plethoria of diverse applications that are pre-programmed
to be close to what they want.  People prefer 0 complexity and one-click
use (no learning curve, nothing to memorize!). People don't want to waste
a single second of their life programming their TV remote control. Webapps
are the only way to converge users away from Windoze.
What you have now is mashups, not deep apps that are needed.
======================================

Turn on your Wake up on LAN folks.  Get the big picture.