Subject: [hybi] Summary of open issues regarding Framing Proposal Take VII

From: John Tamplin <[email-masked]>
Date: Thu, 19 Aug 2010 19:28:59 -0400

I am trying to summarize here what seem to be issues up for further
discussion -- forgive me if I leave something out:

   1. the use of a 2-level length field vs a fixed 64-bit length or a
   variable-length-length field (as in v76)
   2. the use of continuation opcode
   3. putting close/ping/pong in the control frame payload rather than as
   separate opcodes
   4. the presence of the complete message length field in the base protocol

If there are any others, feel free to reply to this thread.

#1 - length field
=================
For #1, I think a fixed 64-bit length is unacceptable because of the penalty
it imposes on small messages.  As discussed previously, existing WebSocket
apps make heavy use of small frames, compression will make more frames
smaller, and I suspect many future WebSocket apps will make heavy use of
small frames.  Finally, existing mobile networks pay a penalty for larger
frames, so we should strive to make as many as possible fit in small frames.

Regarding the variable-length length, I prefer the current proposal for the
following reasons:

   - it saves a bit in the header, which gives us more options for future
   expansion
   - it has two steps of length rather than 9, and looking at the overhead
   at various packet lengths there is little justification for more than 3
   - reading one byte at a time imposes an (admittedly small) penalty on the
   performance of the reader
   - the variable-length length has additional complications which need to
   be defined away, namely leading 0x80 bytes and the maximum length of the
   field.  I would suspect that many naive implementations use a 32-bit field
   for the length and would fail to notice overflow when shifting the current
   length.  Being able to convince some implementations that the frame length
   is different than others would be a key way to exploit vulnerabilities.

That said, I would not block consensus over this issue -- if others feel the
variable-length length is better I could live with it -- we would just move
RSV2 into the length field, and define that additional length bytes follow
if the high bit is set (while outlawing leading 0x80 bytes and limiting the
number of bytes in the length to 9, giving the same 63 bits of length as the
current proposal).

#2 - the use of the continuation opcode
=======================================
What I was trying to avoid here was defining how changing opcodes across
fragments should be defined.  What do you do when the first frame has a
different opcode than later frames?  Should the last one win even if the
receiver has already started decoding it based on the first opcode?  Should
the receiver check it and drop the message if the opcode changes?

I would not object to leaving this out and simply defining it as a protocol
error to use a different opcode in a continuation frame, though I worry that
might open up more opportunities for bugs where an implementation doesn't
verify that and opens an attack vector (admittedly, if an implementation
doesn't verify the continuation opcode there may still be attacks, but if it
is specifically expected I suspect that would make implementations more
robust - they might ignore the opcodes of subsequent packets which would be
mostly ok).

#3 - putting close/ping/pong/etc in a control frame rather than giving them
their own opcode
============================================================================================
These frames will be infrequent and we have limited opcode space (especially
if we primarily use opcodes for extensions).  I would strongly prefer
defining a control opcode and moving all out-of-band operations into the
control-frame payload.

Here too, I would not block consensus over this issue, so if most people
prefer them as opcodes I could go with that, though we would likely need to
allocate one of our precious reserved bits to the opcode field to do so, as
otherwise we would be leaving little room for growth.  Note that while we
could later add an extended opcode that lets you allocate additional
opcodes, that would be no different than using the control frame as
proposed, other than we would have reserved first-class opcodes for less
frequent frames while delegating presumably more frequent opcodes to
second-class status.

#4 - the presence of the complete message length field in the base protocol
===========================================================================
If we are supporting fragmentation in the base protocol, it seems that an
intermediary that fragments the message should be able to provide the
receiver the original message length, without requiring that recipient
support extensions.

Again, I would not object strongly to removing this (leaving CMLP as a
reserved bit) as long as that didn't bring back the objections about
fragmentation it was intended to address.

So, while I prefer the proposed solution in each of the above cases, I am
not entrenched against other options.  Can we get a feel from the group on
the following questions:

   - are there other divisive issues which need to be addressed before we
   can declare the framing for the base protocol done?
   - does anyone feel strongly enough about the above issues that they want
   to make a case for why an alternative should be chosen?  Or are there a lot
   of people who would be happier if a different choice was made?

If the answer to both of the above are no, can we accept this as the base
protocol?

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: gustav trede <[email-masked]>
Date: Fri, 20 Aug 2010 02:24:39 +0200

On 20 August 2010 01:28, John Tamplin <[email-masked]> wrote:

> For #1, I think a fixed 64-bit length is unacceptable because of the
> penalty it imposes on small messages.  As discussed previously, existing
> WebSocket apps make heavy use of small frames, compression will make more
> frames smaller, and I suspect many future WebSocket apps will make heavy use
> of small frames.  Finally, existing mobile networks pay a penalty for larger
>    - reading one byte at a time imposes an (admittedly small) penalty on
>    the performance of the reader
> Seriously,  should the assumed incompetence of programmers be an argument
for how to define protocols ?.
You cant protect idiots from themselves, they will find 1000 other ways to
screw things up regardless.
What is the future of software engineering , if the standards should be
defined by that kind of metrics instead of sound technical reasoning ?.
People who cant handle bits and bytes should perhaps not be implementing at
that level, but use some point and click tool instead.

regards
  gustav trede

---

From: John Tamplin <[email-masked]>
Date: Thu, 19 Aug 2010 21:51:00 -0400

> that level, but use some point and click tool instead.

If two options are otherwise identical, yet one of them is more likely to
induce errors by implementors, surely you would prefer the other one, right?
 I am not saying that should be the determining factor (I argued strenuously
against the amateur programmer requirement earlier on this list), but
certainly it should be considered.

You can certainly write secure code using strcpy/sprintf/etc, yet today
those methods have been implicated in enough security flaws that their use
is discouraged, even when you "know" it is safe.  Many software engineering
practices are about reducing the opportunity for bugs to arise as much as
anything else -- DRY is only marginally about saving time, the real benefit
is that you don't have multiple pieces of code to maintain, where inevitably
they will drift out of sync with each other and cause bugs, or where writing
it slightly differently multiple times gives more opportunities for bugs to
be written.

If you would like to push for the variable-length length instead of what I
proposed, I suggest writing an argument for how it is beneficial over the
current proposal, taking into consideration arguments that have already been
made.

---

From: gustav trede <[email-masked]>
Date: Fri, 20 Aug 2010 11:07:57 +0200

On 20 August 2010 03:51, John Tamplin <[email-masked]> wrote:

>> People who cant handle bits and bytes should perhaps not be implementing
>> at that level, but use some point and click tool instead.
>>
>

Where is the technical sound reasoning for the 64 bit frame length in the
first place ?.
There is a lot of talk that the expected frame lengths are to be small, so
why have 64 bit at all ?, it just penalizes whenever stuff happens to not
fit in the tiny length, at least make that penalty reasonable - 32 bits
should be ok, the few with larger needs can use multi frame in 4gig blocksm
they for sure dont notice that overhead - so let them have it instead of the
majority that is the smaller frames.

Its fundamentally broken to specify a limit will break at random lower
limits in the real world, we don't offer a path-mtu so its a guessing game.

It would be prudent to have a standard way for a peer to know the upper
frame length limit it can safely use,
currently we cant detect if a used frame length is too large, we don't know
if that's the reason the connection closed when we did send the super huge
frame.

A standard header stating the peer upper frame length limit during the
existing handshake could perhaps solve this problem ?.

IPV6 uses 32bit -1 as the largest jumbo frame size, why do we need more ? -
that argument should technically justify the overhead it brings,
so please have something better then just repeating "other layers overhead
justifies us to spew more overhead on top",
what if every layer spec was designed by that motto ?!,
Also using percentage comparisons vs protocols below us as tool to make it
look better then it is in absolute terms is just a bad marketing trick.
Peoples expected incompetence is not a much better argument (.

regards
 gustav trede

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 20 Aug 2010 06:23:05 -0400

> if that's the reason the connection closed when we did send the super huge

Isn't that handled by refragmentation? Or do you mean the final end or an
intermediary chokes on buffering and re-assembling the fragments and
closes the connection? Would a dynamic mechanism be better where the
failure is reported back as "resend partial frame", then the sender can
scale its frame size dynamically (TCP scales dynamically)?  Because the
issue of maximum may be changing dynamically no (load, etc)?

---

From: Pieter Hintjens <[email-masked]>
Date: Fri, 20 Aug 2010 15:39:59 +0200

There is no significant benefit of 32 vs 64 bits for longer frames and it
introduces a special case at 4GB that will affect every single
implementation. That is a poor tradeoff. Simpler is better and here it is
effectively for free.

-Pieter

Sent from my Android mobile phone.

On Aug 20, 2010 11:11 AM, "gustav trede" <[email-masked]> wrote:
>> On Thu, Aug 19, 2010 at 8:24 PM, gustav trede <[email-masked]
>wrote:
>>

---

From: John Tamplin <[email-masked]>
Date: Fri, 20 Aug 2010 11:08:51 -0400

> majority that is the smaller frames.

There were significant discussions about wanting to be able to write a
header and call sendfile() to write the contents of a large file [1], rather
than having to stay involved and write blocks repeatedly.  Granted, this is
not useful today given the JS API where the entire message has to be
buffered, but: 1) there will be non-JS WebSocket clients that could easily
support a streaming API, 2) the framing protocol has to be defined now in
such a way that implementations of the initial spec can interoperate with
ones implementing a future extension of this spec.  As the frame size is
something that has to be understood to even understand frame boundaries,
that means the way to obtain the frame size, even for frames used by future
extensions, must be present in the base protocol.

Yes, it is true that I expect a significant percentage of the frames of real
WebSocket apps to fit in the small frame size (especially when compressed),
and that there is a real desire to be able to exchange WebSocket frames at
3G FACH power levels rather than having to go up to DACH (see [2] for some
details), but we still need to be able to accommodate larger frames.

When I posted an message summarizing the various positions on fragmentation
and lengths [3], there were no objections to the general principles - only
quibbles about the details.  That and the discussions referenced in [1]
suggests any fixed length shorter than 64-bits would not be acceptable to
large numbers of people on the list, which is why I relented and suggested a
large enough frame size that any message could be sent unfragmented if
desired.

> limits in the real world, we don't offer a path-mtu so its a guessing game.

Sure, lower layers will fragment the large frames, but that happens with
TCP/IP now.  Is SMTP broken because it doesn't try and avoid fragmentation
at the network layer and instead just streams TCP data?

> if that's the reason the connection closed when we did send the super huge

If we were to negotiate it, it would not be at the framing layer but at the
connection handshake.  As far as signalling an error, we haven't really
nailed down the control frames and I would not be opposed to having an
optional error indication in the close control frame which could indicate
why it was dropped.

> existing handshake could perhaps solve this problem ?.

Correct.

> justifies us to spew more overhead on top",

WebSockets is intended to be a minimal message-oriented protocol on top of
TCP.  TCP itself has no notion of maximum message size -- you can just write
arbitrary number of bytes, regardless of the MTU of the underlying network,
and they get there.  I am not sure why you see that WebSockets should care
more about the underlying MTU than TCP does.

> Peoples expected incompetence is not a much better argument (.

I think percentage does make sense because if you are sending 4G of data,
does a wasted 4 bytes in the length field mean anything?  No.

Initially, v76 proposed a variable-length length field for binary frames and
sentinel framing for text frames [4].  There was near universal agreement on
this list that we should move to a single length-based framing for both text
and data, and there was enough support against the variable-length length
field that Ian Hickson switched to a fixed 64-bit length field in v00 [5].
 This was also based on discussions where the frame length was proposed as
16 bits, requiring larger messages to be fragmented, and there was
significant opposition on the grounds that it required simple senders to
have to implement fragmentation and inhibited use of sendfile.  Greg also
noted that the initial Jetty implementation used a signed 32-bit length
internally and failed in a real-world case, so it seems more than 31 bits
are required today, not to mention what will become common during the life
of this protocol [6].

Personally, I don't find a fixed 64-bit length acceptable because it imposes
a significant penalty on small frames.  Thus, the proposal of having two
sizes of lengths: a small one where we care about preserving small frames
for small messages, and a larger one that will meet every need.

I am less concerned about choosing between a v76-style variable length and
as in my most recent proposal -- I prefer what I proposed for the reasons
listed in this thread, but I would not object strongly to switching.

What is the best way to decide between these?   Does anyone feel very
strongly between 7/63 bit lengths and {1-9}*7 bit lengths?  Should we take a
vote?

Related threads:

   - [1] Discussions about length choices
      - http://www.ietf.org/mail-archive/web/hybi/current/msg02974.html
      - http://www.ietf.org/mail-archive/web/hybi/current/msg02977.html
      - http://www.ietf.org/mail-archive/web/hybi/current/msg03025.html
      - http://www.ietf.org/mail-archive/web/hybi/current/msg03039.html
      - http://www.ietf.org/mail-archive/web/hybi/current/msg03051.html
      - http://www.ietf.org/mail-archive/web/hybi/current/msg03061.html
      - http://www.ietf.org/mail-archive/web/hybi/current/msg03064.html
      -
      - [2] http://xmpp.org/extensions/inbox/mobile.html#sect-id129052
   - [3] http://www.ietf.org/mail-archive/web/hybi/current/msg03069.html
   - [4] http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-76
   - [5] http://www.whatwg.org/specs/web-socket-protocol/
   - [6] http://www.ietf.org/mail-archive/web/hybi/current/msg03105.html

---

From: gustav trede <[email-masked]>
Date: Fri, 20 Aug 2010 18:34:16 +0200

On 20 August 2010 17:08, John Tamplin <[email-masked]> wrote:

> Ok. fair enough.

> Yes, it is true that I expect a significant percentage of the frames of
> real WebSocket apps to fit in the small frame size (especially when
> compressed), and that there is a real desire to be able to exchange
> WebSocket frames at 3G FACH power levels rather than having to go up to DACH
> (see [2] for some details), but we still need to be able
> to accommodate larger frames.
>
As long as some end must buffer the entire frame , its broken to promise
support for impossible to attain frame lengths.

>> currently we cant detect if a used frame length is too large, we don't
>> know if that's the reason the connection closed when we did send the super
>> huge frame.
>>
>

>> existing handshake could perhaps solve this problem ?.
Should we then add such negotiation and error signaling ?

>
>> IPV6 uses 32bit -1 as the largest jumbo frame size, why do we need more ?
>> - that argument should technically justify the overhead it brings,

Of course it does not matter in that case, that's my point against the need
for the rare larger then 4 gig frames.
The much more common small frames, why should they suffer a so for them
large overhead ?.
Just to give those rare 4gig+ frames a few single bytes free ?.

Or was it the sendfile() argument ?!,  what does userland logic have to do
with protocol implementations ?,
that exists to abstract away the cleverly designed stuff from users and is
not the place to be lazy and demand easy mode in, that's a userland
attitude.

Its a genera problem in the IT land:
Personalities based on fear  vs  inspiration and innovation kinda makes an
extreme differences at design decisions,
WHY should the fear and incompetence win and enforce their idiocy to the
others? !, it should be the other way around imo.. teach them instead..
At least i refuse to budge for fear.

> Initially, v76 proposed a variable-length length field for binary frames
> and sentinel framing for text frames [4].  There was near universal
> agreement on this list that we should move to a single length-based framing
> for both text and data, and there was enough support against the
> variable-length length field that Ian Hickson switched to a fixed 64-bit
> length field in v00 [5].  This was also based on discussions where the frame
> length was proposed as 16 bits, requiring larger messages to be fragmented,
> and there was significant opposition on the grounds that it required simple
> senders to have to implement fragmentation and inhibited use of sendfile.
>  Greg also noted that the initial Jetty implementation used a signed 32-bit
> length internally and failed in a real-world case, so it seems more than 31
> bits are required today, not to mention what will become common during the
> life of this protocol [6].
>
> Personally, I don't find a fixed 64-bit length acceptable because it
> imposes a significant penalty on small frames.  Thus, the proposal of having
> two sizes of lengths: a small one where we care about preserving small
> frames for small messages, and a larger one that will meet every need.
>

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 20 Aug 2010 13:10:47 -0400

> real

Thanks for providing the links to prior discussion, that clarified many
things for me.

So maximized (and thus variable) OS sendfile() size (and thus variable
size WS framing, because we don't want to move the data in memory mapped
file to insert fixed size WS frame headers, same files may be used in
other than WebSockets) is necessary in order to minimize the TCP window
and WS frame window boundary overlap, which thus maximizes DMA -> NIC
(avoids overloading the CPU with network traffic).

With variable size WebSocket (WS) framing we need fragmentation, as I
explained here:

http://www.ietf.org/mail-archive/web/hybi/current/msg03392.html
http://www.ietf.org/mail-archive/web/hybi/current/msg03395.html

But we don't need WS multiplexing, as that is specific to certain classes
of network ends (usually intermediaries at the server end). Sure one day
the client and connection will so multitasking, but is that the mainstream
case today?  Certainly not here on EDGE/GPRS, 3G is only in the cities and
accumulated bandwidth limited. Multiplexing can sit on top of the base
protocol and be negotiated separately by interested ends.

It seems like we need variable size WS framing even if we didn't care
about the OS sendfile() optimization, simply because any fixed WS frame
size is arbitrary and gains nothing over variable sizing.  Even with fixed
size, we might still need fragmentation unless we made the size tiny
enough to be sure all intermediaries can handle.

>> know

As I said above, if we did fixed size WS framing, this upper limit might
be lower than our arbitrary fixed size, thus we still need fragmentation.

> more about the underlying MTU than TCP does.

If I understand your links, actually sending larger frames allows the
lower level decisions to be best optimized.

[snip]

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: John Tamplin <[email-masked]>
Date: Fri, 20 Aug 2010 13:46:57 -0400

> protocol and be negotiated separately by interested ends.

Which, among other reasons, is why fragmentation is built into the base
protocol and multiplexing is not.  However, enough people are convinced that
multiplexing will be necessary that we need to make sure the protocol as
defined now can be extended to support multiplexing without breaking v1.0
implementations.

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 20 Aug 2010 14:12:48 -0400

Oh and the variable 1 or 8 byte header seems logical.

Also 2 or 8 bytes wouldn't give as significant a % of overhead savings on
the smallest packets where it matters most for latency.

So if my calculation is correct, 2^64 length frame would take over a year
at Terabit speed. So seems unlikely we could use more than 64 bits now?

But why not allow 3 levels, 1, 8, or N bytes? It isn't that difficult to
write a routine that can do add and subtract on an N byte counter in any
language. If we decided to make the N byte optional, the N byte capability
could be negotiated on connection handshake. You wrote we need the maximum
frame size negotation any way.

Would it be any advantage versus upgrading the protocol when computers
move to 128 bits and channels speeds increase?

---

From: John Tamplin <[email-masked]>
Date: Fri, 20 Aug 2010 14:21:45 -0400

> frame size negotation any way.

There is a significant increase in implementation complexity having to use
multiprecision arithmetic for lengths.  One problem is the OS has no way of
expressing read/write lengths in multiprecision values.  If we were going to
have 3 levels, it would probably be 1/2/8 bytes (using 126=2 length bytes,
127=8 length bytes), since I suspect many buffered implementations will use
under 64k buffers and could make use of a 2-byte length.

> move to 128 bits and channels speeds increase?

There are so many things that would need to change before that would be
feasible, it seems likely WebSocket would be obsolete by then.  If not, then
they can fragment into 2^63 byte fragments :).

---

From: Roberto Peon <[email-masked]>
Date: Fri, 20 Aug 2010 11:28:14 -0700

On Fri, Aug 20, 2010 at 10:10 AM, Shelby Moore <[email-masked]> wrote:

> > large numbers of people on the list, which is why I relented and
> suggested
> > a

If this protocol becomes widely used, it will likely be so used for at least
twenty years.
We need to consider the problems that we'll have (at least scalability-wise)
5 years or 10 years out from now.
It isn't specific to certain classes of network ends, depending on
carrier-grade NAT growth it may be a concern to everyone.

-=R

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 20 Aug 2010 14:37:05 -0400

I have no objection to the variable framing and fragmention you are
proposing.

I reserve my future judgement on multiplexing extension impacts on base
protocol.