Subject: [hybi] Framing and design philosophy

From: Ian Hickson <[email-masked]>
Date: Thu, 29 Jul 2010 23:26:23 +0000 (UTC)

It has been suggested that I should try to explain the reasoning behind 
the current design of the framing in the WebSocket specification.

As background, the idea behind WebSockets is to provide TCP for scripts 
running in Web browsers. There are several issues with doing that that 
prevent us from literally just using TCP; most are security-related and 
are dealt with by the handshake, which I won't discuss here.

There is one other issue, though, with exposing raw TCP to scripts running 
in Web browsers: TCP provides a stream, whereas the Web browser platform 
operates on an event-based system with discrete units of content. Browsers 
need to know when to notify the script that more content is available.

We could simply expose a stream interface to the scripts, in particular 
with a "read" method that returns all data collected so far. However, this 
is the kind of model that leads to subtle bugs: in testing (on reliable 
networks), one will find packets do not get split and take minimal time, 
making it likely that authors will rely on a "read" method always 
providing a single complete packet of information. In the real world, 
packets get fragmented, packets arrive out of order, packets get delayed, 
and thus a "read" can easily include either only part of a packet, or all 
of a packet plus more data from another.

As part of making the Web platform a reliable experience for authors, we 
have to try to avoid exposing authors to this kind of issue. Web 
developers have repeatedly shown that they are likely to depend on the 
weirdest of quirks; this is not a minor problem. This suggests that we 
should have a market in the stream to indicate when the UA is to notify 
the script that data is available; such a marker would also allow the data 
to be provided in predictable units selected by the server.

If the data is UTF-8, an easy way to demark different events of data is to 
use a 0xFF byte between each event's worth of data.

On its face, this seems fine, but going forward we will need to support 
binary data, and in binary data you can't reserve a particular byte in 
that way, instead we need length-prefixed frames. They can be chunked or 
fixed size or any number of other similar mechanisms, but they are based 
on size and not on delimiters.

Now we could use this for the text data too, but this leads to two 
unrelated problems.

The first is that the browser will need to distinguish text data and 
binary data, because they need to be exposed to scripts differently. This 
is easily solved; simply prefix each event's data with a byte saying what 
kind of data it is.

The second is that if we use length prefixing for text data, it is very 
easy for author to end up measuring the length of their ASCII strings in 
characters and outputting that as the byte length of the data, which ends 
up leading to very subtle bugs when non-ASCII UTF-8 characters are used. 
Since kind of bug is very hard to spot if you don't think anyone is going 
to be trying to mess around with your data, and yet can lead to very 
drastic security problems: one can effectively smuggle in new frames by 
tricking the server into sending the wrong length.

This latter problem is not present when we simply use 0xFF markers to 
delimit frames, because it is much harder to smuggle in a 0xFF byte (you 
can't do it using Web Sockets, you have to use another vector), and 
there's no length to confuse.

Thus, the framing in the current spec. Each frame is prefixed by a byte 
saying whether it's a text frame or a binary frame; text frames are 
marker-delimitted, and binary frames a length based.

If we could assume that programmers don't make silly mistakes, we wouldn't 
need all of this: we would just use raw TCP after the handshake and not 
have framing at all. Simply expose the stream. But we can't assume that. 
Programmers make mistakes, and we have to take that into account and 
design around it.

This isn't hypothetical. Consider the postMessage() API. It is designed on 
the assumption that the authors using the API will not make mistakes. It 
is a completely secure API in principle, but it relies on the author 
carefully checking the origin of incoming messages and carefully setting 
the target of outgoing messages. Usage of this API was examined by some 
security researchers at Berkeley:

   http://www.eecs.berkeley.edu/~sch/w2sp2010ena.pdf

They studied two systems in particular, Google Friend Connect and Facebook 
Connect. Both were found to have all kinds of security problems. The study 
concludes that these problems are in part to blame on the design of the 
API -- it did not "minimize the liability that the user undertakes to 
ensure application security".

We have to realise, as protocol designers, that this applies to us too. We 
have to design our protocols so that it is harder to implement them 
insecurely than to design them securely. I don't pretend to think that Web 
Sockets is perfect in this regard; but I think it behooves us to start 
from that point of view.

For example, it would be easy to suggest that we should simplify the 
protocol and not have the marker-delimited frames. However, if we did this 
we would have to find some other way to mitigate the issue with string 
measurements. (I've considered several, but I haven't found one that would 
be as effective as a marker. For example, requiring all strings to have a 
three-byte UTF-8 character at the start would simply get implemented by 
hardcoding those bytes and outputting the length+3, which doesn't help.)

In closing, here are some examples of people writing Web Socket code today 
that are running afoul of these kinds of problems. Here is someone's 
attempt at writing a server using read() to read the whole handshake at 
once into a buffer and then perform simplified parsing on that buffer:

   http://codepad.org/YBQD16oO

Notice how the code assumes that the handshake isn't fragmented into two 
packets that arrive with a delay.

Here is another example of a server, this one written in JavaScript:

   http://bohuco.net/blog/2010/07/html5-websockets-example/

Notice how it assumes the client is trustworthy and happily "eval"s code 
sent from the client without any input verification.

Here is another, written in PHP:

   http://webreflection.blogspot.com/2010/06/websocket-handshake-76-simplified.html

This one uses regular expressions to parse the headers. If we had just one 
key in the handshake ("Sec-WebSocket-Key" instead of "Sec-WebSocket-Key1" 
and "Sec-WebSocket-Key2") and if we didn't have the crazy "count spaces 
and divide by the number of spaces" thing, it would be vulnerable to the 
key being stuffed into the resource name in an XHR request, a cross- 
protocol attack from HTTP to WebSockets.

I don't see any way to avoid the first two bugs by protocol design (I'm 
glad the design we have does catch the third). But we need to think about 
this kind of thing when designing our protocol.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Matthew Wild <[email-masked]>
Date: Fri, 30 Jul 2010 01:05:15 +0100

> to be provided in predictable units selected by the server.

I'm certainly in agreement that websockets should be message, rather
than stream, based - avoiding that whole class of issues that raw TCP
brings with it.

> up leading to very subtle bugs when non-ASCII UTF-8 characters are used.

Which author? The browser, server author or the API user? I'm assuming
the latter, but why would they receive the raw length via the API?

> tricking the server into sending the wrong length.

I think that's a good reason to simplify the framing into simple TLV
(for all types), and forget the 0x00/0xFF.

> there's no length to confuse.

If we're to keep the distinction between binary and text, I don't see
why 0xFF might be more difficult to sneak into a text stream since it
is already being validated by the browser/server.

> hardcoding those bytes and outputting the length+3, which doesn't help.)

Mmm, are you worried about the browsers/servers doing this? I'm really
not sure why receiving in length-prefixed or marker-delimited blocks
makes any difference to this issue - the wire protocol would lie below
any prior processing done by the recipient (much of the existing
byte-by-byte logic in the current draft would still be applicable to
TLV-framed streams).

> this kind of thing when designing our protocol.

Agreed, but there's only so much that can be done at the protocol
level. The protocol can't be responsible for every security flaw in an
application... eval() on data from the client for example - we cannot
and should not try designing around things like that, let's just give
them a simple API.

Matthew

---

From: Adam Barth <[email-masked]>
Date: Thu, 29 Jul 2010 17:51:05 -0700

> and should not try designing around things like that, let's just give

One of the fun things about working on browser security is that we try
not to pass the security buck.  If it's within our power to do
something about, it's our problem.  In cases like this, we do try to
design around the issue.  We don't always succeed (for sure!), but
that's no reason to give up without trying.

Adam

---

From: Mike Belshe <[email-masked]>
Date: Thu, 29 Jul 2010 18:19:47 -0700

I don't think the examples are relevant.  If anything, they demonstrate that
no matter how "simple" you make the protocol, someone can still gum it up.
 You're not advocating for even simpler, but demonstrating well that even in
the 'simple' form,  it's not simple enough.  BTW - an argument could be made
that length-prefixed frames would help those protocol writers - instead of
thinking "oh, it all arrives in one packet", they'd read the spec which
says, "read the length; now read that many bytes"...  But I won't make that
claim.  Never under-estimate the ability for someone (even me!) to gum it
up! :-)

A couple of things about framing:
  a) The lack of error handling in websockets today is a problem.  Errors
where "if you can't handle this, just close the connection" will be a
problem - it's like HTTP/0.9.  Unfortunately, if you want to handle error
cases, I don't see a way that doesn't have some framing to carry those
errors.  (even HTTP has framing, albeit text-based)
  b) length-prefixed, typed frames offer a lot of flexibility.  It provides
a solid boundary between the application layer and the protocol which
enables extensibility, scalability, and robustness.  Specifically:
   extensibility:  you can add new frame types going forward.
   scalability: Fixed length frames make buffering and flow control
possible, even if added later.  As any server guy will tell you, having to
buffer data of unbounded size is not a good idea.  With length-prefixed
frames, servers know the size up front and can work properly, error out,
stream the data, etc.
   robustness:  with frames, endpoints have a checkpoint for discrete
messages for verification of sanity.  random bytes in the middle are
unlikely to be misinterpreted.

Finally, here is the use case I'd like to see solved.  I believe websockets
today is weak on this case, and a framing based approach solves it well:

    -  Client wants to send a 64KB string to a server.   The client is
unaware that this server is so busy that it can only allocate 4KB to each
client.  What happens?  The server reads until it's buffer is exhausted, and
can do nothing except hang up.  The client has no idea why, and can only
retry, ignore, or tell the user.

Mike

On Thu, Jul 29, 2010 at 4:26 PM, Ian Hickson <[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 30 Jul 2010 13:27:17 +1000

Ian,

thanks for explaining the motivation behind the current design. I think most
of it is not controversial - specially about having messages rather than
streams.

I understand your desire for a simple solution that avoids bugs.  However I
fail to see that sentinel framing of utf-8 data is going to be significantly
less bug prone than length encoded framing.

Sentinel framing allows does allow a very simple send logic:
    out.writeByte(0x00);
    out.writeString(mystring);
    out.writeByte(0xFF);

But  length framing code is not that much more complex:

   byte[] buffer = mystring.getBytes("utf-8");
   out.writeByte(0x00)
   out.writeInt(buffer.length);
   out.writeBytes(buffer);

Both of these are pretty simple.   However, I think the complexity is
different if we look at the receiving side.
The receiver cannot avoid dealing with bytes, because they are reading non
UTF characters.

For sentinel framing, you need to loop and  expand a byte buffer

    byte[] buffer = new buffer[1024];
    int pos=0;
    while(connection is open)
    {
        if (pos>buffer.length)
        {
           if (buffer.length is too large) throw exception
           byte[] tmp = new buffer[1024+buffer.length];
           Array.copy(buffer,0,tmp,0,pos);
           buffer=tmp;
        }
        int len = in.readBytes(buffer,pos,1024);
        pos+=len;
        for (int i=pos-len;i<pos;i++)
           if (buffer[i]==0xff)
              return new String(buffer,1,pos-2);
    }

Of course you could have a fixed buffer size, but it is a significant
inefficiency to limit the maximal message to the minimal buffer size. Even
with a fixed buffer, the need to loop and refill the buffer from a marked
position is bug inducing code.

For length framing, this is considerably simpler

   byte[] header = new byte[3];
   byte op = in.readByte();
   int len = in.readInt();
   if (len is too large) throw exception
   byte[] buffer = new byte[len]
   in.readBytes(buffer,len);
   return new String(buffer,0,len);

No loops, no sliding position marks, optimal buffer sizes.

More importantly, even if length encoding was more complex, the developer is
going to have to write it anyway because binary messages will eventually be
part of the protocol.  So using sentinel encoding does not protect a
developer from having to deal with the complexities of length encoding.

On top of this, we have to add the complexity of switching between two
different framing mechanisms on a message by message basis on the same
connection.

Finally, sentinel framing is vulnerable to injection.  If there is a flaw in
the UTF-8 library used, or if there websocket code is passed user bytes that
are represented as already having been converted to utf-8 encoded data,
then implementations are likely to just send those bytes without scanning
for 0x00 or 0xFF in the data.  Thus an attacker who can influence the the
UTF-8 encoded payload can inject new messages into the message stream.

So in summary while the sentiment to have a simple solution is admirable, I
think that the current solution fails to provide, because:

   -   sentinel is not simpler, it is at best equivalent.
   -   length framing has to be implemented anyway.
   -   there is additional complexity of switching between too framing
   mechanisms message by message.
   -   the is an additional security vulnerability.

regards

PS. if there are bugs in any of my pseudo code above, that just proves Mikes
point that there is no protection against dumb mistakes.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 30 Jul 2010 07:17:00 +0200

On Fri, Jul 30, 2010 at 01:27:17PM +1000, Greg Wilkins wrote:
(...)
> The receiver cannot avoid dealing with bytes, because they are reading non

I agree that the job on the receiver side is more complex when having
to parse each byte. For instance, it's a lot more convenient in HTTP
to parse the chunked transfer-coding than to parse multipart.

Also, it makes a big difference on server processing. The TLV method
implies that the server does not have to parse every byte. It can
very well read the length, read a block and forward that block to
whatever internal sub-component. The sentinel framing implies that
every subcomponent which sees the message has to parse all of its
bytes, and wait until the message is complete. This starts with the
server itself which might very well not care about its contents.
That's simply a waste of CPU power.

And if multiple messages are sent at once, it's even harder to
parse them because reading buffer by buffer implies that some
parts of message 2 can be found after message 1 in the buffer,
implying further copies to move them around. The TLV method has
the advantage that if you know your message is 17kB, you read
17kB and not one more byte, then you know that the next read()
will be for next message. So TLV puts clear delimitations on the
receiver side.

Willy

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 06:18:46 +0000 (UTC)

>    out.writeInt(buffer.length);

Many subprotocols will be initially (or forever, indeed, e.g. a game 
protocol where there's no chat) ASCII only. In such an environment, this 
would work fine (Perl):

   print("\x0");
   print(pack('w', length($buffer)));
   print($buffer);

...but if $buffer contains any non-ASCII, you've got a security bug.

>               return new String(buffer,1,pos-2);

Yeah, in some languages it's a more involved than in others (pseudocode, 
but something like this is how you'd do it in most scripting languages):

   var buffer = '';
   var start, pos = 0;
   while (connection is open) {
     buffer += readBytes();
     while (pos < length(buffer)) {
       if (buffer[pos] = 0xff) {
         yield decode(substr(buffer, start, pos-1);
         start = pos + 1;
       }
       pos += 1;
     }
     if (start > maxBufferSize)
       buffer = substr(buffer, start, length(buffer));
     if (pos > maxBufferSize)
       throw DoS;
   }

> More importantly, even if length encoding was more complex, the 
> developer is going to have to write it anyway because binary messages 
> will eventually be part of the protocol.

Only if they need binary support. I wouldn't at all be surprised if many
implementations never added binary support, since for many applications 
it's not necessary.

> So using sentinel encoding does not protect a developer from having to 
> deal with the complexities of length encoding.

It's not about protecting developers from the complexities of length 
encoding (whatever they might be -- as you say, it's often simpler). It's 
about protecting developers from introducing security flaws by designing 
the protocol in such a way as to make it naive implementations of the 
protocol safe.

In particular, we're not trading features for simplicity. We're trading 
potential subtle security bugs for more obvious bugs that are more likely 
to be caught during simple testing.

> Finally, sentinel framing is vulnerable to injection.  If there is a 
> flaw in the UTF-8 library used, or if there websocket code is passed 
> user bytes that are represented as already having been converted to 
> utf-8 encoded data, then implementations are likely to just send those 
> bytes without scanning for 0x00 or 0xFF in the data.  Thus an attacker 
> who can influence the the UTF-8 encoded payload can inject new messages 
> into the message stream.

Yes, I mentioned this in my e-mail, and explained why using a delimiter 
means that such bugs require a second vector, rather than being able to 
use WebSockets exclusively in an exploit. In general, the more bugs one 
needs to have an exploitable vulnerability, the better. In this case, 
we're trading a single bug (mismeasuring lengths on output) for two bugs 
(incorrect filtering of content on input in an unrelated component, and 
incorrect encoding of data on output).

> So in summary while the sentiment to have a simple solution is admirable

With all due respect, I think you have missed the point. The e-mail to 
which you replied does not describe a sentiment to have a simple solution 
as the reason for the current design. You are correct that if simplicity 
in the protocol was the ultimate goal, we would have a different design. 
Probably something really simple, like a raw stream, with no framing at 
all. Unfortunately simplicity is not our only constraint.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 30 Jul 2010 17:02:50 +1000

Ian,

the same goes for sentinel encoding (forgive my probably wrong perl):

  print("\x0");
  open(HANDLE,"somefile.txt");
  print(<HANDLE>)
  print("\xff");

if somefile.txt contains non UTF-8, you've got a security vulnerability

The main difference between the two is that the length encoding code has a
bug, while the sentinel encoding code has a vulnerability.
Typically bugs are much easier to find as they will fail if given valid
data, while vulnerabilities need to be tested with crafted invalid data to
find them. There failure may also be very dependent on environment and work
with some libraries and be vulnerable on others.

I just don't see that sentinel encoding is sufficiently more secure that it
justifies having two framing mechanisms.

regards

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 07:27:12 +0000 (UTC)

> if somefile.txt contains non UTF-8, you've got a security vulnerability

somefile.txt needs to be something under the attacker's control before you 
have a security vulnerability, but sure, assuming that's the case, you do. 
I mentioned this, and explained why I believe this to be the lesser of two 
evils, in both of my previous e-mails on this thread (the original one and 
my subsequent reply to you).

> data, while vulnerabilities need to be tested with crafted invalid data 
> to find them.

That's a wild oversimplification. Security vulnerabilities are just bugs, 
there's nothing special about them that makes them more subtle. What's 
important for us, as protocol designers, is to make sure that we make the 
likely vulnerabilities either as obvious as possible, or involve as many 
layers as possible, so as to reduce the likelihood them being undetected 
by the author or exploitable if undetected by the author.

> I just don't see that sentinel encoding is sufficiently more secure that 
> it justifies having two framing mechanisms.

So you have said. I have explained why I believe it is.

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Fri, 30 Jul 2010 09:38:54 +0200

On Fri, Jul 30, 2010 at 5:27 AM, Greg Wilkins <[email-masked]> wrote:

>
>
> thanks for explaining the motivation behind the current design. I think
> most of it is not controversial - specially about having messages rather
> than streams.
>
>

The given explanation means that the complexity of rebuilding the context is
performed by the server rather than the client. It's probably fine but
considering it sometimes feels the whole protocol discussion (aside from
security reason) was based on having the simplest client's API.

If we were to use a length framing, then I'd say, why not use directly TCP ?
Servers have dealt with the single .read(length) method for decades, not
perfect at any rate but seems to have worked out rather well. However since
the motivation is to have a straight client API, well then the sentinel
framing is probably easier for the client to deal with.

Now regarding the security concerns Ian exposes, I'm a little worried that
devising a protocol, that deal both with "how to transfer" and "how to
secure", may conflate two responsibilities that don't necessarily fit well
together. Let's assume I'm a bank system wishing to use WebSockets, perhaps
the security protection introduced by WebSockets don't fit the bill and I
still need another layer of security, making the one embedded in WS rather
redundant and costly. Maybe this example is contrived but it'd be great if
the proposal was tested more harshly from a security purpose to clearly
point out whether or not it's a valid approach. If I missed such discussion
then all my apologies.

-- 
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 30 Jul 2010 01:10:03 -0700

On Jul 30, 2010, at 12:38 AM, Sylvain Hellegouarch wrote:

> thanks for explaining the motivation behind the current design. I think most of it is not controversial - specially about having messages rather than streams.
> 
> 
>  
> The given explanation means that the complexity of rebuilding the context is performed by the server rather than the client. It's probably fine but considering it sometimes feels the whole protocol discussion (aside from security reason) was based on having the simplest client's API.
> 
> If we were to use a length framing, then I'd say, why not use directly TCP ? Servers have dealt with the single .read(length) method for decades, not perfect at any rate but seems to have worked out rather well. However since the motivation is to have a straight client API, well then the sentinel framing is probably easier for the client to deal with.

It's not especially easier for the client (in the sense of the client API in the browser) to read a sentinel-framed message instead of a length-framed one. But it's not especially harder either. Either way you have to loop until the message is complete, because a client isn't going to do blocking I/O on a socket. It will most likely use select() or a similar mechanism, and will read only as many bytes as available, without blocking.

> 
> Now regarding the security concerns Ian exposes, I'm a little worried that devising a protocol, that deal both with "how to transfer" and "how to secure", may conflate two responsibilities that don't necessarily fit well together. Let's assume I'm a bank system wishing to use WebSockets, perhaps the security protection introduced by WebSockets don't fit the bill and I still need another layer of security, making the one embedded in WS rather redundant and costly. Maybe this example is contrived but it'd be great if the proposal was tested more harshly from a security purpose to clearly point out whether or not it's a valid approach. If I missed such discussion then all my apologies.

In general, it's not bad to have security mechanisms that operate at multiple levels, so long as they are truly independent of each other (i.e. you have to break each to have a vulnerability). Security practitioners call it "defense in depth". It's hard to do analysis on a general scenario without details though.

Cheers,
Maciej

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 02:54:16 -0700

> incorrect encoding of data on output).

There are bugs that are accidental, and there are deliberate attempts to
exploit bugs.  In both length-prefixed and the sentinel cases, exploits are
possible.  With length-prefixed, it is harder, I'd argue, but I believe this
is fairly subjective.   But for accidental bugs, fixed-length is far better
than sentinel based.  With the sentinel case, a missing sentinel is not
detectable as an error.  But with prefixed lengths, it is totally detectable
and will often generate an error immediately.

But the current proposal has two framing mechanisms - one for binary (which
we all stipulate has to be length-prefixed) and a different one for text -
sentinel encoding.

Surely we agree that, from a simplicity argument, using one mechanism,
rather than both, is simpler?  Now, I think Ian's going to say, "but some
servers only want text, so they'll only use that".  That would probably be
true.  Which would mean a proliferation of clients and servers which, when
you send them legal binary frames, completely choke.   "But these simple
servers will never do that", you might argue.   Probably true, but it leaves
protocol extensibility on the floor- because we've now got half-implemented
protocols strewn about in thousands of endpoint implementations.

I just think a single format for framing is easy to implement for all,
simpler, and more extensible.

Mike

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 Jul 2010 09:06:37 -0700

Getting sentinel framing right while keeping it efficient is very tricky.
Getting length-prefixing framing right is utter simplicity, even when done
with efficiency in mind.

The receiver-side is most often where the complexity resides. Greg's
examples earlier in the thread are a perfect demonstration-- getting the
sentinel framing right just takes more code and more complexity. And that
was a naive implementation! A fast implementation might have to load
everything into SSE registers (or equivalent) and do comparisons of 16 bytes
at a time in order to get some semblance of efficiency. The c-library
functions don't always do the fastest job of this, and in other languages
the ability to do this level of optimization may not even be possible. In
order to do the scanning efficiently, one then also has to maintain a
contiguous buffer with the input data, else one must suffer scanning
byte-at-a-time MUCH more often.

The naive implementation for length-prefix framing, on the other hand, is
essentially already the optimal algorithm. Shorter, faster, less complex!

Last argument as to why length-prefixing is superior (in addition to others'
arguments), you can implement sentinel framing INSIDE of length-prefixed
messages, but not the other way around.
If one placed 0xFF as the last character of the data payload for the utf-8
frametype, you could always instead decide to use sentinel framing. You'd
skip the number of bytes you know isn't payload then scan for 0xFF, then
skip the fixed number of bytes (if any) in the footer of the data frame.

Now, if you were writing a really big utf-8 string and you were doing
sentinel based framing, you'd have to ensure that you always wrote the
string as one frame (which is bad if we're doing multiplexing of any kind as
it induces lots of latency for any of the other messages/streams whatever).
If you were doing length-prefixing you could break that message up into
multiple frames to allow for other data to be sent and received and thus not
starving anyone else from the channel.
One could verify that the utf-8 messages was a valid message by checking
that the last byte of the last frame of the message was indeed 0xFF. This
offers the same benefit as sentinel framing, but also ensures that the
channel need not be discarded when the sender makes a mistake-- the framing
of the protocol itself is still intact and an error can be sent for that
message/stream/whatever.

-=R

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 30 Jul 2010 20:00:39 +0200

> essentially already the optimal algorithm. Shorter, faster, less complex!

I'd also add that sentinel framing is particularly sensible to buffer
overflows on the receiver side because length checks are very hard to
perform before copying data from the buffer to their final locations.
C language uses sentinel framing BTW and I've not observed any significant
decrease of buffer overflows during the last 15 years, despite the
availability of more reliable string primitives. With a length prefix,
you know whether your data will fit your buffer *BEFORE* parsing and
copying it, which makes a huge difference.

For this reason I expect a lot more exploitable buggy implementations
with sentinel framing and with length prefix, whatever the experience
and qualification of the developer.

Regards,
Willy

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 18:34:31 +0000 (UTC)

On Fri, 30 Jul 2010, Roberto Peon wrote:
> sentinel framing right just takes more code and more complexity. And 
> that was a naive implementation! A fast implementation might have to 
> load everything into SSE registers (or equivalent) and do comparisons of 
> 16 bytes at a time in order to get some semblance of efficiency. The 
> c-library functions don't always do the fastest job of this, and in 
> other languages the ability to do this level of optimization may not 
> even be possible. In order to do the scanning efficiently, one then also 
> has to maintain a contiguous buffer with the input data, else one must 
> suffer scanning byte-at-a-time MUCH more often.
> 
> The naive implementation for length-prefix framing, on the other hand, 
> is essentially already the optimal algorithm. Shorter, faster, less 
> complex!

On the medium term, I would expect anyone who is concerned about 
efficiency to this level to do everything using compressed data, which 
would automatically mean only using binary frames (and would be immune to 
the string vs bytes length measuring problem). So efficieny of the 
sentinel-based framing on the server side is a moot point.

Not everyone is going to be worrying about efficiency to the point of 
talking about which registers to decode the protocol with, though. The 
reality of the long tail is that most servers of Web Sockets won't be 
seeing high throughput. We shouldn't optimise for the head of the adoption 
curve at the cost of the tail. We should -- and can, in this case -- 
provide for both.

---

From: Dave Cridland <[email-masked]>
Date: Fri, 30 Jul 2010 19:38:57 +0100

> sentinel-based framing on the server side is a moot point.
That only applies if compression operates at a lower level than  
framing. I see no particular reason why it should - I'd actually run  
it above the framing layer, because it matches every other  
stream-based protocol, and in addition, it becomes radically simpler  
to code properly. Otherwise you need to ensure that the framing of  
the compression protocol matches the WS framing, and/or that you  
compress frames individually (which is drastically less efficient).

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 Jul 2010 11:53:20 -0700

> curve at the cost of the tail. We should -- and can, in this case --
Number of server implementations != volume of traffic.
I've already shown that with length prefixing you can have the receiver
chose to parse using either sentinel scanning or byte-counting with
semantics that provide for non-channel-close error semantics where
appropriate.
I've yet to see where requiring a different framing type is useful.
Also, you're ignoring the argument about parsing via sentinel scanning being
far more complex, even when we're not talking about efficient
implementations (which it makes FAR more complex).

I see the following high level issues:
Ease of implementation:
  sending messages: length-prefix+sentinel about tie for complexity
  receiving messages: length-prefix is far less code, and that code is
simple
  duplication of code: We'll always need length-prefix if we assume that any
implementation will handle binary data ever. Thus, we'll be maintaining more
code if we have two separate framing types.
Speed:
  length-prefix requires fewer allocations and deallocations since required
buffer size is known before payload arrives.
  length-prefix requires zero scanning of the bytes, enabling zero-copy
semantics.
  sentinel requires byte-by-byte or word-by-word scanning at best.
  sentinel requires more allocations and deallocations since efficient
implementations need to keep data mostly contiguous
  sentinel requires copying into userspace for *all* data in order to frame
things properly.
Safety/security:
  length-prefix allows for sentinel parsing, thus any perceived advantages
of sentinel parsing exist for length-prefix
  sentinel could do the same so long as you're putting a length in there
which you're required to skip
  .. thus, code complexity and implementation difficulty aside (addressed
above), these are equivalent
Scalability:
  sentinel does not allow for strings to be easily and safely split up on
anything less than message boundaries without addition of another sentinel
or additional flagging.
    This is problematic when users have large messages and small messages
simultaneously sharing the channel. For reasons of scalability we must be
able to share these channels.
  length-prefix makes solving this problem trivial

-=R

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 12:00:44 -0700

> talking about which registers to decode the protocol with, though. The

Talking about registers is a dramatization.  Nobody was talking about that.
 Let's not pretend here.

> curve at the cost of the tail. We should -- and can, in this case --

You're dangerously close to reopening the subjective "amateur programmer"
requirement which everyone else agreed is not a requirement :-)  It's
certainly not an objective argument :-)

Trying to steer back to objectivity, here are two problems I haven't heard
addressed with the use of UTF-8 framing in websockets:
a) How will we handle error cases?  When something goes awry, I don't want a
simple "close the connection".  How does the side detecting the error convey
the error to the remote?
b) Doesn't using the sentinel based approach mean that we have two types of
framing in the same protocol?  There is no sentinel based approach proposed
for binary data.  We all agree that binary data should be length-prefixed.
 Each type of framing has its own edge cases.  Why would we want to expose
the protocol to both types of framing and hence both types of edge cases?
 Use of length-prefixed frames can solve both binary and utf-8 data
transport.  Sentinel-based framing doesn't.

Mike

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 19:09:35 +0000 (UTC)

On Fri, 30 Jul 2010, Roberto Peon wrote:
> > The reality of the long tail is that most servers of Web Sockets won't 
> > be seeing high throughput. We shouldn't optimise for the head of the 
> > adoption curve at the cost of the tail. We should -- and can, in this 
> > case -- provide for both.
>

I'm glad we agree on at least one thing! :-)

> semantics that provide for non-channel-close error semantics where 

Unfortunately this kind of redundancy in practice leads to implementation 
differences that become very relevant. For example, suppose IE implements 
just the length measuring, and ignores the sentinels. A server that 
doesn't correctly send the sentinels then will work with IE and fail with
browsers that do stricter checking. If IE has high market share when this 
happens, the net result will be that we can no longer do the stricter 
checking, and now we have a vestigial feature in the protocol.

This isn't hypothetical. We're seeing this exact thing happen with 
history.pushState() in the HTML API -- before the spec is even finalised, 
we already have a vestigial argument to the function, and there's nothing 
we can do about it because changing the API would break existing pages.

> I've yet to see where requiring a different framing type is useful.

When the data is UTF-8, but the server only expects ASCII, length framing 
runs the risk of measuring the wrong thing, leading to security 
vulnerabilities. I explained this in detail here at the top of this 
thread:

   http://www.ietf.org/mail-archive/web/hybi/current/msg02605.html

> Also, you're ignoring the argument about parsing via sentinel scanning 
> being far more complex, even when we're not talking about efficient 

In the case of low-throughput implementations, it's trivial. You can just 
do things one byte at a time, for example.

In the case of high-throughput implementations, I don't see why the server 
would need to implement this ata ll on the medium term. Once we have 
compression, why would you use anything uncompressed?

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 12:19:14 -0700

> checking, and now we have a vestigial feature in the protocol.

Maybe we should call a truce on the use of hypothetical bugs as an argument
for one way or the other.  There is no shortage of hypothetical bugs for any
syntax.

> we can do about it because changing the API would break existing pages.

Sure.  As a counter hypothetical bug, we could talk about smuggling attacks
made possible by sentinel framing.  Maciej and others already pointed out
that these are real problems we see today with HTTP.

There is an infinite supply of hypothetical bugs.....

Mike

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 Jul 2010 12:21:06 -0700

> I'm glad we agree on at least one thing! :-)

I'm hoping to agree on more! :)

> checking, and now we have a vestigial feature in the protocol.

Isn't this an argument for not having two framing types as well?

>   http://www.ietf.org/mail-archive/web/hybi/current/msg02605.html

I read and comprehended that!
That is why you like a sentinel, which I said you could still do, but you
said would lead to redundancy leading to implementation rot/fracturization.
You can't argue that it is redundant/unnecessary and then argue that you
need it, or rather, one shouldn't argue that!

> do things one byte at a time, for example.

For low-throughput implementations, memory becomes the limiting factor
(because we end up putting many, many more simultaneous sessions on the
server or proxy in these cases).
That means that one should worry about how to retain memory efficiency,
which should immediately suggest that knowing how many bytes you should
allocate for is a good idea.

> compression, why would you use anything uncompressed?

Compression can and will be done at different layers. Sometimes you want to
compress HTML/XML. That is obviously a win. Attempting to compress a video,
image, etc. likely would likely result in an expansion of data size as well
as lots of needless CPU cost.  Thus, the sender should always be deciding
whether or not to send compressed based on the data that is being sent.

In any case, compression requires length-prefixed framing, thus if you
believe that high-throughput servers should always use compressed data, then
they should always use length-prefixing.

-=R

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 19:26:44 +0000 (UTC)

On Fri, 30 Jul 2010, Mike Belshe wrote:
> > > A fast implementation might have to load everything into SSE 
> > > registers (or equivalent) and do comparisons of 16 bytes at a time 
> > > in order to get some semblance of efficiency.
> >
> > talking about which registers to decode the protocol with, though.
> 

Roberto was.

> You're dangerously close to reopening the subjective "amateur 
> programmer" requirement which everyone else agreed is not a requirement 
> :-)  It's certainly not an objective argument :-)

Not a requirement for what?

It may be, again, that this working group is interested in a different 
protocol than WebSockets. It's hard to tell.

WebSockets is a protocol designed with these goals in mind (in order, most 
important first):

 - does not introduce new attack vectors to the Web
 - be a good match for the Web browser scripting model
 - be as close to raw TCP as possible
 - be designed so that the likely implementation errors (mismeasuring 
   strings, doing regexp on the handshake instead of parsing it, etc) do
   not lead to security vulnerabilities
 - being trivial to implement for amateurs

This protocol was being designed at the WHATWG and later the W3C. The IETF 
then came along and said that they were interested in this protocol, and 
formed the hybi group to work on it.

Now, if the above requirements aren't what this group is interested in 
working on, then this group is clearly not interested in working on Web 
Sockets. That's fine, I have no problem with this group working on 
something else. But we should make sure we're all on the same page here.

If a standards organisation suggested to the IETF that they were 
interested in working on HTTP, and the IETF agreed to move HTTP to that 
working group, and that working group started saying HTTP's requirements 
didn't include being able to transfer files, I think we'd all agree that 
really they weren't interested in HTTP.

> simple "close the connection".  How does the side detecting the error convey

The specs already define this in detail. Unknown frame types cause an 
'error' event to be fired, which can then result in the script sending 
back a notification at the application layer. Handshake errors cause the 
connection to close, for security reasons. Non-UTF-8 bytes in text frames 
are converted to U+FFFD. I think that's it.

> b) Doesn't using the sentinel based approach mean that we have two types 
> of framing in the same protocol?  There is no sentinel based approach 
> proposed for binary data.  We all agree that binary data should be 
> length-prefixed. Each type of framing has its own edge cases.  Why would 
> we want to expose the protocol to both types of framing and hence both 
> types of edge cases?

Ideally we'd only have length-based framing, but that fails to handle this 
case:

 - be designed so that the likely implementation errors (mismeasuring   
   strings, doing regexp on the handshake instead of parsing it, etc) do
   not lead to security vulnerabilities

...for the reasons I discussed in detail here:

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Fri, 30 Jul 2010 12:28:59 -0700

For what it's worth, at the face to face (IETF78) there appeared to be
strong consensus among those present to prefer a single method of framing --
length based, not sentinel based.

---

From: James Graham <[email-masked]>
Date: Fri, 30 Jul 2010 21:30:00 +0200 (CEST)

On Fri, 30 Jul 2010, Mike Belshe wrote:

> Maybe we should call a truce on the use of hypothetical bugs as an argument for one way or the other.  There is no shortage of
> hypothetical bugs for any syntax.

It seems to me that looking at the type of bugs that each implementation 
strategy could cause is an essential part of the process. Otherewise how 
will we avoid fragile designs, or designs that have major security 
problems in the real world, or designs that are particularly susceptible 
to interoperability problems.

Apologies for the lack of a technical point in this post, but I think 
examining possible bugs and problems in various designs is one of the most 
effective ways to weigh up the possibilities, and something we should do 
much more, not less.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Fri, 30 Jul 2010 12:32:07 -0700

>   http://www.ietf.org/mail-archive/web/hybi/current/msg02605.html

FWIW we discussed this exact point at the f2f. The counter argument raised
was that in most languages, you have some sort of write() function call that
takes a number of bytes to be written, as opposed to passing in a string,
and so most implementers of the protocol would have to know the length of
the data anyways. The other point that came up is that if you aren't capable
of counting the length of data you are sending, you are likely to have more
serious errors anyways and perhaps should not be writing code for any sort
of real use/reach.

---

From: James Graham <[email-masked]>
Date: Fri, 30 Jul 2010 21:54:27 +0200 (CEST)

On Fri, 30 Jul 2010, Ian Fette (イアンフェッティ) wrote:

> FWIW we discussed this exact point at the f2f. The counter argument raised was that in most languages, you have some sort of
> write() function call that takes a number of bytes to be written, as opposed to passing in a string, and so most implementers of
> the protocol would have to know the length of the data anyways.

Is that true (it might be, I don't know)? At least in modern scripting 
languages having to specify the number of bytes to write out seems very 
unusual. Since the type of authors we are concerned with making unicode vs 
bytes mistakes are presumably more likely to be using Python than C, it 
seems that the behaviour of these languages should be weighted more 
highly.

> The other point that came up is that if you aren't capable of
> counting the length of data you are sending, you are likely to have more serious errors anyways and perhaps should not be writing
> code for any sort of real use/reach.

I don't think "if you get this wrong you are too dumb to be writing 
websockets servers" is a reasonable argument, especially since the state 
of the web suggests that many people will write and deploy bad code 
whether we lik it or not. It is certianly not just the small sites that 
are full of WTF level javascript, for example.

For the particular case of failure-to-convert-unicode-to-bytes-correctly 
type errors, I suggest Googling for UnicodeDecodeError or 
UnicodeEncodeError. These are Python exceptions thrown when converting 
to/from byte strings goes wrong. Note that Python 2.x has the flaw that it 
will work for ascii without the author explicitly converting, but break 
once non-ascii data is involved; therefore these errors occur in the same 
kinds of situations as the websockets error we are concerned with. In my 
experience - which I think Google backs up - these errors are *very* 
common; I have got them from well known packages and, sadly, have made 
the same mistake myself more than once.

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 30 Jul 2010 21:54:31 +0200 (CEST)

> length based, not sentinel based.

I sense a similar spirit on this mailing list. There aren't that many voices 
arguing for the sentinel approach.

I'll add a +1 on length based for all the good reasons already raised.

-- 

  / daniel.haxx.se

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 30 Jul 2010 22:12:28 +0200

>    http://www.ietf.org/mail-archive/web/hybi/current/msg02605.html

I understand your point here but in my opinion, you're opening a new
can of bugs while trying to avoid another one.

A more robust design that would reconciliate both requirements would
probably be to have a length-prefix + the UTF-8 text + 0xFF, and
state that any UTF-8 text MUST always be suffixed with one 0xFF.
Doing this allows to control it both ways :

   - too short messages are detected because the N-1th byte is not 0xFF
   - too long messages are detected because 0xFF is present before the
     end, causing an UTF-8 decoding exception

And this certainly helps reading data by blocks and not allocating too
much memory in advance.

What do you think ?

Willy

---

From: Roberto Peon <[email-masked]>
Date: Fri, 30 Jul 2010 13:24:35 -0700

> And this certainly helps reading data by blocks and not allocating too
I take it that we're talking about appending 0xFF to the end of the UTF-8
message, as opposed to requiring it for a UTF-8 frame?
If so, SGTM!
-=R

---

From: James Graham <[email-masked]>
Date: Fri, 30 Jul 2010 22:36:25 +0200 (CEST)

>     end, causing an UTF-8 decoding exception

At least in the curent spec, an invalid unicode character would be 
replaced with a replacement character, not throw an exception.

> And this certainly helps reading data by blocks and not allocating too

Hixie already made one argument against this which I tend to agree with 
(although I previously wondered if it would be a good idea myself). The 
problem is that some implentations would always use the length and some 
would use the marker and some would use both. This will lead to 
interoperability problems which eventually lead to browsers following the 
market leader rather than the spec in order to improve compatibility. 
Hixie explained this in more detail, along with some examples of previous 
situations where something similar has occurred.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 30 Jul 2010 22:40:06 +0200

(...)
> this kind of thing when designing our protocol.

In fact, the first one is still at a PoC level, it does not read any
data and does not even close the FD but happily overrides it. I think
the guy it trying to parse the handshake right now.

I see one way to fix it, it's precisely by specifying that the handshake
is a series of lines, not a block. That's how we parse HTTP after all,
and basic HTTP is easily parsable even in shell scripts (I'm not considering
multi-line headers).

If we had told the guy he had to parse every single line, most likely
he'd have used fgets() to more easily read strings one at a time, because
that would have made his job a lot simpler. Have you noticed how he does
strstr() on the whole block without even ensuring it's zero-terminated ?
Also, he happily ignores everything in the handshake, even the Upgrade
header. Doing line-based parsing in C with fgets() is a lot simpler than
his code and would have allowed him better control, even if he still
forgot some length checks. The loop could have looked like this :

	for (;;) {
		clientfd = accept(sockfd, (struct sockaddr*)&caddr, &caddrlen);
		client = fdopen(clientfd, "rw");

		while (fgets(buffer, sizeof(buffer)-1, client) != NULL) {
			if (strcmp(buffer, "\r\n\r\n") == 0)
				break;
			else if (strncmp(buffer, "Sec-WebSocket-Key1: ", 20) == 0)
				key1 = key(buffer);
			else if (strncmp(buffer, "Sec-WebSocket-Key2: ", 20) == 0)
				key2 = key(buffer);
			... various possibly ignored headers ...
		}
		/* body now */
	}

Notice how he has not implemented the framing yet ? Probably that at this
level, it already looks hard to manage the buffering, reading too much and
then not easily being able to know what to do with it.

For the second case, there's a (more complex) way to solve that using typed
data. Then the API would offer some functions to automatically return the
type of next data and to return the typed data itself. We could then have
had something approximately like this :

	if (msg.type == LIST_OF_WORDS) {
		var data = msg.data.list_of_words;
		...
	}

But eventhough this can make the job easier and more secure to the developer,
there is also a risk that many types get added in non-standard implementations
that finally become incompatible.

Regards,
Willy

---

From: Mike Belshe <[email-masked]>
Date: Fri, 30 Jul 2010 13:42:33 -0700

>  - being trivial to implement for amateurs

Ian -

Please go back and read the group's comments about "amateur programmers" as
a requirement.  This is your requirement only; the rest of the group was
pretty unanimously against this so-called requirement.  I wish there were a
formal way to settle this.

At this point, I can't continue to rehash the same arguments.  The arguments
are on the table, so others can weigh in.  You are now citing your own
document (the first msg of this thread) as proof against my criticism of it.
 It's circular.  And, for the issues where you really can't justify your
desired outcome, you switch from objective criteria to subjective ones by
re-raising the "amateur programmer" requirement.  That's just squirrelly.

Cyclical arguments and refusal to listen to the decisions of the group make
it impossible for me to engage you in further debate on this.

Mike

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 30 Jul 2010 13:47:15 -0700

James Graham wrote:
>> counting the length of data you are sending, you are likely to have 
>> more serious errors anyways and perhaps should not be writing
> websockets servers" is a reasonable argument, especially since the 
> state of the web suggests that many people will write and deploy bad 
> code whether we lik it or not. It is certianly not just the small 
> sites that are full of WTF level javascript, for example.
The WTF javascript authors don't write servers. They're going to write 
WTF php scripts behind Apache.

Look at Ian's example links. They got the parsing right, but they 
botched the networking and concurrency. Writing a FSM-based non-blocking 
server is more complicated than parsing a length-framed packet.

Writing even the most basic websocket server is more complicated than 
parsing a length-framed packet. Writing a minimally-secure one is more 
complicated than that. Ian's own server has two major DOS 
vulnerabilities that are trivial to exploit.

And that complexity is before adding basic security requirements like 
authentication. Using the current draft, the servers need to roll their 
own authentication sub-protocol. I can't see how anyone can argue that 
designing authentication is easier than counting the length of a string.

-- Scott

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 30 Jul 2010 22:48:08 +0200

> situations where something similar has occurred.

In this precise case, I don't see how that could switch to only one use
or another depending on a market leader, because many components will
use the length, at least for performance reasons, making buggy implementations
quickly incompatible with them. Many naive implementations would probably
ignore the check on the 0xFF at the end, but this will have a limited
impact because the additional bytes if any would be between the unchecked
FF and the length of the advertised buffer. I don't see how current spec
protects against that class of errors anyway.

Regards,
Willy

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 21:09:18 +0000 (UTC)

On Fri, 30 Jul 2010, Mike Belshe wrote:
> >
> > WebSockets is a protocol designed with these goals in mind (in order, 
> > most important first):
> >
> Please go back and read the group's comments about "amateur programmers" 
> as a requirement.  This is your requirement only; the rest of the group 
> was pretty unanimously against this so-called requirement.

Then the group isn't talking about Web Sockets. That's fine, but we should 
clearly decide whether this group is still interested in working on the 
protocol that the WHATWG was working on when the IETF asked to be allowed 
to work on it, or whether this group wants to work on something else. It 
would be a waste of everyone's time to be trying to design two protocols 
with conflicting requirements at the same time in one spec.

> At this point, I can't continue to rehash the same arguments.  The 
> arguments are on the table, so others can weigh in.  You are now citing 
> your own document (the first msg of this thread) as proof against my 
> criticism of it. It's circular.

With all due respect, I only did so because the arguments presented were 
already countered by that e-mail.

> Cyclical arguments and refusal to listen to the decisions of the group 
> make it impossible for me to engage you in further debate on this.

Simply ignoring our responsibilities, like making a protocol that is 
designed so that the likely implementation errors (mismeasuring strings, 
doing regexp on the handshake instead of parsing it, etc) do not lead to 
security vulnerabilities, isn't an option. It really doesn't matter how 
many people decide that we should write a protocol that will lead to 
insecure implementations. Doing so is irresponsible on the Web.

---

From: Ian Hickson <[email-masked]>
Date: Fri, 30 Jul 2010 21:23:00 +0000 (UTC)

On Fri, 30 Jul 2010, Roberto Peon wrote:
> > >
> > > I've already shown that with length prefixing you can have the 
> > > receiver chose to parse using either sentinel scanning or 
> > > byte-counting with semantics that provide for non-channel-close 
> > > error semantics where appropriate.
> >
> > Unfortunately this kind of redundancy in practice leads to 
> > implementation differences that become very relevant. For example, 
> > suppose IE implements just the length measuring, and ignores the 
> > sentinels. A server that doesn't correctly send the sentinels then 
> > will work with IE and fail with browsers that do stricter checking. If 
> > IE has high market share when this happens, the net result will be 
> > that we can no longer do the stricter checking, and now we have a 
> > vestigial feature in the protocol.
> 

I don't see how. An implementation can't ignore part of the framing 
mechanisms as the spec is written today, so interoperability isn't at 
risk as far as I can tell. If you can come up with a scenario where this 
could occur, though, maybe the spec needs changing.

> > history.pushState() in the HTML API -- before the spec is even 
> > finalised, we already have a vestigial argument to the function, and 
> > there's nothing we can do about it because changing the API would 
> > break existing pages.
> >
> > When the data is UTF-8, but the server only expects ASCII, length 
> > framing runs the risk of measuring the wrong thing, leading to 
> > security vulnerabilities. I explained this in detail here at the top 
> > of this thread:
> >
> That is why you like a sentinel, which I said you could still do, but 
> you said would lead to redundancy leading to implementation 
> rot/fracturization. You can't argue that it is redundant/unnecessary and 
> then argue that you need it, or rather, one shouldn't argue that!

Having two simultaneous mechanisms used for framing the same data is 
redundancy, redundancy leads to interoperability problems.

Maybe I misunderstood what you were proposing? Could you give an example 
of what you imagined the wire format would look like?

> > > Also, you're ignoring the argument about parsing via sentinel 
> > > scanning being far more complex, even when we're not talking about 
> > > efficient implementations (which it makes FAR more complex).
> >
> > In the case of low-throughput implementations, it's trivial. You can 
> > just do things one byte at a time, for example.
> 

By "low-throughput" I mean like sites that have a few hundred WebSocket 
connections a day; something an phone could handle without blinking. I 
didn't mean a situation where we had a hundred million users only sending 
a few bytes a day.

> > In the case of high-throughput implementations, I don't see why the 
> > server would need to implement this at all on the medium term. Once we 
> > have compression, why would you use anything uncompressed?
> 
> Compression can and will be done at different layers. Sometimes you want 
> to compress HTML/XML. That is obviously a win. Attempting to compress a 
> video, image, etc. likely would likely result in an expansion of data 
> size as well as lots of needless CPU cost.  Thus, the sender should 
> always be deciding whether or not to send compressed based on the data 
> that is being sent.

Sure, but the only time we're talking about sentinels is for text data, 
which would always be compressed, right?

> believe that high-throughput servers should always use compressed data, 
> then they should always use length-prefixing.

That's what I'm saying, yes.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 30 Jul 2010 23:26:44 +0200

> insecure implementations. Doing so is irresponsible on the Web.

But you're not the only one to be able to write bugs. Several people
suggest various types of bugs that are more or less addressed by each
proposal. In some situations your proposals work, and in some they
fail (as in the two you have pointed, eventhough one was more of an
incomplete code in progress than a vulnerable one). The examples of
failure you propose don't convince everyone, sometimes because they
require special skills or stacking multiple bugs. But you regularly
decide to dismiss other bugs that are suggested, as if you knew
better than anyone else what the bugs will be.

All we know is that the current design has already been put into
publicly available implementations that are buggy and/or vulnerable,
and that the handshake itself has interoperability issues with HTTP
components and is already not correctly handled in one implementation
you pointed.

I'm not saying that the design it at fault, but it's a bit exagerated
to claim a successful handling of failure conditions that way. Basically,
the will to support "amateur programmers" is already proven not to work
as expected. So maybe we should humbly accept that we don't know what
programmers are capable of, and consider that whatever effort we put
into securing some of them just causes difficulties to other ones,
causing failures.

A simple protocol has less chances of being wrongly implemented than
a complex one apparently designed never to fail.

Willy

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 31 Jul 2010 00:41:40 +0200

>       buffer = substr(buffer, start, length(buffer));

Assuming typical conventions for the syntax and semantics you are using,
that code would seem to have a rather large number of problems, e.g., a
closing parenthesis is missing, you are using assignment where you mean
to use comparison, for text frames you are using the frame type as part
of the string, you treat binary frames as text (short frames in case of
a fault tolerant UTF-8 decoder, which are common, long otherwise [1]),
you treat closing frames as zero-length text frame followed by another
text frame (whose sentinel byte will probably never come), and you might
be reading out of bounds of the buffer (length seems to return the first
index outside it, and initially the third argument to substr is the last
index .inside. that you want to extract, then you pass in the length,
which is outside; if you start with a closing frame and have not checked
for that beforehand, you even pass -1 as third argument to substr).

You might have meant to have a higher level caller that takes care of
the frame type recognition, but then you cannot buffer within the gene-
rator unless readBytes just so happens to stop on 0xFF, and you would
still be passing a negative offset to substr if you have a text frame
with zero characters, whatever effect that has in your pseudocode. Now,
that would seem to be too many problems, and I haven't put much tought
into it, so I am probably mistaken on some of the points, but then your
code certainly demonstrates that having multiple ways to indicate the
length of a frame leads to implementations that are difficult to audit,
as far as I am concerned anyway. Greg's code is incorrect aswell, if I
assume conventional semantics, since he reads past the end of the frame
into a buffer, and then the buffer goes out of scope, so he cannot read
more than a single frame correctly (again, unless readBytes stops at
the sentinel value).

[1] Oh my goodness, your draft even requires using a fault tolerant
    UTF-8 decoder for clients .and. fails to define how to resynchro-
    nize after the decoder hits bad bytes, not to mention that what-
    ever you specify is unlikely to be what the library function
    (non-browser) implementers use does, if it can recover at all.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Patrick McManus <[email-masked]>
Date: Fri, 30 Jul 2010 19:23:48 -0400

Related to the syntax of framing and chunking, I also urge the group to
tackle multiplexing in the first round of the draft. Connection
management has long been a thorn in the side of HTTP - leading to
arbitrary numbers of parallel connections and their assorted problems
interacting poorly with buffering and TCP congestion control.

Multiplexing multiple websocket messages simultaneously over a single
TCP channel (using chunks as necessary) allows them to share a single
TCP congestion control block as well as a single TCP handshake. Both of
those things will be significant wins in terms of reliable efficient
performance. 

There is a downside in terms of fate sharing across messages during loss
recovery events, but frankly a single busy TCP session makes loss
recovery much more efficient than if the same data is spread of
unrelated TCP flows. More importantly the number of client induced
losses due to inaccurate aggregate cwnd's across the large number of
parallel connections is reduced. That's probably the most important
thing.

Not tackling it now basically makes repeating the mistakes of http
forseeable, and I would not support the IETF endorsing such a
specification.

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 31 Jul 2010 12:05:52 +1000

Ian,

So it's "If you don't let me win, I'm taking my bat and ball and going
home"!!

Really???

When you make statements like the one above, are you speaking  as an
individual or as the spokesperson for WHATWG, which is a consortium of
browser vendors?   Are you really saying that if you can't get your way 100%
with the framing design, that you are going to lead the browser vendors out
of the IETF process?

The WHATWG does not have the right constituency to define new standards for
internet protocols.  This protocol is being put through the IETF process
because the vast majority of internet protocols are standardized through
this process and the constituency has good representation and equal access
for vendors, users, client side, intermediaries, server-side, newbies and
grey beards with battle scars from past protocol battles.

Your concerns have been considered by the discussions and proposals here.
They have resulted in much simplification of proposals and a general desire
to select only the minimal required feature set.  The proposals are better
for your contributions and we are pretty much converging on the semantics
that you have argued for.  But it appears that WG is not in agreement with
the exact syntax you want.

Is the inclusion of sentinel framing such an important issue that you are
prepared to break the process and send the protocol back to factional based
evolution?

Browser Vendors?  Do you support Ian as your spokesperson in his
intransigence on this issue?

---

From: Ian Hickson <[email-masked]>
Date: Sat, 31 Jul 2010 06:36:09 +0000 (UTC)

On Sat, 31 Jul 2010, Greg Wilkins wrote:
> > > Please go back and read the group's comments about "amateur 
> > > programmers" as a requirement.  This is your requirement only; the 
> > > rest of the group was pretty unanimously against this so-called 
> > > requirement.
> >
> > Then the group isn't talking about Web Sockets. That's fine, but we 
> > should clearly decide whether this group is still interested in 
> > working on the protocol that the WHATWG was working on when the IETF 
> > asked to be allowed to work on it, or whether this group wants to work 
> > on something else. It would be a waste of everyone's time to be trying 
> > to design two protocols with conflicting requirements at the same time 
> > in one spec.
> 

No, not at all. I'm just describing the protocol that I am writing. 
There's no reason I have to be the editor of the spec the hybi group 
writes; there's no reason the Web can't have another protocol that does 
things differently. There's plenty of space for multiple protocols on the 
Internet. I was asked to develop this protocol in the IETF. If the IETF 
doesn't want to write develop that protocol after all, that's really fine; 
it just means that there was a misunderstanding about what the protocol 
was when the IETF asked for the protocol to be published through the IETF.

As an analogy, suppose WebSockets was an electric locomotive. The IETF 
asked if they could work on this vehicle. Now there appear to be some 
people who think that WebSockets should be able to travel on water. 
Electric locomotives don't travel on water, they travel on rails. Does 
that mean the people who want a water vehicle are wrong? No, not at all. 
It just means they want a boat, not a train. Does that mean the people 
designing the train should stop making their train and make a boat 
instead? Again, not at all. We can have both a boat and a train. They're 
not the same vehicle; we shouldn't try to make one vehicle that is both, 
or we'll just end up with a Frankensteinesque monster.

It's not clear to me if the hybi group wants an electric locomotive or a 
boat or something in between. I'm happy to develop WebSockets here if 
that's what people want. If it's not, then why would we start with an 
electric locomotive? Let's start with what the working group wants.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 31 Jul 2010 09:18:14 +0200

> electric locomotive? Let's start with what the working group wants.

Ian, people here are not trying to make you develop something different,
they're trying to show you that the current design will inevitably
involve some issues that are already easily identified, and that by
considering them right now and seeing what can be done to circumvent
them, you can achieve something more robust satisfying the same original
goals.

The problem is, you simply see the issues that *you* want to address,
and don't want to consider as possible the other issues people try
show you. You pointed yourself some code showing that it was not
easy to get current design right, so why pretend the opposite. We
all know it's not easy and problems should be solved in the protocol
to address these issues, but problems have to be considered first.

Regards,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 31 Jul 2010 17:21:22 +1000

On 31 July 2010 16:36, Ian Hickson <[email-masked]> wrote:

> things differently.

The only reason that you appear to think there are two different protocols
is that you are artificially con straining  the requirements to be "the
protocol that I designed for my own purposes".  Because this group wishes to
solve the problems in a slightly different way does not mean there are two
protocols.

Besides, I think that  the IETF process needs to participation of the
browser vendors to be able to create any meaningful protocol.    Also, the
browser vendors need buy-in from server-side, intermediary and other
interests that are represented in the IETF.

Together we can do something. Apart we are all just pissing in the wind.

> Electric locomotives don't travel on water, they travel on rails.

To stretch this analogy, we have some potential passengers who wish to
travel between the client and the server.  They don't really care how they
get there, but they have two simple requirements:

   - they can travel from a town called Server to the village called Client
   just as easily as they can travel from Client to the Server.
   - all the passengers that left together will arrived together.

Some in the IETF are saying that we could meet this demand with a fleet of
mini buses that can also pick up passengers at the village called Desktop,
stop at the Gateway interchange and is better at clearing customs at the
border with the country DataCenter.  There is a bit of debate about if light
rail would be better or if perhaps the buses should be solar powered.

You're saying that this can only be done with a vehicle that can run on
tracks, is painted green, goes toot toot and answers to the name of
Thomas.

regards

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 31 Jul 2010 09:42:01 +0200

Also I'd like to add something.

> No, not at all. I'm just describing the protocol that I am writing. 

I know you get lots of personal attacks (and you even got a few from
me at times you managed to get me nervous, though I deplore this). This
is something which should not normally happen in a WG and which happens
precisely because you're just offering *your* opinion based on *your*
own intuition. Facts and errors of the past unfortunately have too
little place here, while we're trying to build something that people
expect to be able to use with a high interoperability and securely
without repeating failures of the past.

For instance, you think nobody will use libraries, though you force
them to rely on MD5. You think they must be able to implement a handshake
parser without knowing a word of HTTP, but the examples running in HTTP
servers got difficulties implementing the protocol due to it not being
exactly HTTP, and the ones implementing it from scratch result in
completely buggy implementations as in the code you pointed out.

Some persons try to explain what type of problem may happen if we
don't do X or Y, but that gets wiped with a simple "I don't buy that".

You should consider that it's not easy to work as a group that way.

You may very well write *your* own protocol, as you call it, but
blindly putting it into browsers and servers with a W3C or IETF stamp
will result in broad adoption, and  many important interoperability
or security issues can surface. For this reason, it's not correct to
ignore the issues some people are trying to solve.

For instance, Adam sometimes reports very unlikely security issues,
that noone else thinks about but which can happen. Since he provides
examples of misuse which can result in X or Y with an estimation of
the impact, his suggestions are considered with a lot more interest
and debated in a constructive way.
 
Issues must be debated, not flushed with little consideration based
on pure guess or a wet finger in the wind.

> writes;
> there's no reason the Web can't have another protocol that does 
> Internet.

Except that this one has to become part of a widely adopted standard
and that it's important that we ensure that most implementations will
get it right with very little risks of undesired side effects or even
security issues.

Right now the hanshake is not compatible with the protocol it tries
to imitate, and the framing is difficult to handle on the receiver
side. Those are facts and you shown such examples yourself.

Last, you're considering this list as being IETF driven. But some
people subscribed to report issues as users. It's the case for me.
And I find it spectacular that real issues from the field get ignored
while they're a proof of failure. I personally don't mind if it's the
WHATWG, IETF, W3C, HYBI or whatever who supports the protocol, what I
want is something that works.

Thanks,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Sat, 31 Jul 2010 09:42:40 +0200

On 31.07.2010 04:05, Greg Wilkins wrote:
> browser vendors?   Are you really saying that if you can't get your way
> 100% with the framing design, that you are going to lead the browser
> vendors out of the IETF process?
> ...

The WhatWG is a group of people who happen to work for browser vendors. 
Unless I'm mistaken, I don't think it can be said that these people 
represent their company's opinions. Which, btw, is also the case for any 
IETF WG - everybody speaks as individual.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 31 Jul 2010 09:48:18 +0200

On 31.07.2010 09:21, Greg Wilkins wrote:
> ...
> Some in the IETF are saying that we could meet this demand with a fleet
> of  mini buses that can also pick up passengers at the village called
> Desktop, stop at the Gateway interchange and is better at clearing
> customs at the border with the country DataCenter.  There is a bit of
> debate about if light rail would be better or if perhaps the buses
> should be solar powered.
> ...

Given the last discussion on the ietf78 attendees mailing list, we 
really need to be careful with ground transport analogies.

Best regards, Julian

(Sorry, but I really couldn't resist...)

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Sat, 31 Jul 2010 01:12:50 -0700

> intransigence on this issue?
Wow, this is a bit of a loaded question. I'm not sure that Hixie actually
tried to claim a title of spokesperson for browsers... That said, here's my
perspective. First off, I have great respect for Hixie as he's been able to
pull off a lot with HTML5 and WHATWG, more than I think most other people
would stand a chance of accomplishing. What I see happening is this: I think
Hixie is describing a possible outcome where IETF continues work on a
protocol given the current expressed interest and direction/consensus on the
hybi list, while Hixie continues to write a separate spec/protocol
(presumably in WHATWG), and then it's up to Hixie on one hand and IETF
members on the other hand to convince people that their protocol is
"better", e.g. to implement it. (Presumably the JS API would not be
fractured.) People go off and create new protocols/specs all the time, and
not all of them get implemented / see any wide adoption. It happens all the
time, even in places like W3C - such is life. If he wants to do that, I
would be sad to lose his perspective on the work that would presumably
continue in IETF, but that is certainly his right. I presume in that case
the IETF spec would need a new editor.

-Ian F

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 31 Jul 2010 18:29:25 +1000

Julian

from the bottom of   http://www.whatwg.org/charter

Membership

Anyone can contribute by subscribing to the mailing
list<http://www.whatwg.org/mailing-list>.
The list of subscribers to the mailing list are termed the contributors.

Membership is by invitation only, and consists of a number of
representatives from various browser manufacturers. This group, which is
referred to as the members, will provide overall guidance as described in
the charter above. The members currently consists of:

   - Anne van Kesteren
   - Brendan Eich
   - David Baron
   - David Hyatt
   - Dean Edwards
   - Håkon Wium Lie
   - Ian Hickson
   - Johnny Stenback
   - Maciej Stachowiak

"Queries should be directed either to the mailing
list<http://www.whatwg.org/mailing-list>or to Ian
Hickson <[email-masked]>, who is acting as a spokesman for the group."

So they do describe Ian as a spokesman, so it would be good to know when he
is speaking in that role when suggesting that the two groups should not work
together.

cheers

---

From: Rob Sayre <[email-masked]>
Date: Sat, 31 Jul 2010 01:31:01 -0700

On 7/31/10 1:12 AM, Ian Fette (イアンフェッティ) wrote:
> I think Hixie is describing a possible outcome where IETF continues 
> work on a protocol given the current expressed interest and 
> direction/consensus on the hybi list, while Hixie continues to write a 
> separate spec/protocol (presumably in WHATWG), and then it's up to 
> Hixie on one hand and IETF members on the other hand to convince 
> people that their protocol is "better", e.g. to implement it.

I think it is more like a wager that the IETF process will be so 
ineffective as to be irrelevant. As long as that's a good bet, Ian's 
version will get implemented, vacuous requirements and all. Personally, 
I find the situation pretty disheartening. The requirement at issue has 
been used to advocate many positions I agree with, but I find the term 
meaningless nevertheless.

Two days ago, I thought the WG had settled on a reasonable schedule and 
plan proposed by Ian. Then we got a huge thread full of theoretical 
requirements (bad behavior, close to spam), and the people who hang out 
on the #whatwg IRC channel reverted to clinging to their meaningless 
"amateur programmer" requirement. Then a whole bit about "what is the 
WHATWG". Hint: every time you go there, you lose.

Get it together.

- Rob

---

From: Ian Hickson <[email-masked]>
Date: Sat, 31 Jul 2010 08:33:37 +0000 (UTC)

> tracks, is painted green, goes toot toot and answers to the name of 

I don't know what "this" is. That's the problem.

If "this" is just "a protocol for sending data bidirectionally between two 
peers", then there are dozens of existing solutions for this, and many 
more possible ones. WebSockets is but one of many. (Others would include 
such protocols as XMPP, BEEP, Greg's various proposals, etc.)

> The WhatWG is a group of people who happen to work for browser vendors.

The WHATWG is a group of currently over 1500 people. I don't think all the 
browser vendors put together employ that many people. The vast majority of 
people on the WHATWG list have no relationship to browser vendors.

On Sat, 31 Jul 2010, Willy Tarreau wrote:
> 
> For instance, you think nobody will use libraries

No, I think we shouldn't force people to use libraries. Of course people 
will use libraries. But it should also be easy to implement the protocol 
from scratch.

> though you force them to rely on MD5.

I would love to see a solution that avoids reliance on a cryptographic 
library, but unfortunately designing cryptographic hashes is not something 
that one should take on lightly. Using MD5 is a tradeoff; we lose 
simplicity in return for a better understanding of the characteristics of 
at particular defence-in-depth aspect of our security story.

> You may very well write *your* own protocol, as you call it, but blindly 
> putting it into browsers and servers with a W3C or IETF stamp will 
> result in broad adoption, and many important interoperability or 
> security issues can surface.

I'm not sure what you mean here, but if you're implying that browser 
vendors implement anything with a W3C or IETF stamp, I assure you that 
they do not. They implement what they think is best.

---

From: Julian Reschke <[email-masked]>
Date: Sat, 31 Jul 2010 10:47:24 +0200

On 31.07.2010 10:33, Ian Hickson wrote:
> ....

I was referring to what the WHATWG charter 
(<http://www.whatwg.org/charter>) calls "members", not "contributors".

Best regards, Julian

PS: I think it would be good to separate meta discussions like this from 
the actual design discussions. Also, the WG did some hums related to 
framing on Friday, and it might be wise to wait for the WG chairs to 
post their summary before spending too much time on this issue over the 
weekend.

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 31 Jul 2010 10:58:43 +0200

Ian,

Your defense of the amateur long tail is admirable but you're not
doing anyone a favour - not your long tail, not WHATWG, not
implementors - by ersatzing the IETF process and treating your peers
to patronizing misanalogies about steam trains and electric boats
while insisting on "your" designs.

One of two things will happen here.  Either you will abandon your
insistence on being the author of WebSockets, and you will allow your
peers to take over the design process. Or, this group will conclude
that your only acceptable outcome is a rubber stamping of your text,
and will reject that as incompatible with its mission and process.

It seems you distrust others and dislike working with people unless
you are the thought leader.  Presumably in WHATWG this is how it
works.  You seem to believe you possess almost supernatural abilities
to understand reality to a depth that mere mortals cannot.  This is a
common mindset among very clever people but it's not useful except for
playing chess.  Which this is not.

What will absolutely not happen is that this group will suddenly see
the light, accept that you were right all along, and publish your
protocol.  Nor will the IETF publish your protocol simply because it's
gotten adoption.  Nor will server implementors ever build on top of a
non-IETF WebSockets, because it would not form part of any real
contract they can trust.  WebSockets will work, for a while, and then
be replaced by a real protocol once the experiment is over.  You will
be remembered vaguely and with little love.

Which is what this is all about: long term trust that our investments
in protocol stacks are safe from people who decide that they know
better and want to overthrow the whole world.

My advice is to swallow your pride (in small UTF8-encoded chunks,
possibly) and stop designing WebSockets, perhaps remain as editor, but
allow others to make 100% of the design decisions through processes
that are trusted, proven, and form part of the whole Internet
contract.

Being the lone genius is an anti-pattern, when it comes to protocol
design.  Very clever people, working alone, make very clever mistakes.

-Pieter

---

From: Scott Ferguson <[email-masked]>
Date: Sat, 31 Jul 2010 10:34:02 -0700

> was when the IETF asked for the protocol to be published through the IETF.

That's reasonable. It is unfair to ask Ian to be the editor of a 
protocol he doesn't believe in.

-- Scott