Subject: [hybi] Questions on WebSocket RFC draft 76: "Early" data, whitespace, and integer sizes

From: Brad Spencer <[email-masked]>
Date: Tue, 4 May 2010 21:24:29 -0300

Hi.  I've been implementing the server side of the WebSocket RFC
draft, and I'm happy to report that not only is the specification very
easy to read and understand, but I also haven't really come across any
obvious errors or inconsistencies.  The protocol seems very logical,
easy to deal with, and sensible.  So, nice work :)

[Oh, and BTW, being very strict about exactly what the handshake may
and may not contain has made this much easier to implement than HTTP.
I can't say how happy I was to see rules like "there must be exactly
one space here", and so on ;)]

I've been referring to the draft served up here:

  http://www.whatwg.org/specs/web-socket-protocol/

It is dated April 15, 2010, and I presume it is "version 76".  

However, I do have some questions.  (My apologies if they have been
answered already.  I read some of the recent postings to this list,
but didn't find an easy way to search it.)

1. Can a client send any data to the server after the eight byte
   "key3" data, but before the server has responded to the client's
   initial handshake request?  In other words, is the client allowed
   to "pipeline" the first message?

  I am hoping the answer is "no" because:

    a) I don't think it makes sense to send messages before the
       handshake is complete.  This is probably especially true when
       the subprotocol needs to be confirmed, etc.
       
    b) I am using at is part of the validation of the handshake,
       sort of.  If I have _more_ than what should be in a valid
       handshake, I'd prefer to reject the connection out of hand.

    c) Preventing such data might make things simpler, although it
       doesn't appear to be a practical problem for me to actually
       interpret the any such "early" data as a frame data.

2. In HTTP, IIRC, header values can have arbitrary amounts of
   whitespace between the "field-value" productions (i.e. leading
   and trailing whitespace in the header value itself ends up
   getting ignored).  In WebSocket, this doesn't appear to be the
   case (good), but it does mean that if a client sends, say,

     Connection:_Upgrade__

   or

     Host:__www.example.com___

   where _ represents a space, then the server is going to end up
   failing the handshake, as I understand it.  Is this intended?

   (To be clear, I am quite happy to have the server fail such
   handshakes, as long as they are truly invalid. :)

3. When computing the server's /response/ to the /challenge/ in
   Section 5.2, the RFC draft is explicit about the fact that /part1/
   and /part2/ are 32-bit numbers, but it isn't clear how large
   /key-number_1/ and /key-number_2/ are allowed to be.  I've presumed
   they are also 32-bit numbers based on what's going on with regards
   to the multiplication specified in Section 4.2 Item 19.

   Is it worth saying so explicitly in Section 5.2 Item 5 that the
   /key-number_*/ numbers are also 32-bit?

   Also, I presume that all of these numbers are 32-bit _unsigned_
   numbers, but perhaps that goes without saying.
   
I'd appreciate any guidance on these questions.  Thanks!

-- 
----------------------------------------------------------
Brad Spencer - [email-masked] - www.starscale.com

---

From: "Simon Pieters" <[email-masked]>
Date: Wed, 05 May 2010 10:25:30 +0200

> but didn't find an easy way to search it.)

I'll try to answer some of your questions based on my reading of the spec.

>    to "pipeline" the first message?

No.

[[
Once a WebSocket connection is established, but before the WebSocket  
closing handshake has started, the user agent must use the following steps  
to send data using the WebSocket:
]]

In the API, send() will throw an exception if invoked before the  
connection is established:

[[
The send(data) method transmits data using the connection. If the  
readyState attribute is CONNECTING, it must raise an INVALID_STATE_ERR  
exception.
]]

>        handshake, I'd prefer to reject the connection out of hand.

I think you're allowed to do that:

[[
The client handshake consists of the following parts. If the server, while  
reading the handshake, finds that the client did not send a handshake that  
matches the description below, the server should abort the WebSocket  
connection.
]]

[[
Servers may close the WebSocket connection whenever desired.
]]

>    failing the handshake, as I understand it.  Is this intended?

I think so, yes.

>    /key-number_*/ numbers are also 32-bit?

If we look at the client's construction of the handshake:

[[
16. Let spaces1 be a random integer from 1 to 12 inclusive.
Let spaces2 be a random integer from 1 to 12 inclusive.

For example, 5 and 9.

17. Let max1 be the largest integer not greater than 4,294,967,295 divided  
by spaces1.
Let max2 be the largest integer not greater than 4,294,967,295 divided by  
spaces2.

Continuing the example, 858,993,459 and 477,218,588.

18. Let number1 be a random integer from 0 to max1 inclusive.
Let number2 be a random integer from 0 to max2 inclusive.

For example, 777,007,543 and 114,997,259.

19. Let product1 be the result of multiplying number1 and spaces1 together.
Let product2 be the result of multiplying number2 and spaces2 together.
]]

product1 and product2 are the numbers sent in -Key1 and -Key2.

I agree it would be good if the server section stated the expected range  
of key-number*.

> I'd appreciate any guidance on these questions.  Thanks!

HTH,
-- 
Simon Pieters
Opera Software

---

From: Julian Reschke <[email-masked]>
Date: Wed, 05 May 2010 11:24:22 +0200

On 05.05.2010 02:24, Brad Spencer wrote:
> ...

By the way: Ian, it is a REALLY REALLY bad idea to publish something 
that has the format and the name of an Internet Draft and a fixed draft 
number, but which changes over time. By any means, don't call it "76" 
unless you're planning to submit it as "76". Otherwise, name it 
"latest", "current", or something like that.

Best regards, Julian

---

From: Adam Barth <[email-masked]>
Date: Wed, 5 May 2010 09:38:40 -0700

On Wed, May 5, 2010 at 2:24 AM, Julian Reschke <[email-masked]> wrote:
> By the way: Ian, it is a REALLY REALLY bad idea to publish something that
> has the format and the name of an Internet Draft and a fixed draft number,
> but which changes over time. By any means, don't call it "76" unless you're
> planning to submit it as "76". Otherwise, name it "latest", "current", or
> something like that.

One thing I've done in the past is use a "pre" postfix.  So, I'd label
the draft draft-hixie-worlddomination-76pre and then bump the version
to -76 right before pushing the bits to the IETF server.

Adam

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 6 May 2010 10:29:44 +0100

> >       the subprotocol needs to be confirmed, etc.

If you don't care about extra time delays, it's easier with "no".

With high latency links, such as mobile data, unnecessary extra
round trip times add up quickly to be annoying - hence the SPDY folks
interested in minimising connection setup latency in every way possible.

Comet-style messaging over HTTP may be faster to start up
than WebSocket in those scenarios.  It would be perverse if good
WebSocket JavaScript frameworks had to open a comet-style connection
_and_ a WebSocket connection to get good all round performance :-/

> Servers may close the WebSocket connection whenever desired.

Does that mean in normal operation, or only in error / timeout scenarios?

If that's normal server behaviour, are we repeating the design flaw in
HTTP pipelining that stops it being safe to use for non-idempotent
messages?

> >I'd appreciate any guidance on these questions.  Thanks!

Is there a reason to presume they are 32-bit, 31-bit, 64-bit, or
anything else?

"32-bit unsigned" seems to be contrary to the debated requirement of
"easy to write by amateurs", as many "scripting" languages don't have
such a type conveniently available.

-- Jamie

---

From: "Simon Pieters" <[email-masked]>
Date: Thu, 06 May 2010 13:37:26 +0200

On Thu, 06 May 2010 11:29:44 +0200, Jamie Lokier <[email-masked]>  
wrote:

> round trip times add up quickly to be annoying

Yes, but it's not unnecessary to wait with sending the first frame from  
the client, since if it were allowed to send a frame along with the  
opening handshake, then you would be able to send arbitrary HTTP requests  
by carefully crafting your first message so that an HTTP server sees first  
an HTTP upgrade request that it rejects, then a bogus HTTP request that it  
rejects (starting with the 8 random bytes), then the attacker's carefully  
crafted HTTP request which could do anything without the user's consent.

Therefore the client needs to wait for the handshake to complete before  
sending its first message. However the server could send its first message  
together with the handshake response if it wants to (and indeed the  
closing handshake also in the same packet if it wants to and it fits).

> - hence the SPDY folks

If that's the case, that seems like a good reason to lower the maximum  
number in the keys to something that all scripting languages support  
without the developer having to jump through hoops. Maybe we should make  
the range 0-32767 to allow usage of 16-bit signed integers.

-- 
Simon Pieters
Opera Software

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 06 May 2010 09:25:26 -0700

Simon Pieters wrote:
>
> Yes, but it's not unnecessary to wait with sending the first frame 
> from the client, since if it were allowed to send a frame along with 
> the opening handshake, then you would be able to send arbitrary HTTP 
> requests by carefully crafting your first message so that an HTTP 
> server sees first an HTTP upgrade request that it rejects, then a 
> bogus HTTP request that it rejects (starting with the 8 random bytes), 
> then the attacker's carefully crafted HTTP request which could do 
> anything without the user's consent.
That scenario doesn't really make sense.

If a server sees random junk, it doesn't keep processing until it sees 
something that looks like a HTTP header. Instead, it needs to close the 
connection, possibly after returning a 4xx response. (Broken servers are 
irrelevant here. If the server is broken, it's a security hole no matter 
what the spec says. Servers are not like browsers, which try to keep 
processing after bogus data.)

As long as the frame immediately following the upgrade can't simulate a 
valid HTTP request,  there's no actual attack.

 From the HTTP spec, a valid request starts with:

  (whitespace)* token ...

where

  token = any US-ASCII char but CTL or separator

In particular, a HTTP request can never start with 0x80 - 0xff.

The server can detect this kind of attack as long as the initial frame 
must start with a byte from the set:

  start-chars ::= [\x80-\xff] | [\x00-\x08] | [\x0e-\x1f] | '(' | ')' | 
'<' | ... etc

The cleanest way to protect a user from this attack is to require an 
initial frame with the high bit set, probably some kind of control frame.

-- Scott
>
> Therefore the client needs to wait for the handshake to complete 
> before sending its first message. However the server could send its 
> first message together with the handshake response if it wants to (and 
> indeed the closing handshake also in the same packet if it wants to 
> and it fits).
>
>
>> - hence the SPDY folks
> without the developer having to jump through hoops. Maybe we should 
> make the range 0-32767 to allow usage of 16-bit signed integers.
>

---

From: Brad Spencer <[email-masked]>
Date: Thu, 6 May 2010 20:27:09 -0300

> interested in minimising connection setup latency in every way possible.

I'll just mention that I haven't seen any implementation reason (at
least with the approach I am using) that prevents the server from
looking at data after the eight bytes as protocol data.  It will have
already decided whether to accept the connection and responded or not,
and it doesn't make a difference if the data for the first message has
arrived already or not.

> Is there a reason to presume they are 32-bit, 31-bit, 64-bit, or

The server has to perform integer division on the numbers, so I'd
really rather to be able to do that with a built-in type.  I don't
want to have to deal with arbitrarily large dividends (it seems
undesirable to require some kind of big-number package just to do
this).  Plus, the rules seem to require that the results of the
division are (unsigned) 32-bit numbers, so there's not much point in
making them much bigger than (2^32 - 1) * 12 (where 12 is the maximum
number of spaces), as far as I can see.

> such a type conveniently available.

The numbers have to non-negative since they can only be made up of the
ASCII digits collected from the headers (no minus sign), of course.
But, I can see why it would be useful to restrict them to the [0,
2^31) limit of some languages' basic integer type.  I can't think of a
language that would need something as restrictive as [0, 2^15),
though.

BTW, thanks to everyone for their help!

---

From: "Simon Pieters" <[email-masked]>
Date: Fri, 07 May 2010 08:43:28 +0200

On Thu, 06 May 2010 18:25:26 +0200, Scott Ferguson <[email-masked]> wrote:

>> opening handshake, then you would be able to send arbitrary HTTP  
>> server sees first an HTTP upgrade request that it rejects, then a bogus  
>> HTTP request that it rejects (starting with the 8 random bytes), then  
>> the attacker's carefully crafted HTTP request which could do anything  
>> without the user's consent.

Ok.

> initial frame with the high bit set, probably some kind of control frame.

I thought what followed the upgrade request from a HTTP server's point of  
view are the eight random bytes. Those bytes could happen to result in a  
start of a valid HTTP request, but that seems extremely unlikely.

>> sending its first message. However the server could send its first  
>> message together with the handshake response if it wants to (and indeed  
>> the closing handshake also in the same packet if it wants to and it  
>> fits).
>>
>>
>>> - hence the SPDY folks

-- 
Simon Pieters
Opera Software

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 07 May 2010 09:02:02 +0200

> and it doesn't make a difference if the data for the first message has

I've previously outlines some complications in implementation that
a server must go into to accept the connection with these extra
bytes in the stream.  In summary those concerns are around poor
software encapsulation resulting from this poor software design.

There is a connection - that prior to the HTTP 101 response should
contain only HTTP framed bytes.  After the HTTP 101 response it
should contain only websocket framed bytes.   We currently have
these extra bytes that are neither HTTP nor Websocket that arrive
and and sent during the handshake.   The result is that it is
difficult to right server software than just knows about HTTP
and other software that just knows about Websocket.

The extra bytes are after the request only to test the websocket
connection - so why not send a websocket frame after the
handshake response?  Why does it need to be yet another type
of data on the wire (we already have HTTP and 2 types of
websocket framing)?  It complicates clients, servers and
intermediaries.

Also there is another problem with the bytes after the request (but
not after the response).  If the server (or any intermediary) chooses
to ignore the Upgrade header, these bytes prevent the HTTP connection
from being used as a HTTP connection for subsequent requests.

We have seen from the google research that RTT and number of RTs
is very important when establishing a page.   If we adhere to the
upgrade mechanism as specified in RFC2616, then a connection that
is not upgraded can be used as a normal HTTP connection.  Indeed
the request that contained the upgrade can return useful content
or even become a hanging GET.     OK this is not currently possible
with the JS API, but that is not the only use-case and we should
preserve the possibility of future usage of compliant HTTP.

If for example, we eventually decide to permit BASIC and/or DIGEST
authentication with websockets (reasonable as most clients
support it, most existing servers support it and it is simple
to implement for perl/python do-it-your-self servers), then these
extra bytes will force a new TCP/IP connection to be used after
the 401, with another RTT to setup the connection.

regards

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 7 May 2010 17:07:07 +0100

Greg Wilkins wrote:
> or even become a hanging GET.

It's a good plan.  Upgrade is a sort of hanging POST already... with
the advantage of both upstream and downstream data simultaneously not
being an error.

A better, more future-proof plan to reduce RTTs is "conditional
speculatively sent data", where a limited amount of initial data
stream is included in the negotiation request, either encoded into the
headers (base64 :-/) or in the following bytes (better), and the
server skips over it unless negotiation confirms that it should process it.

That lets the first WebSocket messages piggyback on the negotiation
request, saving one whole RTT, plus TCP delayed-ack delay (which can
be larger than RTT on LANs).

Using the WebSocket rule - already in the spec - that unknown frame
types must be dropped, that _ought_ to be workable already.

> OK this is not currently possible with the JS API, but that is not
> the only use-case and we should preserve the possibility of future
> usage of compliant HTTP.

Yes; it would be possible to extend the browsers' built-in JS
implementation to take advantage of this, if it's useful enough.

Even current browsers can run _non_ built-in WebSocket clients
implemented in Flash and Java (because those offer TCP sockets
directly).  Such clients can implement protocol extensions where
beneficial, and work in most browsers already in use.

> the 401, with another RTT to setup the connection.

And it won't work at all with NTLM authentication, so the inevitable
NTLM version will end up doing something different from the spec to
workaround the problem.

-- Jamie

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 07 May 2010 09:42:43 -0700

Simon Pieters wrote:
>
> I thought what followed the upgrade request from a HTTP server's point 
> of view are the eight random bytes. Those bytes could happen to result 
> in a start of a valid HTTP request, but that seems extremely unlikely.
Yes. The eight random bytes model is broken. That random-byte model was 
not discussed in this list.

-- Scott
>>> first message together with the handshake response if it wants to 
>>> (and indeed the closing handshake also in the same packet if it 
>>> wants to and it fits).
>>>
>>>
>>>> - hence the SPDY folks
>>> If that's the case, that seems like a good reason to lower the 
>>> maximum number in the keys to something that all scripting languages 
>>> support without the developer having to jump through hoops. Maybe we 
>>> should make the range 0-32767 to allow usage of 16-bit signed integers.
>>>
>>
>
>

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 7 May 2010 18:12:32 +0100

Simon Pieters wrote:
> sending its first message.

This a valid security concern, but it is solvable therefore not a good
reason to force a long delay to connection setup.

(I acknowledge basic implementations which prefer simplicity to
minimising delays can do that.)

The easy-sounding solution is to ensure the first bytes are not a
valid HTTP request.

However, we should never depend on the behaviour of invalid HTTP
framing to a HTTP receiver or proxy.

So a "proper" solution is to ensure that any messages the client sends
after the HTTP headers, prior to the server response, _are_ valid HTTP
with no adverse effect.

That can be done in many ways:

   1. Encode initial messages as text in one or more HTTP headers in
      the request.  Unaware servers will just ignore them.  Aware
      servers will recognise and handle them.  Client can tell what
      happened from the negotation response.

      Con: Not ideal due to text encoding.

      Con: It must still delay a message given to the client in the
      time after sending the headers and before negotation returns.

      Pro: It will work with existing WebSocket servers, and even ones
      which follow the advice to abort on seeing bytes after the
      request headers.  Unaware servers will ignore the messages, but
      the client can resend them if needed after negotation.

   2. Wrap initial messages in a "innocuous", compact HTTP request,
      such as OPTIONS with minimal headers, which will be harmless to
      an HTTP server/proxy which does not upgrade.

      Con: A fully general version needs support in all WebSocket
      servers from the start, to skip over the HTTP request.

      Con: Will not be acceptable as an "amateur implementor" strategy
      because that's a bit complex to skip an HTTP request and body.

      Pro: This is the method most likely to behave with all HTTP
      servers and proxies that fail the upgrade.

      Pro: This can send messages given to the client at any time
      after sending the Upgrade request, without delay.

   3. Like method 2 but the compact HTTP request is required to have a
      UTF-8 compatible text body.  If there are initial binary
      messages, or multiple text messages, they'll have to be encoded
      somehow.  It sounds magic, but that makes the whole request a
      valid WebSocket text frame with unallocated type.

      Pro: It will work with existing WebSocket servers, so long as
      they follow the WebSocket spec and drop unknown UTF-8 frame
      types, and don't abort if bytes are received after the Uprade
      request, but just process them.

   4. Wrap initial message in the body of the Upgrade request, using
      Content-Length.

      Con: I'm not sure if a body is allowed with an Upgrade request,
      if it is universally skipped when the upgrade is failed, and if
      it is universally forwarded by proxies when the upgrade is passed.

      Con: Like method 1, it must still delay a message given to the
      client in the time after sending the request and before
      negotation returns.  This is due to the fixed length.

   5. Wrap initial message in the body of the Upgrade request, using
      chunked transfer-coding.

      Con: I'm not sure if a body is allowed with an Upgrade request,
      if it is universally skipped when the upgrade is failed, and if
      it is universally forwarded by proxies when the upgrade is passed.

      Con: HTTP chunked requests are poorly tested, and probably don't
      work everywhere, including over proxies.

      Pro: This can send messages given to the client at any time
      after sending the Upgrade request, without delay.

Out of them all, only 1, 2 and 3 seem likely to be compatible with all
existing HTTP servers acting as front-ends to a WebSocket server, and
all existing HTTP proxies (including those which intercept connections).

Only 1 and 3 seem likely to be compatible with near-future WebSocket servers.

Only 2 and 3 have good timing properties (no delays at any
point).

That leaves 3 as the best option, which is why I included it despite
it sounding uncomfortably magical.

For clients and servers which want _simplicity_, it is simple.
The client does nothing, and the server just has to obey the current
WebSocket spec by dropping unknown frame types, and be sensible about
queuing bytes received before it's sent the negotation response.

For clients and servers looking to minimise round-trips, this provides
a future-compatible mechanism for eliminating one round trip.

> >Is there a reason to presume they are 32-bit, 31-bit, 64-bit, or

My point there was that the range mustn'y be _presumed_.  If it must
fit in 32-bit unsigned range or anything else, that should be in the spec.

> the range 0-32767 to allow usage of 16-bit signed integers.

That sounds small enough to be susceptible to simple brute force
connection attempts.  But I haven't studied the handshake proposal
enough to know what it's really protecting against.

Signed 32-bit non-negative, that is 0..2^31-1 is probably universal
enough these days.  But why are we reinventing authentication
mechanisms when there are so many already available?  This is so ad-hoc.

-- Jamie

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 07 May 2010 10:45:55 -0700

>       negotation returns.  This is due to the fixed length.
The first Con isn't  a big issue, because the body is governed by the 
method: "GET", "POST", etc. and not the Upgrade head. This option is 
possible as long as the method isn't a GET. POST would be fine.

The second Con shouldn't be a big problem because the body-encoded 
pipelining would only apply to data the client already has available. In 
other words, it should be fine to require a fixed buffer size for this 
data because it's a client optimization. If the client wants to pipeline 
a bigger chunk, it can allocate a bigger buffer.

This option is the only clean one of the 5, really. (Well, #5 is clean, 
but HTTP chunked is overkill in this situation.)

-- Scott
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>
>
>
>

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 07 May 2010 20:43:11 -0700

> possible as long as the method isn't a GET. POST would be fine.
Nevermind on the POST because of the security issues, although HTTP does 
allow methods not enumerated in the spec like the WebDAV method 
PROPFIND. It would technically be possible to use "WEBSOCKET" as the 
method, although I'm not sure if intermediaries would reject that method.

-- Scott
> pipelining would only apply to data the client already has available. 
> In other words, it should be fine to require a fixed buffer size for 
> this data because it's a client optimization. If the client wants to 
> pipeline a bigger chunk, it can allocate a bigger buffer.
>
> This option is the only clean one of the 5, really. (Well, #5 is 
> clean, but HTTP chunked is overkill in this situation.)
>
> -- Scott
>>> maximum  number in the keys to something that all scripting 
>>> languages support  without the developer having to jump through 
>>> hoops. Maybe we should make  the range 0-32767 to allow usage of 
>>> 16-bit signed integers.
>>>     
>>

---

From: Brad Spencer <[email-masked]>
Date: Sat, 8 May 2010 11:46:28 -0300

I am new to this list, but I thought I would offer some observations
on recent discussions.

First, let me check that I understand the intended interactions with
legacy HTTP proxies.  I reviewed some of the relevant parts of RFC
2616, and it would seem that

  a) the Connection header is per link

       The Connection general-header field allows the sender to
       specify options that are desired for that particular connection
       and MUST NOT be communicated by proxies over further
       connections.  (Section 14.10.)
  
  b) the Upgrade header will be also removed by the proxy because
     of the Connection header's contents.

I think that this means that there is no chance in getting through any
non-WebSocket-aware HTTP proxies (without CONNECT, as suggested by the
WebSocket draft).  So, presumably this means that the intended
behaviour when a client sends a WebSocket handshake directly to a
non-WebSocket aware proxy is to fail.

> valid HTTP request,  there's no actual attack.

With regards to the eight random bytes, I agree that there is a remote
chance that it could look like the start, middle, or end of an HTTP
Request-Line.  However, since there are only eight bytes, it can't
look like a complete Request-Line.

For example, even "G / HTTP" is already eight bytes, and it doesn't
match the Request-Line production.  Even allowing for leading
whitespace (which seems to be allowed in practice even though I can't
find it in RFC 2616), you still can't produce a valid Request-Line
(you need more non-whitespace, not less).

Now in order to make the connection "fail fast", because I imagine
most HTTP servers are willing to wait for the expected CR LF for quite
some time, then it may be wise to require an additional CR LF after
the eight bytes.  This will "force" those servers to look at the
"Request-Line" and fail it.

Continuing with our "G" method example above, imagine that we allow
early frames from the client, and the client uses frame type 0x2f
('/'), and sends the payload "1.1\r\nHost:...".  Now suddenly we do
have something that looks like a valid HTTP request.  I think what
Scott Ferguson suggested deals with this.

> initial frame with the high bit set, probably some kind of control frame.

Or perhaps a control pseudo-frame.  If we use "nine bytes", and make
the first byte 0xff (which has the benefit of never being valid UTF-8
and thus never valid HTTP, AFAIK), then we force a broken Request-Line
even in the most bizarre circumstances.  Combining this with a
trailing CR LF ("eleven bytes") to force a fast fail, and maybe that
is enough.

A non-WebSocket aware server (or proxy) will, AFAIK, interpret this
line as the first line of a pipelined HTTP request, and fail it,
probably closing the connection (perhaps after responding to the
handshake GET request, but with something other than a valid handshake
reply, of course).

> framing to a HTTP receiver or proxy.

Aren't the only reasonable behaviours to close the connection or send
some response back?  In either case, the handshake fails.  In the
latter case it fails because of the response to the client's handshake
(the initial "GET request") not being a valid response handshake, so
the client has already given up.

(BTW, I think that the two requirements of a) not depending on server
behaviour in the face of an invalid Request-Line, and b) allowing
"early", normal WebSocket frames from the client are incompatible for
the reasons discussed above.)

Furthermore, if there is a desire to protect non-WebSocket aware
servers from malicious browser scripts pointing the user agent at
random resources on the Internet, I think WebSocket is relying on the
behaviour of _all_ TCP servers (not just HTTP servers) when faced with
invalid initial messaging; it is implicitly expecting them to close
the connection without taking any action.

So, even though I believe it is safe to rely on at least one of the
non-WebSocket servers or the client/browser to abort the connection
for us when a handshake fails, I think I have come to agree with the
WebSocket "draft 76" Section 1.6.  I don't think untrusted browser
script should be allowed (by the browser) to send a arbitrarily large
amount of data to a server that has not been shown an intent to
communicate (by sending a valid handshake).  Those with specialized
needs to send an initial frame to reduce RTT can probably just use the
/query/ string in the /url/, the old fashioned way.

Thanks, again.  I hope this perspective helps.

---

From: "Simon Pieters" <[email-masked]>
Date: Mon, 10 May 2010 10:51:42 +0200

On Sat, 08 May 2010 16:46:28 +0200, Brad Spencer <[email-masked]>  
wrote:

> communicate (by sending a valid handshake).

FWIW, untrusted browser script can send arbitrary large amount of data to  
a server anyway. For instance, by including an arbitrary large string as  
the subprotocol in a WebSocket. Or by submitting a form, or using  
XMLHttpRequest, etc.

> Those with specialized

Yeah, I agree with this. I've done this a lot in my test cases. I think  
this is more reliable than any solution we come up with to send the  
client's first frame along with the handshake. The subprotocol can also be  
abused to send data with the handshake.

Cheers,
-- 
Simon Pieters
Opera Software

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 10 May 2010 13:39:39 +0100

> abused to send data with the handshake.

Sorry, no, neither /query/ nor subprotocol are suitable.

They don't allow the RTT reduction to be implemented as a transport
optimisation "below the API", because the URL and subprotocol fields
are owned by the application above the WebSocket API, and the
WebSocket implementation can't safely transform tem until the initial
negotiatation is complete.  Which is too late for any benefit.

Maybe it's clearer with an example:

Even if standardised, Opera/Firefox/Chrome could not implement a
transparent "WebSocket setup RTT reduction" feature using the URL
/query/ or subprotocol mechanisms.

It would need the JavaScript application to explicitly enable the
extension, or implement it itself.

Both options make the API unnecessarily complicated.

In contrast, the mechanisms I talked about could be implemented in the
browsers transparently, with no change to the JS API and no work by
the application authors at either end.

Another option that would work is a "transport subprotocols" header,
which the below-API generic WebSocket implementations are allowed to
modify for negotiating at that level.  We've got other reasons to want
that anyway.  Initial data could be fudged into a large entry in that,
although I would worry about intermediaries and poor server
implementations with fixed-size buffers breaking on very large
headers, which although their fault, may render such a mechanism
unusable speculatively.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 10 May 2010 13:59:20 +0100

> possible as long as the method isn't a GET. POST would be fine.

Depends how such requests are handled by existing deployed
implementations.  Do we have any data on Upgrade header behaviour at
the moment?  Do we even know if Upgrade works at all through
widespread proxies?

> a bigger chunk, it can allocate a bigger buffer.

No, the second Con is about data that the client doesn't know about
when it sends the headers, but gets from the application in the
following seconds.

For example, think of small HTTP-like requests
being generated from the client app (e.g. due to user's typing).

The first one triggers the WebSocket startup.  The next ones may be
ready to send after that, and before negotiation is done.  If the next
ones can't be sent, we've only removed the extra RTT from the first
one; the others are still subject to the extra (unnecessary) RTT or
some fraction up to 100% of it, so the job is only half done.

> but HTTP chunked is overkill in this situation.)

I don't consider #5 suitable because I'm fully expecting half the world
to break when sent a chunked request.  Theoretically clean isn't enough.

Besides, #5 is too complicated for "amateur implementors" of WebSocket
servers.

Both #4 and #5 will break with existing WebSocket servers, but I'm not
sure if that's a legitimate concern given the changes still being
done.  It'll become more of a concern with time.

#4 depends on real world behaviour that I don't know if anyone's
tested.  Do you have any data on proxy and server behaviour with
Upgrade headers?

Thanks,
-- Jamie

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 11 May 2010 11:29:22 -0700

> the moment?  Do we even know if Upgrade works at all through
But any issues with the Upgrade header applies to all the proposals. I 
don't see how the Upgrade issues are specific to sending pipelined data.
> being generated from the client app (e.g. due to user's typing).
Gotcha. But that applies to any option proposed so far except the 
chunking one (and #3, the OPTIONS header).

(#3, the "options" header, is more complicated than chunked encoding. 
Since a websocket server can't reuse the normal HTTP parsing for the 
embedded options, it would have to duplicate the full HTTP parsing. In 
contrast, HTTP chunked is just a trivial length encoding.)

#1 and #2 have the additional problem, by the way, of encouraging DOS 
attacks, because headers take up memory, while the body of a request can 
be buffered.
> to break when sent a chunked request.  Theoretically clean isn't enough.
Well, since only servers that implement the WebSocket protocol matter in 
this case, I'd think it would be reasonable to require that WebSocket 
servers actually implement HTTP/1.1 correctly.

If you send a WebSocket request with pipelined data in a chunked body to 
a non-WebSocket HTTP/1.1 server that doesn't implement chunked encoding, 
well, at that point I don't think it's a big issue.
> Besides, #5 is too complicated for "amateur implementors" of WebSocket
Whatever :)
> done.  It'll become more of a concern with time.
Since we're at the prototype stage, that doesn't matter, although it 
should be resolved sooner rather than later.
> tested.  Do you have any data on proxy and server behaviour with
I'm still puzzled by your question. The HTTP content-length payload has 
nothing to do with the Upgrade header. Upgrade is not part of the server 
logic in deciding how to handle uploaded data. If there is an issue with 
content-length, the issue is that some server/proxies may not allow 
payload data for a GET.

If GET is a problem, we can use WEBSOCKET as the method, which would 
allow for a content-length payload. (And GET isn't quite right anyway 
because it's not supposed to have side-effects, and WEBSOCKET would have 
a benefit of fitting nicely into existing authorization frameworks.) 
Servers can certainly handle custom methods, because they support custom 
methods for WebDAV. I don't know about proxies, although they should 
pass the method through (depending on security configuration, of course.)

So here's my current preference list:

  1. chunked encoding (with a custom method if possible to avoid GET 
problems), i.e. #5.
  2. content-length (again with a custom method if possible), i.e. #4
  3. start sending WebSockets frames immediately without waiting
  4. OPTIONS (although I don't like the extra complexity, or a custom 
WSDATA method), i.e. #3.

The other two are non-viable because of the memory/DOS issues.

I'd really like to avoid the extra RTT, because if we do force the extra 
RTT, WebSockets may be a worse option than plain HTTP for many applications.

-- Scott

---

From: Brad Spencer <[email-masked]>
Date: Tue, 11 May 2010 18:47:20 -0300

On Mon, May 10, 2010 at 01:39:39PM +0100, Jamie Lokier wrote:
> optimisation "below the API"

I see what you are saying, and in many respects, I agree.  

Here's a crazy thought.  What if accepting early messages is
_optional_ behaviour?  After all, you can't point random clients and
random WebSocket servers and expect their application-level protocols
to just work.  The server and client must agree on what they are going
to exchange anyway.

So, if some server never wants to accept messages from a client before
the handshake is complete, it can reject clients that violate that
agreement.  But, if some other server is fine with it, all is well.

> Both options make the API unnecessarily complicated.

Perhaps my crazy thought makes it just as complicated :(
 
> the application authors at either end.

I can't remember which mechanisms you were talking about; I'm not too
keen on having to slurp some initial message out of the headers.  If
an initial message is allowed, I'd rather just read it as usual from
the wire (i.e. not fail the handshake if there is more than the "eight
bytes" already available).

> Initial data could be fudged into a large entry in that,

FWIW, in line with the "the server and client must agree on the
protocol", I plan to reject any handshakes over a certain size
regardless of what the RFC says, because I know that no legitimate
clients of mine will ever use ridiculously large handshakes.  From a
server's point of view, I consider this prudent behaviour.