Subject: Re: [hybi] Questions on WebSocket RFC draft 76: "Early" data, whitespace, and integer sizes

From: Ian Hickson <[email-masked]>
Date: Wed, 21 Jul 2010 00:35:19 +0000 (UTC)

On Tue, 4 May 2010, Brad Spencer wrote:
>
> Hi.  I've been implementing the server side of the WebSocket RFC draft, 
> and I'm happy to report that not only is the specification very easy to 
> read and understand, but I also haven't really come across any obvious 
> errors or inconsistencies.  The protocol seems very logical, easy to 
> deal with, and sensible.  So, nice work :)

Thanks! :-)

> [Oh, and BTW, being very strict about exactly what the handshake may and 
> may not contain has made this much easier to implement than HTTP. I 
> can't say how happy I was to see rules like "there must be exactly one 
> space here", and so on ;)]

I'm glad to see at least one person appreciates it. :-)

> However, I do have some questions.  (My apologies if they have been 
> answered already.  I read some of the recent postings to this list, but 
> didn't find an easy way to search it.)
> 
> 1. Can a client send any data to the server after the eight byte
>    "key3" data, but before the server has responded to the client's
>    initial handshake request?  In other words, is the client allowed
>    to "pipeline" the first message?
> 
>   I am hoping the answer is "no" because:
> 
>     a) I don't think it makes sense to send messages before the
>        handshake is complete.  This is probably especially true when
>        the subprotocol needs to be confirmed, etc.
>        
>     b) I am using at is part of the validation of the handshake,
>        sort of.  If I have _more_ than what should be in a valid
>        handshake, I'd prefer to reject the connection out of hand.
> 
>     c) Preventing such data might make things simpler, although it
>        doesn't appear to be a practical problem for me to actually
>        interpret the any such "early" data as a frame data.

The answer is indeed "no": the first thing the client is required to do 
after sending "key3" is wait for the response from the server.

> 2. In HTTP, IIRC, header values can have arbitrary amounts of
>    whitespace between the "field-value" productions (i.e. leading
>    and trailing whitespace in the header value itself ends up
>    getting ignored).  In WebSocket, this doesn't appear to be the
>    case (good), but it does mean that if a client sends, say,
> 
>      Connection:_Upgrade__
> 
>    or
> 
>      Host:__www.example.com___
> 
>    where _ represents a space, then the server is going to end up
>    failing the handshake, as I understand it.  Is this intended?
>
>    (To be clear, I am quite happy to have the server fail such
>    handshakes, as long as they are truly invalid. :)

Yes, it is intended (and required as written). It is intended to reduce 
the number of "synonyms" for a particular semantic, thus helping increase 
the likelihood that UAs and servers will all act identically, reducing the 
chance of a security bug.

> 3. When computing the server's /response/ to the /challenge/ in
>    Section 5.2, the RFC draft is explicit about the fact that /part1/
>    and /part2/ are 32-bit numbers, but it isn't clear how large
>    /key-number_1/ and /key-number_2/ are allowed to be.  I've presumed
>    they are also 32-bit numbers based on what's going on with regards
>    to the multiplication specified in Section 4.2 Item 19.
> 
>    Is it worth saying so explicitly in Section 5.2 Item 5 that the
>    /key-number_*/ numbers are also 32-bit?
> 
>    Also, I presume that all of these numbers are 32-bit _unsigned_
>    numbers, but perhaps that goes without saying.

I've tried to clarify this, thanks.

On Thu, 6 May 2010, Jamie Lokier wrote:
> > On Wed, 05 May 2010 02:24:29 +0200, Brad Spencer <[email-masked]>  
> > wrote:
> > >  I am hoping the answer is "no" because:
> > >
> > >    a) I don't think it makes sense to send messages before the
> > >       handshake is complete.  This is probably especially true when
> > >       the subprotocol needs to be confirmed, etc.
> 
> If you don't care about extra time delays, it's easier with "no".
> 
> With high latency links, such as mobile data, unnecessary extra round 
> trip times add up quickly to be annoying - hence the SPDY folks 
> interested in minimising connection setup latency in every way possible.
> 
> Comet-style messaging over HTTP may be faster to start up than WebSocket 
> in those scenarios.  It would be perverse if good WebSocket JavaScript 
> frameworks had to open a comet-style connection _and_ a WebSocket 
> connection to get good all round performance :-/

We don't really have a choice; we'd have to encrypt everything (to make it 
appear like random noise) to be safe from cross-protocol attacks if we 
didn't wait for the handshake. (If the content is random noise, we have 
plausible deniability -- it's a pretty poor protocol indeed that is 
vulnerable to attack from just random noise.)

> > [[
> > Servers may close the WebSocket connection whenever desired.
> > ]]
> 
> Does that mean in normal operation, or only in error / timeout scenarios?

It is unrestricted in scope.

> If that's normal server behaviour, are we repeating the design flaw in 
> HTTP pipelining that stops it being safe to use for non-idempotent 
> messages?

Web Socket cannot do anything on its own, so no. Your application layer 
protocol (which does do something) is welcome to add its own restrictions, 
e.g. requiring a closing handshake.

> "32-bit unsigned" seems to be contrary to the debated requirement of 
> "easy to write by amateurs", as many "scripting" languages don't have 
> such a type conveniently available.

That's news to me; which languages did you have in mind that can't 
represent all integers up to 4294967295? If this is indeed the case, then 
we should fix this.

On Thu, 6 May 2010, Scott Ferguson wrote:
> Simon Pieters wrote:
> > 
> > Yes, but it's not unnecessary to wait with sending the first frame 
> > from the client, since if it were allowed to send a frame along with 
> > the opening handshake, then you would be able to send arbitrary HTTP 
> > requests by carefully crafting your first message so that an HTTP 
> > server sees first an HTTP upgrade request that it rejects, then a 
> > bogus HTTP request that it rejects (starting with the 8 random bytes), 
> > then the attacker's carefully crafted HTTP request which could do 
> > anything without the user's consent.
>
> That scenario doesn't really make sense.

Nonetheless, it is the scenario for which the spec is designed.

> If a server sees random junk, it doesn't keep processing until it sees 
> something that looks like a HTTP header. Instead, it needs to close the 
> connection, possibly after returning a 4xx response. (Broken servers are 
> irrelevant here. If the server is broken, it's a security hole no matter 
> what the spec says. Servers are not like browsers, which try to keep 
> processing after bogus data.)

Broken servers are most definitely relevant if they are deployed.

On Fri, 7 May 2010, Jamie Lokier wrote:
> 
> So a "proper" solution is to ensure that any messages the client sends 
> after the HTTP headers, prior to the server response, _are_ valid HTTP 
> with no adverse effect.

How does the client know if the server has responded yet or not?

(Also, HTTP is not the only possible victim protocol here.)

If getting the data from the Web Socket server ASAP is really that 
critical, it seems better to simply have the HTTP server include the first 
batch of data in the initial HTTP response that contains the JS that then 
opens the Web Socket connection. No need to play weird games of trying to 
make frames look like HTTP or some such.

> > If that's the case, that seems like a good reason to lower the maximum 
> > number in the keys to something that all scripting languages support 
> > without the developer having to jump through hoops. Maybe we should 
> > make the range 0-32767 to allow usage of 16-bit signed integers.
> 
> That sounds small enough to be susceptible to simple brute force 
> connection attempts.

The values each time are random, so I don't really see how that would 
help. It's not like you can try each value in turn.

On Sat, 8 May 2010, Brad Spencer wrote:
>
> So, presumably this means that the intended behaviour when a client 
> sends a WebSocket handshake directly to a non-WebSocket aware proxy is 
> to fail.

Right.

> Now in order to make the connection "fail fast", because I imagine most 
> HTTP servers are willing to wait for the expected CR LF for quite some 
> time, then it may be wise to require an additional CR LF after the eight 
> bytes.  This will "force" those servers to look at the "Request-Line" 
> and fail it.

Even that wouldn't really guarantee a fast failure -- I would imagine that 
it'd be pretty common for the proxy to not return anything until the 
server has responded, and the server would likely wait for hte 8 bytes 
(which aren't going to come) for a long time.

The goal isn't so much to fail fast, so much as to not appear to succeed. 
The earlier handshake (what people refer to as "75") would appear to 
succeeded, only to then fail when the JS tried to send a frame.

> Continuing with our "G" method example above, imagine that we allow 
> early frames from the client, and the client uses frame type 0x2f ('/'), 
> and sends the payload "1.1\r\nHost:...".  Now suddenly we do have 
> something that looks like a valid HTTP request.  I think what Scott 
> Ferguson suggested deals with this.

The odds of this happening are pretty miniscule (somewhere in the range of 
1 in 1e16), and that's assuming we ever define the 0x2f frame type. I 
don't think it's a viable enough attack scenario for anyone to ever use 
it. Even if the intermediary is truly willing to try hard to interpret 
anything as a request, so that you only need to get the first four 
characters to be a case-insensitive match for "GET " and the 8th bit is 
ignored, say, it's still way less than a one in a million chance with each 
request. The user is going to notice if you try to open half a million 
WebSocket connections one after the other (you can't do them all at once, 
since that is also stopped).

But this is all moot, since you can't send any frames until the connection 
is established anyway.

> So, even though I believe it is safe to rely on at least one of the 
> non-WebSocket servers or the client/browser to abort the connection for 
> us when a handshake fails, I think I have come to agree with the 
> WebSocket "draft 76" Section 1.6.  I don't think untrusted browser 
> script should be allowed (by the browser) to send a arbitrarily large 
> amount of data to a server that has not been shown an intent to 
> communicate (by sending a valid handshake).  Those with specialized 
> needs to send an initial frame to reduce RTT can probably just use the 
> /query/ string in the /url/, the old fashioned way.

Agreed.

On Mon, 10 May 2010, Simon Pieters wrote:
> 
> Yeah, I agree with this. I've done this a lot in my test cases. I think 
> this is more reliable than any solution we come up with to send the 
> client's first frame along with the handshake. The subprotocol can also 
> be abused to send data with the handshake.

It's also possible to just open the connection 200ms earlier than you 
would normally, e.g. by just preemptively opening a connection when the 
page is opened. I don't really see what concrete cases exist where the 
handshake will actually be a latency issue.

The server can send the data before the connection is made, along with the 
script that makes the connection. The client isn't going to have data to 
send before the handshake is established, if it establishes it straight 
away. What other cases are there?

On Mon, 10 May 2010, Jamie Lokier wrote:
> 
> They don't allow the RTT reduction to be implemented as a transport 
> optimisation "below the API", because the URL and subprotocol fields are 
> owned by the application above the WebSocket API, and the WebSocket 
> implementation can't safely transform tem until the initial 
> negotiatation is complete.  Which is too late for any benefit.

I don't understand what that means. Both the URL and the subprotocol are 
under the control of the same script that started the connection and that 
is going to send the frames.

> Maybe it's clearer with an example:
> 
> Even if standardised, Opera/Firefox/Chrome could not implement a 
> transparent "WebSocket setup RTT reduction" feature using the URL 
> /query/ or subprotocol mechanisms.
> 
> It would need the JavaScript application to explicitly enable the 
> extension, or implement it itself.

What's the problem with that?

> Both options make the API unnecessarily complicated.
> 
> In contrast, the mechanisms I talked about could be implemented in the 
> browsers transparently, with no change to the JS API and no work by the 
> application authors at either end.

Anything that requires a change on the server obviously requires some work 
on the server side, by definition.

If what you want is for a future feature wherein browsers can opt in to 
sending some frames before the handshake, though, we can easily extend the 
protocol later if that's really necessary. We'd just include the payload 
in a field of the handshake, and then require the server to confirm 
receipt in its handshake; if the server doesn't confirm, send the frames 
once connected. But it's not at all clear that we need this; I strongly 
suggest we wait until we know we need this before doing it.

On Tue, 11 May 2010, Brad Spencer wrote:
> 
> FWIW, in line with the "the server and client must agree on the 
> protocol", I plan to reject any handshakes over a certain size 
> regardless of what the RFC says, because I know that no legitimate 
> clients of mine will ever use ridiculously large handshakes.  From a 
> server's point of view, I consider this prudent behaviour.

That's fine.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'