Subject: [hybi] Redesigning the Web Socket handshake

From: Ian Hickson <[email-masked]>
Date: Mon, 1 Feb 2010 23:47:46 +0000 (UTC)

There have been a number of suggestions recently for changing how the 
handshake works in Web Sockets. I'd rather avoid doing this unless we can 
get a significant improvement out of it, so I'd like to solicit feedback 
on how much of an improvement such changes would result in.

The main suggestions are:

* Using a challenge instead of a fixed sequence of bytes for the 
  handshake. Basically the client would send a nonce, and the server would 
  have to send back some value generated from that nonce.
  - this means the server has to read the handshake, which avoids the 
    cross-site message forgery attack.
  - we could make it even more secure by saying that you have to xor 
    something derived from this nonce with the frame types (or maybe every 
    byte) of all the data sent and received, so that even if the server 
    still muddles through without a nonce, you can't possibly inject 
    anything meaningful into the stream.
  - makes it less likely that an attack can be found where you can cause 
    the server to echo text back to do a cross-protocol attack.
  - means that people who are using infrastructures with limitations on 
    header order or whatnot can still use it.
  - it has been suggested that we'd still want the first few bytes of 
    the response to be fixed (and unique) even with the nonce; this would 
    lose most of the legacy intermediary benefits, though.
  - it would dramatically increase the complexity of the simplest possible 
    websocket server. Right now you can have a server that doesn't even 
    read the client handshake at all; this would force you to parse it.
    You could no longer write simple test servers that just send back a 
    bunch of frames, for instance.

* Not using HTTP at all is something I'd like to do, but since people are 
  going to reuse port 443 with this regardless of what we say, it seems 
  better to make it usable with HTTP-based parsers.

* Using ports 81/815 instead of 80/443 would be ideal, but IANA said that 
  if we look like HTTP, we must use ports 80/443.

* Using CONNECT instead of GET with an Update has been suggested by some 
  people. From the point of view of Web Sockets it doesn't matter either 
  way, it's just an opaque handshake. From the point of view of HTTP, it 
  doesn't seem CONNECT is very well defined, so I'm not sure what the 
  implications of this are. As far as I can tell, we wouldn't do an 
  Upgrade, we'd just be using the HTTP server like a proxy. It seems a bit 
  dubious from an HTTP semantics point of view. As far as I can tell, it 
  would mean that we don't have to worry about the server sending back 
  something that looks like HTTP, though, which would get around the
  issue of limitations in legacy intermediaries.

I'm especially interested in feedback regarding actual effects these 
changes would have on concrete deployed softare (like, how much easier 
would this make deploying Web Socket on a specific load balancer) and 
protocols (like, what actual attacks does this prevent or allow).

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 02 Feb 2010 11:15:08 +1100

> The main suggestions are:

I think you've left out the main suggestion that's been made on this
list.

Leave the handshake more or less as it is, but specify it as a real
HTTP request and real HTTP response, rather than a binary
almost-equivalent.     I've got no objections to websocket
correctly using the existing HTTP upgrade mechanism on 80/443.

The immediate benefits of this change are:

 + that it allows the handshake to work with existing HTTP servers and
   intermediaries.

 + No need to explicitly specify behaviour such as HttpOnly cookies.
   that behaviour is inherited from the HTTP specs.

 + If HTTP/1.2 or HTTP/2.0 ever come out, then websocket will
   continue to work with new servers as the binary response
   line will not need to be changed.

It would allow some simple extensions.  For example redirection
could be supported with a 302 response.   For example is the server
responded with

   HTTP/1.1 302 Found
   Location: ws://foo.com:81

then the ws client could open port 81 and immediately start
talking websocket without any handshake at all (because of the
ws://).  If the server responded

   HTTP/1.1 302 Found
   Location: http://foo.com:81

Then it should open 81 and do another HTTP upgrade request.

This would allow both configuration of a shared port 80
or separate ports for HTTP and WS.

regards

---

From: Ian Hickson <[email-masked]>
Date: Tue, 2 Feb 2010 03:15:11 +0000 (UTC)

On Tue, 2 Feb 2010, Greg Wilkins wrote:
> Leave the handshake more or less as it is, but specify it as a real HTTP 
> request and real HTTP response, rather than a binary almost-equivalent.  

That's basically a non-starter, IMHO. The HTTP rules are orders of 
magnitude more complicated than necessary here. Given the goal of making 
this trivially implementable, we can't require that servers implement 
every last detail of the HTTP rules.

>    line will not need to be changed.

We can handle all three of these points without going anywhere _near_ the 
full HTTP rules with all their complications.

> It would allow some simple extensions.

Extensions should be done by changes to the spec, no need for the whole 
HTTP ruleset. Even with the current strict rules it's easy to extend the 
spec in the future.

> For example redirection could be supported with a 302 response.  For 
> example is the server responded with
> 
>    HTTP/1.1 302 Found

The spec actually has redirect support already, but it is commented out 
because it introduces a number of security problems. In practice it's 
unnecessary; just do the redirect at the subprotocol level if you need it.

> then the ws client could open port 81 and immediately start talking 
> websocket without any handshake at all (because of the ws://).

That would be a pretty big security hole!

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 01 Feb 2010 19:29:34 -0800

> correctly using the existing HTTP upgrade mechanism on 80/443.

This proposal doesn't address address the security vulnerabilities that Ian mentioned, and it seems make the protocol more vulnerable. Note that having newlines in the hardcoded part of the handshake is no longer particularly beneficial if you adopt the nonce scheme, thus the status line would be the only part of the headers subject to special rules, and no one here has argued that it's remotely hard to make a server send the status line of their choice. Thus, I don't see the practical benefit of your proposal, in exchange for the increase in security risk.

 - Maciej

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 02 Feb 2010 14:38:20 +1100

Ian,

you are not listening to the server side developers.

Apache is telling you that they can't share port 80
with websocket and you're giving them nothing back!

For many of us, we are already "near the HTTP spec" and
it's full rules and we've already dealt with the complications.
It is your attempt to simplify HTTP that is causing the
complications.

We've already parsed the HTTP request by the time that
we know it is a websocket upgrade. We're deep inside
our containers... potentially not even on the same
machine that terminated the HTTP request.

Nobody is saying that a compliant websocket server has
to also be a fully compliant HTTP server.

We are simply saying don't impose additional constraints
on the format of the HTTP request/response.   Allow
them to be order and case insensitive and you're more
or less done.

There is no need to specify HTTP-lite within the websocket
specification.    You just refer to the HTTP upgrade mechansim
and describe the extra requirements for websocket.  You
can constrain header values all you like - but don't
constrain what the definition of a header is!

regards

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 01 Feb 2010 19:50:35 -0800

> you are not listening to the server side developers.

It looks to me like Ian just made a bunch of changes to the draft in response to comments from server side developers. On top of the fact that the WebSocket protocol was essentially designed by server-side developers. Could we please stick to substantive points instead of debating who is listening to whom?

> Apache is telling you that they can't share port 80

Did anyone really say that? Isn't mod_pywebsocket a proof that it is in fact possible: <http://code.google.com/p/pywebsocket/>?

> machine that terminated the HTTP request.

As far as I can tell, the WebSocket spec does not require special hardcoded handling of the request handshake. It's totally allowed to run it through a normal HTTP stack. In fact, it's totally allowed for servers to do no checking of the client request handshake at all, but that creates a security hole (possibility of violating integrity of a WebSocket service with XDomainRequest, cross-site XHR or cross-site form submission), one of the security holes we are trying to fix.

> them to be order and case insensitive and you're more

FWIW I think my nonce proposal would allow doing this for the response at no cost in security - getting your victim to echo back a hash of an unpredictable nonce is so unlikely that there's no need to count on newlines in the handshake.

I wish you could consider the nonce handshake proposal and report whether it addresses your concerns. Instead, you are just repeating your own suggestion. I listened to you and gave you feedback. I even made a proposal that tries to address the requirements you have identified. Can you please do me the same courtesy?

Would it be helpful if I spelled out the nonce proposal in more detail and explained how it fixes security vulnerabilities in the current handshake, while also making it easier to integrate with existing servers? I admit both my explanation and Ian's later summary were somewhat roughly sketched out.

Regards,
Maciej

---

From: Justin Erenkrantz <[email-masked]>
Date: Tue, 2 Feb 2010 09:24:24 -0800

> Did anyone really say that? Isn't mod_pywebsocket a proof that it is in fact possible: <http://code.google.com/p/pywebsocket/>?

I did.

If you want it in the core and done in a clean manner (which is my
intent), then I feel that you have to amend the draft so sharing of
the port is feasible until you know the protocol is absolutely WS.

Also, that implementation in mod_python doesn't meet the strictness
requirements set out in the draft.  So, if that is the exemplar, then
it certainly just reiterates the points that Greg and I and others
have been making.  Their "memorizing" line hack (which enables the
strictness requirements as *mandated* by the current drafts in the
name of security) only works on their standalone server - not httpd.
There is no way for httpd to provide that functionality - well,
without violating about a billion design rules; yes, I could code
something up, but I'm sure you could do nasty things to WebKit if you
wanted that would never pass muster and get you laughed out of the
room when you show that patch to the rest of the team...  -- justin

---

From: Francis Brosnan Blázquez <[email-masked]>
Date: Tue, 02 Feb 2010 22:37:07 +0100

Hi Greg,

> you are not listening to the server side developers.

Please don't fall into using "server side developers" where you really
want to say server side HTTP developers...

> It is your attempt to simplify HTTP that is causing the

Not all server side developers interested in Websocket are working with
HTTP servers/products...
-- 
Francis Brosnan Blázquez <[email-masked]>
ASPL

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 03 Feb 2010 09:21:24 +1100

> want to say server side HTTP developers...

Sure.

And I shouldn't have said Ian is not listening.  my apologies.

He is listening. He's just not agreeing that it's a problem for
some HTTP developers to add extra constraints to the upgrade
response.

regards

---

From: Justin Erenkrantz <[email-masked]>
Date: Tue, 2 Feb 2010 14:31:21 -0800

> Would it be helpful if I spelled out the nonce proposal in more detail and explained how it fixes security vulnerabilities in the current handshake, while also making it easier to integrate with ex…

Yes, I admit it'd be a bit helpful if the nonce proposal was little
more concrete.

I'm still not exactly clear what the security risks are (I know you
tried to clarify a bit, but it didn't really hit what you were talking
about) - to me, expecting a hard-code byte sequence just to do the
initialization doesn't seem to add any real security benefits.  I'm
not quite sure what a nonce would add, but if I saw it at a slightly
more concrete level, perhaps that'd trip the light bulb.

I'm for pretty much anything which lets any conformant HTTP/1.1 server
process the upgrade request (or a HTTP/1.1 client issue it!) and then
hand-off to a new protocol upon completion of the HTTP/1.1 response.
-- justin

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 02:44:41 +0000

> Did anyone really say that? Isn't mod_pywebsocket a proof that it is in fact possible: <http://code.google.com/p/pywebsocket/>?

Curious: Is it compliant?  I.e. does it reject requests which aren't
matching the exact octets needed, and create a response which does
match the exact octets needed?

Or does it just work without being WebSocket spec compliant?

> As far as I can tell, the WebSocket spec does not require special
> hardcoded handling of the request handshake. It's totally allowed to
> run it through a normal HTTP stack. In fact, it's totally allowed
> for servers to do no checking of the client request handshake at
> all, but that creates a security hole (possibility of violating
> integrity of a WebSocket service with XDomainRequest, cross-site XHR
> or cross-site form submission), one of the security holes we are
> trying to fix.

Is it permitted to not check the client request and be WebSocket spec compliant?
(Not just work, but be able to claim compliance.)

> FWIW I think my nonce proposal would allow doing this for the
> response at no cost in security - getting your victim to echo back a
> hash of an unpredictable nonce is so unlikely that there's no need to
> count on newlines in the handshake.

hash(nonce) + regular HTTP would go through proxies.

Current proposal intentionally fails at compliant HTTP proxies, on port 80,
doesn't it?

That would be a significant change in behaviour.

Other than that, I do like the idea, as an "are you a WebSocket server" check.

(I'm wondering why the CORS check used for XmlHttpRequest is
unsuitable, though.)

Maybe we can take the nonce idea further: An "are you a WebSocket
server that supports subprotocols X,Y,Z that I must have" check, where
a WebSocket server that doesn't support X,Y,Z would fail it.  That'd
be infinitely more reliable than HTTP's Expect mechanism, which relies
on proactive implementation by server developers who sometimes don't,
making it unusable.

If you like that idea, I'll describe how it would work.

(Starts getting ideas about nonce-verified proxy opt-in too... Hmm.)

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 02:57:36 +0000

> protocols (like, what actual attacks does this prevent or allow).

I think the question of CONNECT vs. Upgrade must come down to:
What works over real proxies?

CONNECT works over port 443, we know that.  We also know it forwards
the data in a timely fashion, without depending on the byte values.
It is widely used.

Upgrade probably hasn't been tested much.  It ought to use much the
same implementation in a proxy, after the handshake, but you know... I
wouldn't count on Upgrade working, without testing it against a wide
range of real proxies.  Is there any data about this?
Are you listening, Google? :-)

An interesting idea I've looked at, which would be tricky but _really_
improve widespread support (and compatibility), would be to send
something which looks like a valid HTTP long-polling request, and
allow the server and any proxies to proactively signal that they'll
convert it to the more efficient WebSocket - when they will.  When
they won't, communication continues over HTTP long-polling, carrying
the same messages and subprotocols over that.  I guarantee if that
could be pulled off, everyone would start using WebSocket asap ;-)

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 03:12:47 +0000

> every last detail of the HTTP rules.

That's understandable.  As a wannabee non-HTTP WebSocket server writer,
I sympathise.

The header syntax rules for HTTP aren't all that complicated.  Heck,
I've seen it done in one or two lines of Perl (wrongly of course! ;-)

Maybe it's worth extracting out the essential header syntax into
straightforward form.

HTTPbis is a bit more precise on the subject, and might be simpler than RFC2616
which meanders rather.

   - Request line / Status line syntax is quite simple, one regexp.
   - Header: value (no space before the colon, no space before Header)
   - Header names must only use characters in "token" list.
   - Discard whitespace before and after value (it defines what whitespace)
   - Line beginning with whitespace just appends to previous line, (possibly
     after trimming whitespace before and after, and joining with a space)
   - Duplicate header may be merged with previous by joining with a comma
   - Lines ends with CRLF, suggests accepting just LF too.
   - Empty line finishes the lot.

On top of that, a server should:

   - Reject Expect values that it doesn't recognise
   - Reject methods it doesn't recognise
     (Sending a rejection message is easy, no need to follow many rules)
   - Reject HTTP/a.b versions it can't handle

And both sides should:

   - Include a Date header with the current time

Because WebSocket clients and servers are not proxies, many of the
rules in RFC2616 don't apply.

It's not much really, when you condense it down, is it?

(I probably missed a few things.  Finding them might involve a tedious
read of RFC2616 or HTTPbis.  But once found, I doubt if they demand
much from a WebSocket implementor, nor take much space in the spec.)

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 03:27:40 +0000

> some HTTP developers to add extra constraints to the upgrade

I don't think that's true.  As far as I remember, Ian hasn't disagreed
with that, only argued that it isn't sufficient reason to change the spec.

Remember, what makes life easier for those piggybacking on HTTP
implementations also makes life *harder* for those writing WebSocket
clients and servers from scratch.

However I do basically think the right balance has not been struck,
and that HTTP-compatible (or HTTP-similar) header flexibility will
benefit WebSocket-only agents in the long run too.

HTTP parsing isn't a tenth as complex as it looks in the HTTP spec,
and most HTTP rules don't need to be followed in this situation.  Even
most headers can be ignored.  Nobody would care if continuation lines
were rejected as bad syntax.

-- Jamie

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 02 Feb 2010 20:06:42 -0800

> match the exact octets needed?

I don;t actually know. Note that the spec does not require

> (Not just work, but be able to claim compliance.)

As far as I can tell, yes.

> Current proposal intentionally fails at compliant HTTP proxies, on port 80,

I think they would still fail because of the upgrade request. 

> (I'm wondering why the CORS check used for XmlHttpRequest is

I think CORS might have a potential weakness too. But not quite the same as the vulnerabilities I'm worried about. WebSocket won't start transmitting actual messages until it sees the handshake, while XHR will transmit the message body without checking for an HTTP response first. However, it can only send cross-site what you can already send with forms, so it does not open any new attack surface.

> If you like that idea, I'll describe how it would work.

I'm not sure I fully understand it from your description.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 03 Feb 2010 15:37:11 +1100

> could be pulled off, everyone would start using WebSocket asap ;-)

Cometd-2 (in alpha state) is more or less doing this already.

It should be in beta release shortly and then any cometd client
will be able to use websockets if the browser has it and the
server handshake says that the server supports it.  Otherwise
it is long polling business as usual.

regards

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 02 Feb 2010 21:22:15 -0800

> Yes, I admit it'd be a bit helpful if the nonce proposal was little

Let me start by laying out the security risks in the current handshake. Then I will explain my proposed change and how I believe it addresses them.

1) Hostile JavaScript could use WebSocket for a cross-protocol attack against vanilla HTTP resources or non-HTTP servers.

If the attacker can trick a non-WebSocket server into echoing back chosen text (for example through something in the URL part of the request), then they could make it give what appears to be a valid WebSocket handshake response. This could result in unauthorized access.

The current handshake attempts to mitigate this, by requiring some fixed parts of the response after the status line; thus, the fixed part of the handshake response effectively includes newlines. Since none of the parts of the request controlled by client JS can contain literal newlines, this somewhat increases the difficulty of echoing back exactly the right handshake. However, this is weaker than it could be. The exact correct handshake response is fully predictable, and furthermore consists only of fixed parts and verbatim repetition of parts of the request.

While we do not know of any service that is currently vulnerable, if the correct handshake response could not be predicted by client JS, that would be much more robust. And we do know that cross-protcol attacks from Web browsers are a real issue in general.

2) Cross-site XMLHttpRequest (using CORS or XDomainRequest) could be used for a cross-protocol attack against WebSocket resources, potentially violating integrity (though not confidentiality).

The WebSocket protocol currently does not require any checking of the client handshake. However, any WebSocket server that performs any side effects in response to messages from the client has a security vulnerability if it does not check correctness of the handshake request from the client. A cross-site XMLHttpRequest facility could be used to send a request to the victim an HTTP request with a body that looks like one or more valid WebSocket messages. This request would be a POST instead of a GET, and would lack the required WebSocket headers. But if the server is not checking, it would happily chug along.

The server would, of course, not respond with the right CORS headers needed for the attacker to actually read the response, so confidentiality cannot be violated. However, if the server treats incoming client messages as commands, then there is no protection against violating integrity.

To give a concrete example, consider a chat service over WebSocket. Let's say it doesn't check correctness of the handshake, and does authentication in-band. An attacker could construct an HTTP POST body which would look like an authentication message plus a couple of submitted sent messages. True, the attacker cannot read the victim's chat messages, but could send chat messages as the victim. Clearly this would be a significant security breach.

Admittedly, the spec could be changed to require servers to check some aspects of client handshake correctness. But because they can give a correct response without any processing, it remains relatively easy to forget to do this. A handshake protocol that required the server to read the client's request would reduce the risk.

Ian argued that services which do not read messages from the client at all (perhaps a data stream service like a stock ticker) would face an extra burden without gaining any safety from such a requirement. However, pure data stream services are easier to do via EventSource anyway; EventSource deals with intermediaries and can work with a completely unmodified Web server. So I don't think WebSocket needs to put a lot of weight on that use case.

Suggested solution:

We could mitigate both of these risks, and also reduce the implementation difficulty for servers and proxies, by changing the handshake. We would remove the requirement for newlines or exact capitalization of the headers. Instead, we could have the browser generate a unique random nonce, which it includes in the handshake request. The server would be required to read it, and respond with a hash of the nonce value. 

This has two benefits: (a) client JS can no longer predict the exact text of a valid handshake response; and (b) servers are essentially forced to look at the handshake, and must look for at least one element that cannot be forged with cross-site XMLHttpRequest. Thus, this mechanism mitigates risks (1) and (2) above. The reason to respond with a hash of the nonce, instead of the nonce literally, is so the response includes something that is both unpredictable *and* not a verbatim echo of any part of the request. There is no need for the hash to be cryptographically strong.

There are many possible variants, but here is an example of what the handshake request and response could look like (the hash used here is MD5; we could use a weaker hash that is faster to compute however):

Handshake from the client:

        GET /demo HTTP/1.1
        Upgrade: WebSocket
        Connection: Upgrade
        Host: example.com
        Origin: http://example.com
        WebSocket-Protocol: sample
        WebSocket-Nonce: 2d1283cf01e0e9f562989f0781450e7e

Response from the server:

        HTTP/1.1 101 Web Socket Protocol Handshake
        Upgrade: WebSocket
        Connection: Upgrade
        WebSocket-Origin: http://example.com
        WebSocket-Location: ws://example.com/demo
        WebSocket-Protocol: sample
        WebSocket-Nonce-Hash: 8ba7ca1e53376d29842e88d0f9db6978

The status line must come first, but order and capitalization of all request and response headers would be free. If we wanted to, we could even allow the status line to use any HTTP version.

A possible variant would be to include the nonce hash in the status line instead of in a header. But I think header is probably better.

Regards,
Maciej

---

From: Justin Erenkrantz <[email-masked]>
Date: Tue, 2 Feb 2010 22:37:48 -0800

> Let me start by laying out the security risks in the current handshake. Then I will explain my proposed change and how I believe it addresses them.

Thanks!  For the security stuff, I think it's best if I ponder that
for a few days to see if I can grok where the risks are.  In the

> To give a concrete example, consider a chat service over WebSocket. Let's say it doesn't check correctness of the handshake, and does authentication in-band. An attacker could construct an HTTP POS…

I am trying to understand how a properly implemented WS or HTTP server
would fall into this trap.  For httpd, this would not be feasible -
the code that parses request bodies isn't the same that handles the
protocol.  I'm trying to think of any real server implementations that
would attempt to process an HTTP body that also includes "fake"
requests (but in a different protocol).  Maybe there's a bit of
crafting that I'm just not seeing...

> We could mitigate both of these risks, and also reduce the implementation difficulty for servers and proxies, by changing the handshake. We would remove the requirement for newlines or exact capita…

While I'm not 100% sold on the security rationale for nonce yet (still
pondering it), at the least, the nonce is pretty straightforward to
implement - so, I could certainly be willing to admit I have a blind
spot there and go with the nonce if it means I can implement this in
httpd in an otherwise-straightforward manner.

Thanks again.  -- justin

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 02 Feb 2010 23:54:05 -0800

On Feb 2, 2010, at 10:37 PM, Justin Erenkrantz wrote:

> protocol.

A standalone WebSocket server could, per the WebSocket protocol draft, do no validation whatsoever of the WebSocket request handshake. I believe it would be completely conforming to skip until it sees two newlines in a row and start reading messages. Well, not quite - there seems to be a should-level requirement to abort if the third token of the first line does not contain a "/" character.

http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-74#section-5.2

A recent change also adds that "If a server reads fields for authentication purposes ... then the server should also verify that the client's handshake includes the invariant "Upgrade" and "Connection" parts of the handshake".

This is an improvement, but I think it should be a MUST not a should, and should be a requirement for all WebSocket servers. The spec also does not say that a WebSocket server MAY disconnect from the client if it finds the client handshake is invalid in any way other than lacking a "/".

A WebSocket module for an HTTP server would be less likely to fail to validate the handshake at all. However, whether the HTTP server would pass off invalid requests to a WebSocket module depends on how exactly it does dispatch. If it dispatches to the WebSocket module based on the relevant Connection and Upgrade headers, then it can't be tricked by XHR. But if it dispatches based on the path in the request, then it could pass a non-WebSocket request to the WebSocket module. In that case, if the WebSocket module doesn't do checking, it is similarly vulnerable.

>  I'm trying to think of any real server implementations that

The HTTP body wouldn't "also" include fake requests, it would *just* include fake requests. The HTTP header of the HTTP request itself might be enough to fool a client that did nothing but check for "/" and skip until it finds two newlines. And after that, you can have well-formed WebSocket messages in the HTTP body.

> httpd in an otherwise-straightforward manner.

Thanks, that's useful feedback.

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 00:20:40 -0800

> A possible variant would be to include the nonce hash in the status line instead of in a header. But I think header is probably better.

I asked a security expert to review this and he had two suggestions:

1) Include the nonce hash in the status line. The strongest protection against cross-protocol attacks comes in the first few bytes, according to him, so it should be as early as possible.

2) The hash should also include the request origin and some fixed WebSocket-specific string (e.g. "WebSocket::"). (He actually suggested 'HMAC the string "Web
Socket::" and origin of WebSocket request using the nonce as a key' but I'm not sure if he was serious.)

Here is an example of the server response with the hash in the status line, and it is actually the MD5 hash of "WebSocket::<ORIGIN> <NONCE>" (i.e. "WebSocket::http://example.com 2d1283cf01e0e9f562989f0781450e7e").

       HTTP/1.1 101 Web Socket Protocol Handshake 2daff5fe4d1295d48697c3f1f4ab9538
       Upgrade: WebSocket
       Connection: Upgrade
       WebSocket-Origin: http://example.com
       WebSocket-Location: ws://example.com/demo
       WebSocket-Protocol: sample

Regards,
Maciej

---

From: Roberto Peon <[email-masked]>
Date: Wed, 3 Feb 2010 01:16:47 -0800

> were rejected as bad syntax.

That reminds me that we've open sourced yet another HTTP parsing library.
-=R

---

From: Justin Erenkrantz <[email-masked]>
Date: Wed, 3 Feb 2010 07:38:48 -0800

On Wed, Feb 3, 2010 at 12:20 AM, Maciej Stachowiak <[email-masked]> wrote:
> 1) Include the nonce hash in the status line. The strongest protection
> against cross-protocol attacks comes in the first few bytes, according to
> him, so it should be as early as possible.

The drawback with doing it in the reason is that it is going to be
substantially harder to trust that HTTP/1.1 clients and such get
proper access to the reason.  (And, again, relying upon any bits in
the reason is a violation of HTTP - the reason is *only* intended for
human eyes.  I've seen hands slapped by Roy many times when folks try
to expect that the reason is machine-parsable.)  Due to that, I think
it's an acceptable tradeoff to just put it in the header as originally
suggested.  It will likely still be in the first packet that the
client receives.

> 2) The hash should also include the request origin and some fixed
> WebSocket-specific string (e.g. "WebSocket::"). (He actually suggested 'HMAC
> the string "Web

Using the origin IP introduces a lot of problems when there are
reverse proxies involved.  (Since it seems that prior discussions have
belittled NATs and transparent forward proxies as not being
"relevant", I'll set those aside - but I feel the implementation
challenges are just as relevant there too.)  Try as some might to
fight against it, we should simply realize that most large-scale WS
deployments are going to see some type of reverse proxy sitting in
front of it.  Therefore, the server generating the nonce probably
won't know the origin easily - and will have to rely upon custom
information set by its reverse proxy in front of it to generate the
proper nonce.  On the server-side, knowing the "real" origin IP is a
very error-prone process and probably would give us more grief due to
naive server implementations getting it wrong than we really intend.
There is no HTTP standard for setting the origin IP on the hops, but
most big sites end up setting some custom header out of necessity, but
I expect many basic deployments do not do so.

My $.02.  -- justin

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 07:52:21 -0800

> suggested.  It will likely still be in the first packet that the

The goal is more like "first few bytes" than "probably first packet". Remember, we're trying to defend against cross-protocol attacks, so what well-behaved clients and servers will do is not super relevant. I am more concerned with the security of the protocol than pleasing the aesthetic sense of HTTP gurus.

> reverse proxies involved.

The origin is not an IP address. I'm talking about what goes in the WebSocket-Origin header. See the spec for an explanation and examples.

>  (Since it seems that prior discussions have
> won't know the origin easily

The nonce is actually generated by the client, not the server. I assume you mean the hash (produced from the nonce among other things). But see above. This doesn't relate to IP addresses.

> - and will have to rely upon custom

 - Maciej

---

From: Justin Erenkrantz <[email-masked]>
Date: Wed, 3 Feb 2010 08:06:04 -0800

> The goal is more like "first few bytes" than "probably first packet". Remember, we're trying to defend against cross-protocol attacks, so what well-behaved clients and servers will do is not super…

No, but what matters is what well-behaved servers will do when they
accidentally break the only security mechanism when they reset the
status reason and clear those hash'd bytes.

Many servers and proxies will allow the status reason to be overriden
and replaced (think localization).  The failure case is going to be
abysmal to track down and completely fatal - the handshakes will fail
because some intermediary is rewriting the status reason and there
won't be *anything* at all that a client can do about it.  Suck.  At
least if it is in a separate HTTP header, it's a pretty good bet that
it won't be touched and will survive most interactions with reasonable
servers and intermediaries.

> The origin is not an IP address. I'm talking about what goes in the WebSocket-Origin header. See the spec for an explanation and examples.

Oops.  My bad.  (Perhaps we should consider another name besides
"origin" - that's a very overloaded term?)  -- justin

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 08:31:52 -0800

> status reason and clear those hash'd bytes.

This won't lead to a security failure, it will lead to a connection failure.

> servers and intermediaries.

Do you have examples of servers that will actually do this to a 101 response? Will they do this even to modules designed to handle a protocol upgrade?

As for intermediaries, the kind that like to rewrite requests and responses will not work with WebSocket anyway, unless they are updated. Only HTTPS proxies (i.e. CONNECT type) are likely to work. For proxies that are updated to handle WebSocket over non-SSL connections, they can also be updated to make sure not to mangle the status line.

If this is a real problem that will actually occur, then sure, we should design around it, but there's not much we can do with "it might happen".

> "origin" - that's a very overloaded term?) 

"Origin" is the usual term of art for this in web technology. As in, "same origin-policy". It's also used in the XMLHttpRequest spec, the WebSocket client spec, HTML5, and probably other specifications as well.

Regards,
Maciej

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 16:33:59 +0000

> it is long polling business as usual.

Nice!  But that's not really what I meant.

What I meant is two things, I suppose.  One protocol, one client API:

Protocol: Establishing a connection, and then each proxy in the chain
either opts in to making it a WebSocket (or similar in style)
resulting in single connection with bidirectional messaging, or if
there are any proxies detected which won't participate, there will be
HTTP long-polling - at least for the hops on either side of that proxy
(perhaps not the other hops).

I *think* what you described is: Use WebSocket if the whole path
supports it, otherwise long-polling for the whole path.

The difference from what that is the intentional proxy-friendliness
and friendliness to all other deployed HTTP infrastructure.  It'll
pass through everything, including the internals of xrandomhttpd <->
CGI, etc. and upgrade to WebSocket opportunistically on those hops
where it can, while working transparently to the user where it can't.

For the API, you've probably understood that it'd be nice if the
WebSocket API had a fallback to HTTP long-polling implementation,
Of course that can be written in Javascript today.

-- Jamie

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 08:39:08 -0800

> Of course that can be written in Javascript today.

That would be cool if it worked. But I'm not sure how you would detect that all your intermediaries understand WebSocket. Our best idea so far was hop-by-hop headers. But as someone else mentioned, there are intermediaries out there that just pass along hop-by-hop headers, but won't deal with arbitrary bidirectional non-HTTP traffic. So there doesn't seem to be a reliable mechanism for this.

Regards,
Maciej

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 21:56:10 +0000

Maciej Stachowiak wrote:
> That would be cool if it worked. But I'm not sure how you would
> detect that all your intermediaries understand WebSocket.

The main idea is opportunistic upgrade at each hop _without_ requiring
it for every hop.  So you could possibly have HTTP long-polling from
your client to your local proxy, for example, because one of the other
doesn't support the upgrade, but somewhere along the path it gets
converted to and from WebSocket.  (And other things, such as
opportunistic multiplexing (must be done very carefully)).

You don't try to check if every intermediary supports it - in fact,
they don't have to.  The idea is that agents/intermediaries recognise
a particular style of HTTP request passing through, and opt in by
adding something to say "I'm up for upgrading to the more efficient
method".  The next hop (in either direction) picks up the added note,
and if it understands it, then that specific hop can upgrade,
transparently to the other hops.

If hops on both sides of an intermediary upgrade, then the
intermediary would probably switch into a more efficient mode, because
it wouldn't need to translate between the different framing styles.

It can't quite be a single HTTP header, because we're talking about
bidirectional HTTP long-polling here which needs multiple connections,
so the intermediaries need to be able to recognise those connections.

The reliability problem comes from that added note being picked up
further away.  But that's not necessarily a problem, as long as the
intermediaries it passes through are able to carry the byte stream
reliably.  For which:

> Our best idea so far was hop-by-hop headers. But as someone else
> mentioned, there are intermediaries out there that just pass along
> hop-by-hop headers, but won't deal with arbitrary bidirectional
> non-HTTP traffic. So there doesn't seem to be a reliable mechanism
> for this.

There is and there isn't...

What we want from "dumb" intermediaries is that they _either_ are
detectable as regular HTTP proxies who would mangle, _or_ they behave
as a transparent TCP relay, after setting up.  ("Smart" intermediaries
are upgrade participants).

It's not possible to guarantee that, so sometimes there will be problems.

_But_, that is exactly the same problem which WebSocket faces already:
If a proxy cannot be detected and it mangles bidirectional binary
data, then basic (i.e. current draft) WebSocket is screwed as well.

The two problems are equivalent.  Anything which helps basic
WebSocket, helps with opportunistic participation.  Any time the
latter wouldn't work, would also break basic WebSocket.

All strategies used to make basic WebSocket work reliably apply:
Detecting bad proxies with rigid header checks, putting good proxies
into a helpful state (i.e. CONNECT?).

So it's the same problem and has the same expected success and failure
rate, and the same strategies to minimise that failure rate and
try something else when it's detected.

The idea of deliberately inserting your own "smart" proxies, at the
client or server end, offers an interesting alternative approach to
implementation too.  Existing HTTP long-poll clients and servers could
be used, where that is all infrastructure can handle for the time
being, and "smart" proxies would upgrade itto WebSocket over the
Internet, where it detects that is possible.

Conversely, standalone and simple "micro" WebSocket clients and
servers would be able to use non-WebSocket routes over the Internet
too, by virtue of that same "smart" proxy - making their
implementation a lot simpler.  (That's not dissimilar to cometd as a
way of deploying web apps, but this applies at the client side too.)

-- Jamie

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 04 Feb 2010 18:52:46 +0900

>    if we look like HTTP, we must use ports 80/443.

Well, the port space is pretty crowded these days. But IANA is not going 
to tell the *IETF* that they need the same ports as another protocol in 
case the IETF decides that it needs different ports. IANA's job 
description, for the protocol registries, is essentially: Register 
everything the IETF tells you, the way they tell you.

So for the moment, please assume that if we (as an IETF WG, followed by 
IETF last call) decide that we need new ports, IANA will try their best 
to give us new ports. The exact number may be treated as a detail.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Lars Eggert <[email-masked]>
Date: Fri, 5 Feb 2010 10:23:47 +0200

Hi,

>>   if we look like HTTP, we must use ports 80/443.

I'm trying to find out from IANA if this is really what the IANA Expert Reviewer said. There might have been a misunderstanding here.

> everything the IETF tells you, the way they tell you.

IANA assumes that requests coming through the IETF have been more carefully vetted than port requests that arrive from elsewhere, so there is no Expert Review for those, because the IETF process should have eliminated/corrected nonsensical requests before they hit IANA.

> to give us new ports. The exact number may be treated as a detail.

Yes.

Lars

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 05 Feb 2010 03:53:32 -0800

>> to give us new ports. The exact number may be treated as a detail.

Notwithstanding what IANA may say, sharing the standard HTTP and HTTPS ports is rather useful for getting through firewalls. Even if they were not the defaults, I bet a lot of people would manually choose 443 in particular. Given this, we really want to make the protocol look enough like HTTP that you can share that port on the same host between HTTP and WebSocket. If there was no need to look anything like HTTP, we could use a handshake that's much more secure n the face of cross-protocol attacks.

Regards,
Maciej

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 07 Feb 2010 18:09:14 +0900

Hello Maciej,

> On Feb 5, 2010, at 12:23 AM, Lars Eggert wrote:
[I wrote.]

> Notwithstanding what IANA may say, sharing the standard HTTP and HTTPS ports is rather useful for getting through firewalls. Even if they were not the defaults, I bet a lot of people would manually…

Yes. I didn't want to imply that we need separate ports, or that we 
don't need to be able to bootstrap from HTTP, or anything. I just wanted 
to say that "we won't get new port numbers" isn't true for development 
in the IETF, so we shouldn't take this as a constrain of the design.

So we could conclude to go with new ports full force, or we could 
conclude that we need to stay with the HTTP ports, and only these, or we 
could conclude that we want to be able to use the HTTP ports when 
nothing else is available, but to get new ports allocated and used 
because that's better in the long term, or ... whatever we think will 
actually work best.

Regards,    Martin.

P.S.: It may also pay out to look at http://tools.ietf.org/html/rfc3205. 
Although we are not using HTTP as a substrate, some of the 
considerations there in one way or another may apply to our case. I hope 
the WG Chairs and the ADs can check out early on how this might affect us.