Subject: Re: [hybi] Web sockets and existing HTTP stacks

From: Mark Nottingham <[email-masked]>
Date: Mon, 1 Feb 2010 20:58:01 +1100

Indeed. HTTP is *designed* to allow upgrades to other protocols. If anyone wants to argue against that, they're more than welcome to come argue that in HTTPbis, although I have a feeling they'll be needing an asbestos suit...

From this perspective, the only issue I* have with WebSockets is how it constrains the initial interchange to a subset of full HTTP; this may cause problems with some HTTP implementations (e.g. a server stack that wants to parse the request with an existing library before handing it off to the WebSockets code, where it doesn't provide enough fidelity in its abstractions). That's easily fixed.

Cheers,

* and, AFAICT, most of the HTTP community. YMMV.

On 01/02/2010, at 6:40 PM, Roberto Peon wrote:

> I am a server developer
> I do expect that if something goes over port 80, it should conform to HTTP spec until the spec agrees that it no longer applies.
> We have a significant and large population of users who have no understanding of the underpinnings of the technology-- all they want is their web pages to work. 
> 
> If you're seriously saying that we should not acknowledge that the web is broken over anything other than port 443 (heck, I don't even think that you can claim that port 80 works all the time, especially if you want to use UPGRADE, etc.), I'm interested in hearing about hard data that supports that assertion.
> 
> In other words, If deployment using a new port isn't a problem, what is the solution? Whatever it is, should it exist, I'd be happy to use it. 
> -=R
> 
> On Sun, Jan 31, 2010 at 10:32 PM, Mridul Muralidharan <[email-masked]> wrote:
> ----- Original Message ----
> 
> > From: Jamie Lokier <[email-masked]>
> > To: Pieter Hintjens <[email-masked]>
> > Cc: [email-masked]
> > Sent: Mon, 1 February, 2010 6:59:14 AM
> > Subject: Re: [hybi] Web sockets and existing HTTP stacks
> >
> > Pieter Hintjens wrote:
> > > On Sun, Jan 31, 2010 at 10:22 AM, Ian Hickson wrote:
> > >
> > > > Well, yeah. That's going to be the case with any protocol that shares its
> > > > port with HTTP. Web Socket tries to make this easier by making it at least
> > > > _possible_ to parse the header with an HTTP stack, if not necessarily
> > > > easy.
> > >
> > > Do you not understand the impact of breaking (cheerfully or not) a
> > > 30-year standard respected by the entire Internet?
> > >
> > > Post 80 is not shared by protocols.  Port 80 IS HTTP by definition, by
> > > contract.
> >
> > It is a fact that internet access is only granted over ports 80
> > and/or 443 at some locations.
> >
> > This is why WebSocket uses those.  It is for that practical reason,
> > not from a desire to break the port convention.  Thus there is no
> > point complaining on port convention grounds.
> 
> 
> This is an oft-repeated argument in this list, and in some xmlrpc/WS lists - and I am not very sure I buy the argument.
> There is a reason why only http is allowed by a lot of firewalls and proxies - and it is a deployment choice the customer makes : whether driven by security concerns, convention, or other - it is a conscious decision at times (not always, I admit).
> 
> At times? What percentage of web users out there know how it all works? .. and driven by "convention" is a poor reason to do something.
> 
> 
> 
> The reasoning that - port 80 is not blocked, so let us tunnel protocol xyz over it, is not a very good line of reasoning: it is not an invitation for protocol designers to tunnel arbitrary protocols on top of http.
> Either you send http on top of port 80, or use something else - so that a deployment has control : both in terms of security concerns, intermediaries which can be deployed, contractual validation, etc.
> 
> All of those things can still do the right thing over port 80. They see what they don't like, then they can drop it.
> .. and much of the difficulty in deploying anything "new" on the web today is because of intermediaries that the user doesn't know exist (and thus couldn't desire the presence of..)
> 
> -=R
> 
> 
> My 2 cents.
> Regards,
> Mridul
> 
> 
> >
> > An earlier version of WebSocket proposed port 81, but that was changed to 80.
> >
> > I do wonder if those sites only allowing port 80 all run intercepting
> > proxies on port 80 which would prevent WebSocket using it, so that it might
> > as well use port 81 anyway. Anybody know?
> >
> > -- Jamie
> > _______________________________________________
> > hybi mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/hybi
> 
> 
> 
>      Your Mail works best with the New Yahoo Optimized IE8. Get it NOW! http://downloads.yahoo.com/in/internetexplorer/
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
> 
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

--
Mark Nottingham     http://www.mnot.net/

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 1 Feb 2010 11:32:39 +0100

> Indeed. HTTP is *designed* to allow upgrades to other protocols. If anyone wants to argue against that, they're more than welcome to come argue that in HTTPbis, although I have a feeling they'll be…

HTTPbis sounds like a fun list.  Clearly since port 80 is often the
only way in or out of a network, upgrades over HTTP are essential for
pragmatic reasons.

Reading the discussion here over the last months, I see three sets of concerns:

1. Concerns about the quality of the web socket protocol spec itself,
which are normal and expected.
2. Concerns about the status of this spec as The Official spec for (?)
bidirectional transport over/beside HTTP.
3. Concerns about the impact that this spec has on existing HTTP stacks.

Clearly these concerns are interconnected, and that is a problem.  Us
server implementors become hypersensitive to the quality of the spec
because it appears to have an excessive sense of status AND it plays
too loose with the contracts we've invested years of work in.

It should be trivial to make Web Sockets work *properly* with the HTTP
upgrade mechanism, at which point these concerns become properly
orthogonal, and we can all relax somewhat.

-Pieter

---

From: Mridul Muralidharan <[email-masked]>
Date: Mon, 1 Feb 2010 16:10:05 +0530 (IST)

There is a difference between sending an upgrade request, which is conforment with http spec - and so intermediaries/others being able to make a decision about whether to allow/honour it or not, and doing it by fudging requests by making it look like http.

I am all for using http in the way it is meant to be used - even for bootstrapping non-http protocols : if done right, which is not opaque : but not when trying to piggyback on 'port 80 is open, let us use it since admin cant stop us'. 
[Think using CONNECT to start sending arbitrary protocols : it is done a lot, but reason for having CONNECT enabled in proxies is not for arbitrary protocols, but just https].

I am reserving my comments about xmlrpc and the like : that is a long, and definitely irrelevant (in this context) line of discussion.

Regards,
Mridul

----- Original Message ----
> From: Mark Nottingham <[email-masked]>
> To: Roberto Peon <[email-masked]>
> Cc: Mridul Muralidharan <[email-masked]>; [email-masked]
> Sent: Mon, 1 February, 2010 3:28:01 PM
> Indeed. HTTP is *designed* to allow upgrades to other protocols. If anyone wants 
> to argue against that, they're more than welcome to come argue that in HTTPbis, 
> although I have a feeling they'll be needing an asbestos suit...
> 
> From this perspective, the only issue I* have with WebSockets is how it 
> constrains the initial interchange to a subset of full HTTP; this may cause 
> problems with some HTTP implementations (e.g. a server stack that wants to parse 
> the request with an existing library before handing it off to the WebSockets 
> code, where it doesn't provide enough fidelity in its abstractions). That's 
> easily fixed.
> 
> Cheers,
> 
> > I do expect that if something goes over port 80, it should conform to HTTP 
> spec until the spec agrees that it no longer applies.
> > We have a significant and large population of users who have no understanding 
> of the underpinnings of the technology-- all they want is their web pages to 
> work. 
> > 
> > If you're seriously saying that we should not acknowledge that the web is 
> broken over anything other than port 443 (heck, I don't even think that you can 
> claim that port 80 works all the time, especially if you want to use UPGRADE, 
> etc.), I'm interested in hearing about hard data that supports that assertion.
> > 
> > In other words, If deployment using a new port isn't a problem, what is the 
> solution? Whatever it is, should it exist, I'd be happy to use it. 
> > -=R
> > 
> > On Sun, Jan 31, 2010 at 10:32 PM, Mridul Muralidharan 
> wrote:
> > > From: Jamie Lokier 
> > > To: Pieter Hintjens 
> > This is an oft-repeated argument in this list, and in some xmlrpc/WS lists - 
> and I am not very sure I buy the argument.
> > There is a reason why only http is allowed by a lot of firewalls and proxies - 
> and it is a deployment choice the customer makes : whether driven by security 
> concerns, convention, or other - it is a conscious decision at times (not 
> always, I admit).
> > 
> > At times? What percentage of web users out there know how it all works? .. and 
> driven by "convention" is a poor reason to do something.
> > 
> > 
> > 
> > The reasoning that - port 80 is not blocked, so let us tunnel protocol xyz 
> over it, is not a very good line of reasoning: it is not an invitation for 
> protocol designers to tunnel arbitrary protocols on top of http.
> > Either you send http on top of port 80, or use something else - so that a 
> deployment has control : both in terms of security concerns, intermediaries 
> which can be deployed, contractual validation, etc.
> > 
> > .. and much of the difficulty in deploying anything "new" on the web today is 
> because of intermediaries that the user doesn't know exist (and thus couldn't 
> desire the presence of..)
> > 
> > -=R
> > 
> > 
> > My 2 cents.
> > Regards,
> > Mridul
> > 
> > 
> > >
> >      Your Mail works best with the New Yahoo Optimized IE8. Get it NOW! 
> http://downloads.yahoo.com/in/internetexplorer/

---

From: Mridul Muralidharan <[email-masked]>
Date: Mon, 1 Feb 2010 16:16:47 +0530 (IST)

>
>From: Roberto Peon <[email-masked]>
>To: Mridul Muralidharan <[email-masked]>
>Cc: Jamie Lokier <[email-masked]>; Pieter Hintjens <[email-masked]>; [email-masked]
>Sent: Mon, 1 February, 2010 1:10:08 PM

I am not sure what is the assertion being referred to here.
My point, in a nutshell was, trying to get arbitrary protocols running on top of port 80 (pls note : not on top of http) just cos it is opened by almost all deployments [proxies, firewalls, etc] is subverting/breaking intermediaries.
443 has been a prime candidate for this for quite a while now.

>In other words, If deployment using a new port isn't a problem, what is the solution? Whatever it is, should it exist, I'd be happy to use it. 

Use 80 if it is HTTP, else dont.
HTTP leading to XYZ is a different case (through upgrade, etc) - since it is done explicitly. From what I understand, websockets tries to not take this route, but masquerade as http. Any intermediary (corporate firewall for example) cant control this sort of access transparently.

>At times? What percentage of web users out there know how it all works? .. and driven by "convention" is a poor reason to do something.

I am not referring to endusers - but to deployers and admins. ISPs, admins and the like.
Hope that clarifies.
HTTP-like is not HTTP.

Thanks,
Mridul

>>     Your Mail works best with the New Yahoo Optimized IE8. Get it NOW! http://downloads.yahoo.com/in/internetexplorer/

      The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 1 Feb 2010 10:52:40 +0000

SM wrote:
> The registration request was rejected.  If the protocol requires a 
> port to be allocated, there should be proper justification for the 
> request.  It is unlikely that two ports will be assigned as the 
> practice of having a different port number for the secure version of 
> a protocol is discouraged.

I'm confused by your apparently contradictory position.

I think you first said WebSocket should use a different port from HTTP
because it's a different protoocol, yet here you're saying a different
port for WebSocket does not have proper justification.

Also I don't see what putting WebSocket on port 81 has to do with secure
versions of a protocol.

Can you clarify what you mean?

Thanks,
-- Jamie

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 1 Feb 2010 12:12:51 +0100

> Can you clarify what you mean?

This was discussed some time ago on this list.  IANA will hesitate to
allocate two ports to a new protocol when one is sufficient.  SSL/TLS
is easy to handle as an upgrade.  We had this exact discussion on the
AMQP working group.  Further, asking for two ports in the sub-4096
range (especially port 81!) is asking for rejection.  (It seems
unbelievably optimistic-slash-something to ask for port 81...!)

Failing to get port 81 does not give Web Socket the right to break the
contract on port 80.  Thus this feeble argument:

* We followed the rules and asked for our own ports 81 and 444 (iirc) :-)
* IANA said no to our request! :-(
* We were pointed to port 80 by IANA :-/
* So we're allowed to do what we like on port 80! :-)  Yay!

Sorry to be critical but I have arguments like this with my six year
old daughter and even she knows it's untenable.

-Pieter

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 1 Feb 2010 11:20:52 +0000

Mridul Muralidharan wrote:
> For sharing multiple logical channels over a single underlying
> channel, I had suggested BEEP [1] to the list initially.
> 
> For a more http compliant server side push (simulated ofcourse),
> there is XEP 124 [2] which uses a variant of more robust version of
> long poll - which has been used quite successfully in xmpp space :
> with the caveat that, one of clients connections would need to be
> blocked at the server (within timelimits) for notifications.
> 
> While implementing xmpp on top of http, we had more practical
> reasons why we chose xep 124 (it was a fledging standard when we
> looked at it initially), which was how well you can interact with
> intermediaries which you have no control over : which is what led to
> not pursuing a comet channel approach (reliability issues, proxy
> caching of response (non-compliant at times), etc). We can wish away
> a lot of these issues, but for real world customer deployments - you
> will have to address them anyway.
>
> I am not sure if proper due diligence was done on these, but I
> assume the group has - since I have not followed the discussions of
> the past couple of months.

Don't assume!

Some of us have looked at BEEP in some detail, and based on experience
deploying simular protocols, picked out some issues where it can be
much improved as a multiplexing channel.  In particular, opportunistic
multiplexing, where a user-agent or proxy combines incoming message
streams into fewer connections, doesn't work well within BEEPs
structure - due to channel setup latency, mainly.  BEEP-over-TCP does
offer an approach to the head of line blocking problem though, one I
think all multiplexing protocols should take note of.

There's an attempt to document best practice and issues with HTTP
long-polling and variations in general.  After all, everyone's
developing proprietary protocols over that; it might as well be guided
by collective knowledge.  I'm not sure how that's going.  I haven't
had time to contribute to it myself.

I've never heard of XEP 124, and I don't think I've seen it mentioned
on the list before.  How does it differ from comet style - in
particular, why do you think XEP 124 is robust and comet
insufficiently so?  That could be very useful input into the
long-polling best practice document.

Thanks,
-- Jamie

---

From: SM <[email-masked]>
Date: Mon, 01 Feb 2010 07:26:05 -0800

>I'm confused by your apparently contradictory position.

My comment was about the points that may be raised.

>port for WebSocket does not have proper justification.

If you are writing a protocol which requires a specific port, use a 
different port.  If you can do HTTP Upgrade, run it on the HTTP port 
(this should not be read as an encouragement for all applications to 
run over the HTTP port).  As I mentioned previously, you will have to 
provide proper justification if you require an assignment in the 
well-known port range.  That generally means documenting why the 
alternatives are unsuitable for your application.

>Can you clarify what you mean?

Two ports were requested.  The second one was for the secure version 
of the protocol.

Regards,
-sm

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 02 Feb 2010 05:29:29 +1100

Maciej Stachowiak wrote:

>> Then if we ever get HTTP/1.2 or HTTP/2.0, websocket will not break!
>
> If the client tries to upgrade an HTTP/1.1 request to WebSocket and the
> server gives an HTTP/1.2 or HTTP/2.0 response, then it had better break!
> We'll need to adjust the client side of the protocol before we can let
> servers respond with different versions of HTTP.

Why?

Saying HTTP/1.2 in a response line does not make the response
version 1.2. The version of the response is controlled by the version of
the request. The version in  the response says what the server is capable
of.  A HTTP/1.2 capable server should be able to server 1.0 websocket
clients without further breaking the http protocol by hacking about its
version for one particular upgrade protocol.

So this is a perfect example how websockets will break future server
code by not following the http contract.

regards.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 02 Feb 2010 09:06:25 +1100

Maciej Stachowiak wrote:
> On Jan 31, 2010, at 8:05 PM, Greg Wilkins wrote:

>> But then the sentinel framing of websocket is completely vulnerable to
>> injection attacks.
>> All websocket endpoints will have to validate that utf-8 data given to
>> them really is utf-8 data.
> 
> I believe the only component that absolute has to check for violations
> of the sentinel framing to maintain security is the browser - it needs
> to prevent script code running in the browser from doing shady things to
> an existing connection, so it can't let scripts inject a sentinel. 
>
> For robustness, of course, all parties should check both incoming and
> outgoing framing for correctness.

For robustness, it is best to have a design that is not susceptible
to injection attacks.      Sentinel framing (include CRLF used by
HTTP) is simply not a good way to wrap user provided data.

The length encoding is there and free from injection vulnerabilities.
It would be trivial to switch to using it transparently to the ws API.

regards

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 02 Feb 2010 14:55:35 +1100

Ian Hickson wrote:

> Similarly, the server, when parsing 
> the headers in "HTTP" mode, is unaffected by the order -- and indeed, the 
> Web Socket spec doesn't require _anything_ from the server in terms of 
> parsing the client request. You can completely ignore it for all the spec 
> cares. All that matters is that you send back a specific handshake. But if 
> you're sending back the handshake, then you're a Web Socket server, so why 
> do we need to follow HTTP rules? We've already established the client is a 
> Web Socket client, so what on earth is the point of using HTTP rules?

We need to follow HTTP rules because we are in a HTTP server
that has implemented the HTTP standard and until the CRLF
is sent after the 101 response, HTTP rules OK!

You think we are making up the difficulties we are having with this?

---

From: Roberto Peon <[email-masked]>
Date: Mon, 1 Feb 2010 21:39:40 -0800

> is sent after the 101 response, HTTP rules OK!

Ian-- I know that I'd kick scream and generally raise hell about putting
that in our little proxy were it not HTTP compliant until the end of the 101
response.
-=R

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 01 Feb 2010 22:11:00 -0800

On Feb 1, 2010, at 9:39 PM, Roberto Peon wrote:

> Ian-- I know that I'd kick scream and generally raise hell about putting that in our little proxy were it not HTTP compliant until the end of the 101 response.

The current handshake response *is* HTTP compliant, it's just that the HTTP response needs to meet some additional restrictions, or it will be rejected by the client. Those restrictions include requirements to include particular response headers (apparently not controversial), to include particular text in the status line (only mildly controversial), and to require that the first two response headers headers must be two specific ones, with an exact required order and capitalization (this appears to bug people the most and perhaps creates the most headaches inside existing server code bases).

Are these restrictions a practical problem for you? If so, which are problematic, and can you describe the concrete issue? Knowing the intensity of your response is less useful than knowing the reason for it.

Regards,
Maciej

---

From: Roberto Peon <[email-masked]>
Date: Mon, 1 Feb 2010 23:43:50 -0800

Simple-- the proxy won't know that the response from the server is websocket
specific, and may reorder things.
You should not assume that the termination point (the "host") is a server.
It could be a reverse proxy. In such cases, the proxy needs to treat the
response as HTTP until it has been fully read and interpreted.

This response IS HTTP, so HTTP rules apply. It isn't a some random
more-restrictive subset of the response rules. It is still HTTP at that
point. That means that you don't get to add additional restrictions on that
request. If you do, then you're creating headaches for everyone.

Now, I may kick and scream, etc. but I certainly can be convinced that the
effort to change is worthwhile if the reasons are not arbitrary.
What advantage do we get from requiring special behaviors from the
servers/proxies in this case? I can't think of any benefits, thus my
response!

-=R

On Mon, Feb 1, 2010 at 10:11 PM, Maciej Stachowiak <[email-masked]> wrote:

> The current handshake response *is* HTTP compliant, it's just that the HTTP
> response needs to meet some additional restrictions, or it will be rejected
> by the client. Those restrictions include requirements to include particular
> response headers (apparently not controversial), to include particular text
> in the status line (only mildly controversial), and to require that the
> first two response headers headers must be two specific ones, with an exact
> required order and capitalization (this appears to bug people the most and
> perhaps creates the most headaches inside existing server code bases).
>
> Are these restrictions a practical problem for you? If so, which are
> problematic, and can you describe the concrete issue? Knowing the intensity
> of your response is less useful than knowing the reason for it.
>
> Regards,
> Maciej
>
>

---

From: Francis Brosnan Blazquez <[email-masked]>
Date: Tue, 02 Feb 2010 18:50:31 +0100

> We already have BEEP for framing over TCP and to avoid
> head-of-line blocking. 

Right, but Websocket design should be closer to a TCP connection than an
application protocol so people can layer whay they want...

> What would be wrong with that, apart
> from the fact that BEEP is already dead?

Why you state this? 

-- 
Francis Brosnan Blazquez <[email-masked]>
ASPL

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 02 Feb 2010 17:25:28 -0800

On Feb 1, 2010, at 11:43 PM, Roberto Peon wrote:

> Simple-- the proxy won't know that the response from the server is websocket specific, and may reorder things.
> You should not assume that the termination point (the "host") is a server. It could be a reverse proxy. In such cases, the proxy needs to treat the response as HTTP until it has been fully read and interpreted.
> 
> This response IS HTTP, so HTTP rules apply. It isn't a some random more-restrictive subset of the response rules. It is still HTTP at that point. That means that you don't get to add additional restrictions on that request. If you do, then you're creating headaches for everyone.

OK, just to clarify, are you saying that it's specifically the constraint on header ordering that is a problem? I'm trying to figure out which specific requirements are problematic and why. For example, is the requirement to have some special text in the status line acceptable?

Regards,
Maciej

> 
> Now, I may kick and scream, etc. but I certainly can be convinced that the effort to change is worthwhile if the reasons are not arbitrary.
> What advantage do we get from requiring special behaviors from the servers/proxies in this case? I can't think of any benefits, thus my response!
> 
> -=R
>

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 01:44:32 +0000

Mridul Muralidharan wrote:
> There is a difference between sending an upgrade request, which is
> conforment with http spec - and so intermediaries/others being able to
> make a decision about whether to allow/honour it or not, and doing it
> by fudging requests by making it look like http.
>
> I am all for using http in the way it is meant to be used - even for
> bootstrapping non-http protocols : if done right, which is not opaque
> : but not when trying to piggyback on 'port 80 is open, let us use it
> since admin cant stop us'.

In case you hadn't checked, WebSocket actually does use the HTTP
Upgrade mechanism.  Firewalls which block HTTP Upgrade requests based
on the request and response headers can include blocking WebSocket, or
not as they choose, in their rules.

In this regard, there is no fudging.

The rigidity of WebSocket's spec forces senders to be more rigid in
what they send, and receivers to be more strict in what they accept.
This has the side effect that proxies which modify the request will
sometimes break it.  But it does not prevent HTTP firewall function,
as long as it doesn't modify the messages when it's allowing them to
pass.

> [Think using CONNECT to start sending arbitrary protocols : it is
> done a lot, but reason for having CONNECT enabled in proxies is not
> for arbitrary protocols, but just https].

Just a random FYI, CONNECT and port 443 are blocked in some locations,
because they want to be able to inspect all HTTP requests and block
those they can't inspect.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 01:47:27 +0000

Pieter Hintjens wrote:
> Clearly since port 80 is often the

Pragmatically, if upgrade over HTTP isn't used, application developers
will just carry on using long-polling over HTTP instead.  Which is
perfectly valid, if unpleasant.

(Btw, you should have seen the network where ICMP ECHO was the only
way in and out.  The administrators were curious why I had fast web
access and nobody else did, and also curious why their packet stats
showed more ICMP ECHOs than any other kind of packets :-)

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 01:48:09 +0000

Mridul Muralidharan wrote:
> HTTP leading to XYZ is a different case (through upgrade, etc) -
> since it is done explicitly. From what I understand, websockets
> tries to not take this route, but masquerade as http.

Your understanding is mistaken.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 01:49:40 +0000

> different port.  If you can do HTTP Upgrade, run it on the HTTP port 

WebSocket does use HTTP Upgrade, so there's no problem.

-- Jami

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 02:08:39 +0000

Maciej Stachowiak wrote:
>    to prevent script code running in the browser from doing shady things
>    to an existing connection, so it can't let scripts inject a sentinel.

Every component which forwards data and needs to track the frame
boundaries, including those which just use it to control buffering,
need to scan every byte and are vulnerable to injection attacks.

A simple forwarding component may track frame boundaries to buffer up
a whole message before forwarding it.  A frame injection attack can
fool such a component to cause an unbounded delay in delivering that
or later messages, breaking application expectations.

Other things which have been discussed such as orderly close,
keepalives and so on can be very broken by a frame injection attack on
an earlier frame.

Web pages with multiple Javascript components sharing a connection
through a simple framework - which is the primary target for WebSocket
I believe - may be tricked into things like retrying a state changing
request sent to the server when it is not safe to do so, causing the
state change twice, despite the application protocol being carefully
coded to make that "impossible".  Because of a frame injection attack
breaking protocol assumptions.

If a receiver supported the gzipped binary frame which
was mentioned, a sentinel frame injection attack could inject a
gzipped frame which may decode in all sorts of ways... denial of
service at least.

If anything, the *sender* has to be especially careful to check for
violations, otherwise the whole delivery chain may go wrong.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 02:20:56 +0000

> version for one particular upgrade protocol.

This is not an issue, because a WebSocket server knows it's responding
to a WebSocket/1.0 client and can send the required response.

> code by not following the http contract.

It doesn't follow the HTTP contract to the letter, but it's not
illegal to "become" a 1.0 server :-)  (Though it would affect part of
RFC2616 which requires proxies to remember the HTTP version of recently
used servers).

It makes it hard to reuse existing server and proxy components,
but doesn't make it impossible to share ports with HTTP.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 3 Feb 2010 02:22:11 +0000

> to a WebSocket/1.0 client and can send the required response.

The real trouble is how will a WebSocket/1.1 client interoperate with
a WebSocket/1.0 server, without trying, failing, then starting another
connection to try again which is wasteful and not entirely reliable.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 03 Feb 2010 15:51:15 +1100

Maciej Stachowiak wrote:

> OK, just to clarify, are you saying that it's specifically the
> constraint on header ordering that is a problem? I'm trying to figure
> out which specific requirements are problematic and why. For example, is
> the requirement to have some special text in the status line acceptable?

Maciej,

It is mainly the header ordering and somewhat the case sensitivity.

I've got no problem with the websocket server requiring special
content in the status line reason.    However some existing servers
may have an issue as I have seen that some - in response to CRLF
injection attacks on the reason string - ignore any passed value
for the reason string.   But I think fixes for that will in general
be a lot less ugly than a fix for header ordering.

cheers

---

From: Mridul Muralidharan <[email-masked]>
Date: Wed, 3 Feb 2010 12:16:29 +0530 (IST)

----- Original Message ----
> Cc: Mark Nottingham <[email-masked]>; Roberto Peon <[email-masked]>; [email-masked]
> Sent: Wed, 3 February, 2010 7:14:32 AM

I will need to recheck the latest draft spec later today, but my understanding was the it does not honour redirects, auth requests (proxy or server), insertion of arbitrary headers (c -> p1 -> p2 -> s) , reordering of headers, etc.
Hopefully these are resolved now.
If they are indeed resolved, I dont see the problem with bootstrapping using UPGRADE.

On a related note, an oft repeated assumption is that client and server would be developed by the same 'person/team/company' - this, I believe, is very short sighted.
As we have seen it in xmpp space, well defined protocols leads to explosion of client side libraries for various languages, frameworks, etc : allowing you to talk to any compliant server.

> those they can't inspect.

Exactly my point, I would not like it as a user (if I cared about it  : since this means I cant connect to my bank),  but I would definitely want this ability if I were an admin.
The ability to monitor & control access is very useful and probably fundamental.

Regards,
Mridul

---

From: Mridul Muralidharan <[email-masked]>
Date: Wed, 3 Feb 2010 12:21:03 +0530 (IST)

----- Original Message ----
> Cc: Roberto Peon <[email-masked]>; Pieter Hintjens <[email-masked]>; [email-masked]
> Sent: Wed, 3 February, 2010 7:18:09 AM

Unfortunately, I am not mistaken.

- Mridul

---

From: Roberto Peon <[email-masked]>
Date: Wed, 3 Feb 2010 00:00:49 -0800

On Tue, Feb 2, 2010 at 5:25 PM, Maciej Stachowiak <[email-masked]> wrote:

> Simple-- the proxy won't know that the response from the server is
> websocket specific, and may reorder things.
> OK, just to clarify, are you saying that it's specifically the constraint
> on header ordering that is a problem? I'm trying to figure out which
> specific requirements are problematic and why. For example, is the
> requirement to have some special text in the status line acceptable?
>

Having some required header ordering is a problem.
As for special text in the status line, I'd have to examine the spec again
to be sure-- if proxies have the ability to drop the status text (the third
part of the first line of the HTTP response, which I'm assuming we're
talking about here), then it shouldn't be relied upon.

-=R

>> The current handshake response *is* HTTP compliant, it's just that the
>> HTTP response needs to meet some additional restrictions, or it will be
>> rejected by the client. Those restrictions include requirements to include
>> particular response headers (apparently not controversial), to include
>> particular text in the status line (only mildly controversial), and to
>> require that the first two response headers headers must be two specific
>> ones, with an exact required order and capitalization (this appears to bug
>> people the most and perhaps creates the most headaches inside existing
>> server code bases).
>>

---

From: Julian Reschke <[email-masked]>
Date: Wed, 03 Feb 2010 09:07:55 +0100

Roberto Peon wrote:
> 
> 
> On Tue, Feb 2, 2010 at 5:25 PM, Maciej Stachowiak <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> 
>>     You should not assume that the termination point (the "host") is a
>>     server. It could be a reverse proxy. In such cases, the proxy
>>     needs to treat the response as HTTP until it has been fully read
>>     and interpreted.
>>
>>     more-restrictive subset of the response rules. It is still HTTP at
>>     that point. That means that you don't get to add additional
>>     restrictions on that request. If you do, then you're creating
>>     headaches for everyone.
> 
>     constraint on header ordering that is a problem? I'm trying to
>     figure out which specific requirements are problematic and why. For
>     example, is the requirement to have some special text in the status
>     line acceptable?
> 
> 
> As for special text in the status line, I'd have to examine the spec 
> again to be sure-- if proxies have the ability to drop the status text 
> (the third part of the first line of the HTTP response, which I'm 
> assuming we're talking about here), then it shouldn't be relied upon.
> ...

I don't think HTTP specifies this. As Status-Phrase is just 
informational, I wouldn't surprised when it happened, or when it was 
rewritten. I'm pretty sure that there are server frameworks where it 
can't easily be set.

Anyway: if another line of defense is added (and that's how I understand 
Maciej's proposal), is there still a point to insist on the exact 
Status-Phrase?

Best regards, Julian

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 00:23:24 -0800

On Feb 3, 2010, at 12:07 AM, Julian Reschke wrote:

> Roberto Peon wrote:
>> On Tue, Feb 2, 2010 at 5:25 PM, Maciej Stachowiak <[email-masked] <mailto:[email-masked]>> wrote:
>> As for special text in the status line, I'd have to examine the spec again to be sure-- if proxies have the ability to drop the status text (the third part of the first line of the HTTP response, which I'm assuming we're talking about here), then it shouldn't be relied upon.
>> ...
> 
> I don't think HTTP specifies this. As Status-Phrase is just informational, I wouldn't surprised when it happened, or when it was rewritten. I'm pretty sure that there are server frameworks where it can't easily be set.

I am pretty sure a lot of popular servers do let modules/extensions/scripts running inside them set it, so I don't think this is a huge practical problem.

> 
> Anyway: if another line of defense is added (and that's how I understand Maciej's proposal), is there still a point to insist on the exact Status-Phrase?

According to a security expert I asked, the strongest defense against cross-protocol attacks is in the first few bytes. So I think removing all requirements on the status-phrase would make the protocol less robust against cross-protocol attacks.

Regards,
Maciej

---

From: Roberto Peon <[email-masked]>
Date: Wed, 3 Feb 2010 01:07:48 -0800

On Wed, Feb 3, 2010 at 12:23 AM, Maciej Stachowiak <[email-masked]> wrote:

> >> On Tue, Feb 2, 2010 at 5:25 PM, Maciej Stachowiak <[email-masked]<mailto:
> [email-masked]>> wrote:
> again to be sure-- if proxies have the ability to drop the status text (the
> third part of the first line of the HTTP response, which I'm assuming we're
> rewritten. I'm pretty sure that there are server frameworks where it can't
> easily be set.
>
> I am pretty sure a lot of popular servers do let modules/extensions/scripts
> running inside them set it, so I don't think this is a huge practical
> problem.
>
> >
> According to a security expert I asked, the strongest defense against
> cross-protocol attacks is in the first few bytes. So I think removing all
> requirements on the status-phrase would make the protocol less robust
> against cross-protocol attacks.
>

How exactly does the random bit of text in the first line help?
You still have to frame the HTTP response, which means that any errors in
that framing are going to be causing problems.

In other words, the response will always be as insecure as HTTP. No way
around that while still using HTTP. And, as we've already covered, not using
HTTP isn't a terribly good option while on port 80 until the upgrade has
finished.

-=R

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 02:02:29 -0800

> How exactly does the random bit of text in the first line help?

In the case of a cross-protocol attack against HTTP, it's likely harder to inject text into the status line than it is to inject a response header, so that's the reason why it is in the first line. In the case of a cross-protocol attack against other protocols, in general it's harder to mess with the very beginning of the server response than the end (though not always).

The text is not random, it is a hash of some information from the request. The reason for the particular nature of the bit of text is to further increase the challenge of cross-protocol attacks by requiring the handshake response to contain something that (a) cannot be predicted before client JS tries to initiate the connection; and (b) is not a verbatim echo of anything in the handshake request. So even if you can find a server that will echo back exact text of your choice, or pieces of the request of your choice, you cannot make it return what appears to be a valid handshake response.

> You still have to frame the HTTP response, which means that any errors in that framing are going to be causing problems.

I don't understand what kinds of problems you have in mind, could you clarify?

> In other words, the response will always be as insecure as HTTP. No way around that while still using HTTP. And, as we've already covered, not using HTTP isn't a terribly good option while on port 80 until the upgrade has finished.

Again, I don't know what you mean by "as insecure as HTTP". Specifically what we're worried about here are cross-protocol attacks (both using WebSocket and against a WebSocket service). The fact that the handshake format is an HTTP request does increase the risk in both directions, but we can do things to mitigate it. If you think my proposed defense is ineffective, could you explain how?

Regards,
Maciej

---

From: Roberto Peon <[email-masked]>
Date: Wed, 3 Feb 2010 09:56:02 -0800

On Wed, Feb 3, 2010 at 2:02 AM, Maciej Stachowiak <[email-masked]> wrote:

> In the case of a cross-protocol attack against HTTP, it's likely harder to
> inject text into the status line than it is to inject a response header, so
> that's the reason why it is in the first line. In the case of a
> cross-protocol attack against other protocols, in general it's harder to
> mess with the very beginning of the server response than the end (though not
> always).
>
> The text is not random, it is a hash of some information from the request.
> The reason for the particular nature of the bit of text is to further
> increase the challenge of cross-protocol attacks by requiring the handshake
> response to contain something that (a) cannot be predicted before client JS
> tries to initiate the connection; and (b) is not a verbatim echo of anything
> in the handshake request. So even if you can find a server that will echo
> back exact text of your choice, or pieces of the request of your choice, you
> cannot make it return what appears to be a valid handshake response.
>

You can still do this as an HTTP header.

> I don't understand what kinds of problems you have in mind, could you

HTTP header injection problems: If it is possible, you can get interesting
things to happen. This true of anything which is attempting to frame HTTP,
and thus isn't limited to WebSockets, however, when it happens on a server
there is no way to recover safely. The real problem with these is that the
server has no idea that it happened without scanning its own output to see
that it is framed the way it expects (and I don't know of any server which
does this).

> Again, I don't know what you mean by "as insecure as HTTP". Specifically
> what we're worried about here are cross-protocol attacks (both using
> WebSocket and against a WebSocket service). The fact that the handshake
> format is an HTTP request does increase the risk in both directions, but we
> can do things to mitigate it. If you think my proposed defense is
> ineffective, could you explain how?
>

MitM still works just fine even with the hash value.
What is the attack vector that you see mitigated? I may just not be seeing
it.

-=R

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 14:17:57 -0800

> You can still do this as an HTTP header.

Indeed! And that was my original proposal. But I have been advised that the status line could afford stronger protection.

> HTTP header injection problems: If it is possible, you can get interesting things to happen. This true of anything which is attempting to frame HTTP, and thus isn't limited to WebSockets, however, when it happens on a server there is no way to recover safely. The real problem with these is that the server has no idea that it happened without scanning its own output to see that it is framed the way it expects (and I don't know of any server which does this).

My proposal (both the header version and the status line version) should be effective at protecting normal Web servers from WebSocket even if there is a header injection vulnerability; likewise it should still help protect WebSocket servers from HTTP. Even if you can trick the server into injecting a header in its response, you can't predict the nonce, so you can't make it return a correct handshake.

> What is the attack vector that you see mitigated? I may just not be seeing it.

Cross-protocol attacks, either using WebSocket from the browser to attempt to talk to plain HTTP another protocol; or using vanilla HTTP (via cross-site XMLHttpRequest) to attempt to talk to WebSocket. I explained both in detail in an earlier message.

Regards,
Maciej

---

From: Vladimir Katardjiev <[email-masked]>
Date: Wed, 3 Feb 2010 23:51:24 +0100

For reference, linking the message from Maciej with the problem statement. http://www.ietf.org/mail-archive/web/hybi/current/msg01198.html  

On 3 feb 2010, at 23.17, Maciej Stachowiak wrote:

>> 
>> You can still do [the nonce] as an HTTP header.
> 

I can appreciate the logic in having the mandatory response before the request is able to manipulate the output. On the other hand, if the request has anyway successfully manipulated the first bytes of the output to state "HTTP/1.1 101 Web Socket Protocol Handshake", it stands to reason that a couple of more bytes at the end of that won't pose much of a difficulty for an attacker. (Alternatively, the remote host is a WebSocket aware endpoint, at which point you don't have a cross-protocol vulnerability anyway).

In any event, assuming the nonce is unpredictable, the only case in which an unaware server can be tricked into sending back a correct response is if the client is able to make the server compute it. Since this is tantamount to executing arbitrary code on the server, I'd say that's a case that can't be combated, as arbitrary code can always send back the proper response.

For the second case, I don't see at all why this would help on a websocket server that doesn't validate input data... or on a websocket server at all. Since the nonce is opaque to the server, why can't I just send something that pretends to be a nonce from my xhr? It should properly parse, as long as it's [a-zA-Z0-9]+ and the server will happily reply. But it stands to reason if you don't validate your input you're open to attacks. The best you can do is always force the WebSocket server to validate the handshake (which, if I'm not mistaken, you're arguing anyway?)

Vladimir

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 04 Feb 2010 10:03:28 +1100

Vladimir,

I think that the idea is that the attacker is not able to pick the nonce and thus not able to
predict the response.

The attacker would trick some webuser to run their js, it would open a websocket connection
to some URL, but it would be the browser that thinks of the nonce, so even if the
attacker was able to inject arbitrary text into the response, they would not know the
nonce and would not be able to insert the correct hash.   The browser would see the
wrong hash and not complete the handshake.

I still think that this is putting too much responsibility on websocket to not be
usable to exploit an insecure server.   The security problem here is that the server
is injectable, not that websockets can be used to exploit it.    But then the nonce
proposal is really easy to implement, so there is little harm.

I'm also not convinced that if the browser was tricked into opening a websocket
to a HTTP server, that it could send any messages that would look like valid
HTTP requests.    If in doubt, then perhaps a few more fixed bytes on every
WS frame are needed.  If the bytes "WS" were sent before every message, then
nobody could spoof a "GET ..." or a "POST ..." - which I think is impossible
anyway with both length and sentinel framing.

But if anybody from chrome/mozilla/opera wants to experiment with the nonce,
I can have it implemented in the jetty server in about 10 minutes.

cheers

Vladimir Katardjiev wrote:
> ------------------------------------------------------------------------
>

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 03 Feb 2010 15:47:21 -0800

On Feb 3, 2010, at 2:51 PM, Vladimir Katardjiev wrote:

> 
> For the second case, I don't see at all why this would help on a websocket server that doesn't validate input data... or on a websocket server at all. Since the nonce is opaque to the server, why can't I just send something that pretends to be a nonce from my xhr?

XDomainRequest cannot send custom headers, so it cannot send WebSocket-Nonce at all. Cross-site XHR using CORS cannot send custom headers without the server explicitly opting into that header via a preflight request. So it would be unable to send WebSocket-Nonce unless there was a major configuration error that led to replying to preflight with permission to send a WebSocket-Nonce header. We could also blacklist all headers starting with WebSocket- in CORS.

> It should properly parse, as long as it's [a-zA-Z0-9]+ and the server will happily reply. But it stands to reason if you don't validate your input you're open to attacks. The best you can do is always force the WebSocket server to validate the handshake (which, if I'm not mistaken, you're arguing anyway?)

I am indeed arguing that. But part of the reason for the nonce is to protect non-WebSocket servers from being exploited using WebSocket.

Regards,
Maciej