Subject: Re: [hybi] [whatwg] Web sockets and existing HTTP stacks

From: Ian Hickson <[email-masked]>
Date: Thu, 3 Dec 2009 23:57:02 +0000 (UTC)

On Tue, 17 Nov 2009, Christian Biesinger wrote:
> 
> Is it intentional that it is impossible to implement this spec over an 
> existing HTTP stack, as currently specified?

Only to the same extent that it is "intentional" that it's impossible to 
implement Telnet or SSH over an existing HTTP stack.

> In particular, due to the strict requirements on the headers to send, it 
> seems like you can't really implement it with an existing HTTP stack. On 
> the other hand, you really do want to reuse your HTTP implementation, 
> because otherwise you have to reimplement all your HTTP proxy 
> authentication, including NTLM, GSSAPI, etc. Obviously another problem 
> would be that the actual bidirectional communication might be hard to 
> fit into your HTTP implementation, but that might be easier to work 
> around.

Yeah, it's likely that the code reuse theoretically made possible by 
reusing the HTTP proxy and cookie infrastructure is more theoretical than 
practical in many cases. I had hoped that it would be easier.

I guess it might still be possible if the relevant code can be abstracted 
out somehow, but that might result in code that feels a little twisted.

It's not clear what to do about this; we could limit the proxy support to 
just SOCKS, but that seems suboptimal.

(Similar problems exist on the server-side if you try to use an HTTP stack 
to implement WebSocket, though on that side there is far less reason to do 
so, since there's no proxy code to reuse and the cookie code that could be 
reused is minimal and more easily abstractable. The problem only really 
exists in the more advanced case of sharing a port with an HTTP server. 
The complexity there is considered acceptable because few people should 
ever have to write that code, compared to the number of people writing a 
straight WebSocket server.)

HTH,
-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Alexey Proskuryakov <[email-masked]>
Date: Sat, 05 Dec 2009 23:43:50 -0800

> out somehow, but that might result in code that feels a little twisted.

FWIW, we got a quite reasonable amount of code reuse with HTTP cookie and proxy support. This is mostly because our networking layer already had support for establishing streaming connections over proxies, but the point stands, as the actual implementation is shared.

- WBR, Alexey Proskuryakov

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 19 Dec 2009 11:24:50 +1100

> implement Telnet or SSH over an existing HTTP stack.

Telnet and SSH don't use the HTTP port and don't start with a "legal"
HTTP request.

To support websocket in Jetty, we had to use our existing HTTP stack because
we don't know that the request is a websocket upgrade until we have
already accepted it and parsed it using our existing HTTP stack.

> practical in many cases. I had hoped that it would be easier.

With the Jetty Websocket server, we were able to reuse the HTTP stack
for the websocket upgrade.

But we had to put a number of hacks into the base HTTP code to handle
the restrictions of the websocket spec with regards to the ordering
of headers etc.

These restrictions add nothing but pain and complication.

regards

---

From: Justin Erenkrantz <[email-masked]>
Date: Fri, 18 Dec 2009 16:40:02 -0800

> Telnet and SSH don't use the HTTP port and don't start with a "legal"

Exactly.

> already accepted it and parsed it using our existing HTTP stack.

Ditto - this is a critical problem with the current websocket RFC.  I
would place more credence into the argument that websocket RFC has
nothing at all to do with HTTP if it didn't rely upon the HTTP/1.1
Upgrade semantics.  Either it's related, or it's not.  Let's please
not make the mistake of trying to tunnel protocols thru HTTP firewalls
just because you think it's fun.  You only do so because the semantics
are so similar and there's a lot of code reuse that can be done (which
is the appeal to me of BWTP).

On a server, it's not trivial to switch protocol mechanisms halfway
into the request flow.  Witness all the contortions Apache HTTP Server
has to do when SSL renegotiation is enabled and a request is split
across the renegotiation.  Fun!  Not.  For a client, it's a bit easier
to handle as the client *knows* it intends to upgrade.  But, a real
server won't know that.  -- justin

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 19 Dec 2009 00:54:42 +0000

Justin Erenkrantz wrote:
> Let's please not make the mistake of trying to tunnel protocols thru
> HTTP firewalls just because you think it's fun.

Ian has said several times that WebSocket is not intended to tunnel
through HTTP proxies, and that it is intentionally rigid so that it's
likely to fail early when passing through a proxy.

Note that I don't particularly like that goal; I'm just stating what I
think it is.

> You only do so because the semantics are so similar and there's a
> lot of code reuse that can be done (which is the appeal to me of BWTP).

For protocols which don't look like HTTP, you mainly do it because
sometimes TCP port 80 is the only available outgoing port in
firewalled environments.  Ironically, there is often an intercepting
proxy in such environments, so it won't actually work.

Given WebSocket's stated goals (on this list) of:

   - Not working through HTTP proxies, including intercepting proxies.
   - Not being HTTP compatible.
   - Not being correctly parsable by a correct HTTP request parser.

I'm thinking it would be easier to just use port 81 and be done with it.

-- Jamie

---

From: Justin Erenkrantz <[email-masked]>
Date: Fri, 18 Dec 2009 17:11:35 -0800

> I'm thinking it would be easier to just use port 81 and be done with it.

Proxies (mainly reverse proxies) and caching are incredibly important
to any real scalability efforts.  I'm not aware of many high-traffic
web setups that don't rely upon massive amounts of front-ends.  So,
any port 80 traffic to any big sites is very likely to hit an hardware
box (F5, etc.) or a software balancer (a la mod_proxy or varnish or
squid).  If the current WebSockets ID is intended to *fail* in those
situations, the deployment story is likely to be a real non-starter...

I'm curious what the makeup of this hybi@ list/WG is - how many server
or intermediary devs are on this?  From browsing the archives, it
looks like it is mostly browser/user-agent developers so far.  --
justin

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 19 Dec 2009 12:12:30 +1100

> proxy in such environments, so it won't actually work.

In countries with repressive regimes that are against free
speech ( eg. Australia), there will always be a proxy on port
80 making sure that you don't read about religions (eg falun gong)
that offend our major trading partners (eg china) etc.

So websocket is specified to not work at all in Australia.
Of course it probably will work - but not because it is
specified to do so, but because the proxies just happen
to be implemented in certain ways.   Happen stance is
not a great basis for a protocol.

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 19 Dec 2009 05:28:06 +0000

> situations, the deployment story is likely to be a real non-starter...

(Note that I prefer BWTP myself; I'm just describing WebSocket).

It fails early because the alternative, for a pure HTTP proxy, is
unpredictable behaviour when it sees bidirectional messages later.
Failing early is a good thing!

It won't fail if the proxy explicitly supports WebSocket, so adding
that support is the intended solution, I believe.

A pure HTTP proxy cannot carry any of the protocols under discussion,
except for those designed to look like normal HTTP unidirectional
request-response pairs ("polling", "long GET" and "streaming GET"),
which are already widely implemented.

If you are looking for an efficient bidirectional messaging protocol
which works with pure HTTP proxies, not modified to support the
bidirectional protocol, what do you suggest?

> looks like it is mostly browser/user-agent developers so far.

I'd say there are many server and application developers on the list
(including me), but the WebSocket protocol appears to be more driven
from the browser API side.  It is explicitly designed to support a
Javascript API, and pays little attention to issues like network
performance and scalability.

-- Jamie

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 22 Dec 2009 12:46:02 +1100

There was a fairly long thread initiated by Ian on squid-dev, FYI:
  http://marc.info/?t=124700786100001&r=1&w=2

That discussion is very similar to this one.

Cheers,

On 19/12/2009, at 12:11 PM, Justin Erenkrantz wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

--
Mark Nottingham     http://www.mnot.net/

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 22 Dec 2009 13:22:20 +1100

> That discussion is very similar to this one.

This is looking more an more like a hostile takeover
bid of port 80.

Whenever the objection "it breaks my existing HTTP
stuff" is raised, the response appears that the HTTP
infrastructure should stop trying the "share" the port
with websocket.

Does ANYBODY else other than Ian think that the
websocket upgrade request/response should be anything
other than a semantically legal HTTP upgrade
request/response?

regards

---

From: Ian Hickson <[email-masked]>
Date: Tue, 22 Dec 2009 02:35:16 +0000 (UTC)

On Tue, 22 Dec 2009, Greg Wilkins wrote:
> This is looking more an more like a hostile takeover bid of port 80.

Just so we're clear, Web Sockets (then TCPConnection) was originally 
completely independent of HTTP. It was then requested that the handshake 
be made HTTP-compatible such that an HTTP server could share its port with 
a Web Sockets server, so that port 443 could be used (needed to get around 
man-in-the-middle proxies). Even then, though, the Web Sockets spec 
originally specifically discouraged the use of port 443 and recommended 
the use of ports 81 and 815. It was only in response to feedback 
suggesting that discouraging port 443 was a bad idea that the spec was 
changed to be neutral on the matter. Then I tried registering ports 81 and 
815 and the IANA told me to just use ports 80 and 443.

So describing this as a "hostile takeover bid" is rather revisionist. It's 
only through three years of responding to feedback that we've reached this 
point, and I've done my best to ensure that the handshake really is 
compatible with HTTP to the extent necessary to handle the use cases that 
people have brought to the table.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 22 Dec 2009 19:32:18 +1100

Ian Hickson wrote:
> So describing this as a "hostile takeover bid" is rather revisionist.

OK fair cop - a bit of an exaggeration.

But now that you have been encouraged to use port 80, then you need
to play by the existing rules on port 80 - up until the time the
upgrade response is sent.   After that, you can go crazy with the
cheeze-whiz on new protocol and any intermediary that does pass
the upgrade, but does not handle the subsequent traffic is in the wrong.

But prior to the upgrade response, you are in HTTP and should be
governed by RFC2616 and not just pretend HTTP.

> It's 
> compatible with HTTP 

Why "compatible with HTTP"?   Why not just be real HTTP prior to the
upgrade?

> to the extent necessary to handle the use cases that

This list and the squid list and other forums are full of people
raising use-cases that are not supported.    This thread was
started by a post about a C HTTP stack that had difficulty
with the "compatible HTTP", I've posted similar issues with
the Jetty implementation.   The squid list shows how
a caching proxy is having problems with just "compatibility"
and there have been plenty of other use-cases listed
about load balancers, SSL offload, etc. etc.

You have been entirely unsuccessful in convincing anybody
else of the validity of your assertion that binary specified
upgrade requests/responses increases security or serves
any purpose at all.  I've not seen 1 word of support for
that approach from anybody else.

So I'm sorry about the hostile take over bid comment, but
it just reflects my frustration at your intransigence on
this issue.    When existing HTTP infrastructure on port
80 is incompatible with your new protocol, your response
is that we should not use that infrastructure on port 80!

But that is simply not an option.

regards

---

From: Pieter Hintjens <[email-masked]>
Date: Fri, 25 Dec 2009 13:12:39 +0100

>> 815 and the IANA told me to just use ports 80 and 443.

You won't get ports in this range in 2009, easily.  If you are going
to use ports that are governed by existing RFCs you MUST obey those
RFCs.  HTTP is not special in this respect.

> governed by RFC2616 and not just pretend HTTP.

Do we really need to say this?

-Pieter