Subject: [hybi] Fwd: Web sockets and existing HTTP stacks

From: Christian Biesinger <[email-masked]>
Date: Fri, 20 Nov 2009 16:08:41 +0100

[resent from [email-masked], with slight additions here]

Hi,

so I was reading the web sockets spec as well as a Firefox patch to
implement it.

Is it intentional that it is impossible to implement this spec over an
existing HTTP stack, as currently specified? In particular, due to the
strict requirements on the headers to send, it seems like you can't
really implement it with an existing HTTP stack. On the other hand,
you really do want to reuse your HTTP implementation, because
otherwise you have to reimplement all your HTTP proxy authentication,
including NTLM, GSSAPI, etc. Obviously another problem would be that
the actual bidirectional communication might be hard to fit into your
HTTP implementation, but that might be easier to work around.

I'll note that the implementation on Chromium trunk skips
authentication completely at the moment
(http://src.chromium.org/viewvc/chrome/trunk/src/net/websockets/websocket.cc?view=markup,
search for 407)

Do you as the spec authors have any thoughts on this?

-christian

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 20 Nov 2009 16:04:23 +0000

Christian Biesinger wrote:
> really implement it with an existing HTTP stack.
> On the other hand, you really do want to reuse your HTTP
> implementation, because otherwise you have to reimplement all your
> HTTP proxy authentication, including NTLM, GSSAPI, etc. Obviously
> another problem would be that the actual bidirectional communication
> might be hard to fit into your HTTP implementation, but that might
> be easier to work around.

This has been the cause of quite some discussion on this list.

The main spec editor believes WebSocket should intentionally fail when
encountering a proxy etc. and the web application should fall back to
some other method in that case, or simply not support those cases.

Needless to say, several people take the opposite view, that it should
work over general HTTP infrastructure, but as that would make the
protocol more complicated and simplicity is also a goal, there is no
consensus.

Originally WebSocket was just a wrapper around raw TCP sockets, like
you have in Flash and Java, which is where it gets this from.

-- Jamie

---

From: Christian Biesinger <[email-masked]>
Date: Fri, 20 Nov 2009 17:49:55 +0100

> some other method in that case, or simply not support those cases.

Well, I'd support that view since it simplifies implementation a lot
:) But a lot of people are using proxies... And the spec in its
current form does require dealing with proxies.

-christian

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 21 Nov 2009 11:26:32 +1100

> protocol more complicated and simplicity is also a goal, there is no

All,

As an example of the problems resulting from this spec, I had some
hassle implementing websocket support within the Jetty codebase.

When processing headers of a response, jetty mostly encodes them directly
into the output buffer as it iterates over them.  However a few headers
are held back until all the other headers have been processed as
the values of other headers might influence the values imposed on
transport headers.    For example, the lack of a content-length
header might mean that a "close" value needs to be added to the
"Connection" header.    Thus Jetty delays generation of the connection
header until all the other headers have been processed.

This means the for Jetty responses, the connection header is normally
one of the last headers generated.

Unfortunately this breaks the websocket specification, which insists
on the Connection header being immediately after the Upgrade header
and before the other WebSocket- headers

I thus have had to put a special purpose hack in place deep in the
HTTP generation code for jetty - to not delay Connection header handling
if the response looks like a websocket upgrade response.  Alternately
I could refactor the generator to do two passes over the headers, or
use a special purpose generator just for this response.

However, it is my expectation that such special purpose hacks or
refactoring of existing working code will result in far more breakage
and security risks than any possible gain resulting from a strict
adherence to header ordering.

Existing HTTP code allows great flexibility in handling of headers
and most code will be written with this assumption.

Is there any consensus at all that strict header equivalence is
a reasonable thing for this specification to have and is worth
breaking the vast majority of existing HTTP code?

regards

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 21 Nov 2009 01:15:55 +0000

Greg Wilkins wrote:
> into the output buffer as it iterates over them.

Why are you using HTTP code to implement WebSocket?

> and before the other WebSocket- headers

WebSocket specifies more than that.  It specifies the exact sequence
of octets to send.  So why hack around HTTP header sending code, just
to send a fixed sequence of octets?  Forcing the order isn't enough:
you have to ensure the upper/lower case choices are exactly right too.
Of course most HTTP header senders will pass through what you ask from
the application (but not all).

> breaking the vast majority of existing HTTP code?

I suspect if it were put to a representative vote of informed
citizens, there'd be a near consensus _against_ the strict
octet-sequence specification :-)

But let's not pretend it 'breaks' existing HTTP code.

WebSocket, in it's current form, has virtually no connection with HTTP
except the initial octet sequence can be mistaken for a HTTP request,
so (imho) it's a mistake to try to repurpose HTTP code at all for it.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 21 Nov 2009 18:41:04 +1100

> Why are you using HTTP code to implement WebSocket?

Because I have a HTTP server that is listening on port 80
and until I read the request and interpret it, we have no
idea if the request is going to be a websocket upgrade or not.

By the time we do know that it is going to be a HTTP upgrade
to websocket we are deep inside our HTTPConnection handling.

We implement support for the upgrade by sending a "normal"
response from our HTTPConnection and then switching the
socket endpoint over to a WebSocketConnection for ongoing
processing.

> the application (but not all).

Because either I have to hack my way out of our whole HTTP handling
or I have to hack our HTTP handling to send a well ordered response.

Also we already have existing code that does authentication and origin
handling, which I believe can be applied to the upgrade request, so we
don't want to have to replicate that code in the websocket connection
handling.

> But let's not pretend it 'breaks' existing HTTP code.

I'm not pretending.

This is real life code used by literally millions already for
server push.  I'm upgrading the server so that it can take
advantage of websocket that is being rolled out in browsers
now.

The header criteria have caused me real additional complications.
I'm not looking for problems - because as much as I dislike the
protocol it is being deployed so we have to work with it and
I would prefer not to have to put special case handling in my
HTTP code just to hand over control to new websocket code.

> so (imho) it's a mistake to try to repurpose HTTP code at all for it.

I'm only trying to "repurpose" the HTTP code that is already
listening on port 80  and handling the HTTP request that initiates
the upgrade. Once the upgrade is processed, the connection will be
handled by entirely new code (albeit sharing an async IO
infrastructure).

It is precisely to have good separation of concerns that I
want our HttpConnection code to send the 101 response and deal
with any HTTP headers, and for the  new WebSocketConnection code
to deal only with WebSocketFraming and not sending pseudo HTTP
responses.

If this was just a standalone websocket server, I would not
start from a HTTP server code base.   But port 80 is already
owned by HTTP, so websocket on port 80 is going to meet existing
HTTP code.

Modifying that code to pass the connection to new websocket code is
made more difficult by the restrictions put on the 101 response.

regards