Subject: [hybi] #4: handshake does not work properly with HTTP reverse proxy.

From: "hybi issue tracker" <[email-masked]>
Date: Thu, 08 Jul 2010 17:06:09 -0000

#4: handshake does not work properly with HTTP reverse proxy.
-------------------------------------------+--------------------------------
 Reporter:  salvatore.loreto@…             |       Owner:     
     Type:  defect                         |      Status:  new
 Priority:  critical                       |   Milestone:     
Component:  thewebsocketprotocol           |     Version:     
 Severity:  Active WG Document             |    Keywords:     
-------------------------------------------+--------------------------------
 the  handshake in draft-ietf-hybi-thewebsocketprotocol-00 (as in draft 76)
 does not work properly in presence of HTTP reverse proxy.

 The 8-bytes nonce from the client is not advertised as a content-length,
 so it is not forwarded by the reverse proxy as it is either part of a next
 request or pending data for when the handshake completes. Unfortunately,
 the server wants those bytes to complete the handshake, so we have a dirty
 deadlock not even detectable by the end user.

 for more information:
 http://www.ietf.org/mail-archive/web/hybi/current/msg02149.html

-- 
Ticket URL: <http://trac.tools.ietf.org/wg/hybi/trac/ticket/4>
hybi <http://tools.ietf.org/hybi/>
The Hypertext-Bidirectional (HyBi) working group will seek
standardization of one approach to maintain bidirectional
communications between the HTTP client, server and intermediate
entities, which will provide more efficiency compared to the current
use of hanging requests.

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 17 Jul 2010 03:37:49 +0100

>From discussions on the hybi list several months ago, several times, I
got the impression that failure to work through any kind of HTTP proxy
was *intentional* in the handshake design.

I don't personally agree with that as a design goal, but if it is the
design goal than this issue #4 isn't a bug.

If the goal is to work through proxies, there are some other issues
that also need to be addressed, in particular what are the forwarding
requirements that a proxy must satisfy after handshake is complete -
partial frames delayed indefinitely, every byte forwarded eagerly, or
something else?

-- Jamie

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 17 Jul 2010 08:15:57 +0200

On Sat, Jul 17, 2010 at 03:37:49AM +0100, Jamie Lokier wrote:
> >  Reporter:  salvatore.loreto@???             |       Owner:     

If that goal really is to be supported, then the initial HTTP-like
handshake is overly complicated and expensive, and will serve no
purpose, as nobody will be able to use the protocol that way on shared
HTTP ports. Then let's go on with the framing protocol alone without the
fake initial handshake and let's have another draft later suggest how to
use WS over HTTP as a standard protocol Upgrade request.

Regards,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 20 Jul 2010 10:28:25 +1000

>From those same discussion, I concluded that the intent of this
"feature" was to fast fail in the presence of any kind of HTTP proxy.
Given that this "feature" is now resulting in a hang if there is such
a proxy, then I think it is still a bug whatever way you look at it.

More importantly, the best way to test if a connection will support
websocket packets, is to just send websocket packets.   Sending 8/16
random bytes immediately after the HTTP handshake may succeed for a
huge variety of implementation reasons that still do not mean that
websocket packets will succeed.   More importantly, to detect the
non-success of sending of the 8/16 random bytes will still involve the
same timeout handling that would be required to check the transmission
of websocket packets.

Even successfully sending websocket packets is not a fool proof way of
predicting future success of websocket packets, thus we already have
started discussions about keep alives and other mechanisms to check
the health of a connection.   The checking of the initial health of
the connection should really just be an edge case for these mechanism
and not some special purpose exchange pseudo secure algorithms.

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 20 Jul 2010 06:43:52 +0200

> "feature" was to fast fail in the presence of any kind of HTTP proxy.

It was to fail in the presence of any kind of WebSocket-incompatible
HTTP transparent proxies, to give opportunity to spot them and upgrade
them to support the protocol. But now the requirements are not compatible
with HTTP, meaning that those proxies or reverse proxies will never be
able to be upgraded.

> websocket packets will succeed.

You make a good point here, because very old versions of haproxy would
consider that whatever was sent after headers was data and could be
immediately exchanged (there was no difference between GET and POST),
so that in practice that would have worked, despite the brokenness of
the product.

Also, I would really insist that we add a "Connection: close" in the
initial HTTP handshake, whatever it looks like in the end, so that we
protect the servers against any form of request injection after the
first one.

Regards,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 20 Jul 2010 16:40:33 +1000

> protect the servers against any form of request injection after the

I think that if we fail to come up with a HTTP compliant handshake, then a
Connection:close would be a good get out of goal free card.

However, it does prevent one possible good use case of using the HTTP
connection as a HTTP connection in the case that websocket is rejected.
Until websocket is widely deployed, many sites will want to fall back to a
HTTP transport if websockets is rejected.  If the connection can be kept
open, then it may be used by the browser for subsequent HTTP requests,
saving a round trip.  If a failed handshake response can contain other useful
content, then you may be able to save 2 round trips in the establishment of
HTTP transports.

This is already an issue with cometd-2, now that we support websocket as
an optional transport.  When it is enabled, there is noticeable additional
establishing a comet web page (eg logging into a chat room), while websocket
hand shake is tried and fails, before the next handshake is tried.   This is a
discouragement to enabling websocket support in the framework, which is
then does not encourage infrastructure providers to support websocket.

regards

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 20 Jul 2010 11:06:32 +0200

> then does not encourage infrastructure providers to support websocket.

Agreed. Also, keeping the connection open is required with some authentication
schemes such as NTLM which authenticate the connection instead of the request.
While this will not be a problem on the Internet, it could make it difficult
to integrate websocket into enterprise applications. This is another reason not
to send any unadvertised data in the handshake by the way !

Regards,
Willy

---

From: Mike Belshe <[email-masked]>
Date: Tue, 20 Jul 2010 11:38:49 -0700

> to send any unadvertised data in the handshake by the way !

+1.

>
> Regards,
> Willy
>
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: "hybi issue tracker" <[email-masked]>
Date: Wed, 21 Jul 2010 21:49:24 -0000

Changes (by ian@…):

 * cc: ian@… (added)

Comment:

 This is a new protocol, so network infrastructure naturally needs to be
 updated to handle it, as discussed in this reply:
    http://www.ietf.org/mail-archive/web/hybi/current/msg02274.html

 No change to the protocol is proposed to resolve this; it is easily
 resolved by implementing Web Socket support in the proxy.

-- 
Ticket URL: <http://tools.ietf.org/wg/hybi/trac/ticket/4#comment:1>

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 21 Jul 2010 23:22:10 +0100

> protect the servers against any form of request injection after the

I agree, that's good advice.

-- Jamie

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 00:28:51 +0200

On Wed, Jul 21, 2010 at 09:49:24PM -0000, hybi issue tracker wrote:
> Changes (by ian@???):
>  * cc: ian@??? (added)

No, as explained in many other messages, it CANNOT be updated and
remain HTTP-compatible at the same time. The protocol needs be fixed.

Willy

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 21 Jul 2010 23:30:10 +0100

> a proxy, then I think it is still a bug whatever way you look at it.

I agree with something implied here: fast failing is more desirable
than slow failing, where it's realistically feasible to trigger fast
failing without causing problems.  It can't be guaranteed, and there's
no point overcomplicating it, but it's useful to try.

> same timeout handling that would be required to check the transmission

The argument against just sending WebSocket packets is that some
WebSocket packets may be valid HTTP and therefore more likely to be
misinterpreted or be used as another attack vector, or simply fail to
fail fast.

That's not possible with the current WebSocket frame types, but it
could happen with frame types from the currently reserved range.

That's also an argument against making the bytes uniformly random;
instead they should also be constrained to avoid looking like HTTP.

-- Jamie

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 00:39:47 +0200

> misinterpreted or be used as another attack vector, or simply fail to

Indeed, but even the first 8 random bytes can currently be used as a
valid HTTP request or beginning of such, as some servers still accept
HTTP/0.9 requests after a keep-alive request, and basically anything
terminated with a line feed may constitude an HTTP/0.9 request. Also,
the line feed is not even required, you can just send 8 normal bytes
which will prefix the method of the next request.

So in fact, relying on the server's 101 response to the Upgrade request
is the most reliable thing to do, as it translates the server's intent
to switch and to respect the negociated protocol.

Someone suggested that regularly sending keep-alives in the connection
would be nice. I second this, especially with long-lived connections
that will regularly get dropped on firewalls after too long idleness.
This would also make it possible to ensure that the connection is OK.

Regards,
Willy

---

From: "hybi issue tracker" <[email-masked]>
Date: Thu, 22 Jul 2010 13:29:49 -0000

Comment(by salvatore.loreto@…):

 No, as explained in http://www.ietf.org/mail-
 archive/web/hybi/current/msg02276.html,
 it CANNOT be updated and remain HTTP-compatible at the same time.

 The protocol needs to be fixed!

-- 
Ticket URL: <http://tools.ietf.org/wg/hybi/trac/ticket/4#comment:2>

---

From: Ian Hickson <[email-masked]>
Date: Thu, 22 Jul 2010 20:44:12 +0000 (UTC)

On Thu, 22 Jul 2010, Willy Tarreau wrote:
> 
> Ian, I don't know how to explain it to you now. I've exhausted every bit 
> of possibility a normal human is able to understand, so I think that you 
> are deliberately acting to refuse to understand the facts :-(
> 
> I've told you *several* times that it's not a matter of "updating" 
> server- side components, but that your cross-dressed protocol will not 
> be mergeable with HTTP on reverse-proxies. So even if the 
> reverse-proxies are "updated" to use your terms, then they will have to 
> be either configured to support WebSocket *OR* configured to support 
> HTTP, but not both on the same IP:port couple.

I've already explained several ways that this could be done. Saying that 
it simply can't be done seems quite fatalistic.

> > > The first obvious solution that comes to mind is to comply with the 
> > > HTTP protocol which will be implemented along the whole chain and to 
> > > simply add a "Content-Length: 8" header in the request.
> > 
> > As far as I can tell there is nothing here that contradicts the HTTP 
> > spec. If there is a specific requirment in the HTTP spec that is being 
> > contradicted, please cite it.
> 
> I really think you're trying to make all of us waste our time while 
> we're trying to help you release something which works instead of it 
> becoming a major failure that will be attempted to be used for 6 months 
> then abandonned due to massive failures everywhere. That's a real pity. 
> So here it comes, from RFC2616 (and this part remained unchanged in 
> http-bis-p1-10) :
> 
> 4.4 Message Length
> ...
>   HTTP/1.1 requests containing a message-body MUST include a valid Content-Length
>   header field unless the server is known to be HTTP/1.1 compliant. If a request
>   contains a message-body and a Content-Length is not given, the server SHOULD
>   respond with 400 (bad request) if it cannot determine the length of the message,
>   or with 411 (length required) if it wishes to insist on receiving a valid
>   Content-Length.
> 
>   All HTTP/1.1 applications that receive entities MUST accept the "chunked"
>   transfer-coding (section 3.6), thus allowing this mechanism to be used for
>   messages when the message length cannot be determined in advance.
> 
>   Messages MUST NOT include both a Content-Length header field and a non-identity
>   transfer-coding. If the message does include a non-identity transfer-coding,
>   the Content-Length MUST be ignored.
> 
>   When a Content-Length is given in a message where a message-body is allowed,
>   its field value MUST exactly match the number of OCTETs in the message-body.
>   HTTP/1.1 user agents MUST notify the user when an invalid length is received
>   and detected.

Here are the requirements given in the above four paragraphs:

R1: HTTP/1.1 requests containing a message-body MUST include a valid 
    Content-Length header field unless the server is known to be HTTP/1.1 
    compliant.

R2: If a request contains a message-body and a Content-Length is not 
    given, the server SHOULD respond with 400 (bad request) if it cannot 
    determine the length of the message, or with 411 (length required) if 
    it wishes to insist on receiving a valid Content-Length.

R3: All HTTP/1.1 applications that receive entities MUST accept the 
    "chunked" transfer-coding

R4: Messages MUST NOT include both a Content-Length header field and a 
    non-identity transfer-coding.

R5: If the message does include a non-identity transfer-coding, the 
    Content-Length MUST be ignored.

R6: When a Content-Length is given in a message where a message-body is 
    allowed, its field value MUST exactly match the number of OCTETs in 
    the message-body.

R7: HTTP/1.1 user agents MUST notify the user when an invalid length is 
    received and detected.

Of these, R2 applies only to servers and can be ignored. R3 and R7 apply 
to receiving messages and thus aren't relevant here. R4 and R5 are 
irrelevant since there's no trasfer-coding included in the message. R1 is 
irrelevant since we know the server is an HTTP/1.1 server (if it isn't, 
it's either a WebSocket server, in which case the HTTP spec is irrelevant, 
or a server that doesn't support WebSockets at all, in which case it 
doesn't matter what we do since we're just going to terminate the 
connection ASAP anyway). (R1 would also be irrelevant since there's no 
message body, the 8 bytes are part of the WebSocket data, not part of the 
HTTP data.) Finally R6 is irrelevant because there's no Content-Length 
given. So again, I ask, if there is a specific requirment in the HTTP spec 
that is being contradicted, please cite it. Ideally specifically the 
requirement you think is being violated and not an entire section.

> In short, there is a message body, so either you advertise it using 
> Content-Length or you advertise it using Transfer-Encoding: chunked, 
> though the later requires that you're certain that the server supports 
> HTTP/1.1 which is not necessarily the case.

There's no message body in this case. The 8 bytes are part of the Web 
Socket connection, not part of the HTTP connection. They couldn't be part 
of the HTTP connection; if they were, their entire purpose would be made 
moot since man-in-the-middle proxies would simply pass the data along with 
the original request and thus give a false-positive result.

> > We can't add Content-Length: 8, since that would mean the data would 
> > be sent through with the first request even in non-WebSocket-aware 
> > man-in- the-middle proxies, which defeats the point.
> 
> No, this is *what you need*. You need any HTTP-compliant component to 
> reliably deliver this request because HTTP is the medium over which 
> WebSocket will be used, like it or not. What you're currently doing is 
> ensuring that HTTP-compliant software that are already working correctly 
> everywhere will not be able to pass WebSocket requests to their peers, 

The entire point here is to make man-in-the-middle proxies _not_ forward 
this data. What you are proposing defeats the entire point of this part of 
the handshake.

> which will result in the protocol never being used beyond what it's 
> always been since the beginning : tests between your local browser and 
> your local hand-written server.

Given that it's already being used, I don't see this as being a realistic 
concern.

> No, the handshake does not work because the server does never get the 
> first 8 bytes so it does not respond. I would have no problem if those 8 
> bytes were not required before the 101 response, but it happens the 
> server needs them before responding, which is causing the 
> chicken-and-egg problem :
> 
>   client : hey, I'm sending you my handshake, and don't care about my extra bytes
>   rev-proxy: hey server, I'm sending you a handshake, and if you reply with 101,
>              I will send you the extra bytes
>   server : I won't complete my handshake until you send me those bytes.

The reverse proxy and the server in this scenario are both implementing 
the Web Socket specification's concept of "server". If they don't do it 
correctly, then they're not conforming to the specification. There is 
nothing that stops the "server" part in your component from sending the 
101 line to the reverse proxy before it gets the last 8 bytes of the 
handshake. This is an artificial requirement you are placing on your 
setup. It would be like saying "you server send the character 'W', 
therefore the handshake can't be implemented".

> As long as the server has not responded with 101, it *IS* HTTP.

I disagree with the premise of this statement.

> > All of these problems come from thinking of Web Sockets as a 
> > subprotocol of HTTP. It isn't. Web Sockets is its own high-level 
> > protocol built on top of TCP. It just happens to look enough like HTTP 
> > that you can reuse the port, but that doesn't mean it's an HTTP-based 
> > protocol. Thinking of Web Sockets as having anything to do with HTTP 
> > is a mistake.
> 
> Please stop denying the initial goals, you know it won't be used it you 
> can't share the port, and it's even still written in draft-76 :
> 
>    When a connection is to be made to a port that is shared by an HTTP
>    server (a situation that is quite likely to occur with traffic to
>    ports 80 and 443), the connection will appear to the HTTP server to
>    be a regular GET request with an Upgrade offer.  In relatively simple
>    setups with just one IP address and a single server for all traffic
>    to a single hostname, this might allow a practical way for systems
>    based on the WebSocket protocol to be deployed. 

Just because you can share the port doesn't mean it's the same protocol. I 
can point my SSH client to an SSH server listening on port 80 also, that 
doesn't make SSH into HTTP.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 23:38:21 +0200

> it simply can't be done seems quite fatalistic.

It's not fatalistic, it's just that you seem to be refusing to admit that
what is written in draft 76 is causing that to happen. As we said in several
earlier exchanges, if you slightly change the spec so that the server does
not have to wait for the 8 additional bytes in order to respond, the problem
is gone. But right now the HTTP reverse proxy must not send those bytes until
to sees the server's response and the WS server must not respond until it
gets those bytes.

(... rules ...)
> So again, I ask, if there is a specific requirment in the HTTP spec 

I'm happy, you've demonstrated that there is no body, so those bytes
MUST not be forwarded to the server as they're not part of the message.
So you now agree that your repeated suggestion of "just upgrading the
reverse proxy to make it support websocket and forward those bytes to
the server" is not possible.

> the original request and thus give a false-positive result.

But that's what I'm stating since the beginning !!! Those bytes will not
be forwarded because they're not part of the message. So either you make
them part of the message by specifying a content-length OR you make the
server respond without waiting for those bytes so that once the handshake
completes, they're correctly forwarded.

> this data. What you are proposing defeats the entire point of this part of 

I understood, so indeed, let's make the server respond without waiting for
them ! That was proposed several times, the only drawback is that it adds
round trips. In my opinion, round-trips between a reverse-proxy and a server
are not much big trouble because generally they're on the same LAN.

> Given that it's already being used, I don't see this as being a realistic 

It's being tested. And people already report issues, otherwise I would not
have joined the list to try explain them.

> therefore the handshake can't be implemented".

No I don't agree that *I* am placing this requirement. *Your* spec is
worded so that the request handshake is understood as a single-piece
message and the response handshake is as well. Example :

   The concatenation of the number obtained from processing the |Sec-
   WebSocket-Key1| field, expressed as a big-endian 32 bit number, the
   number obtained from processing the |Sec-WebSocket-Key2| field, again
   expressed as a big-endian 32 bit number, and finally the eight bytes
   at the end of the handshake, form a 128 bit string whose MD5 sum is
   then used by the server to prove that it read the handshake.

...

   The server can also set cookie-related option fields to _set_
   cookies, as in HTTP.

   After the fields, the server sends the aforementioned MD5 sum, a 16
   byte (128 bit) value, shown here as if interpreted as ASCII:

        fQJ,fN/4F4!~K~MH

   This value depends on what the client sends, as described above.  If
   it doesn't match what the client is expecting, the client would
   disconnect.

   Having part of the handshake appear after the fields ensures that
   both the server and the client verify that the connection is not
   being interrupted by an HTTP intermediary such as a man-in-the-middle
   cache or proxy.

And if you're telling me that it's not explicitly stated that it's a
single piece message, I'd like to know where it's indicated what the
server must wait for before sending each line of response.

Now please add to your handshake description that "the server may have
to send all the fields in order to get the 8 bytes from the request
that are needed to compute the MD5 sum", and it will suddenly work
better.

> I disagree with the premise of this statement.

I know.

> Just because you can share the port doesn't mean it's the same protocol.

No, but they must be reliably distinguishable enough to leverage any
intentional or accidental ambiguity. Otherwise one has to be backwards-
compatible with the other one.

There are many examples of shared ports between HTTP and HTTPS. There's
no ambiguity between those, and any routing decision error will be detected
wherever it happens without dangerous side effects.

> I can point my SSH client to an SSH server listening on port 80 also, that 

No, but it least it will not pretend to be the other one either.

BTW, I'd like to ask you a simple question, if the handshake is not HTTP.
How do you recognize an HTTP request when you get an incoming connection
on a port with some data ? Personally, when I see a line with a method
matching a well-defined one in the HTTP protocol, followed by exactly one
space, followed by a URI, followed by exactly one space, followed by the
word "HTTP" followed by a "/" followed by two integers delimited with a
dot, then I can say for sure that it's an HTTP/X.Y request signature.

In other words, what in the request handshake can be used to know that
this is *NOT* HTTP ? If the whole handshake 100% matches a valid HTTP
request, there is no way to know it is in fact not. The "Upgrade:"
header is a perfectly valid and well-documented HTTP header which says
that once the server returns 101, then we're switching to the specified
protocol. So this cannot be used to distinguish between your imitation
of HTTP and a valid HTTP request. All other headers cannot be used as
such either because they could very well be used by any private
implementation. So either you admit that it's HTTP at the beginning, or
you must explain how to unambiguously distinguish between that imitation
and a real HTTP request that would make use of the same fields.

Thanks,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 26 Jul 2010 11:05:57 +1000

This is the road block that we have come up against for month after month
after month.

I think we have to assume that you are unconvincing that the handshake MUST
be HTTP compliant before the 101. But can you explain why you think that it
MUST be HTTP non-compliant before the 101?

Why can't we just make it HTTP compliant before the 101 so it will get us
past this road block.  What are the problems that result from doing this?
Surely you can give us the chance to address them by other means than by
being HTTP non-compliant?

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 26 Jul 2010 22:26:30 +0100

Willy Tarreau wrote:
> terminated with a line feed may constitude an HTTP/0.9 request.

According to RFC1945, a HTTP/0.9 request begins with "GET ".

> This would also make it possible to ensure that the connection is OK.

Many people have suggested it.
I see three basic purposes for keepalives:

   - To tell the application at one end that it is up to date with
     state from the other end, with a certain amount of slack (the
     keepalive interval).  For example, a site intending to show live
     auction prices might display "information may be out of date" if
     it hasn't had a confirmation in the last 20 seconds.

   - To keep the connection open with numerous types of proxy, router,
     firewall etc.

   - To tell the server not to close the connection.  Servers must
     close connections after a timeout to avoid running out of resources.

Each of these has different timing requirements.

With WebSocket it's possible for an application to perform all the
necessary keepalives.  Strictly speaking, there is no need for them in
WebSocket itself.  However there are potential benefits to generic
assistance.  Applications might not make efficient choices for the
network context; it will be impossible for aggregators to combine
keepalives to reduce traffic because they cannot interfere with
unlabelled application messages, and some applications might (annoying
to some users) choose intervals which are too large for some networks,
or too small for others, and which drain power excessively and continuously.

That said, those are difficult to solve at any level, and it's not
immediately clear that WebSocket implementations would do a
particularly good job either.

-- Jamie