Subject: [hybi] Proposal for a clean way to detect non-HTTP compliant transparent proxies

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 00:52:10 +0200

I've just thought about a way to fix the initial handshake so that it
works with proper HTTP-compliant implementations but still fails through
non-compliant transparent proxies.

Let's have the client send the initial handshake request to the server.
This handshake is only made of HTTP headers. The client completes the
request with "Connection: close" so that a transparent intermediate
does not wait for a possible second request in case it does not understand
the upgrade response :

   GET /xxxx HTTP/1.1
   Upgrade: WebSocket
   Connection: close
   ... ws headers ...
   ...
   \r\n

The server will send :

   HTTP/1.1 101 Swithing protocols
   Upgrade: WebSocket
   Content-length: 0
   Connection: close
   ... ws headers ...
   ...
   \r\n
   any-form-of-hello-message (eg: the server's protocol version)

That way, if a transparent proxy does not have explicit support for 101,
it will only forward the headers with an empty body (CL: 0) and close the
connection towards the client.

The client then gets an empty response without the in-protocol HELLO
message and knows that an intermediate is in the way and is not compatible.

The only way for such a response to pass unmolested to the client is
for intermediates to have a proper understanding of the 101/Upgrade
mechanism which implies that any data after the empty line is to be
tunnelled, despite Content-length: 0.

It also has the nice advantage of being compatible with HTTP, which
means that ISPs making use of transparent proxies will have no problem
adding reliable cache-bypass rule which relies on the Upgrade: header.

Any thoughts ?

Willy

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 22 Jul 2010 09:23:36 +1000

Willy,

if we insist on trying to fail on infrastructure that might work (even if it
doesn't understand
websocket), then I think this is indeed an good proposal.

But I fundamentally do not agree with the premise that we should try to fail
simply
because our infrastructure is working, but not the way we want it to work.

I don't see what this buys us - say we do establish that every bit of
infrastructure between UA and server has been replaced with something shiny
and new that understands websocket - does this mean that we no longer need
to monitor the health of the connection and just assume that everything will
magically work?

No!

Networks can and do fail - even shiny new ones that understand websocket.
And they can fail in all sorts of nasty ways - black holes, repeated frames,
horrible latencies etc etc.

No matter what, any reasonable websocket based application is going to need
something to monitor the health of the connection and to make a continual
assessment of "it's working" vs "it's not working".     Even if we could
knowing that all intermediaries are websocket aware does not help this in
any way, shape or form.  And we can't know if all intermediaries are
websocket aware: your suggestion will indeed detect a class of non-ws
intermediaries, but there will also be other ones that are semi-HTTP away
but don't respect Connection:close or that operate at TCP/IP level etc.

Currently the protocol is designed to try to fail and only work if
everything is perfect.
I think we should instead make it try to work and fail only if there is a
problem.

regards

On 22 July 2010 08:52, Willy Tarreau <[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 01:46:54 +0200

Hi Greg,

On Thu, Jul 22, 2010 at 09:23:36AM +1000, Greg Wilkins wrote:
> doesn't understand websocket), then I think this is indeed an good proposal.

Please note that I'm deliberately checking if the connection is
HTTP-compliant enough so that we can use it for WS, I'm not trying
to see if there are WS-specific crap implemented half-way. Assuming
that we finally fix the handshake to be HTTP-compliant, we just need
from the intermediates to support Upgrade+101, nothing else.

> because our infrastructure is working, but not the way we want it to work.

Indeed, I don't either and that's what I hate in the current handshake :
standard-compliant components fail while the most crappy and insecure
devices will still work.

> to monitor the health of the connection and just assume that everything will

No, that's not what I'm saying at all. I'm making a big distinction between
detecting compatibility during handshake and detecting end-to-end link
reliability during communications, which should involve regular keep-alives.

> horrible latencies etc etc.

... and silent timeouts !

100% agreed, and that's my every day job to spot such issues too. That's
why I'm a lot in favor of supporting some keep-alives or "ping" messages.
In fact all my SSH clients and servers have such keep-alives enabled for
a reason ;-)

> assessment of "it's working" vs "it's not working".

Agreed too.

> Even if we could

In fact, the ones that act at TCP/IP level are currently the most likely
to work despite their incomplete support for HTTP, because they either
have support for content-length (for the high-end ones) or simply switch
to tunneled mode after the request where both sides can freely exchange.
That was how the first versions of haproxy did work, that was how Alteon
did work for a lot of time, this is the most naive approach but it freely
supports POST and CONNECT methods, so that makes it implicitly compatible 
with upgrade+101. The really nasty ones (we still have to get ones pointed
out) are the ones which would not understand 101 and still forward the
response to the client but not the data. That's what my proposal tries to
spot in an HTTP-compatible manner.

> I think we should instead make it try to work and fail only if there is a

100% agree here too. Also, some "ping" type messages will help spot another
class of hard to debug network issues : the small MTU paths. It happens
that due to poor VLAN or PPPoE settings on some equipment, some devices
can't receive full-sized ethernet frames and will silently drop them.
It's not easily spotted at all from either end, since it rarely strikes. 
That would mean that a long connection could work for some time then
suddenly block during a larger exchange. For instance because the client
is resending aggregated TCP segments after a few drops, or because the
user has performed a copy-paste of some text in a box. Neither side will
be able to spot this. The client will fill system buffers and the server
will wait for the client to talk. A "ping" would notice this quickly
because it would simply not get any response due to the segment that
cannot go through.

For the same reason such a "ping" would also detect broken connections
caused by firewall timeouts, NAT gateways, local IP address changes,
etc... So I'm all in favor of that, and I agree that my proposal doesn't
address these issues. It just helps quickly detecting if we can use WS
over a path or switch to something else without having the user wait
for ages.

Regards,
Willy

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 22 Jul 2010 01:38:01 +0100

> connection towards the client.

I think this won't work.  RFC2616 (HTTP 1.1):

   "Proxies MUST forward 1xx responses, unless the connection between the
    proxy and its client has been closed, or unless the proxy itself
    requested the generation of the 1xx response."

It's not spelled out explicitly, but the expectation is that proxies
forward 1xx responses generically, and then forward the next response.

If the proxy doesn't know how to handle 101 responses, then it may
simply forward the 101 as a generic 1xx, and then try to forward the
*following* bytes as a HTTP response.

I would expect proxies to ignore Content-Length in 1xx:

   "1. Any response message which "MUST NOT" include a message-body (such
       as the 1xx, 204, and 304 responses and any response to a HEAD
       request) is always terminated by the first empty line after the
       header fields, regardless of the entity-header fields present in
       the message."

In other words, a relatively stupid proxy, but which is aware of the
1xx rule, will not close the connection, and will treat the hello
text and subsequent bytes as a HTTP response to modify and forward.

There's some discussion on another list about using a 1xx response to
indicate "request is being processed", for when the processing is slow
to produce a response.  The fact that is being taken seriously
suggests that at least some deployments do forward 1xx messages in the
way the standard requires.

-- Jamie

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 02:58:45 +0200

Hi Jamie,

first, thanks for taking the time to review the proposal in depth and
to match it with standards.

> *following* bytes as a HTTP response.

Yes but those will fail as the following bytes won't start with "HTTP/1.1",
and the proxy will close the connection.

>        header fields, regardless of the entity-header fields present in

The content-length I proposed was precisely for those which are not
even aware of 1xx, nothing more.

> text and subsequent bytes as a HTTP response to modify and forward.

Due to the next non-HTTP data, they must close with a "502 Bad Gateway"
upon receiving the first data following the 101 message that does not
look like "HTTP/1.1 XXX reason".

> to produce a response.

Nice idea, I was not aware of this discussion, I only initiated the one
about 101 not working like the general 1xx case ;-)

> The fact that is being taken seriously

Yes, at least the 100 are forwarded and heavily used with POST requests
("Expect: 100-continue").

But I agree with what you said in your other mail, the best way to know
is precisely to test !

As of now I have just tested via a Squid 2.6, Bluecoat proxy-sg v4 and
netcache (I don't remember the version), and all failed cleanly on the
101 by closing the connection. Apache 2.2.15 remains open after the 101
but immediately closes once it gets the first following byte. Also, none
of them sent the "Upgrade: WebSocket" in response, which is another good
reason to declare a failure ;-)

I'm not pretending the scheme covers every case (and I've not tested them),
but I think it addresses various generations of simplified implementations.

Regards,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 22 Jul 2010 11:01:02 +1000

> way the standard requires.

We have users that definitely use the 102 processing response with success.
So there must be some network segments where this is possible.
I've often thought that if 102 responses could have bodies and were exposed
to the browser, then they would make an useful transport for comet style
applications.

cheers

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 21 Jul 2010 18:02:48 -0700

One cannot send the Upgrade header field without also
sending "Connection: Upgrade".

   The Upgrade header field only applies to the immediate connection.
   Therefore, the upgrade keyword MUST be supplied within a Connection
   header field whenever Upgrade is present in an
   HTTP/1.1 message.

....Roy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 03:07:07 +0200

> sending "Connection: Upgrade".

Yes Roy, sorry about it missing in my example, it's late here and
what I wanted to do was to spot the proposed changes with current
spec. Having it in both messages is essential and is part of the
reason of the fast clean failures on non-compatible proxies.

Thanks,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 03:23:40 +0200

> We have users that definitely use the 102 processing response with success.

Nice to know. I think that it's simply forwarded as the 1xx rule just
like the 100 is.

> to the browser, then they would make an useful transport for comet style

Very clever idea. You could use headers though, or even shorter messages
by playing with the reason string after the status code ;-)

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 22 Jul 2010 03:27:46 +0200

I don't like to reply to myself, but I'm just realizing that using a
POST instead of a GET as was already suggested in the past would most
likely match most of the already in-place cache-bypass rules in front
of transparent proxies, precisely because those proxies have no business
processing such POST requests. I have a feeling that doing just that
should already improve the success rate.  It may also unlock the
client-to-server direction on very low-level devices that would otherwise
fail, allowing them to work.

Willy