Subject: Re: [hybi] Experiment comparing Upgrade and CONNECT handshakes

From: Zhong Yu <[email-masked]>
Date: Tue, 30 Nov 2010 21:33:50 -0600

With Upgrade handshake, only 1 (out of 50K) firewall circumvention
attack was successful. While with POST/Java/Flash, the number is
around 1000. The Upgrade handshake is a huge improvement! But why? I
assume that the random bytes and framing bytes in the -76 handshake
corrupted HTTP streams and busted 99.9% of HTTP parsers.

In the singular successful case, the transparent proxy ignored these
non-http bytes and constructed a http request anyway, at least, it
extracted the "Host" header. For this attack to be really useful, it's
not enough that the bytes are tunneled to target.com. The proxy must
strip non-http bytes and send a compliant http request to target.com.
Or, the proxy forwards all bytes to target.com, and target server
strips non-http bytes and reconstructs the intended http request. Did
one of these two things actually happened in the experiment?

Even worse for the attacker, it's highly unlikely that the http
response sent back by target.com will be accepted by the WebSocket
client as valid WebSocket stream. If the attacker cannot read the
response from target.com, this attack is less useful than sending a
simple HTTP request.

The combined odds that this attack works over Upgrade handshake is
extremely small.

With CONNECT handshake the same transparent proxy did not pass through
the firewall circumvention attack. It is very likely that the proxy
routes by Host for each request. The handshake contains an invalid
Host, the routing failed, the connection was aborted, and the attack
couldn't be carried out. Most likely CONNECT wins over Upgrade on this
single case because of the bogus Host header, not because the proxy
understands CONNECT and gives up parsing the rest of stream.

Altering the semantics of Host is a big deal. Even though it turns out
to be "helpful" in this singular case of the experiment, the price far
more exceeds the benefit.

It should be stressed that the paper contains no empirical evidence
that the bogus Host would be helpful in other threat models.

If Host really is a problem, we should simply remove it from
handshake. Having no Host header is not worse than having a bogus Host
header. The Host header was a hack anyway because the request URI
wasn't absolute. We don't have to keep that hack for WebSocket.

I'm voting for CONNECT method with real Host header.

= Inconsistent data regarding cache poisoning attack =

Although the Upgrade handshake reduced firewall circumvention attack
to 1/1000, it does not have the same luck with cache poisoning attack,
according to the paper. 50% attacks still got though. That is *very*
surprising. Caching proxies must demarcate requests precisely, how do
they have such a higher tolerance of corrupt stream? How come 99.9%
host-based-routing proxies are busted by the stream, yet 50% of
ip-based caching proxies are not busted? I must respectfully
disbelieve the result for now.

- Zhong Yu

---

From: Adam Barth <[email-masked]>
Date: Wed, 1 Dec 2010 09:00:35 -0800

> corrupted HTTP streams and busted 99.9% of HTTP parsers.

We're worried about the cache poisoning attacks, not the firewall
circumvention attacks.

> one of these two things actually happened in the experiment?

We didn't testing the framing in this experiment, just the handshakes.
 The target server was a stock Apache server.  That means the proxy
forwarded the request in such a form that stock Apache was willing to
respond to it.

> The combined odds that this attack works over Upgrade handshake is

Indeed.  That's why we're worried about the cache poisoning attack instead.

> I'm voting for CONNECT method with real Host header.

IMHO, we should adopt the CONNECT handshake first and discuss the
details of what Host header to send second.

> disbelieve the result for now.

I'm not sure where you got the 50% number from.  You're reading the
table incorrectly.  Each of the different handshakes (POST, Upgrade,
and CONNECT) are different experimental conditions (technically
within-subjects conditions).  To understand the data in Table II, you
should read the data vertically, not horizontally.

One way to think about the effectiveness of the cache poisoning attack
is in exploits per dollar.  We show that (without any targeting) an
attacker can achieve 8 exploits for $100, which is concerning.

Adam

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 1 Dec 2010 18:14:50 +0100

> details of what Host header to send second.

+1

What concerns there are about CONNECT are probably best evaluated with
working code, test deployments and without the distraction of the
debate about bogus host headers.

cheers

---

From: John Tamplin <[email-masked]>
Date: Wed, 1 Dec 2010 12:18:16 -0500

> debate about bogus host headers.

I do have working code for Adam's first proposed handshake at the
websocket-draft-eval project, and it could easily be updated to a more
recent version if anyone wanted to test another implementation against it.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 1 Dec 2010 19:35:40 +0100

> details of what Host header to send second.

I agree that this work should be incremental. The difficulty as we're
working today is that you presented a replacement proposal. It is very
hard to agree on multiple changes at once.

If we want to progress, we should consider replacing just the GET+Upgrade
with a CONNECT and not change anything else (meaning we keep the host and
Hello framess). Only then we can study what else can be done.

Willy

---

From: John Tamplin <[email-masked]>
Date: Wed, 1 Dec 2010 13:43:59 -0500

> Hello framess). Only then we can study what else can be done.

AFAIK, the Hello frames do not appear in any draft and only in Greg's
proposal.  Personally, I am not sure what exactly they buy us and I
don't know if we want to pay the extra round trip for them.

So, if we really want this to be incremental, it would be a change
from Ian's last draft.

As I understand it, these are the components of Adam's latest proposal:
 1) use CONNECT instead of GET+Upgrade
 2) use a fixed, bogus host header and mask the real headers
 3) mask all payload data
 4) encode the headers as JSON

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 1 Dec 2010 19:48:28 +0100

> don't know if we want to pay the extra round trip for them.

I'm sorry, I thought we had that in -03. At some point I'm getting lost
between proposals and drafts :-)

> So, if we really want this to be incremental, it would be a change

That's what I wanted indeed.

>  4) encode the headers as JSON

I think we agree on the process then !

Cheers,
Willy

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 1 Dec 2010 11:00:05 -0800

> >  2) use a fixed, bogus host header and mask the real headers

I thought that we already agreed on this point weeks ago.

-Ekr

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 1 Dec 2010 20:01:19 +0100

> don't know if we want to pay the extra round trip for them.

As proposed, the Hello frames do not cost an extra round trip.

The Hello frames are proposed to replace the current unframed bytes.
There primary purpose is  to test the connection after an upgrade to
see if WS frames really can be exchanged.  The secondary purpose was
to carry nonces and hashes.

I think with CONNECT there is probably less need for such Hello
packets, as I think it much more likely that if a CONNECT succeeds
then the path will be good.   But there could still be some failure
cases, so there may still be value.   For example, consider a  really
dumb intermediary that reads a HTTP request in full, then forwards it
and then reads the response in full.  Such an intermediary would be
able to forward a CONNECT and the 200 response, but would not then
function as a WS intermediary.  Sending the Hello frame from the
server would detect this as the client would timeout waiting for it.
Without the Hello frame, the client would never know if the connection
is working or if the server just has nothing to say.

So this is why I think we should have kept incrementally improving the
current handshake, so we could now simply switch to CONNECT.  But we
still have the issue of the unframed bytes and the space encoded
nonces etc.     I think both of these things need to go regardless of
a switch to CONNECT.    Thus I think we should accept the Hello
proposals before switching to CONNECT (but I don't necessarily think
we need a draft inbetween).

regards

---

From: John Tamplin <[email-masked]>
Date: Wed, 1 Dec 2010 14:10:58 -0500

> As proposed, the Hello frames do not cost an extra round trip.

Unless I misunderstood your proposal, doesn't the server have to wait
for the client's Hello reply before it can send any messages?  That
seems like it adds one RTT (not two, as it would be if the first Hello
wasn't piggy-backed with the server response).

> is working or if the server just has nothing to say.

I agree the Hello frames are useful in this case.  However, how often
is this case present, and is getting a WebSocket-layer timeout rather
than app-layer (which in most cases will have to be present anyway)
worth the price for all the other cases?  I don't have any numbers for
it (do you?), but my gut feeling is it is low.

> we need a draft inbetween).

Except that depending on what we switch to, those things you want
removed aren't present at all, such as in Adam's proposals.  So,
moving them somewhere else doesn't seem like an incremental step but
rather wasted effort.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 1 Dec 2010 20:40:55 +0100

On 1 December 2010 20:10, John Tamplin <[email-masked]> wrote:
> wasn't piggy-backed with the server response).'

The server can send messages as soon as it has sent it's Hello.

> it (do you?), but my gut feeling is it is low.

I agree that Hellos may be less useful with CONNECT.  But we have to
do something about the current handshakes unframed bytes and space
encoded nonces.  If we leave the unframed bytes as they are with
CONNECT, then we will probably still have the identified issue of an
intermediary that does not forward the bytes and the server hanging
while it tries to read them.

> removed aren't present at all, such as in Adam's proposals.

Adam's complete proposal has other aspects to it that are not as well
accepted as trying CONNECT.

If we do agree that CONNECT is better than Upgrade, that is not a
reason to completely adopt the first proposal that uses CONNECT
without questioning each of it's elements.

The handshake is in such a mess because we accepted a big bundle of
changes after only agreeing one aspect of them. We have now spent
months talking about the removal of the additional "features" that
crept in with that batch.

> So, moving them somewhere else doesn't seem like an incremental step but
> rather wasted effort.

Incremental changes are always wasted effort if you are 100% sure you
know all the increments in advance.
The whole point of incremental changes is that you can re-evaluate
along the way and perhaps revert some changes made earlier when
considered in the light of subsequent changes.

---

From: Zhong Yu <[email-masked]>
Date: Wed, 1 Dec 2010 13:45:01 -0600

> We didn't testing the framing in this experiment, just the handshakes.

The attack occurs after handshake, doesn't it? The attack data must be
framed, otherwise how do you explain that in the POST case, the attack
succeeds 1376 times, while in the Upgrade case, it only succeeds 1
time. What's the difference if it's not due to framing?

> forwarded the request in such a form that stock Apache was willing to

Apache will response 400 Bad Request to garbage inputs. That shouldn't
be counted as a successful attack. It is unclear from your paper how
it is counted.

> details of what Host header to send second.

How about CONNECT+Upgrade, everybody is happy.

> should read the data vertically, not horizontally.

We can still cross examine the data and find something mysterious.

>From POST to Upgrade column, the firewall circumvention attack
successes decrease from 1376 to 1. If I'm mistaken, please correct me
with the right explanation, but I believe the POST experiment sent
clean/compliant HTTP requests, and the Upgrade experiment sent the
attack data framed - the non-http bytes busted 99.9% parsers used by
the transparent proxies.

Yet, the cache poisoning attack success count only drops from 15 to 8.
This attack also depends on proxies' ability to parse http requests.
If the non-http bytes in the Upgrade protocol would bust 99.9%
parsers, we should see the attack success count drop to 15/1000 = 0.

So I must question the validity of the 8 success attacks. (note I also
questioned the 1 success attack in the firewall circumvention case)
More details are needed to analyze the experiments and the results.

This is important because these 9 cases are the only evidence
presented so far that plaintext pay load in simple framing could be
misinterpreted as compliant HTTP requests although it is not. The
evidence is used to argue for stream obfuscation. As the only
evidence, it should be examined carefully.

---

From: Zhong Yu <[email-masked]>
Date: Wed, 1 Dec 2010 13:57:09 -0600

The Hello frame exchange can be initiated by the client.

client to server
  HTTP handshake
  HELLO

server to client
  HTTP handshake
  HELLO

In most cased this is done in 1 round trip. Some current
intermediaries may withhold the client HELLO until server HTTP
handshake is received. That's all right, it's a minor performance that
will disappear in long term.

The advantage of using WS frame to test the connection is also
short-term. In the long run, that perspective becomes moot.

It is more useful to move majority of handshake data into WS frames,
keeping HTTP handshake short and simple, thereby reducing WS's
dependency on HTTP syntax. We have more design freedom if we are not
subject to HTTP baggage.

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 01 Dec 2010 13:47:13 -0800

> I thought that we already agreed on this point weeks ago.

I think what's new now is that we are approaching consensus on adopting point (1).

It sounds like we don't yet have consensus on (2) - people would rather adopt (1) first, and then examine it further.

I don't recall the state of the discussion on (3). I think the new XOR-masking likely addresses the concerns with the original AES masking.

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 01 Dec 2010 13:51:49 -0800

> while it tries to read them.

There is a reasonable subset of Adam & Eric's proposal that includes CONNECT, but omits the bogus host header, JSON headers and payload masking. It would not have unframed bytes or space encoded nonces. 

I think that would be a decent starting point, after which we could evaluate the remaining aspects.

Regards,
Maciej

---

From: John Tamplin <[email-masked]>
Date: Wed, 1 Dec 2010 17:05:47 -0500

On Wed, Dec 1, 2010 at 4:51 PM, Maciej Stachowiak <[email-masked]> wrote:
> There is a reasonable subset of Adam & Eric's proposal that includes
> CONNECT, but omits the bogus host header, JSON headers and payload masking.
> It would not have unframed bytes or space encoded nonces.
>

Ok, so if we took this approach, the proposed draft changes would be to include:
 - using CONNECT instead of GET+Upgrade
 - uses the real host/port in the CONNECT and Host header
 - all handshake headers would be normal headers in the HTTP request/response
 - keys would be included in headers not unframed bytes, with no space encoding
 - no masking of headers or payload would take place
 - no Hello frames

Is that correct?

If so, then I agree that would be a good step to see if we can get
consensus on the less controversial parts before moving on to areas
where there is more disagreement.

---

From: Zhong Yu <[email-masked]>
Date: Wed, 1 Dec 2010 16:12:21 -0600

If HTTP compliant is important, we cannot arbitrarily separate Method
from headers.

CONNECT with bogus Host is hardly HTTP compliant. A correctly
implemented end point should reject it. It's a flaw if it's accepted.
Should WebSocket be based on such flaws?

CONNECT with real Host seems to be OK. And Adam's experiments did NOT
reveal any security problems caused by a real Host header in the
handshake.

Does Roy have any problem with CONNECT + real Host being used for
WebSocket tunneling?

On Wed, Dec 1, 2010 at 3:47 PM, Maciej Stachowiak <[email-masked]> wrote:
> It sounds like we don't yet have consensus on (2) - people would rather
> adopt (1) first, and then examine it further.
> I don't recall the state of the discussion on (3). I think the new
> XOR-masking likely addresses the concerns with the original AES masking.
> Regards,
> Maciej
>

---

From: Maciej Stachowiak <[email-masked]>
Date: Wed, 01 Dec 2010 15:08:48 -0800

> evidence, it should be examined carefully.

Perhaps Adam & company could publish the experiement code and cite it in the paper, so that anyone who is skeptical of the results could attempt to replicate the experiment or attempt to identify methodological errors. (It's sad that this is not the norm for CS papers.)

It might even be possible, though more work, to attempt replication solely based on the info in the paper.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 2 Dec 2010 06:13:01 +0100

+1

> I think that would be a decent starting point, after which we could evaluate

Good call!

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 2 Dec 2010 06:25:25 +0100

>  - using CONNECT instead of GET+Upgrade

+1 (give it a go at least and see if we can make it not look like an
abomination to those with HTTP eyes).

>  - uses the real host/port in the CONNECT and Host header

To be clear - that would be  something like

      CONNECT server.example.com:80 HTTP/1.1
      Host: server.example.com:80

and not the proposal I made to use a dedicated port (which can be
consider later)

>  - keys would be included in headers not unframed bytes, with no space encoding

Since the current draft contains only a client nonce and server hash,
I think we should start with the same.   We can then consider if a
server nonce is needed or if neither are.

> Is that correct?

It is a good start

> where there is more disagreement.

... where there is more agreement yet to be had.

cheers

---

From: Zhong Yu <[email-masked]>
Date: Fri, 3 Dec 2010 02:27:19 -0600

Adam, is there any chance that the POST experiment poisoned a cache
first, then the Upgrade experiment, which is carried out immediately
afterward, fetched the same poisoned cached result from proxy, and
counted it as a success attack in the Upgrade experiment?

- Zhong Yu

---

From: Adam Barth <[email-masked]>
Date: Sat, 4 Dec 2010 01:11:26 -0800

> counted it as a success attack in the Upgrade experiment?

Nope.  We controlled for this possibility by using different URLs.

Adam

---

From: Adam Barth <[email-masked]>
Date: Sat, 4 Dec 2010 01:25:42 -0800

> Perhaps Adam & company could publish the experiement code and cite it in the paper, so that anyone who is skeptical of the results could attempt to replicate the experiment or attempt to identify m…

Yeah, I'll ask if we can do that.  I'm a big proponent of releasing
the code behind papers.

> It might even be possible, though more work, to attempt replication solely based on the info in the paper.

That should be possible (it's important for reproducibility), but it's
rarely done in practice.  There are some subtitles to getting the
attacks to work that aren't reported in the paper, but I'd be happy to
share with folks off-list if they're interested in replicating the
experiment.

Adam