Subject: Re: [hybi] Experiment comparing Upgrade and CONNECT handshakes

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 04 Dec 2010 01:26:43 +0100

* Adam Barth wrote:
>David Huang, Eric Chen, Eric Rescorla, Collin Jackson, and I have been
>experimenting with the security of the Upgrade-based and CONNECT-based
>WebSocket handshakes.  Please find a paper detailing our findings at
>this location:
>
>http://www.adambarth.com/experimental/websocket.pdf

It would have been helpful if the key metrics had been summarized in a
simple tabular form for easy comparison (so one would not have to com-
pare an absolute X with a relative "all but two"), but from my cursory
reading this does not seem helpful in deciding which handshake is best.

For instance, the success rate of cache poisoning is not helpful as
they apparently compare as 8 vs 0 of about 50 000 attempts. That could
be chance, that could be a result due to the rich media banner that
may not be supported in certain corporate or mobile settings, it could
be due to mistakes in the experiment, and so on. If tomorrow someone
says they repeated the experiment and got 6 vs 3 I would find that en-
tirely plausible, and if then someone comes along saying it's 3 vs 6
for them, I would find that plausible too, leaving no clear winner.

Could you summarize the key metrics in a tabular form so I can make
sure I haven't missed or misread any of them (between the Upgrade and
the CONNECT handshake primarily)?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Adam Barth <[email-masked]>
Date: Sat, 4 Dec 2010 01:23:00 -0800

On Fri, Dec 3, 2010 at 4:26 PM, Bjoern Hoehrmann <[email-masked]> wrote:
> be chance,

We didn't report the p-value for these observations in the paper, but
running a quick chi-square puts the p-value of these observations
being the same distribution around 0.0047, which is certainly
statistically significant.

> that could be a result due to the rich media banner that
> be due to mistakes in the experiment, and so on.

As with any experiment, there are factors we can't control.  However,
we do the best we can, and science moves forward.

> If tomorrow someone

I would be surprised if someone made those observations, purely on
mathematical grounds.  Another thing that's not reported in the paper
is that we did a series of pilot studies to work out the bugs in our
experiment.  Our observations were fairly stable across all these
iterations of the experiment.  Of course, reasoning statistically
about the pilot study is tricky due to post-hoc issues, which is why
we only report the final data in the paper, which we can confidently
analyze pre-hoc.

> the CONNECT handshake primarily)?

Sure.  We show that for $100, an attacker is able to succeed in
mounting 8 cache poisoning attacks against the Upgrade handshake.
Note that this value is somewhat of a lower bound because our attacks
are completely untargeted.  By targeting the attack, an attacker can
likely be even more cost-efficient.  By contrast, we never observed a
single successful cache poisoning attack against the CONNECT handshake
despite 54,534 attempts.

Adam