Subject: [hybi] WebSockets Testsuite Update / Browser Status

From: Tobias Oberstein <[email-masked]>
Date: Mon, 12 Sep 2011 05:24:58 -0700

Autobahn WebSockets 0.4.2 is done, which includes an update of the WS test suite:

  * supports draft 14
  * UTF-8 test coverage
  * now >200 test cases

The UTF-8 test cases both cover UTF-8 per-se (RFC3629) as well as specifics related to WS,
for example testing fragmentation within code points and whether the implementation
fails fast, or only upon complete frames or complete messages.

The underlying UTF-8 validator is incremental, and thus streaming capable. The implementation
is based on the algorithm

"Flexible and Economical UTF-8 Decoder" by  Bjoern Hoehrmann
http://bjoern.hoehrmann.de/utf-8/decoder/dfa/

[Python also includes an incr. validator, but there are unresolved issues related to
UTF-16 surrogates being accepted - which is wrong]

Test reports including current browsers (Firefox/Chrome) can be found here

http://www.tavendo.de/autobahn/testsuite.html

==

Browser status:

Chrome
+ the latest Chrome includes support for binary messaging, all related tests pass.
+ the UTF-8 implementation of Chrome also passes all tests.
+ It does not fail fast on UTF-8 validation, but only when complete message was received.
+ 2 test cases are excluded (6.4.3/6.4.5) since they crash Chrome (this has probably nothing to do with UTF-8 though)

Firefox:
+ Firefox 7 still has a couple of bugs found by the test suite, which are fixed on 8/9.
+ Firefox handles a couple of cases "non-strict", which is allowed by spec, but well - non-strict
+ all versions of Firefox seem to deny a couple of octet sequences which are valid UTF-8

==

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 12 Sep 2011 14:35:11 +0200

2011/9/12 Tobias Oberstein <[email-masked]>:
> Autobahn WebSockets 0.4.2 is done

~# cat /etc/issue
Ubuntu 11.04 \n \l

~# easy_install -U Autobahn
Searching for Autobahn
Reading http://pypi.python.org/simple/Autobahn/
Reading http://www.tavendo.de/autobahn
Best match: autobahn 0.4.2.win32
Downloading http://pypi.python.org/packages/any/a/autobahn/autobahn-0.4.2.win32.zip#md5=e9b0f78b1e58ba37b8c94fb4bf9a73a1
Processing autobahn-0.4.2.win32.zip
error: Couldn't find a setup script in
/tmp/easy_install-Dsd_RR/autobahn-0.4.2.win32.zip

The new version is just available for Windows? :)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 12 Sep 2011 14:38:34 +0200

> The new version is just available for Windows? :)

Forget it. After 2 minutes it's already available for Linux.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 12 Sep 2011 15:16:45 +0200

> is based on the algorithm

Hi, I don't agree with test case 6.4.3:

"Send invalid UTF-8 text message in 3 fragments plus more. First is
valid, then wait, then 2nd which contains the octet making the
sequence invalid, then wait, then 3rd with rest. Then we send 64k
frames forever."

This test assumes that the server must close the connection after
second frame as it contains illegal UTF-8, and then it sends 64k
frames forever. My server performs UTF-8 validation over the whole
received message so basically this test is a DoS for my server rather
than a test.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Tobias Oberstein <[email-masked]>
Date: Mon, 12 Sep 2011 06:43:42 -0700

> "Send invalid UTF-8 text message in 3 fragments plus more. First is valid, then
> wait, then 2nd which contains the octet making the sequence invalid, then
> wait, then 3rd with rest. Then we send 64k frames forever."
> 
> This test assumes that the server must close the connection after second
> frame as it contains illegal UTF-8, and then it sends 64k frames forever. My
> server performs UTF-8 validation over the whole received message so
> basically this test is a DoS for my server rather than a test.

I agree that it's probably suboptimal to mix those 2 issues.

Maybe split out to  a new test section: "Implementation Limits"

That could then drive the implementation to it's limits - if there are
any (if not, timeout) - and see how it behaves.

==

Wrt to UTF-8 fail fast handling:

The spec does not say when to fail, only to fail

Options for when to fail:

1. only when complete message was received
2. only when complete frame was received
3. only when complete octets for encoded integer was received
4. as soon as invalid octet is received

Currently, there are tests to distinguish between 1., 2. and 4.

So for completeness, there need to be tests to differentiate
between 3. and 4.

==

So, above are essentially 2 new features: pls bare with me that
I only have limited time to invest, so it might take some time
until the stuff arrives.

Or submit patches;)

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Mon, 12 Sep 2011 21:09:35 +0200

> http://www.tavendo.de/autobahn/testsuite.html

I've implemented the C code in the link
http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ to my WebSocket server
so it now performs incremental UTF-8 validation. Updated test results:

  http://oversip.net/public/oversip/websocket-autobahn-results/

Thanks for these nice tests which help *a lot* !!

-- 
Iñaki Baz Castillo
<[email-masked]>