Subject: [hybi] preliminary WebSockets compression experiments

From: John Tamplin <[email-masked]>
Date: Fri, 23 Apr 2010 15:47:24 -0400

[This is a resend of a message sent on Monday that never appeared to show
up - apologies if this is a duplicate]

I have run some experiments on existing WebSockets applications to see what
could be gained if compression were enabled.

To do this, I captured traces with tcpdump, then wrote a small libpcap-based
program to strip out individual TCP streams to separate files.  I then wrote
another program which read these files, removed the WebSocket framing, and
tried compressing them in various ways using zlib and analyzed the results.

The applications tested were:

   - A hacked version of Google Wave that used WebSockets for communication
   with the server, with Chrome 5 as the browser.  The messages in this case
   are JSON consisting of ASCII text.  (Note that this was a relatively small
   trace, I am getting a larger trace to make sure the results hold).
   - GWT Quake, which is an HTML5 app that gets up to 60fps running in the
   browser using WebSockets for communication with the server.  The messages
   are largely binary, with individual bytes being encoded as UTF8 characters
   (so bytes 0x80-0xFF are encoded as a two-byte UTF8 character), and those
   binary values are largely IEEE floating point values.  The sample taken was
   from two players, one using Chrome 5 on Windows and the other using WebKit
   nightly on Mac playing a multiplayer game.

The compression methods tried were:

   - GZIP - using zlib with a gzip header (ie, deflateInit2(zstr,
   Z_DEFAULT_COMPRESSION, Z_DEFLATED, 15 | 16, 8, Z_DEFAULT_STRATEGY) - this
   should closely model the gzip encoding used for HTTP responses.
   - DEFLATE - similar, but using deflateInit(Z_DEFAULT_COMPRESSION) so only
   the zlib header is included
   - DEFLATE/STREAM - the above methods compress each frame separately, so
   the compression dictionary has to be rebuilt for each message.  However,
   this method maintains compression state across messages in the same stream
   (using deflate(Z_SYNC_FLUSH) to finish an individual frame), so later
   messages can exploit redundancy from previous messages.  This does have the
   downside of maintaining state for the duration of the connection, but there
   is already significant state due to keeping a TCP connection up

For each stream, I collected the following information:

   - total payload bytes transferred (note that I did not count TCP/IP
   overhead or WebSockets framing overhead)
   - packet sizes at the 25th, 50th 75th, and 90th percentiles
   - percent reduction
   - in each case, I assumed that if the compression resulted in an increase
   in the frame size, it would be sent uncompressed instead (but I include the
   count of such frames)

Google Wave

client->server:
   31 frames totalling 9189 bytes, percentiles: 248/249/396/397 bytes

   - GZIP: 6193 total bytes, percentiles 183/184/225/228, 32.60% reduction
   - DEFLATE: 5937 total bytes, percentiles: 175/176/217/220 bytes, 35.39%
   reduction, none under 63 bytes
   - DEFLATE/STREAM: 1274 total bytes, percentiles: 26/28/36/76 bytes,
   86.14% reduction, 88.8% under 63 bytes

server->client:
   42 frames totalling 8962 bytes, percentiles: 90/91/111/657 bytes

   - GZIP: 5485 total bytes, percentiles: 87/88/109/282 bytes, 38.80%
   reduction
   - DEFLATE: 5149 total bytes, percentiles: 79/80/101/274 bytes, 42.55%
   reduction
   - DEFLATE/STREAM: 1427 total bytes, 25/26/28/37 bytes, 84.08% reduction

Conclusion:

   - Contrary to expectations, the wave protocol has sufficient redundancy
   to get savings from compression - even basic gzip compression provides
   significant benefits for this trace of the wave protocol.
   - Sharing compression state across messages results in 5-6x reduction in
   frame sizes, which would be very important in mobile environments

GWT Quake

client 1 -> server:
   3837 frames totalling 212653 bytes, percentiles: 50/54/60/66 bytes

   - GZIP: total 210242 bytes, percentiles: 50/54/60/66 bytes, 1.13%
   reduction, 3833 frames grew larger
   - DEFLATE: total 210184 bytes, percentiles: 50/54/60/66 bytes, 1.16%
   reduction, 3817 frames grew larger
   - DEFLATE/STREAM: total 94902 bytes, percentiles: 21/24/28/31 bytes,
   55.37% reduction

server -> client 1:
   2150 frames totalling 608658 bytes, percentiles: 120/163/476/478 bytes

   - GZIP: total 405572 bytes, percentiles: 112/142/281/282 bytes, 33.37%
   reduction
   - DEFLATE: total 388897 bytes, percentiles: 104/134/273/274 bytes, 36.11%
   reduction
   - DEFLATE/STREAM: total 96093 bytes, percentiles: 23/44/58/68 bytes,
   84.21% reduction

client 2 -> server:
   1996 frames totalling 103091 bytes, percentiles: 44/50/58/64

   - GZIP: total 101887 bytes, percentiles: 44/50/57/64 bytes, 1.17%
   reduction, 1982 frames grew larger
   - DEFLATE: total 101647 bytes, percentiles: 44/50/57/64 bytes, 1.40%
   reduction, 1938 frames grew larger
   - DEFLATE/STREAM: 49229 bytes, percentiles 21/24/28/32 bytes, 52.25%
   reduction, 99.9% under 63 bytes

server -> client 2:
   1880 frames totalling 423606 bytes, percentiles: 110/165/333/380

   - GZIP: total 303437 bytes, percentiles: 102/142/217/255 bytes, 28.37%
   reduction, 250 frames grew larger
   - DEFLATE: total 289864 bytes, percentiles: 94/134/209/247 bytes, 31.57%
   reduction, 77 frames grew larger
   - DEFLATE/STREAM: total 73246 bytes, percentiles: 18/27/53/67 bytes,
   82.71% reduction

Conclusion:

   - It is absolutely critical that uncompressed frames be allowed even when
   compression has been negotiated -- otherwise, the total bytes transferred
   would be much higher (either through loss of compression where useful, or
   by sending compressed data that is larger than the uncompressed data).
   - the client->server stream only compresses if state is maintained across
   frames, which also gives an order of magnitude size reduction on the
   server->client stream.
   - Even small packets benefit -- with persistent compression state,
   traffic that is 90% under 64 bytes still gets 2:1 compression.

Implications for Protocol Changes for Compression:

   - At least some apps will have different characteristics between the
   traffic in each direction.  For example, in GWT Quake, the client->server
   traffic would pay a roughly 13% size penalty if it had to be compressed or
   pay a 33%+ size penalty on the server->client traffic if the connection
   wasn't compressed at all
      - The simple approach is to simply allow compression to be optional
      for each frame and only use it if it reduces the size
      - This decision could be based on heuristics or by simply compressing
      and comparing the size, though the latter is likely to be inefficient for
      mobile devices.
      - A more complicated approach would be to allow asymmetric compression
      algorithms, though it isn't clear how the browser/server could take
      advantage of it without exposing some API for the application to describe
      the likely traffic.
   - Maintaining compression state across frames gives a large benefit, and
   likely overcomes the need to allow optional compression (though
   in pathological cases there are probably still cases where compression
   results in a size increase).  However, this comes at the cost of additional
   state, so striking the proper balance on mobile devices that are constrained
   on both memory and network bandwidth may be difficult.

Next Steps

   - I will get longer Wave traces and verify the measurements made here
   still hold, especially during startup
   - If anyone else has packet traces of actual WebSocket traffic they can
   share (pcap format is fine, but please don't include any sensitive data)
   please email them to me and I can include them in the analysis.  If that is
   a problem I could also send the source for the analysis tools which should
   build on any Unix-based system.
   - I want to get some functional compression test going so I can measure
   actual latency gains on real apps running over real networks, which means
   defining some plausible framing format.  Probably the most straightforward
   would just be to define frame type 0x80 as compressed UTF8 text, and the
   uncompressed bytes are decoded and passed to the app just like the 0x00
   frame.  It looks like it would be easy to add to Jetty7 (though lack of
   Z_SYNC_FLUSH from Java's Deflater makes it harder), though getting a
   functional browser implementation might be a lot of work.

--
John A. Tamplin
Software Engineer (GWT), Google

---

From: Mike Belshe <[email-masked]>
Date: Fri, 23 Apr 2010 14:48:49 -0700

Great analysis!

Two thoughts:

*Memory Consumption:*
BTW - zlib, using the configuration you specified, might use a surprising
amount of RAM (to the tune of ~250KB!).  I suspect you want to decrease the
window size. Brian Olson already ran this experiment a while back with SPDY
and his results are here.  It would be interesting to see if you reach the
same conclusion.
groups.google.com/group/spdy-dev/browse_thread/thread/dfaf498542fac792?pli=1

As a result of Brian's work, SPDY decreased the windowBits from 15 to 11.
 This reduces the memory footprint substantially, while only leading to
modest reduction in compression ratios.

*On Generically Compressing A Stream:*
While we wanted to have SPDY provide required compression, in the end we
find it likely a bad idea.  Compressing the stream is never as good as
compressing the content.  For example, if the content can compress the
image, there is no point in having the stream redundantly (and likely
less-effectively) compress it.  Since most developers will get the content
compressed correctly, that is the right layer to do so; rather than having
the channel blindly compress what is already done.  We might revisit this
approach if new data arises, though.

SSL has taken a similar approach.  When the protocol was designed, the
authors had the foresight to include compression negotiation in addition to
cypher negotiation as part of the handshake.  They did this because
compression should be applied before encryption (compressing encrypted data
is never good).  But, what do browsers today negotiate?  The major browsers
all advertise the empty-set of supported compression algorithms (meaning
don't do compression).  Why?  because compressing the stream blindly when
the application content is already compressed is inefficient.

Anyway - those are two protocols which are opting not to do stream-based
compression and instead requiring the content provider to make the choice
(like HTTP does).  I think this matches your conclusion that you shouldn't
just compress everything.

<http://groups.google.com/group/spdy-dev/browse_thread/thread/dfaf498542fac792?pli=1>
Mike

On Fri, Apr 23, 2010 at 12:47 PM, John Tamplin <[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: John Tamplin <[email-masked]>
Date: Fri, 23 Apr 2010 18:11:18 -0400

> modest reduction in compression ratios.

Thanks for the pointer -- I will try that out.

> just compress everything.

I agree with the basic premise that the application can do a better job
compressing the data than some generic algorithm -- my compression
background is chess endgame databases where application-specific compression
algorithms are much better than generic ones.  However, I don't know about
trying to implement that in JS along with a corresponding server-side
implementation.

For HTTP, most servers are configured to compress things automatically if
the client supports it, typically with a blacklist of file types not to
compress (granted, you can pre-compress static content and serve that, and
some servers may be intentionally configured to not compress other data).
 Clearly using gzip compression on HTTP traffic is generally a good thing,
and especially in the case of mobile or other low-bandwidth networks.

As the GWT Quake example shows, even data that you wouldn't think is
compressible (and isn't if you don't maintain compression state) compresses
quite well.  With the safety valve of allowing uncompressed frames even when
compression has been negotiated, I don't see how it can be a bad thing.  The
server-side API might allow a way for the particular service on the other
end of the socket to deny compression if it knows its data can't benefit,
but that is really beyond the scope of the wire format.

It would be feasible to provide an API for JS to call for compressing data
and sending it itself, but there are a couple of problems:
 - JS currently doesn't have any efficient way to manipulate binary data
 - in the common case, it just adds extra work for the developer and extra
code to download

I think that might be a good idea once the binary data problem is solved,
but I think a default case of allowing reasonable compression on the stream
automatically is a good idea.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 23 Apr 2010 15:23:27 -0700

Here is another thing to note.
If compression is optional and/or negotiated, it is likely that it will
either not be implemented, or it will be turned off by various end-user
devices that wish to inflate benchmark numbers.

SPDY requires compression support over the headers of certain control
frames. To date, we've not seen a case where this has resulted in a larger
request than raw, and it does provide for at least the availability of the
gzip functionality at the endpoint.

For Websockets, if we use gzip, the decision to do so should be made
*solely* by the sender of that data. There should probably be no
negotiation, else we'll end up with liars and cheats again.
-=R

---

From: John Tamplin <[email-masked]>
Date: Fri, 23 Apr 2010 18:33:15 -0400

> devices that wish to inflate benchmark numbers.

How would turning off compression inflate benchmark numbers?  If compression
is a win, then it should make the benchmarks worse to disable it.

> negotiation, else we'll end up with liars and cheats again.

I guess I am just missing the point here.  Say I have a JS client talking to
some backend, and I want to exchange data between them.  Just like I don't
care about the underlying transport, whether a VPN or other tunnel is
involved, or how the name is resolved, I don't care how my data gets to the
other end.  When I hand a frame to the browser, it seems reasonable for it
to send it to the server the best way it sees fit (subject to the agreement
with the server) -- if compressing the data is better than not compressing
it, it should compress it.  The server should then take the data that was
received and convert it back to the form the client code provided, by
decompressing if necessary.

As a programmer who is just sending data back and forth, I would prefer not
to have to worry about it when writing my client/server code -- my server
receives a message and interprets it, and if it happened to be
compressed/decompressed along the way that's fine.  It is no different than
writing a CGI program generating HTML -- let the web server do the
compression if the client supports it.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 23 Apr 2010 15:45:03 -0700

On Fri, Apr 23, 2010 at 3:33 PM, John Tamplin <[email-masked]> wrote:

> How would turning off compression inflate benchmark numbers?  If
> compression is a win, then it should make the benchmarks worse to disable
> it.
>

A win for what? Lets say I'm selling a server which supplies virus filtering
services. I say that it can process X requests per second.
I am strongly incented to make X as high as possible. One way to do this is
to make my proxy highly efficient. Another way is to lie and cheat-- I strip
off every instance where the client indicates that it can accept gzip. The
servers now see that non of the clients that come through this proxy accept
gzip. The clients suffer increased bandwidth and higher latency.... but at
least they bought the proxy with the highest X.. right?

> I guess I am just missing the point here.  Say I have a JS client talking
> to some backend, and I want to exchange data between them.  Just like I
> don't care about the underlying transport, whether a VPN or other tunnel is

I think you're saying what I'm saying-- the sender decides to compress or
not. I'm adding: And the receiver gets no say in the matter (i.e.
gzip/whatever should be included mandatory if you ever want to be able to
use it).

> compression if the client supports it.

The 'client' supporting it is the part that is bad. It is much better to be
assured that the client supports it.

-=R

---

From: John Tamplin <[email-masked]>
Date: Fri, 23 Apr 2010 19:00:19 -0400

On Fri, Apr 23, 2010 at 6:45 PM, Roberto Peon <[email-masked]> wrote:

> A win for what? Lets say I'm selling a server which supplies virus
> filtering services. I say that it can process X requests per second.

I assume you have actually seen this, but it seems counter-intuitive to me
that decompression, which is quite straightforward, would be slower than the
virus scanning.  Since you can pipeline it, decompressing should add very
little time to the time a virus scanner takes to process a message.

> assured that the client supports it.

Given some of the comments I have seen about wanting to make server
implementation easy for amateurs, I think it is likely to be a hard sell to
require it.

Also, imagine some great new compression algorithm is invented after the
spec is published -- clearly support of that algorithm has to be optional,
since you have an installed base that doesn't support it, so the protocol is
going to have to have some mechanism for negotiating compression.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 23 Apr 2010 16:20:49 -0700

On Fri, Apr 23, 2010 at 4:00 PM, John Tamplin <[email-masked]> wrote:

>> I am strongly incented to make X as high as possible. One way to do this
>> is to make my proxy highly efficient. Another way is to lie and cheat-- I
>> strip off every instance where the client indicates that it can accept gzip.
>> The servers now see that non of the clients that come through this proxy
>> accept gzip. The clients suffer increased bandwidth and higher latency....
>> but at least they bought the proxy with the highest X.. right?
>>
>

Whether by laziness or incompetence, the users suffer. The question is if we
wish to enable this behavior or not.

>
>
>> The 'client' supporting it is the part that is bad. It is much better to
>> be assured that the client supports it.
>>
>

It is difficult to mandate something which is not invented yet :) You could
still negotiate other compression. I'm just saying that it is good to always
have a fallback compression option (e.g. gzip) that you can rely to be
implemented, and need not be negotiated.

As for requiring that amateur programmers be the target? I still think that
is crazy.
The protocol should enable reliability, robustness, and scalability. These
traits are, in my opinion (and I don't seem to be the only one) far more
important than enabling amateur programmers to create new implementations.
If we can enable amateur programmers to implement
-=R

---

From: John Tamplin <[email-masked]>
Date: Fri, 23 Apr 2010 19:29:33 -0400

> implemented, and need not be negotiated.

Well, I would not object to having compression required in the spec, but I
think it may be difficult to get enough support to make that happen and that
isn't the fight I am trying to fight at the moment -- I just want to get
compression in the spec at all :).

Also, what about very low-end devices -- is it really a good idea to require
them to support compression?  Granted, what constitutes a low-end device
these days is rapidly moving up, but still I could imagine some
microcontroller wanting to use WebSockets to talk to a server.

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 24 Apr 2010 03:46:06 +0100

John Tamplin wrote:
>    Also, what about very low-end devices -- is it really a good idea to
>    require them to support compression? Â Granted, what constitutes a
>    low-end device these days is rapidly moving up, but still I could
>    imagine some microcontroller wanting to use WebSockets to talk to a
>    server.

Yup, I know someone who is thinking about that on a tiny
microcontroller with less than 64kB of RAM, and who is quite likely to
use it on a device with about 4MB spare.

With that 250kB/connection memory use that was mentioned, and the fact
that deflate isn't all that fast at compression (LZO is better),
if serving a 10GB link on a LAN, it's not hard to imagine some servers
turning it off too.  Why compress if your network is 10 times faster
than your CPU can compress?  That's making it slower.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 26 Apr 2010 15:46:27 +0200

Roberto Peon wrote:
> As for requiring that amateur programmers be the target? I still think
> that is crazy.

I agree - but I still don't think we should mandate compression support.

As a professional programmer I still like to have the ability to
just crack open a socket (eg from a low capability device or even
just from a test harness) and simply talk websocket.

However, I would like to see optional compression supported
in the base protocol as standardized by the IETF.

So at this stage of the process, we should be considering if
we have consensus to make compression a requirement or not.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 26 Apr 2010 16:05:20 +0200

Mike,

some slightly off topic thinking out loud on my idea that websockets
should be able to eventually replace the framing layer of SPDY....

The problem that SPDY poses for efficient compression implemented in the
framing layer is that it's frames may contain both headers and
a content body.

The headers are very compressible, but SPDY will probably have
already compressed them. The content body may be very compressible
(eg HTML) or not (eg JPG).

So if we were to run SPDY over websocket, then it would be unlikely
that the compression mechanism in websocket would be used, as SPDY
will have already compressed what is compressible.   It strikes me
as somewhat duplicated/wasted effort to have compression in both
layers.

Ideally websockets should be able to provide framing compression
sufficient so that SPDY (or any new application layer) would not
be required to have it's own compression mechanisms.

For SPDY, the problem of using framing compression is that the
decision to compress the headers and/or the body is separate.
So perhaps the frame batch concept in BWTP might be useful
after all - so that a HTTP message could be a batch of a
two frames, one (normally compressed) containing the headers
and another (optionally compressed) containing the body.

Note also that compression efficiency means that the current
1 header field per frame approach in the latest BWTP draft
is a non starter.

cheers

---

From: John Tamplin <[email-masked]>
Date: Mon, 26 Apr 2010 10:14:58 -0400

> 1 header field per frame approach in the latest BWTP draft

Actually if you maintain compression state across frames it shouldn't be
that bad -- only a few bits would be lost for padding, and less than you
lose to additional framing anyway.

---

From: Mike Belshe <[email-masked]>
Date: Mon, 26 Apr 2010 09:28:11 -0700

> framing layer is that it's frames may contain both headers and

The most efficient compression will always come from the entity which knows
the most about what it is compressing.   Compressing headers separately from
other content compresses more efficiently than if you mix compression of
headers and data.  (Imagine a gzip buffer with 32KB of window where you
first send 500B of headers and then 50KB of data through it - by the time
your next chunk of headers come through, all state from the prior headers
has been lost.  And the data that you sent through might have been
compressed anyway!)

I welcome you to test out this theory - I've done so and I know the answer
:-)   As you can see, header-specific compression is a feature.

Also - remember the note about zlib memory footprint.  Using an 11bit window
size works just fine for headers because they are generally small, and this
keeps overall zlib memory pretty low (25K or so?).  But, if you want a
general compressor, you'll need the full 15bit compression window if you
want decent efficiency, and then you're looking at 256KB of RAM for each
zlib compression state instance.  I know, you could use a different
compressor - but most compressors will have a similar property.

The point is that a content-specific compressor will almost always
outperform a generic compressor in efficiency, speed, and footprint.

> as somewhat duplicated/wasted effort to have compression in both

Roberto tried to convey why optional compression is a bad thing sometimes.
 Let me use a real world example.

We've wondered here at Google why so many requests come from browsers
claiming not to support compression.  After a lot of investigation (I was
not involved with this work), we ran into client requests which contained
headers like:
     Accept-Xncoding: gzip, deflate

What is that?

Well, it turns out that some anti-virus providers didn't want to support
compression in payloads.  Doing so meant that sometimes they couldn't
inspect the payloads, and they were too lazy to implement the basic,
well-known compressors.  So these anti-virus programs watched for outbound
HTTP requests and intentionally mangled the browser's outbound
"Accept-Encoding" header.  This prevents the server from knowing the browser
supports compression, and thus the anti-virus software can look through the
result without having to do any decompress work.   But what happened here?
 The poor user in this case just had the single-worst-thing happen to his
browser - no compression at all!!!  And all because of some lazy
man-in-the-middle.

This is an example where too much flexibility over an insecure pipe bites
you.  And the best answer is to have the protocol help avoid it.  You can do
so by encrypting everything to avoid tampering, or you can avoid it by
mandating compression.  If compression really is a key feature of the
protocol, then allowing some implementations to turn it off is a bad idea.
 It will come back to haunt you.

Summary:
  * Compression is always best handled by the party that knows what is being
compressed.  (Compress at the highest layer possible)
  * Generic stream-compression of unknown content is okay, but not great.
 (because the content may well already be compressed)
  * Don't allow endpoints to not-support compression.  If they're allowed to
skip it, you end up with a critical feature missing for some poor users.
  * Don't allow intermediaries to disable your compression. This can only be
done by removing configuration flags/bits or by encrypting everything.

> be required to have it's own compression mechanisms.

I don't buy it for the reasons above.  But if you can prove me wrong, then
great!

> decision to compress the headers and/or the body is separate.

Why is this a problem?

> and another (optionally compressed) containing the body.

Okay.  I didn't have a problem with the batch frame - it's just a different
syntax.

Mike

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 27 Apr 2010 13:47:46 +0200

Mike Belshe wrote:
> On Mon, Apr 26, 2010 at 7:05 AM, Greg Wilkins <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Mike,
> 
> The most efficient compression will always come from the entity which
> knows the most about what it is compressing.   Compressing headers
> separately from other content compresses more efficiently than if you
> mix compression of headers and data.  (Imagine a gzip buffer with 32KB
> of window where you first send 500B of headers and then 50KB of data
> through it - by the time your next chunk of headers come through, all
> state from the prior headers has been lost.  And the data that you sent
> through might have been compressed anyway!)  
> 
> I welcome you to test out this theory - I've done so and I know the
> answer :-)   As you can see, header-specific compression is a feature.

I'm not disputing this.   Yet it is still useful to have compression
available when the source of the data does not compress it - otherwise
this would be an argument for HTTP not supporting compression.

If we are to have a multi-layered protocol then it would be
desirable to avoid every layer having it's own compression mechanism.

So my point was that since we appear to recognize that compression
is valuable to have in the framing layer (to handle apps that
do not compress), it would be great if we could design that
with enough flexibility so that SPDY over websocket would
not need to implement it's own compression.    Ideally the
framing compression would be flexible enough so that SDPY
would be able to specify a content-specific compressor
for header frames.

Now the complexity of layers delegating compression to
the framing layer may well be equal to the complexity of
each layer implementing it's own framing.  But I do think
it worth while evaluating if this is true or not.

> Roberto tried to convey why optional compression is a bad thing

Again, I'm not really disputing the validity of Roberto's example.
I understand that there are are intermediaries that act less than
sensibly.

However, I'm not sure that is sufficient reason to mandate a fixed
compression algorithm.    To do so, we would raise the complexity
of the simplest client (and while I agree that is not by much, there
is still a definite sentiment that we should keep it really simple).

Also, if we are to support future compression algorithms (or
content-specific ones), then we must support negotiated.  In
which case, there will be null compressors and lazy
intermediaries will just negotiate a noop compressor.

What Roberto's example does show, is that there in a genuine
desire of intermediaries (even ones baked into the client
computer) to inspect and add value to passing traffic.

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 27 Apr 2010 14:37:24 +0100

Mike Belshe wrote:
  And the best answer is to have the protocol help avoid it.
>     You can do so by encrypting everything to avoid tampering, or you can
>    avoid it by mandating compression.

It's not necessarily "best".

First, if you encrypt everything end to end, that will prevent
intermediaries from applying joint compression to multiple streams.
That is most relevant to opportunistic multiplexing, but it isn't the
only way.

The results which started this thread strongly imply that joint
compression of a stream of small objects is sometimes far better than
individual compression of them - enough to tip the balance in favour
of compression versus not, in some cases.

Second, end to end encryption rules out some devices from using the
protocol at all - they don't have enough processing capacity and/or
memory to do it.

Third, same applies to mandatory compression, if it is one of the
compression types which needs a lot of processing and/or memory.

Fourth, we know that many client locations block many different kinds
of outgoing traffic, and some of them block anything they don't
understand.  That no doubt includes all encrypted traffic from some of
them.  It should not be mandatory to encrypt, for that reason.

Fifthly, forcing implementations to use a compression scheme which was
good in the year 2000 (deflate), is looking a bit dated now for
compression performance (bzip2, lzma are better for many things), and
is too compute intensive for on the fly encryption on slower or
heavily loaded or power-sensitive devices (lzo family are better for
that)... projecting ahead to 2015, 2020, the "mandatory" part of the
spec would probably get ignored or worked around by many implementations.

Sixthly, we're assuming proxy-caches play no role in these protocols.
With WebSocket itself, that's true, but with protocols like SPDY that
might be layered over it, it is likely (imho) that a role will be
found for proxy-caches.  End-to-end encryption is not compatible with
that.

-- Jamie

---

From: Mike Belshe <[email-masked]>
Date: Tue, 27 Apr 2010 07:40:45 -0700

On Tue, Apr 27, 2010 at 4:47 AM, Greg Wilkins <[email-masked]> wrote:

> this would be an argument for HTTP not supporting compression..

> would be able to specify a content-specific compressor

Ok.

> computer) to inspect and add value to passing traffic.
Yes, and they will do so even if it means ruining the poor user's
experience! :-)

Mike

---

From: Mike Belshe <[email-masked]>
Date: Tue, 27 Apr 2010 07:44:29 -0700

> protocol at all - they don't have enough processing capacity and/or

I don't believe this to be true.

> compression types which needs a lot of processing and/or memory.

I don't believe this to be true.  Processing power grows very fast in all
hardware.  You're going to have a quadcore on your mobile phone in 5 years.

> them.  It should not be mandatory to encrypt, for that reason.

This is wrong; encryption protects the user and is valuable.  It will be
coming whether you like it or not.

> spec would probably get ignored or worked around by many implementations.

Fine - I never made a claim that a specific algorithm must be mandated.

> found for proxy-caches.  End-to-end encryption is not compatible with

You're ignoring the problems caused by proxies.  I won't go into them again.
 But the hazards are so large that, for the last 10 years, basic protocol
changes have been unimplementable due to proxies.  New protocols should not
ignore this.

Mike