Subject: [hybi] Proposed Text - Stream Based Compression

From: Patrick McManus <[email-masked]>
Date: Mon, 13 Sep 2010 09:47:41 -0400

Hi All, as promised last week here is an update to the compression
proposal. This one is stream based as that appears to be the more
popular approach. 

The summary is dead simple - basically the whole stream after the
handshake is encoded with deflate plus a requirement to flush() whenever
not doing so would cause part of a message to be buffered and not sent.
1951 includes both compressed and uncompressed block formats which can
be mixed in the stream so advanced senders which would like to turn
compression on and off can use that mechanism to do so without
websockets having to define anything in detail... basic use just
requires plugging in zlib.

-----

8. Known Extensions

Extensions provide a mechanism for implementations to opt-in to
additional protocol features. This section defines the meaning of
well-known extensions but implementations may use extensions defined
separately as well.

8.1. Compression

The registered extension token for this compression extension is
"deflate".

The extension does not have any per message extension data and it does
not define the use of any WebSocket reserved bits or op codes.

Senders using this extension MUST apply RFC 1951 encodings to all
bytes of the data stream following the handshake including both data and
control messages. The data stream MAY include multiple blocks of both
compressed and uncompressed types as defined by RFC 1951.

Senders	MUST NOT delay the transmission	of any portion of a WebSocket
message because the deflate encoding of	the message does not end on a
byte boundary. The encodings for adjacent messages MAY appear in the
same byte if no delay in transmission is occurred by doing so.

---

From: John Tamplin <[email-masked]>
Date: Mon, 13 Sep 2010 11:37:33 -0400

> requires plugging in zlib.

I still think compressing only the payload is better because:
 - you don't clutter up the compression dictionary with framing and
other incompressible data
 - compressing the payload means you can use the shorter payload length formats
 - AFAIK, zlib does not provide a way to send BTYPE=00 blocks (though
maybe you can just fake the BTYPE=00 block on the stream)
   - overhead on uncompressed blocks is at least 5 bytes as we define
the length redundantly, instead of using 1 reserved bit

That said, having reasonable compression built in, even if a better
one is created later, is a good thing.

Regarding extension negotiation, I would like to see the option for
passing additional parameters for the extension.  Perhaps something
like a comma separated list of extensions, and then optional parens
with tag/value pairs (whatever the server replied with is what will be
used, extensions would specify how the server's response is
constrained by what the client sent).  A hypothetical example:

Whatever-Header-We-Use: deflate, mux(maxchunk=65536; combinekeepalives=1)

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Dave Cridland <[email-masked]>
Date: Mon, 13 Sep 2010 17:40:00 +0100

> Whatever-Header-We-Use: deflate, mux(maxchunk=65536;  

I think several existing headers use the same format as the HTTP  
Accept header:

Some-Header: deflate, mux; maxchunk=65535; combinekeepalives=1,  
otherextension

This would seem reasonably sensible.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: John Tamplin <[email-masked]>
Date: Mon, 13 Sep 2010 12:42:49 -0400

On Mon, Sep 13, 2010 at 12:40 PM, Dave Cridland <[email-masked]> wrote:
> I think several existing headers use the same format as the HTTP Accept
> header:
>

Works for me.

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 13 Sep 2010 12:12:08 -0500

John Tamplin wrote:
>  - you don't clutter up the compression dictionary with framing and other
> incompressible data

This assumes that the framing would be incompressible. I think it is too
early to come to that conclusion. In fact, if you can rely on the framing
being compressed then a lot of problems regarding the encoding of frame
headers are pretty much solved automatically. 

>  - compressing the payload means you can use the shorter payload length

DEFLATE implementations do a pretty good job of automatically optimizing
this.

>  - AFAIK, zlib does not provide a way to send BTYPE=00 blocks (though
maybe
> you can just fake the BTYPE=00 block on the stream)

You can use zlib's deflateParams function to set the compression level to
zero. java.util.zip.Deflater.setLevel appears to do the same thing.

>    - overhead on uncompressed blocks is at least 5 bytes as we define the
length
> redundantly, instead of using 1 reserved bit

You might end up paying this 5 byte cost anyway because a sync flush outputs
an ~5.5 byte empty BTYPE=00 block and a partial flush has just slightly less
overhead.

> That said, having reasonable compression built in, even if a better one is
created
> later, is a good thing.

This is exactly my thinking. Compression that is simple to implement and
more efficient than HTTP's compression is already a big win. Applications
that require even more compact encodings are probably better off doing their
own application-specific compression in the application layer.

Cheers,
Brian

---

From: John Tamplin <[email-masked]>
Date: Mon, 13 Sep 2010 14:04:25 -0400

> DEFLATE implementations do a pretty good job of automatically optimizing

That hasn't been my experience -- in general, adding a bunch of
symbols to the encoding table (ie replacing what would have been a
length of 125 with dozens of uncompressed lengths that all compress to
that size) means adding a bit or two to every compressed symbol.
Granted, the leading zeros will get compressed well, probably to a few
bits.

Still, this is fine for a first compression extension.  We can get
something in the spec and then we can easily evaluate performance of
proposals for improved compression extensions against this one.  Since
likely most WebSocket implementation will already have the necessary
code for HTTP compression, it isn't likely to add much work to
implementations that want to support it.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 14 Sep 2010 15:09:35 +0900

I agree that it's good to have something like this as a first extension, 
and if it's indeed true that both spec and implementation are as simple 
as the text sent in, that would be great.

However, I see an important problem on the horizon: Interaction between 
extensions. As an example, how would this extension interact with a 
multiplexing extension? Is it the multiplexing extensions's job to say 
something about this? Or the compression extension's job? How does this 
work in general?

Regards,   Martin.

On 2010/09/14 3:04, John Tamplin wrote:
> On Mon, Sep 13, 2010 at 1:12 PM, Brian Smith<[email-masked]>  wrote:

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: John Tamplin <[email-masked]>
Date: Tue, 14 Sep 2010 02:47:34 -0400

On Tue, Sep 14, 2010 at 2:09 AM, "Martin J. Dürst"
<[email-masked]> wrote:
> I agree that it's good to have something like this as a first extension, and
> if it's indeed true that both spec and implementation are as simple as the
> text sent in, that would be great.
>
> something about this? Or the compression extension's job? How does this work
> in general?

In this case, since the compression is of the entire stream then
clearly the resulting mux streams would be compressed if it was
negotiated on the mux connection (which does pose a bit of a problem
for "upgrading" a single connection to a mux'd connection when you
need a second channel, since you don't want to compress already
compressed data, but that could be worked around).  For the logical
channels, either a multiplexing intermediary could participate in the
negotiation to force no compression (since it could compress the
physical stream), or it could make the outer channel uncompressed and
all the mux'd frames would be compressed according to the negotiation
on their logical channel (though in that case the mux frame would have
to include its own sub-framing rather than reusing the underlying
channel's framing since it wouldn't want to have to uncompress and
recompress at different frame boundaries).

This interaction is one reason I prefer the per-frame compression,
since it seems more flexible for such interactions.

I think the general pattern is that any new extension is going to have
to define how it works with any already-defined extension.  Maybe
after we have a better idea of what these extensions will do and how
they interact with each other, we could write some general rules for
it, but I don't think we can do that to start.

---

From: John Tamplin <[email-masked]>
Date: Tue, 14 Sep 2010 02:48:46 -0400

> The registered extension token for this compression extension is

BTW, I suggest a more specific keyword here to preserve good options
for future compression extensions -- how about deflate-stream?