Subject: [hybi] Payload only compression extension, again

From: Takeshi Yoshino <[email-masked]>
Date: Mon, 28 Feb 2011 23:51:18 +0900

Hi,

Seeing several people preferring payload only compression, I'd like to
resume discussion on it.

Here's requirements and my strawman proposal based on Patrick's proposals
and recent discussion.

I've removed compression indication that might be implemented by some
reserved bits or opcode for now. Please use this as starting point and
propose any extension to add necessary features. Different from the time
deflate-stream was defined, now extension negotiation is well defined in the
spec.

----

Requirements
- Each message must be able to be uncompressed immediately
-- Use Z_SYNC_FLUSH or Z_FINISH at the end of application data
- Want to maintain the compression state (LZ77 sliding window) between
frames
-- Use Z_SYNC_FLUSH
- 4 octet overhead of Z_SYNC_FLUSH is too big
-- Use the same scheme as RFC 1979 PPP Deflate to eliminate it
- Want not to mess up dictionary by compressing incompressible messages
-- Use BTYPE=0b00 block for such messages

Requests I gave up to address in the strawman below for simplicity
- Want to maintain the compression state (LZ77 sliding window and Huffman
table)
- Want to keep separate dictionaries for messages with different
characteristics
-- Case 1: Binary and text mixed
-- Case 2: Short and long mixed
-- Case 3: Application data in data frame and control frame may have
different characteristics
--- Apply DEFLATE only to data frame (Is it worth including control frame?)
- Want to turn off compression for messages that becomes bigger when
compressed e.g. 1-octet message

----

Proposal

Compression

The registered extension token for this compression extension
is "deflate-application-data".

The extension does not have any per message extension data and it does not
define the use of any WebSocket reserved bits or opcodes.

Senders using this extension MUST apply RFC 1951 DEFLATE to all the bytes of
the application data portion of data frames. Senders MAY include multiple
blocks in a frame. Senders MAY use blocks of any type as defined in RFC
1951. Senders MUST use the algorithm described in Section 2.1 of RFC1979 to
align compressed data to byte boundary. Senders MAY keep using the same LZ77
sliding window for multiple frames.

Receivers MUST append 0x00 0x00 0xff 0xff to the application data portion
and decompress it using DEFLATE to decode the original application data.
Receivers MUST keep using the same LZ77 sliding window for all the frames of
the same WebSocket connection. Receivers MUST assume the senders use 32768
byte LZ77 sliding window.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 1 Mar 2011 18:01:57 +1100

> Seeing several people preferring payload only compression, I'd like to

+1

> - Each message must be able to be uncompressed immediately

+1

> - Want not to mess up dictionary by compressing incompressible messages

Is that to say that you don't want to compress fragments?
I'm cautious about this as we don't want to have to buffer up all
fragments before we can decompress.

> - Want to maintain the compression state (LZ77 sliding window and Huffman

+1, although it might be good to have this as an extension option that
can be turned off.

> different characteristics

-1 too complex.

I think that eventually MUX will be available that will keep different
types of messages in different channels.
Until that time, I expect to see very little mixed usage on WS
connections as each usage will just open up a fresh connection.

> --- Apply DEFLATE only to data frame (Is it worth including control frame?)

+1

control frames should be in the clear, so that they can be considered
and handled without the need to track extension or extension state.

cheers