Subject: [hybi] Restarting IESG review on permessage-deflate

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 4 Oct 2013 16:08:52 +0900

Hi HyBi chairs,

After addressing Barry's comments, the spec went through reviews by several
people. It's not receiving any critical issue report recently.

We have two client implementation (Autobahn and Chromium) and three server
implementations (Autobahn, WebSocket++, pywebsocket).

I think we can restart IESG review. What do you think?

---

From: Joakim Erdfelt <[email-masked]>
Date: Fri, 4 Oct 2013 07:24:10 -0700

There is also Eclipse Jetty 9.1 (currently in RC releases) support (server
and client)

We cannot support some of the spec, as written.
Our experiences implementing it, and using chromium + pywebsocket for
testing both the server or client implementation.

The following configuration parameters are not supported, nor can/will be.

s2c_no_context_takeover
c2s_no_context_takeover
s2c_max_window_bits
c2s_max_window_bits

This level of control of the deflate algorithm is just not possible in
Java, sorry.
We are not using our own implementation of Deflate, as the one provided by
Java itself is better performant (both in speed and memory use)

We also find the nature of configuration parameters awkward (even if Java
supported these specific ones)
The whole "s2c" and "c2s" prefix noise is unnecessary.

Our requests for this spec:

* eliminate the "s2c" and "c2s" prefixes.
   "c2s" == request, from client, what it wants to do
   "s2c" == response, from server, what it wants to do
   draft-ietf-hybi-permessage-compression-12, doesn't make the case for the
need for these prefixes, they seem superfluous, duplicating configuration
parameters just to add context on which direction they have an effect on.
 something the standard websocket handshake already accomplishes nicely.
so either the stated reason for them in the spec documentation is unclear,
or they truly are irrelevant prefixes.
   imagine a world where we had ...
   POST /upload HTTP/1.1
     Request-Content-Length: 6000
   HTTP/1.1 200 OK
     Response-Content-Length: 50
* define default deflate behavior better.
   http's content encoding and compression has default behavior, for every
flavor of compression scheme.
   even spdy has default behavior (note: while spdy has these kinds of
optional deflate tweaks, in the wild, and in practice, no server currently
honors them)
* move the existing configuration parameters (for no_context_takeover and
max_window_bits) out into optional advanced configuration tweaking.

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 4 Oct 2013 08:26:30 -0700

> This level of control of the deflate algorithm is just not possible in Java, sorry.

Well, I'm sorry for the Java community not being able to fix this in 10+ years;)

> We are not using our own implementation of Deflate, as the one provided by Java itself is better performant (both in speed and memory use)

I feel your pain, but:

1) Z_SYNC_FLUSH is available in Java now
http://bugs.sun.com/view_bug.do?bug_id=4206909

2) FWIW, as a sidenote, here is a pure Java implementation capable of the respective features:
https://github.com/ymnk/jzlib

>We also find the nature of configuration parameters awkward (even if Java supported these specific ones)

No, it's not.

>Our requests for this spec:

I'd vote against changing the spec at this stage:

- it has proved to work
- the parameters and the control offered makes sense
- just because Java is lacking should not impose limitations on everybody else (C, C++, Python, NodeJS (http://nodejs.org/api/zlib.html#zlib_options), and more all have support for the respective options).

Please: just fix Java.

/Tobias

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 04 Oct 2013 17:29:57 +0200

>>The whole "s2c" and "c2s" prefix noise is unnecessary.

What is your reasoning?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 4 Oct 2013 08:57:25 -0700

>>>We also find the nature of configuration parameters awkward (even if 
>>>Java supported these specific ones) The whole "s2c" and "c2s" prefix noise is unnecessary.
>>
>>No, it's not.

>What is your reasoning?

We need to differentiate between parameter offers and parameter accepts, and do that for both directions.

If there is no prefix, both client and server would only be able to use the same parameters for both directions.

/Tobias

---

From: Joakim Erdfelt <[email-masked]>
Date: Fri, 4 Oct 2013 09:25:56 -0700

> the same parameters for both directions.
That's desired actually, and consistent with behavior of other protocols
and extensions (mux for example).
The websocket handshake is always client initiated to server.
The client always says what it desires on the handshake request.  (this is
the spec "c2s")
The server always says what it can do on the handshake response.  (this is
the spec "s2c")
Where does the need for the "s2c" and "c2s" prefixes come from?
Even the justification for them in the permessage-deflate spec is lacking.

To put it another way:

Handshake Request:
"c2s_no_context_takeover"
"c2s_max_window_bits"

Handshake Response:
"s2c_no_context_takeover"
"s2c_max_window_bits"

The spec use of these prefixes is unclear at best, and downright misleading
at worst.
If the spec is trying to say something like this ...

   Hi, I'm the client, I want to talk to you using 1 set of configurations,
and I want you to talk back to me on a different set of configurations.

... then this is not made clear in
http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-12

And If this is the intention, make it clear, in the spec, that the whole
point of the prefixes is to have different configurations for incoming vs
outgoing frames. (and explain the pros and cons of this)

- Joakim

---

From: Joakim Erdfelt <[email-masked]>
Date: Fri, 4 Oct 2013 09:55:54 -0700

> Well, I'm sorry for the Java community not being able to fix this in 10+

Yet, oddly, with web technologies, this hasn't been a problem since our
initial use of Oak in 1995 (yes, Jetty has existed since 1995)
Also, the opposite is true as well.
The capabilities of one set of libraries (c & python) should not be forced
on all environments as well.
The spec should not have these kinds of things mandated or required.
We have no problem with a spec having advanced features and configurables
such as these, but don't force them.

>
> > We are not using our own implementation of Deflate, as the one provided
> by Java itself is better performant (both in speed and memory use)
>
> I feel your pain, but:
>

This does not operate in the same way as what pywebsocket expects.  However
chromium's behavior is consistent (or resilient, hard to say)

> https://github.com/ymnk/jzlib

Again, not performant, and uses too much memory, and is ultimately at odds
with precedents already set with other web technologies.
Remember, websocket is a web technology at its heart.
Imagine the opposite, a theoretical spec mandated something that would mean
you could not use your native implementations of something common, lets say
for argument that you suddenly were forced to, by a spec, to handle
filesystem metadata tags.
Now what?
do you toss out all of your standard file i/o and filesystem libs and start
anew?
do you force the entire environment ecosystem to catch up to
this theoretical spec?
do you introduce another file system library mechanism just to handle this
theoretical spec? having 2 filesystem libraries at the same time?

As you can see, these concepts might have value, for some, but should not
be forced or required.
These are advanced features, optional features.

- Joakim

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 4 Oct 2013 09:58:16 -0700

>That's desired actually, and consistent with behavior of other protocols and extensions (mux for example).
...
>Where does the need for the "s2c" and "c2s" prefixes come from?

The need comes from having different parameters for up- vs downstream.

The value of having different parameters up- vs. downstream is being able to optimize for the resource restrictions of both (client & server) independently.

/Tobias

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 4 Oct 2013 10:11:37 -0700

>Yet, oddly, with web technologies, this hasn't been a problem since our initial use of Oak in 1995 (yes, Jetty has existed since 1995)

Jetty being around since 1995 is obviously awesome!

As a strange coincidence, zlib has also be around since 1 May 1995 (I haven't checked whether the respective options have been in the first version though).

> The capabilities of one set of libraries (c & python) should not be forced on all environments as well.

Lets collect a list: a) stuff that exposes the complete zlib vs b) stuff that does not;)

a) C, C++, Python, NodeJS, Perl, Ruby, PHP, Erlang, Haskell
b) Java

Caveat: I (obviously) have not used all of above myself. Just had a quick glance at their docs.

Pointers:
http://perldoc.perl.org/Compress/Zlib.html#Deflate-Interface
http://www.ruby-doc.org/stdlib-2.0.0/libdoc/zlib/rdoc/Zlib/Deflate.html
http://www.php.net/manual/de/function.gzdeflate.php
http://erlang.org/doc/man/zlib.html
http://hackage.haskell.org/package/zlib-0.5.4.1/docs/Codec-Compression-Zlib.html

I stop here;)

/Tobias

---

From: Joakim Erdfelt <[email-masked]>
Date: Fri, 4 Oct 2013 10:41:01 -0700

Ok, add to the b) side of the equation:

scala - same classlib and jvm as java
jython - same classlib and jvm as java, no native zlib support here
php - http://www.php.net/manual/en/ref.zlib.php (cannot support
max_window_bits, no_context_takeover might be possible)
c# -
http://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream_methods.aspx
(cannot
support max_window_bits, no_context_takeover might be possible)

--
Joakim Erdfelt <[email-masked]>
webtide.com <http://www.webtide.com/> - intalio.com/jetty
Expert advice, services and support from from the Jetty & CometD experts
eclipse.org/jetty - cometd.org

On Fri, Oct 4, 2013 at 10:11 AM, Tobias Oberstein <
[email-masked]> wrote:

> As a strange coincidence, zlib has also be around since 1 May 1995 (I
> haven't checked whether the respective options have been in the first
> version though).
>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 04 Oct 2013 19:57:22 +0200

>outgoing frames. (and explain the pros and cons of this)

That seems to be in order.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 04 Oct 2013 20:08:43 +0200

>a) C, C++, Python, NodeJS, Perl, Ruby, PHP, Erlang, Haskell

If you have an environment to develop application in C then it should be
relatively easy to compile the zlib library and link it to your program,
but the C language and its standard library and the typical extensions
to it do not usually include zlib by themself; and if you write in Java,
there are comparable libraries available where the same holds true, but
the out-of-the-box experience is different. So this does not seem to be
a fair comparison.

A better comparison would be among independent DEFLATE implementations;
that way we could at least tell how zlib-specific the relevant features
are, and if they are very zlib-specific and not commonly implemented in
independent implementations, they might make bad mandatory features. My
http://bjoern.hoehrmann.de/pngwolf/ for instance uses the DEFLATE imple-
mentation in Igor Pavlov's 7-Zip library, which I believe is the best-
compressing free software implementation; that would be a good candiate,
among others.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 4 Oct 2013 11:34:44 -0700

>If you have an environment to develop application in C then it should be relatively easy to compile the zlib library and link it to your program, but the C language and its standard library and the typical extensions to it do not usually include zlib by themself; and if you >write in Java, there are comparable libraries available where the same holds true, but the out-of-the-box experience is different. So this does not seem to be a fair comparison.

Of course making up such lists is questionable.

My main point is: should IETF protocols be designed against a least-common denominator of todays existing language run-times / implementations / environments? I don't know if there is a general IETF policy with this. My view: it should not.

>A better comparison would be among independent DEFLATE implementations; that way we could at least tell how zlib-specific the relevant features are, and if they are very zlib-specific and not commonly implemented in independent implementations, they might >make bad mandatory features. My http://bjoern.hoehrmann.de/pngwolf/ for instance uses the DEFLATE imple- mentation in Igor Pavlov's 7-Zip library, which I believe is the best- compressing free software implementation; that would be a good candiate, among >others.

Good point. It's one thing which run-time does expose which feature of one and the same Deflate implementation ("zlib"), and another to compare features of independent Deflate implementations.

Here is another point:

The current spec only refers to the original LZ77 paper under normative-references:

http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-12#page-27

The Deflate RFC

http://tools.ietf.org/html/rfc1951

is only listed under informational.

This is strange, since the "permessage-deflate" extension is not only LZ77, but really Deflate (= LZ77 + Huffman Coding + Window Size .. see below).

And the RFC1951 mentions:

" 3.3. Compliance

      A compressor may limit further the ranges of values specified in
      the previous section and still be compliant; for example, it may
      limit the range of backward pointers to some value smaller than
      32K.  Similarly, a compressor may limit the size of blocks so that
      a compressible block fits in memory.
"

Which I read as refering to "window size". But the wording neither has SHOULD nor MUST in it.

Mmh.

/Tobias

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 5 Oct 2013 04:03:03 +0900

Joakim,

I put s2c and c2s prefixes in order to allow response to describe settings
of both direction.

When to allow a client to include multiple permessage-deflate offers, we
need to have some method to have the server to tell the client which offer
has been accepted. I investigated several methods such as adding IDs to
each offer, and then concluded that having the server to send all
parameters (both client -> server and server -> client) is simple and ok.
But to use this method, we need to distinguish parameters for up and down.
So, I introduced these prefixes.

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 5 Oct 2013 04:17:26 +0900

> The following configuration parameters are not supported, nor can/will be.
Sorry, why? s2c_no_context_takeover requests the server to reset context.
The server just need to throw away current deflater, create a new one and
use it.

If the client didn't send s2c_no_context_takeover, you can still do the
same. You don't have to carry over compression context.

The problem is that your server cannot decompress incoming data if the
client carries over compression context. So, you can just reply with
c2s_no_context_takeover. In
http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-12#section-8.1.1.2,
a client is required to implement c2s_no_context_takeover. I made this
mandatory since it's doable. So, you don't have to worry about clients
coming without c2s_no_context_takeover capability.

> This level of control of the deflate algorithm is just not possible in
Again, please explain why you can't support no_context_takeover family
parameter.

---

From: Peter Thorson <[email-masked]>
Date: Fri, 4 Oct 2013 14:56:35 -0500

On Oct 4, 2013, at 11:25 , Joakim Erdfelt <[email-masked]> wrote:

> The spec use of these prefixes is unclear at best, and downright misleading at worst.
>    Hi, I'm the client, I want to talk to you using 1 set of configurations, and I want you to talk back to me on a different set of configurations.
> 
> ... then this is not made clear in http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-12 
> 
> And If this is the intention, make it clear, in the spec, that the whole point of the prefixes is to have different configurations for incoming vs outgoing frames. (and explain the pros and cons of this)

Based on my reading of the draft 12 spec sections for the individual method parameters (sec 8.1.1.1-4), it seemed clear to me that the intention is specifically to allow asymmetric configuration.

This is useful because the nature of the deflate algorithm and the capabilities of endpoints are likely to be asymmetric. Deflate typically takes significantly more memory than inflate. Desktop browsers are likely to be less memory sensitive than a server processing tens of thousands of connections each needing their own sliding window or a server running on an embedded device with limited memory. In these cases the resource rich endpoint may want to use its extra memory for a better outgoing compression ratio while allowing the resource poor endpoint to compress less thoroughly to conserve memory.

While I think the present descriptions of each individual parameter convey this information, I wouldn't be opposed to clarifying the overview part of section 8 to review the asymmetric nature of the parameters. A proposal:

======= Existing =======

For an offer for this extension, the following 4 extension parameters
are defined.  If needed to distinguish from ones for a response,
these parameters are called with a prefix "client-to-server".

- "s2c_no_context_takeover"
- "c2s_no_context_takeover"
- "s2c_max_window_bits"
- "c2s_max_window_bits"

For a response for this extension, the following 4 extension
parameters are defined.  If needed to distinguish from ones for an
offer, these parameters are called with a prefix "server-to-client".

- "s2c_no_context_takeover"
- "c2s_no_context_takeover"
- "s2c_max_window_bits"
- "c2s_max_window_bits"

======= Possible clarification =======

Four extension parameters are defined for permessage-deflate to help 
endpoints manage per-connection resource usage.

- "s2c_no_context_takeover"
- "c2s_no_context_takeover"
- "s2c_max_window_bits"
- "c2s_max_window_bits"

These represent two methods of constraining memory usage that may be 
applied independently to either direction of WebSocket traffic. The prefixes 
c2s and s2c are used to indicate the client-to-server channel and the 
server-to-client channel respectively. All four parameters are defined for both 
a client's offer and a server's response.

=====================

> - Joakim
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Takeshi Yoshino <[email-masked]>
Date: Sat, 5 Oct 2013 10:19:11 +0900

> Again, please explain why you can't support no_context_takeover family

I tried using java.util.zip.Deflater and Inflater.

For compressing, used deflate() and finish() method. We need to strip a
ZLIB header (CMF and FLG) of 2 bytes at the head and an Adler 32 footer of
4 bytes at the tail from output.

For decompressing, we need to prepend a ZLIB header (CMF of 0x78 and FLG of
0x9c (some FLEVEL, FDICT=0, appropriate FCHECK)). Used inflate() method
until getRemaining() becomes 0. If we want to see Inflater.finished()
returning true to make sure all data is processed, append a block with
BFINAL e.g. 0x01 0x00 0x00 0xFF 0xFF (after appending 0x00 0x00 0xFF 0xFF
as specified in the I-D) and a valid Adler 32 footer (use
java.util.zip.Adler32).

---

From: Tobias Oberstein <[email-masked]>
Date: Sat, 5 Oct 2013 06:41:41 -0700

Takeshi,

thanks for all the work you and others put into the spec!

We'd love to see Chrome (both desktop and Android) ship with permessage-deflate enabled in production soon. Of course it'll be great if other browsers follow soon.

We have seen significant value / compression ratios (2-15x) for use with JSON-based protocol over WebSocket - great for mobile in particular!

And: we just released AutobahnPython with permessage-deflate support to the Python package repo: https://pypi.python.org/pypi/autobahn

So users can pick this up easily now. For clients and servers. It's ready.

We soon will ship an updated version of AutobahnTestsuite also that includes a whole new section for testing permessage-deflate (and our own, unofficial permessage-bzip2 and permessage-snappy).

/Tobias

Von: [email-masked] [mailto:[email-masked]] Im Auftrag von Takeshi Yoshino
Gesendet: Freitag, 4. Oktober 2013 09:09
An: [email-masked]; salvatore loreto; Gabriel Montenegro
Betreff: [hybi] Restarting IESG review on permessage-deflate

Hi HyBi chairs,

After addressing Barry's comments, the spec went through reviews by several people. It's not receiving any critical issue report recently.

We have two client implementation (Autobahn and Chromium) and three server implementations (Autobahn, WebSocket++, pywebsocket).

---

From: Adam Rice <[email-masked]>
Date: Mon, 7 Oct 2013 11:37:30 +0900

For PHP, soywiz has worked out that the "$encoding" parameter allows you to
set the number of window bits--see the comment at
http://www.php.net/manual/en/function.zlib-encode.php

Interestingly, the problem with PHP seems to be not supporting
no_context_takeover, but supporting the default context_takeover behaviour.
zlib_decode() creates a new zlib context every time.

There might be a way to abuse the gz* functions to do it, I have not
explored them in depth.

It is always possible to support no_context_takeover in any language by
creating a new compression context for each message.

On 5 October 2013 02:41, Joakim Erdfelt <[email-masked]> wrote:

> http://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream_methods.aspx (cannot

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 8 Oct 2013 11:45:41 +0900

On Sat, Oct 5, 2013 at 10:41 PM, Tobias Oberstein <
[email-masked]> wrote:

> We have seen significant value / compression ratios (2-15x) for use with
> JSON-based protocol over WebSocket - great for mobile in particular!
>
> ** **
>
> And: we just released AutobahnPython with permessage-deflate support to
> the Python package repo: https://pypi.python.org/pypi/autobahn****
>
> ** **
>
>
Great!

> ****
> ** **
> We soon will ship an updated version of AutobahnTestsuite also that
> includes a whole new section for testing permessage-deflate (and our own,
> unofficial permessage-bzip2 and permessage-snappy).
>

Thanks as usual. We'll check it out.

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 8 Oct 2013 12:50:21 +0900

Published a new version -13. Tried to address Joakim's comments. Please
check it out.

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 15 Oct 2013 23:00:52 +0900

On Sat, Oct 5, 2013 at 3:34 AM, Tobias Oberstein <
[email-masked]> wrote:

> This is strange, since the "permessage-deflate" extension is not only
> LZ77, but really Deflate (= LZ77 + Huffman Coding + Window Size .. see
> below).
>

Fixed in -14.

---

From: Takeshi Yoshino <[email-masked]>
Date: Tue, 15 Oct 2013 23:01:04 +0900

Thanks, Peter. I replaced mine with your text in -14.