Subject: [hybi] With deflate-stream, Close frame doesn't work as an end of data marker

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 16 Feb 2011 18:20:18 -0800

It's still under discussion if it's really necessary (see "Clarify the role
of closing handshake" thread), but RST suppression was one of the original
objectives of Close frame. Close frame assures no more data is coming on the
underlying channel to allow the endpoint to close(2) safely.

deflate-stream extension in -05 applies deflate on whole octets of a frame
including the header. If we simply use zlib's flush, we can spoil this
function of Close frame. There can be some data remain not processed/recv-ed
even after getting a Close frame uncompressed.

e.g.
(if Z_SYNC_FLUSH is used) possibly a few octets containing EoB symbol, 3-bit
header, padding, and then BTYPE=00 block "00 00 ff ff".
(if Z_FINISH is used) possibly one octet containing EoB symbol.

Endpoints need to drain these octets to prevent RST. This should be noted.

Takeshi

---

From: [email-masked]
Date: Thu, 17 Feb 2011 16:33:50 -0800

Hi Takeshi,

Good point. There is indeed a legitmate case where trailing bytes is 
present after 
Close frame.

It is important to note that after endpoint sent and received Close frame, 
it is considered 
that WebSocket layer close handshake has been complete ('wasClear' gets 
set to 1, 
etc.).

What Takeshi pointed out is a good motivation for WS-layer to perform 
graceful 
shutdown at TCP layer as well. After Close frame has been exchanged 
successfully, 
the endpoint must start shutting down the local TCP connection with 
shutdown(SHUT_WR), and discard the trailing bytes, if any received, and 
then finally 
close the socket on receipt of EoS.

This works for any trailing bytes that could be caused by artifact of 
another deflate 
algorithm that may be introduced in the future, or just by a bug.

- Yutaka

[email-masked] wrote on 02/16/2011 06:20:18 PM:

> It's still under discussion if it's really necessary (see "Clarify 
> the role of closing handshake" thread), but RST suppression was one 
> of the original objectives of Close frame. Close frame assures no 
> more data is coming on the underlying channel to allow the endpoint 
> to close(2) safely.
> 
> deflate-stream extension in -05 applies deflate on whole octets of a
> frame including the header. If we simply use zlib's flush, we can 
> spoil this function of Close frame. There can be some data remain 
> not processed/recv-ed even after getting a Close frame uncompressed.
> 
> e.g.
> (if Z_SYNC_FLUSH is used) possibly a few octets containing EoB 
> symbol, 3-bit header, padding, and then BTYPE=00 block "00 00 ff ff".
> Takeshi_______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 24 Feb 2011 22:11:09 +0900

> it is considered
> that WebSocket layer close handshake has been complete ('wasClear' gets set
> to 1,
> etc.).
>

> algorithm that may be introduced in the future, or just by a bug.
For this case, we can take another way without depending on TCP FIN
- have deflater compress Closing frame in BFINAL block (flush with Z_FINISH)
- have inflater drain compressed stream until it see EOB

This might work for most environments, but not all library/wrapper of zlib
expose API to recognize EOB/BFINAL immediately. For example, zlib module of
Python doesn't. See the explanation of zlib.Decompress.unused_data. If we
move Close frame out of compressed stream, we can use this, but it
complicates protocol.

I agree that using SHUT_WR + EoS is nice on platforms where they're
available, but I still want to provide a way that deals with this issue
without depending on them if possible.

Takeshi

---

From: [email-masked]
Date: Thu, 24 Feb 2011 21:19:18 -0800

Hi,

If we could apply deflate-stream only to payload, that would solve 
the problem. I was looking for reasons why entire stream has to be
compressed, and found only one here:

http://www.ietf.org/mail-archive/web/hybi/current/msg04117.html

>From layering aspects, I would prefer any compression taking place
on top of framing layer from the other way around:

   +-------------+
   | compression |
   +-------------+
   +-------------+
   | WS framing  |
   +-------------+
   +-------------+
   |     TCP     |
   +-------------+

where, current draft -05 sugggests:

   +-------------+
   | WS framing  |
   +-------------+
   +-------------+
   | compression |
   +-------------+
   +-------------+
   |     TCP     |
   +-------------+

Justifications for the former layering would be:

o  Framing is necessary because of the underlying transport
   layer is in fact 'streaming', or TCP. (TCP and WS-framing 
   are more tightly coupled)
o  WS-layer needs to buffer entire message (passed by app/JS)
   anyway.
o  Text and binary do not interleave fragments, hence there
   needs only one context for compression / decompression in
   memory.
o  Overhead (frame headers and control frames) are much smaller
   than payload of text/binary frames and negligible as to 
   compression ratio.
o  Close frame appears at the end of stream with no traling
   bytes. (Can remove requirement for TCP graceful shutdown).

Please let me know if I am missing something...

Best.
- Yutaka

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 25 Feb 2011 15:58:32 +0900

Hi,

> on top of framing layer from the other way around:
Yeah, Patrick initially proposed compression using that approach
http://www.ietf.org/mail-archive/web/hybi/current/msg04071.html

The options we can take are basically
a) compress only application data not keeping compression state
b) compress only application data while keeping compression state between
multiple frame
c) compress entire stream

John, Dave objected a) since it's degrading compression rate. I agree that.
So, b) and c) are left. Patrick wrote up c) as deflate-stream extension.

The name deflate-stream was given for c) (see
http://www.ietf.org/mail-archive/web/hybi/current/msg04129.html). I think
it's OK to leave deflate-stream extension as is. I'm also fine with this
built-in in the initial version of the WebSocket protocol spec. What we
should do now is try to define some "better-deflate" which convinces people.
These threads is in a kind of suspended state. We should use them or create
a new thread for resuming discussion on some "better-deflate" there.

- Some said we need ability to disable compression on frame with
incompressible contents
- Coordination with mux is also problem
- If frame header is compressible/incompressible is still a question.
etc...

For endpoints/intermediaries which don't recognize deflate-stream, yes, it's
problematic.

Except for this, technically, it's ok since frame boundary can be
immediately told to the other peer by flush-ing e.g. Z_SYNC_FLUSH of zlib.

>    needs only one context for compression / decompression in

People seemed to be more concerned about compression efficiency at that time
than the cost of compression state sharing. This is a point to revisit.

>    than payload of text/binary frames and negligible as to

Not sure.

>    bytes. (Can remove requirement for TCP graceful shutdown).
Yes, it's not a problem if we adopt payload only compression.

---

From: [email-masked]
Date: Fri, 25 Feb 2011 00:02:52 -0800

Hi Takeshi, my comments below...

> For endpoints/intermediaries which don't recognize deflate-stream, 
> yes, it's problematic.

What if we'd like to introduce a new control frame that is intended
to communicate with intermediaries (imagine, trace-route mechanism, 
etc.) in the future, then the 'compression' layer gets in the way 
and all intermediaries have to decompress stream all the time. 

This is clearly another consequence of having compression under the 
framing layer. We should avoid this, IMO.

> People seemed to be more concerned about compression efficiency at 
> that time than the cost of compression state sharing. This is a 
> point to revisit.

Yeah... as now we see downside of the current direction.

>    than payload of text/binary frames and negligible as to 

We may need more quantitative evidence here, though I agree 
larger scope (~32KB in case of gzip) for compression by
references for recurring texts would make compression ratio 
better. I also imagine that recurring texts may appear more in
the same message (or channel) than in the other channel. In mux'd
case for example, I am not sure how the current approach would
help.

>    bytes. (Can remove requirement for TCP graceful shutdown). 

I think this benefit is significant. We do not want to worry
about another trailing bytes in the future again when a new 
compression algorithm needs to be introduced.

Per message compression has another benefit of shrinking the 
size of buffer for storing application messages in memory.

- Yutaka

---

From: Andy Green <[email-masked]>
Date: Fri, 25 Feb 2011 09:34:51 +0000

On 02/25/2011 08:02 AM, Somebody in the thread at some point said:

Hi -

> and all intermediaries have to decompress stream all the time.

The intermediaries have to rewrite the compression too if there's ever 
anything they don't pass through, if the compression has any state held 
at both ends.

I thought about traceroute a bit when I wrote libwebsockets-test-ping 
which uses PING / PONG control frames just like normal ping app.

In fact you can do traceroute just with PING / PONG if the standard 
allowed for

  1) PONG MAY contain additional text after the PING payload echo

  2) PING MUST contain a TTL byte at the start

  3) Intermediaries MAY decrement the TTL when passing on PING

  4) PONG SHOULD be issued by intermediary IF TTL = 0 after the 
decrement and MUST be by the endpoint regardless of TTL

  5) When PONG issued, intermediary or endpoint MAY append its ID text

-Andy

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 08:34:42 -0500

On Fri, Feb 25, 2011 at 4:34 AM, Andy Green <[email-masked]> wrote:

> In fact you can do traceroute just with PING / PONG if the standard allowed
> for
>
>  4) PONG SHOULD be issued by intermediary IF TTL = 0 after the decrement
> and MUST be by the endpoint regardless of TTL
>

I am not sure how useful it would be if all these are MAY's -- in particular
the TTL seems useless if you don't require that it be decremented at each
hop.

However, requiring this would be complicate some simple intermediaries
dramatically -- I don't know if the value gained is worth anything.  Also,
there would seem to be a problem exposing this to JS -- you wouldn't want
hostile JS to be able to get a layout of the network behind the firewall.

What does this gain over using ICMP or UDP-based traceroute?

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Andy Green <[email-masked]>
Date: Fri, 25 Feb 2011 13:58:07 +0000

On 02/25/2011 01:34 PM, Somebody in the thread at some point said:

> dramatically -- I don't know if the value gained is worth anything.

Well if it made trouble it could just be ignored and your intermediary 
is invisible to it without breaking anything.  But I guess most 
intermediaries which could consider it are going to be parsing the 
websocket stuff enough to understand the opcode anyway, it doesn't seem 
like any radical complication to do it from that basis.

>   Also, there would seem to be a problem exposing this to JS -- you
> wouldn't want hostile JS to be able to get a layout of the network
> behind the firewall.

That's why it was a MAY, just like some places filter ICMP ECHO in and out.

Some ninja intermediaries might not want to participate in a headcount 
either for their own reasons.

> What does this gain over using ICMP or UDP-based traceroute?

Well, traceroute is telling you about things that at least perform a 
routing action whereas this websocket-traceroute is telling you about 
websocket intermediaries on your path that might be meddling with your 
websocket stuff AS websocket stuff.

It's not critical but since it's only a small addition to existing 
opcodes, I thought I would describe it since adding opcodes to do it was 
mentioned.

-Andy

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 28 Feb 2011 20:52:25 +1100

I'm +1 for compression within the frame.

If we compress the framing, then intermediaries will have to
decompress in order to see framing and to know when to forward etc.
It also has significant debug issues as it makes it much harder to
decode a partial frame capture.

I understand the argument that compression will not be as good, but I
think some compromise is necessary.

To look at the analogy of HTTP, compression does not change HTTP
framing and intermediaries do not need to be updated to deal with any
new compression algorithms that may be deployed.

regards

On 25 February 2011 16:19,  <[email-masked]> wrote:
> _______________________________________________
> hybi mailing list
> [email-masked]

---

From: John Tamplin <[email-masked]>
Date: Mon, 28 Feb 2011 09:32:03 -0500

[BTW - here is the thread where I first posted results of an
experiment - http://www.ietf.org/mail-archive/web/hybi/current/msg01810.html
]

> think some compromise is necessary.

The framing is pretty small and it is hard to imagine much redundancy
from the header (the opcode and reserved bits will be constant, but
then you will interfere with the compression by all the constantly
changing lengths), so I think little is lost by compressing only the
payload.

However, it does mean the implementation is more complicated, and I
don't believe this would be feasible using the stock inflate/deflate
in Java for example.  I would not agree to just compressing each
frame's payload independently -- my earlier experiments show this will
frequently increase the size of the data and result in only modest
savings compared to maintaining compression state over the life of the
connection (while we are discussing it, we should talk about the
parameters to use for deflate to minimize memory requirements on the
server -- experiments done for compression in SPDY [mentioned in the
linked thread] show you can get most of the benefit with more modest
memory requirements using lower parameters).

It also means we have to agree on the details of how it should be
specified, which is why the super-simple deflate-stream was added to
the standard instead of trying to get agreement on something better.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 1 Mar 2011 07:40:40 +1100

> the standard instead of trying to get agreement on something better.

Well I remember the proposal for deflate stream being made, but I
certainly don't recall any consensus that it should be put into the
draft.  There have been consistent voices against compressing the
framing.

I really think this is an important point of principal, if any
arbitrary extension is free to entirely change what is sent on the
wire, then we are going to have a very difficult environment in which
to develop any kind of meaningful intermediaries.    I keep coming
back to the analogy of HTTP, where it is not legal to just add a
header to define an extension and then change HTTP framing entirely.
The only way to break out of HTTP framing is with something like
Upgrade or CONNECT.

If the first exemplar of an extension we have is something that
completely changes what is sent over the wire, then I think we have
totally devalued WS framing as a standard.

regards

---

From: Brian McKelvey <[email-masked]>
Date: Mon, 28 Feb 2011 13:48:45 -0800

I completely agree.  Extensions should have free reign over the payload, but not be allowed to change the framing at all.  In fact the framing layer itself could be imagined as the first and last stage in the middleware stack, for the receiver and sender, respectively.  

Once all the extensions have executed their transformations on the sending node, then and only then is the framing generated from the state of the internal data structures that describe the current frame.

And since we're already not supporting things like "sendfile" and arbitrarily long messages of unknown length can be fragmented, this works quite cleanly and simply.

Brian

Sent from my iPhone

---

From: [email-masked]
Date: Mon, 28 Feb 2011 23:37:17 -0800

Hi John,

[email-masked] wrote on 02/28/2011 06:32:03 AM:

> experiment 
- http://www.ietf.org/mail-archive/web/hybi/current/msg01810.html
> changing lengths), so I think little is lost by compressing only the

Just to be clear, the compression does not have to be per payload of
a data frame, but can be entire stream of messages application passes in.
At the end of the message, though, F_SYNC_FLUSH is necessary to prevent 
receiver from waiting for the last compressed data to arrive from sender, 
but we should be aware that the compressor still maintains 32kB(or 
whatever 
it is configured for) of reference range in its context.

> However, it does mean the implementation is more complicated, and I

It is actually very simple, in python for example:

        zlibObj = zlib.compressobj()

        def send(in_data):
                buf = zlibObj.compress(in_data)
                buf += zlibObj.flush(zilib.Z_SYNC_FLUSH)
                        :
                (construct frames/fragments and put into a queue)

Note: the 'zlibObj' has the same lifetime as that of WebSocket.

> connection (while we are discussing it, 

Again, lifetime of the compression state/context can be the same. It 
just needs, Z_SYNC_FLUSH operation at the end of the 'WS-frame' to avoid 
unnecessary delay. Even compression was done for whole stream, we still 
need Z_SYNC_FLUSH per frame basis with the exact same reason - we cannot
delay the frame! (Unless we introduce nagle's algorithm like protocol
just for slightly better compression ratio, which I think would not be
feasible.) This, at least to me, looks more complicated.

I am aware that, if application passes in is very small message 
frequently, (Z_SYNC_FLUSH has to be done frequently), resulting 
compression ratio would not be that great. However, since we can not 
delay each frame, we need the Z_SYNC_FLUSH anyways. Therefore, there's 
no benefit in having compression layer under the framing layer.

As I mentioned earlier, the compression state can be shared between 
text and binary messages since they are not interleaved (even fragmented),
though it may be better to have dedicated compression state if we can
afford memory. Either way, we need to define the spec in the document.

- Yutaka

> we should talk about the