Subject: [hybi] Data type transparency in PMCEs

From: Pavel Rappo <[email-masked]>
Date: Wed, 28 Jan 2015 17:28:58 +0000

Hi, I was wondering how this thing from [1]:

>   PMCEs don't change the opcode field.  The opcode of the first frame
>   of a compress message indicates the opcode of the original message.
>
>   The payload data portion in frames generated by a PMCE is not subject
>   to the constraints for the original data type.  For example, the
>   concatenation of the output data corresponding to the application
>   data portion of frames of a compressed text message is not required
>   to be valid UTF-8.  At the receiver, the payload data portion after
>   decompression is subject to the constraints for the original data
>   type again.

coexists with this thing form [2]:

>   Text
>
>     The "Payload data" is text data encoded as UTF-8.  Note that a
>     particular text frame might include a partial UTF-8 sequence;
>     however, the whole message MUST contain valid UTF-8.  Invalid
>     UTF-8 in reassembled messages is handled as described in
>     Section 8.1.

and this [3]:

> 8.1.  Handling Errors in UTF-8-Encoded Data
>
>    When an endpoint is to interpret a byte stream as UTF-8 but finds
>    that the byte stream is not, in fact, a valid UTF-8 stream, that
>    endpoint MUST _Fail the WebSocket Connection_.  This rule applies
>    both during the opening handshake and during subsequent data
>    exchange.

I noticed the "...The payload data portion in frames generated by a PMCE is not
subject to the constraints for the original data type..." comment in [1], but
still. It seems like it requires an additional comment in [2], something like:

>    When[*] an endpoint is to interpret a byte stream as UTF-8 but finds
>    that the byte stream is not, in fact, a valid UTF-8 stream, that
>    endpoint MUST _Fail the WebSocket Connection_.

[*] But not before the extension stack is re-applied.

Otherwise it's too broad, isn't it? What do you think? Thanks.

-Pavel

-------------------------------------------------------------------------------
[1]: https://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-19#section-6.1
[2]: https://tools.ietf.org/html/rfc6455#section-5.6
[3]: https://tools.ietf.org/html/rfc6455#section-8.1

---

From: Joakim Erdfelt <[email-masked]>
Date: Wed, 28 Jan 2015 11:10:12 -0700

Process the frame through the negotiated websocket extension chain before
validating the websocket frame contents.

The validation of frame structure can be done both at the raw network
frames level, and at the post-extension level.
The validation of frame contents is best done at the post-extension level.

With RFC6455, it addresses this with the nuance about "Payload Data"
See https://tools.ietf.org/html/rfc6455#section-5.2

   Payload data:  (x+y) bytes

      The "Payload data" is defined as "Extension data" concatenated
      with "Application data".

   Extension data:  x bytes

      The "Extension data" is 0 bytes unless an extension has been
      negotiated.  Any extension MUST specify the length of the
      "Extension data", or how that length may be calculated, and how
      the extension use MUST be negotiated during the opening handshake.
      If present, the "Extension data" is included in the total payload
      length.

   Application data:  y bytes

      Arbitrary "Application data", taking up the remainder of the frame
      after any "Extension data".  The length of the "Application data"
      is equal to the payload length minus the length of the "Extension
      data".

What this means is that the extension is using "Extension data", but the
validation for UTF-8 is part of the "Application data" section.
You'll need to send that frame through the extension which _might_ produce
"Application data", which you then validate per the rules in RFC6455
Section 8.1

--
Joakim Erdfelt <[email-masked]>
webtide.com <http://www.webtide.com/> - intalio.com/jetty
Expert advice, services and support from from the Jetty & CometD experts
eclipse.org/jetty - cometd.org

On Wed, Jan 28, 2015 at 10:28 AM, Pavel Rappo <[email-masked]> wrote:

> [1]:
> https://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-19#section-6.1
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Kevin <[email-masked]>
Date: Wed, 28 Jan 2015 13:13:35 -0500

> Otherwise it's too broad, isn't it? What do you think? Thanks.
Pavel:
I have to agree and I would like to add something.  I think that 
specification leaves to much room for error.  Why are we hard coding 
everything?  Am I missing something?

-- 
Kevin

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com

---

From: Pavel Rappo <[email-masked]>
Date: Wed, 28 Jan 2015 18:17:24 +0000

Perfect, thanks. I forgot about the fact you cannot extract Application data
from the Payload data. Only a certain extension in the chain should know how to
do that.

-Pavel

On Wed, Jan 28, 2015 at 6:10 PM, Joakim Erdfelt <[email-masked]> wrote:
> The validation of frame structure can be done both at the raw network frames
> level, and at the post-extension level.
> "Application data", which you then validate per the rules in RFC6455 Section
> 8.1
>
>
>
> --
> webtide.com - intalio.com/jetty

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 29 Jan 2015 23:14:44 +0900

Agreed with Joakim's interpretation. To make it work, we need to be careful
about phrasing. I.e. compressed data must be extracted from payload data
part, not application data part. Current PMCE text is written so.

Takeshi

On Thu, Jan 29, 2015 at 3:17 AM, Pavel Rappo <[email-masked]> wrote:

> Section
> >> subject to the constraints for the original data type..." comment in
> [1],
> >> but