Subject: [hybi] thewebsocketprotocol #40 (new): Clarify binary/utf-8 mixed handling

From: "hybi issue tracker" <[email-masked]>
Date: Thu, 10 Feb 2011 00:32:17 -0000

#40: Clarify binary/utf-8 mixed handling

 It can be confusing to mix binary and utf-8 frames in one tcp-like stream.
 For example, what will the current text-only JS Websocket API do if
 suddenly binary starts appearing?
 Additionally, when only partial frames may be available, it is expensive
 to verify that this is indeed a valid UTF-8 stream (protocol
 implementation needs to take into account multi-byte characters and end of
 current data payload).  If binary has been negotiated for this session,
 processing can be optimized accordingly.

 Proposal: allow negotiation to clarify if a stream will not mix binary and
 text in order to enable optimizing for the binary-only case.

-- 
--------------------------------------+-------------------------------------
 Reporter:  g_e_montenegro@…          |       Owner:     
     Type:  defect                    |      Status:  new
 Priority:  major                     |   Milestone:     
Component:  thewebsocketprotocol      |     Version:     
 Severity:  -                         |    Keywords:     
--------------------------------------+-------------------------------------

Ticket URL: <http://wiki.tools.ietf.org/wg/hybi/trac/ticket/40>
hybi <http://tools.ietf.org/hybi/>
The Hypertext-Bidirectional (HyBi) working group will seek
standardization of one approach to maintain bidirectional
communications between the HTTP client, server and intermediate
entities, which will provide more efficiency compared to the current
use of hanging requests.

---

From: "hybi issue tracker" <[email-masked]>
Date: Thu, 10 Feb 2011 00:44:27 -0000

Comment(by jat@…):

 I would assume a user agent with no binary support would fail the
 connection on receiving a binary frame and deliver an appropriate event to
 the JS client.

 Since the JS API provides no streaming API, you can't deliver partial text
 results anyway, so there is no need to validate the UTF8 until you have
 all of it.  If you are processing it as you go, such as converting it to
 UTF16, then you validate it whenever you get a complete character, and
 your state machine maintains where you are.

 I don't see why adding negotiation for the type of frames which may appear
 is useful -- you already have to maintain state about how many bytes are
 left in the current frame in progress and what type it is, and surely you
 can take appropriate actions based on that state as easily as you could
 some flag negotiated during the handshake.

Ticket URL: <http://trac.tools.ietf.org/wg/hybi/trac/ticket/40#comment:1>

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 10 Feb 2011 02:00:07 +0100

> processing can be optimized accordingly.

If you do accept UTF-8 encoded data then you have to validate it, other-
wise you get strange and possibly dangerous failures if you receive mal-
formed data, for instance, you can't trust that the length has been cal-
culated correctly. Anyway, you would seem to have this problem due to
fragmentation anyway if you accept text frames, and if you don't mean to
accept text frames then you just don't, there would seem to be no need
to negotiate for "binary-only". I also note that validating UTF-8 is not
really expensive, it's just a matter of `state = table[state + byte]`
for each byte. Work, sure, but not very "expensive". It's easy too if
you use my http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ decoder for it.

> text in order to enable optimizing for the binary-only case.

I do agree the protocol specification needs to discuss mixing frame
types, like, what if you have a fragmented text message but one of the
frames is not a text frame, but allowing to negotiate this complexity
away will most likely lead to interoperability and security problems, as
people will take shortcuts like not validating the frame type. We've
in fact seen that already on this list. Essentially negotiation binary-
only would be subsetting and fragmenting the protocol, and I don't think
the benefit here warrants that.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Gabriel Montenegro <[email-masked]>
Date: Thu, 10 Feb 2011 09:47:22 +0000

Yes, if one accepts UTF-8, that must be validated, no question about it. The point is not to disallow UTF-8, but to be able to optimize on those websocket connections that use binary only, so this is not a fragmenting of the protocol any more than any other per-session negotiable feature is.

>From the comments, folks seem to be thinking about Javascript Websockets API, and how it does not allow binary nor support streaming. That's true, but as we've emphasized before: we're designing the websockets protocol to be usable with the JS Websockets API, but is not limited to it. Other APIs may offer both binary and streaming.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Bjoern Hoehrmann
> Sent: Wednesday, February 09, 2011 17:00
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: [hybi] thewebsocketprotocol #40 (new): Clarify binary/utf-8 mixed
> handling
> 
> > Additionally, when only partial frames may be available, it is
> > expensive to verify that this is indeed a valid UTF-8 stream (protocol
> > implementation needs to take into account multi-byte characters and
> > end of current data payload).  If binary has been negotiated for this
> > session, processing can be optimized accordingly.
> 
> If you do accept UTF-8 encoded data then you have to validate it, other- wise
> you get strange and possibly dangerous failures if you receive mal- formed data,
> for instance, you can't trust that the length has been cal- culated correctly.
> Anyway, you would seem to have this problem due to fragmentation anyway if
> you accept text frames, and if you don't mean to accept text frames then you
> just don't, there would seem to be no need to negotiate for "binary-only". I also
> note that validating UTF-8 is not really expensive, it's just a matter of `state =
> table[state + byte]` for each byte. Work, sure, but not very "expensive". It's easy
> too if you use my http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ decoder for
> it.
> 
> > Proposal: allow negotiation to clarify if a stream will not mix binary
> > and text in order to enable optimizing for the binary-only case.
> 
> I do agree the protocol specification needs to discuss mixing frame types, like,
> what if you have a fragmented text message but one of the frames is not a text
> frame, but allowing to negotiate this complexity away will most likely lead to
> interoperability and security problems, as people will take shortcuts like not
> validating the frame type. We've in fact seen that already on this list. Essentially
> negotiation binary- only would be subsetting and fragmenting the protocol, and
> I don't think the benefit here warrants that.
> --
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 10 Feb 2011 20:47:42 +0100

* Gabriel Montenegro wrote:
>Yes, if one accepts UTF-8, that must be validated, no question about it.
>The point is not to disallow UTF-8, but to be able to optimize on those
>websocket connections that use binary only, so this is not a fragmenting
>of the protocol any more than any other per-session negotiable feature
>is.

Well, I don't see what optimization you can do by knowing promises have
been made that no text frames will be transmitted, and if there are op-
timizations possible, I don't see why this needs to be communicated in-
side the protocol, and if it does, why the existing mechanisms are not
sufficient or otherwise unsuitable. In other words, I do not see why it
is a matter of conformance with respect to the core specification.

What I do not want is a situation where I tell somebody "Your Websocket
implementation is broken" and they respond "Oh, no, it's okay because
we've negotiated binary-only" for some value of "it". So how would your
proposal affect conformance or, if it does not, why would it matter that
we adopt some negotiation mechanism for this?
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: John Tamplin <[email-masked]>
Date: Thu, 10 Feb 2011 15:28:59 -0500

On Thu, Feb 10, 2011 at 4:47 AM, Gabriel Montenegro <
[email-masked]> wrote:

> websocket connections that use binary only, so this is not a fragmenting of
> the protocol any more than any other per-session negotiable feature is.
>
> From the comments, folks seem to be thinking about Javascript Websockets
> API, and how it does not allow binary nor support streaming. That's true,
> but as we've emphasized before: we're designing the websockets protocol to
> be usable with the JS Websockets API, but is not limited to it. Other APIs
> may offer both binary and streaming.

Ok, but what exactly is the benefit of having a binary-only negotiation?
 Your implementation is always free to make assumptions about the frames
that are sent by another of your implementations, so if you want to fail the
connection on any text frame that is ok.

If you mean that you would swap out a different state machine for
binary-only connections, then why is it not sufficient to choose that state
machine based on the opcode of the first frame?

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Gabriel Montenegro <[email-masked]>
Date: Fri, 11 Feb 2011 21:08:37 +0000

John said:
“If you mean that you would swap out a different state machine for binary-only connections, then why is it not sufficient to choose that state machine based on the opcode of the first frame? “

I realize now that this works. I was confused about the scenario in which one would send a message comprising multiple fragments. I was under the impression that this would allow mixing text and binary fragments. This discussion and another look at 4.4 has now clarified to me that this is not the case. Only the first fragment can specify the type of either “text” or “binary”. All subsequent fragments in that message cannot change the type since the opcode’s only valid value is 0x0 (“continuation”), so neither “text” nor “binary” nor anything else is possible.

I would suggest adding some clarifying text at the end of 4.4 pointing this out as follows:

OLD:
   o  An intermediary MAY change the fragmentation of a message if the
      message uses only opcode and reserved bit values known to the
      intermediary.

NEW:

   o  An intermediary MAY change the fragmentation of a message if the

      message uses only opcode and reserved bit values known to the

      intermediary.

   As a consequence of these rules, all fragments of a message are of the same type,

   as set by the first fragment’s opcode. Since Control frames cannot be fragmented,

   the type for all fragments in a message MUST be either text or binary.

---

From: John Tamplin <[email-masked]>
Date: Fri, 11 Feb 2011 16:49:04 -0500

>    the type for all fragments in a message MUST be either text or binary.
I don't object to this, but I note the general problem of saying essentially
the same thing several places in the document in slightly different ways.
 If we aren't very careful, there will be slight differences and thus
ambiguity as to which one is correct.

---

From: "hybi issue tracker" <[email-masked]>
Date: Mon, 21 Feb 2011 11:20:49 -0000

Changes (by salvatore.loreto@…):

  * status:  new => closed
  * resolution:  => invalid

Old description:

> It can be confusing to mix binary and utf-8 frames in one tcp-like
> stream. For example, what will the current text-only JS Websocket API do
> if suddenly binary starts appearing?
> implementation needs to take into account multi-byte characters and end
> of current data payload).  If binary has been negotiated for this

New description:

--

Comment:

 closed... some clarification text will be added in 06

 see: http://www.ietf.org/mail-archive/web/hybi/current/msg06311.html

-- 
--------------------------------------+-------------------------------------
 Reporter:  g_e_montenegro@…          |        Owner:         
     Type:  defect                    |       Status:  closed 
 Priority:  major                     |    Milestone:         
Component:  thewebsocketprotocol      |      Version:         
 Severity:  -                         |   Resolution:  invalid
 Keywords:                            |  
--------------------------------------+-------------------------------------

Ticket URL: <https://wiki.tools.ietf.org/wg/hybi/trac/ticket/40#comment:2>