Subject: [hybi] Framing redux

From: Greg Wilkins <[email-masked]>
Date: Thu, 26 Aug 2010 18:30:47 +1000

All,

while I think that Johns VII proposal is workable, I still have some
concerns about the approach of a dedicated ext area per frame that is
orthogonal to op-code.

Similarly I'm having concerns about fragmentation as part of the framing.
I'm thinking that fragmentation is an attribute of the payload and not of
the frame itself (ie do we really want fragmented ping frames?), so perhaps
it should be done as an op code.

So here is an alternate proposal that makes extension and fragmentation all
op-code based mechanism, so the basic framing is just length+data.  The new
idea is that the data can itself be ex-code+length+data and we can recurse
down with onion skins of extensions. So every frame can have multiple
extension op codes, each with it's own data.    For those who hate the idea
already, please wait until you see the examples below.

For now I have guessed a couple of standard extensions, but those are not
really part of the proposal, just there for illustration. Formally the
proposal is:

ws-frame      = frame-length frame-data

frame-data    = *(ex-code ex-length ex-data) op-code app-data

op-code       = %x00    ; text message
              | %x01    ; text fragment
              | %x02    ; binary message
              | %x03    ; binary fragment
              | %x04    ; ping
              | %x05    ; pong
              | %x06-3E ; reserved
              | %x3F    ; final
              | %x40-7E ; undefined

ex-code       = %x80    ; total message length in bytes
              | %x81    ; total message length in chars
              | %x82    ; sequence number
              | %x83    ; channel id
              | %x84    ; mime type
              | %x85-BF ; reserved
              | %xC0-FF ; undefined

frame-length  = *(%x00-FF) ; encoding TBD

app-data      = *(%x00-FF)

ex-length     = *(%x00-FF) ; encoding TBD

ex-data       = *(%x00-FF)

For the examples below, I will keep all the lengths < 127, so I can just use
a single byte for the length fields. For longer lengths we can use whatever
length encoding is agreed.

So for a simple text frame the encoding is more or less as it is today but
with length first

0x15 0x00 "Hello Wonderful World"

This can be fragmented by sending a fragment op-codes :

0x06 0x01 "Hello "
0x0A 0x01 "Wonderful "
0x05 0x00 "World"

If the total message length is known when fragmenting, this can be sent as
extension 0x80 on any of the frames :

0x0A 0x80 0x01 0x15 0x01 0x06 "Hello "
0x0A 0x01 "Wonderful "
0x05 0x00 "World"

If we want to add meta-date like a mime type, this can also be added as an
extension :

0x16 0x84 0x0A "text/plain" 0x80 0x01 0x15 0x01 0x06 "Hello "
0x0A 0x01 "Wonderful "
0x05 0x00 "World"

If we want to add a channel ID (eg 0x1234) to each frame, the 0x83 extension
can be used:

0x1B 0x83 0x02 0x1234 0x84 0x0A "text/plain" 0x80 0x01 0x15 0x01 0x06 "Hello
"
0x0F 0x83 0x02 0x1234 0x01 "Wonderful "
0x09 0x83 0x02 0x1234 0x00 "World"

For this formulation of the extension codes, I've preallocated some ex-codes
that might be part of the core. However it would be entirely possible to
negotiate all ex-codes and have them dynamically assigned.  So please when
considering this proposal don't get hooked up by any extension codes you
think we don't need (eg mime-type) and just consider them as a general
example of an op-code that needs meta-data associated with a frame.

The minimal overhead for an extension is 2 bytes (ex-code + zero length),
which is basically a boolean (the frame has the extension or it doesn't).
If an extension needs a numeric value per frame, then the overhead is 3-10
bytes for 8-64 bits. Extensions needing strings are string length +2 bytes.

I think it captures the benefits of Johns proposal (Eg extension data can be
associated with every frame), but gives some additional benefits:

   - less opportunity for extensions to conflict with op-codes
   - you can't fragment control frames like ping, final etc.
   - rather than one blob of extension data, there can be multiple tagged
   extensions per frame.
   - simple implementations can ignore/forward all extended frames by
   looking at first code
   - moderate implementations can scan through the extensions looking for
   any that they understand or at least know they can ignore.  They can always
   find the raw frame under the extensions.

cheers

---

From: Eduardo Lima Mitev <[email-masked]>
Date: Thu, 26 Aug 2010 11:37:54 +0200

Greg,

I like your new approach. IMO it is simpler and slightly more consistent
than previous one. Couple of comments below:

On 08/26/2010 10:30 AM, Greg Wilkins wrote:
> [...]

If we will have different op-codes to signalize fragments, I rather use
a fixed bit for it (e.g the 2nd bit):

              | %x01    ; binary message
              | %x02    ; text fragment
              | %x03    ; binary fragment

This way, implementors can check for fragmentation using a mask, instead
of switching all the cases.

> [...]

So, is ex-data assumed to be limited to 255 bytes? I don't have a use
case that would require longer, it is just a confirmation question.

Thank you for rising your proposal not-too-late.

cheers,
Edu

---

From: Eduardo Lima Mitev <[email-masked]>
Date: Thu, 26 Aug 2010 11:42:10 +0200

> case that would require longer, it is just a confirmation question.

Please, ignore this. I completely overlooked the "econding TBD".

Edu

---

From: James Graham <[email-masked]>
Date: Thu, 26 Aug 2010 11:51:56 +0200

On 08/26/2010 10:30 AM, Greg Wilkins wrote:

> So here is an alternate proposal that makes extension and fragmentation
> all op-code based mechanism, so the basic framing is just length+data.
> The new idea is that the data can itself be ex-code+length+data and we
> can recurse down with onion skins of extensions. So every frame can have
> multiple extension op codes, each with it's own data.    For those who
> hate the idea already, please wait until you see the examples below.

This proposal seems (in general) good to me, at least at first glance.

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 11:16:23 -0400

> concerns about the approach of a dedicated ext area per frame that is

Note that we specifically did not decide whether the reserved bits will be
used to indicate extensions or be part of the opcode, because the list
seemed to be well-divided over those issues.  The whole point on the VI =>
VII changes were intended to try and get agreement on what v1.0
implementations could do and not worry about being broken by whatever we
decide for the extension mechanism.  It seemed separating the base protocol
from deciding on how to implement extensions was the only way we were going
to stop going in circles and make progress.

My personal feeling is that the ultimate solution will be a mix of using new
opcodes for mandatory extensions and metadata for optional ones, but I would
still prefer to separate the exact implementation of extending the base
protocol from the base framing.  If you are now saying that approach is not
viable (and there were others who were nervous about totally punting
extension support in the base protocol), then I guess we have to decide that
now.

> it should be done as an op code.

If the sender is sending a large ping frame, perhaps to attempt to measure
available bandwidth by sending small and large ping frames, why shouldn't it
be fragmented if necessary?  Should an intermediary with frame size limits
just drop the connection instead?  Alternatively, do we have to have
something like "path-mtu" negotiation in the handshake and require all
intermediaries to participate?

Likewise, an intermediary that needs to fragment a message containing an
extension it can't understand can't do so, leaving the only option to drop
the message.

> already, please wait until you see the examples below.

We discussed this extensively in the VI proposal thread.  The basic problem
is that it basically says all extensions are mandatory -- if you don't
understand the top level extension, then you can't do anything further with
the frame.  For an intermediary that doesn't care about the MIME type of a
message and doesn't understand that extension, why should it be prevented
from processing the message?  How does it distinguish an unknown extension
opcode it can ignore (such as MIME type) from one it can't (such as
something that transforms the ultimate payload)?

Fragmentation really seems orthogonal to the opcode -- imagine an
intermediary that wants to fragment the frame.  That means it either has to
have a table to say if I fragment opcodeX, it becomes opcodeY, or we would
have to define a relationship between the fragmented and unfragmented
opcodes, which still works out to a fragment bit and declaring that some
frames must not be fragmented (which I still don't think is a good idea).

Also consider compression - if we add per-frame compression (still up in the
air), then I suppose there will be a compressed text opcode and a
compressed-foo fragment opcode.  If we have new content-bearing opcodes,
would we need a compressed-foo opcode pair as well?

> associated with every frame), but gives some additional benefits:

The most recent proposal doesn't discuss extension data at all, because we
were not able to agree on how it should be represented.

> If extension metadata is separate from payload data, you can have multiple
tagged extensions inside that data.

> As you pointed out previously in the discussion, some extensions alter the
meaning of the following data.  How can an implementation ignore an
extension it doesn't know without knowing that that extension has
transformed the following data?

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 08:20:14 +1000

On 27 August 2010 01:16, John Tamplin <[email-masked]> wrote:

>  It seemed separating the base protocol from deciding on how to implement
> extensions was the only way we were going to stop going in circles and make
> progress.
>

Sorry but I can't evaluate a base protocol without considering how it can be
extended.  Unextended ws does very little for the use-cases that I'm
interested in (and actually makes my pain points worse), so I need to see
the road forward.   If we are not to add extensions, then the current
framing is probably good enough.

> be fragmented if necessary?
>

I understand what ping does for the protocol.  I don't understand what piii
iiiiii iiiiiiii iiiiiiii iiiiiii iiiiiii iiiiii iiiii ing would mean. Pings
need to be atomic to have meaning I think.  Similarly with any control frame
(eg half close, throttle etc.).

> extension it can't understand can't do so, leaving the only option to drop

This is always going to be true for some extensions.  For example if the
extension is to add a checksum or frame sequence number, then fragmentation
without considering that extension will break. With my proposal an
intermediary can scan all the extension codes in a frame, and if all are
known to it then it can fragment any text or binary data segment as
necessary.

> something that transforms the ultimate payload)?

Well this is true even if the extensions are not part of the core protocol.
 If there is an in-data extension that does compression, then a scanning
intermediary still wont be able to scan the content.

With ex-code based extensions, then an intermediary that wants to fragment
can have a list of known extensions codes that it can fragrment - it can
scan for those and if no unknown ex-codes are present, then it can fragment.
 Note that it does not need to actually implement support for all the
extensions, it just needs to know if they are fragmentable or not.   You
could do a similar thing with an intermediary that wants to scan packets, it
can have a list of extensions known not to change the data content.

Punting such problems to the application layer does not make them not
problems,  in fact I think it makes them harder problems.

> frames must not be fragmented (which I still don't think is a good idea).

Mmmm  well I could live with a fragment bit, but I'm still in favour of only
fragmenting data frames and keeping all other frames unfragmented.   The
nuts of my proposal is the extension ex-codes, so perhaps consider that in
isolation of fragmentation, which can be done via data op-code or fragment
bit or even as an extension.

> would we need a compressed-foo opcode pair as well?

I would not do it that way.  I would add a compressed ex-code (eg 0x87) that
has zero length data and convert the text frame (0x00) to a binary (0x01).
So if  you compress the frame:

   0x0B 0x00 "Hello World"

you would get something like (assuming 3 bytes of compressed data):

   0x06 0x87 0x00 0x01 0x4b10a2

> transformed the following data?

This proposal gives implementations the ability to easily scan a frame for
all the ex-codes that are being used.  Because the ex-code is part of the
base protocol, implementations can decide to ignore the frame, continue
ignoring the extensions, or handle the extensions based on those codes.

Punting the problems of extensions out of the core protocol does not solve
the problems - it just makes them somebody elses problem. If we leave out
extensions from the core protocol, then intermediaries will get written that
will not know to scan for extensions before attempting their value add, and
that will make the job or common extensions a lot more difficult in future.

cheers

---

From: Dave Cridland <[email-masked]>
Date: Thu, 26 Aug 2010 23:32:30 +0100

> (eg half close, throttle etc.).
P i n g means that the peer is attempting to discover a max MTU.

P o n g means you just have.

I'm content to fragment anything - who knows what use it might be put  
to.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 19:11:10 -0400

I am responding to these:

http://www.ietf.org/mail-archive/web/hybi/current/msg03710.html
http://www.ietf.org/mail-archive/web/hybi/current/msg03720.html

I already gave you the solution:

http://www.ietf.org/mail-archive/web/hybi/current/msg03707.html

Framing is 100% orthogonal to its payload (data) and any extension in the
payload. Let me explain how that works.

The frame only needs to know how large it is, what sequence of frames it
belongs to, and what its position (index or count) in that sequence.  That
is all it needs to know and nothing more. Let me explain.

What ever intelligience you want to process extensions or opcodes or what
ever you want to call what you put in the payload, sits above the framing
layer. The framing layer needs no knowledge about what is inside its
payload, it is only concerned with moves bytes across the network and
re-assembling them in the proper order.

On top of that framing layer, you build your extensions by embedding data
structures in your payload. You call into the framing layer APIs to send
and receive data.  Your higher layer knows the byte size of the
re-assembled payload is (it may not know that until it reaches the end but
that is okay too), and the framing layer does not know and does not need
to know the size of the re-assembled payload, not what is in the payload.

The framing layer goes about fragmenting and re-assembling bytes.  The
higher layers read this and process it.  Why are you guys making it
complicated when it is so simple?

You don't need any CMLP bit.  The frames do not and should not know how
long the re-assembled payload is.  Only the higher level code that calls
into the framing layer API, needs to know what is in the payload.

The framing can fragment a message at any byte.  There is no need for the
framing layer to know the internal data structure of the payload.  The
higher layer at each intermediary will walk these payloads and parse them
and resend the payloads through the send() API of the framing layer, then
the framing layer will refragment as necessary.

There is no loss of efficiency.  Your higher level code calls read() and
provides a buffer and maximum size to read and callback.  The callback
processes the result and is the total size has not been reached, it calls
read() again and so on. If it is an intermediary, then at any time at its
chosing, it can call send().

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 19:24:04 -0400

On Thu, Aug 26, 2010 at 6:20 PM, Greg Wilkins <[email-masked]> wrote:

> Sorry but I can't evaluate a base protocol without considering how it can
> be extended.  Unextended ws does very little for the use-cases that I'm

The problem is that we kept going around in circles making zero progress.
 What VII was intending to do was take VI and, in addition to the change of
fragmentation bits and making CML optional, was to leave the extension
mechanism reserved so those objecting to the complexity could implement a
base protocol confident they would never have to worry about such things.
 Then when we agreed on the base protocol, we would immediately set out to
define how extensions are implemented, while having made sure that the base
protocol could be extended to either or both of the two main extension
options being argued.

For example, let's say that we do want to take your approach that all
extensions are opcodes.  If the handshake includes negotiation for
extensions, the opcode would then be a 6-bit field (allocating the two
reserved bits next to it) and extensions would be allocated with one of the
two high bits set.  Everything can work exactly as you suggest in this
thread, albeit with fragmentation set aside as a separate bit.  If it makes
you feel better, we could even say that certain frames can't be fragmented.

However, it also gives us the option, after further debate, to decide on a
different extension mechanism.  I believe intertwining the base framing and
the extension mechanism is what was causing us to make no progress, as those
wanting simple implementations worried about having to support all of that.
 If VII could be extended using the basic mechanism you suggest, would you
be willing to defer that discussion for a week or two?

>  I would not do it that way.  I would add a compressed ex-code (eg 0x87)
> that has zero length data and convert the text frame (0x00) to a binary
> (0x01). So if  you compress the frame:
>
>    0x0B 0x00 "Hello World"
>

So what happens when a foo payload type is added?  Would additional
compression opcodes have to be defined for them?  It seems better to either
allocate a bit to indicate the frame is compressed and the opcode indicates
the type of the uncompressed frame, or to have an opcode for compressed
frame and part of the compressed payload is the next-level opcode (though
that adds to the minimum compressed frame size).

> ignoring the extensions, or handle the extensions based on those codes.

We could do the same thing with the VII protocol.  As just an example of how
it could be extended to support what you want -- not making any sort of
proposal -- it could work like this:

   - In the handshake, the foo extension is negotiated.  Since simple
   implementations would never negotiate the extension, they don't have to
   worry about it and can just deal with the VII framing.
   - The foo extension defines opcodes 0x27 and 0x28.  We have
   hypothetically decided that all extension opcodes will have a length field
   at the beginning of the payload defining the data specific to that
   extension, and the following byte is the next-layer opcode.  So, the frame
   looks like:
      - [MORE=0,OPCODE=0x27,LEN=0x0C] 0x03 (length of foo extension data)
      "foo" [OPCODE=TEXT] "Hello"
   - That means intermediaries would still be able to skip over the foo data
   they didn't understand, though it still seems risky to assume the rest can
   be interpreted without knowing what foo does.  Maybe we can use a bit in the
   length (if we went with 7/16/63 for the frame length encoding, we could use
   the same mechanism for extension length and have a bit free) to indicate
   that the extension is critical to further understanding of the payload.

> that will make the job or common extensions a lot more difficult in future.

My expectation is that the specification that is published will include the
extension mechanism, but it will just be defined in such a way that simple
implementations can ignore it.  The proposal is not that the spec would not
have extensions, but rather than we define it separately as a means of
achieving consensus on the base protocol and only those who care about
extensions can argue (I mean debate :) about it.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 09:25:23 +1000

On 27 August 2010 08:32, Dave Cridland <[email-masked]> wrote:

> I'm content to fragment anything - who knows what use it might be put to.
>

So here is a variation that allows any op-code to be fragmented:

frame-data    = *(ex-code ex-length ex-data) op-byte app-data

op-byte       = frag-bit op-code

frag-bit      = %x0-1   ; 1 bit, 1==fragment

              | %x02    ; ping
              | %x03    ; pong
              | %x04-1E ; reserved
              | %x1F    ; final
              | %x20-3F ; undefined

app-data      = *(%x00-FF)

ex-data       = *(%x00-FF)

cheers

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 19:39:38 -0400

> So here is a variation that allows any op-code to be fragmented:

The basic idea of this proposal can be implemented as an extension to VII (I
am assuming that since no objections have been raised in the discussions of
removing CMLP that it is safe to move that functionality to an extension if
it is later deemed useful).  I don't like that the fragment bit is buried at
arbitrary places in the frame rather than the leading bit.  Aside from that,
the main difference is moving control frame types into the opcode (I had
previously suggested having them separate so that less-frequent frame types
are moved out of the opcode leaving more room for expansion).

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 19:47:49 -0400

What you wrote is essentially the same as what I just described here:

http://www.ietf.org/mail-archive/web/hybi/current/msg03722.html

There is no need for the framing layer to have any knowledge of frame-data
grammar.

The ws-frame and the frame-data are two separate layers as follows:

Framing Layer
-------------

 frame-data    = binary bytes

Data Layer
----------

 etc..

The data layer calls into the framing layer with callbacks, as I explained
at the above link.

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 09:51:06 +1000

On 27 August 2010 09:39, John Tamplin <[email-masked]> wrote:

> The basic idea of this proposal can be implemented as an extension to VII
>

I think you may be overloading the word "extension" a bit there :)

I do consider my proposal an evolution of VII. As you say it is essentially
the same thing except that:

   - structure was given to the extension space
   - CML is not a special case and is just and extension
   - I tried a variation on fragmentation (which does not appear to have
   much support).

(I am assuming that since no objections have been raised in the discussions
> of removing CMLP that it is safe to move that functionality to an extension
> if it is later deemed useful).  I don't like that the fragment bit is buried
> at arbitrary places in the frame rather than the leading bit.
>

The fragmentation is only meaningful in regards to the data, so having it
associated with the data is not a bad thing.  Specifically I think we want
to prohibit extension data from being fragmented and this design makes that
very clear.

> Aside from that, the main difference is moving control frame types into the
> opcode (I had previously suggested having them separate so that
> less-frequent frame types are moved out of the opcode leaving more room for
> expansion).
>

I think the other big difference is that by including the ex-code structure
as part of the core, it removes the need for an extra length field (and thus
helps with byte squeezing that we appear keen to do) and makes the ex-code
structure very explicit so that all implementation from day 1 that wish to
handle extension will know about the structure.

I do feel that my proposal is a reasonable evolution of VII, and sorry if
you think I've sent the conversation back around a loop again.

regards

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 19:54:16 -0400

> you think I've sent the conversation back around a loop again.

I just know that we were making no progress when we previously were trying
to define the base framing and the extension mechanism at the same time --
perhaps this time will be different.

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 19:55:48 -0400

Now I see you put a frag bit in the data.  That is wrong.

The fragmentation has nothing to with the data.  Why are you conflating it??

What I have below is correct, if you remove the frag bit from the frame-data.

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 19:59:06 -0400

> (I
> arbitrary places in the frame rather than the leading bit.

It is really stupid to put fragmentation information in the data layer.
You are going to create a very brittle spaghetti if you do that.

I have already given you the solution:

http://www.ietf.org/mail-archive/web/hybi/current/msg03726.html

>  Aside from
> that,

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 10:03:42 +1000

Here are the updated examples for the variation with a fragment bit and
hypothetical frame compression extension.

0x06 0x40 "Hello "
0x0A 0x40 "Wonderful "
0x05 0x00 "World"

0x0A 0x80 0x01 0x15 0x40 0x06 "Hello "
0x0A 0x40 "Wonderful "
0x05 0x00 "World"

0x16 0x84 0x0A "text/plain" 0x80 0x01 0x15 0x40 0x06 "Hello "
0x0A 0x40 "Wonderful "
0x05 0x00 "World"

0x1B 0x83 0x02 0x1234 0x84 0x0A "text/plain" 0x80 0x01 0x15 0x40 0x06 "Hello
"
0x0F 0x83 0x02 0x1234 0x40 "Wonderful "
0x09 0x83 0x02 0x1234 0x00 "World"

A hypothetical compression extension could use 0x8f as the ex-code and could
carry the original frame op-code as the extension payload. So the original
message

could be compressed as:

0x08 0x8F 0x01 0x00 0x01 0x1f5bc345

 could be fragmented as:

 0x06 0x8F 0x01 0x00 0x41 0x1f5b
0x05 0x8F 0x00 0x01 0xc345

cheers

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 20:09:40 -0400

>    - structure was given to the extension space

Good!  It is in the data layer. It has nothing to do with framing.

>    - CML is not a special case and is just and extension

Correct.  The data size must go in the data layer.  It has nothing to do
with framing or the frame size, as I explained here:

>    - I tried a variation on fragmentation (which does not appear to have

Because you are trying to put the fragmentation in the data layer.  The
fragmentation belongs in the orthogonal framing layer.

>> if it is later deemed useful).

There is no such bit needed ever!

The data layer knows the size of its data structures (by inserting a
Content-Length, a length field at start of itself, or waiting for EOM
marker, etc).  It reads from the framing layer.  The framing layer does
not need to know the size of anything but each frame.

Why are you guys complicating this????????????????????????

>  I don't like that the fragment bit is
>> buried
> The fragmentation is only meaningful in regards to the data,

That is absolutely false.  It is the opposite actually.  The fragmentation
is only meaningful to the framing layer.

> so having it

That is your goal, to have the data layer control fragmentation, because
you think you need that feature.  And thus you want to conflate the data
layer with the framing layer.

But that is very wrong.  The fragmentation layer MUST be free to fragment
without respecting data structures in payload.

Cripes! You want to turn this into a brittle spaghetti.

> you think I've sent the conversation back around a loop again.

Everything is okay, except your concept of having data and fragmentation
conflated (know about each other).

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 20:16:11 -0400

On Thu, Aug 26, 2010 at 8:03 PM, Greg Wilkins <[email-masked]> wrote:

> A hypothetical compression extension could use 0x8f as the ex-code and
> could carry the original frame op-code as the extension payload. So the
> original message
>

I am having trouble decyphering this -- ex-length is 8, ex-opcode is 0xBF,
presumably the uncompressed opcode is 0x00 and the compressed data is
011f5bc345, but what is the 01 between 8F and 00?

> 0x05 0x8F 0x00 0x01 0xc345

So where is the fragment bit in this?  I thought it was supposed to be the
high-bit of the unextended opcode?  If so, how do you know when you are at
op-byte instead of ex-code?

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 10:27:59 +1000

> 011f5bc345, but what is the 01 between 8F and 00?

John,

decoded as follows:

0x08 frame length
0x8f hypothetical compression extension ex-code
0x01 length of the extension data
0x00 extension data == original op-code
0x01 binary op-code to hold the compressed data
0x1f5bc345 compressed data.

> op-byte instead of ex-code?

Frag bit is the second top bit (0x40) as the top bit indicates an ex-code.
Of course the exact bit used is not important.  Ah I think I made a mistake
in my BNF, it should have been:

op-byte       = %x0 frag-bit op-code

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 27 Aug 2010 01:30:27 +0100

(The mail around here is too HTML for my poor text mail reader to
understand who'se quoting what.  Lynx can't format the HTML sensibly.
So I'll just reply to a few choice quotes without being sure who said them.)

>    This is always going to be true for some extensions. Â For example if
>    the extension is to add a checksum or frame sequence number, then
>    fragmentation without considering that extension will break.

I proposed a simple, extensible format a few weeks ago that allows you
to include a checksum, frame sequence number etc. and does not break
if fragmented.

>    Mmmm well I could live with a fragment bit, but I'm still in favour
>    of only fragmenting data frames and keeping all other frames
>    unfragmented.  The nuts of my proposal is the extension ex-codes, so
>    perhaps consider that in isolation of fragmentation, which can be done
>    via data op-code or fragment bit or even as an extension.

I know we're not discussing multiplexing, except to evaluate how it
well it might fit over a base protocol, but just an observation:

If a multiplexing node can't fragment some frame type that happens to
be large, whether it knows that frame type of not, it will be
susceptible to channel takeover attacks, livelocks and (in aggregate)
deadlocks.

Those are serious reliability issues, incompatible with the goal of
opportunistic (transparent) multiplexing, which means a multiplexing
extension will end up implementing its own, *separate* fragmentation
method if the base protocol's fragmentation is overly constrained.

There's nothing wrong with that; it's just something to weight up -
does the base protocol need flexible fragmentation, or are the base
protocol's reasons for fragmentation less demanding anyway (i.e. it's
basically for the same reason as HTTP chunking).

There is nothing inherently wrong with layering two types of chunking
on top of each other.  I'm not sure whether it would be more or less
robust that way.

>    As you pointed out previously in the discussion, some extensions alter
>    the meaning of the following data.  How can an implementation ignore
>    an extension it doesn't know without knowing that that extension has

- By sensible layering - so that all implementations know which of the
  following bytes are the "payload" associated with, or affected by, the
  extension.  The number needn't be finite, or fixed in advance.

- Perhaps by distinguishing between "must understand" extensions, "may ignore
  but must pass" ones, and "safe to drop" ones?

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 10:41:26 +1000

> understand who'se quoting what.  Lynx can't format the HTML sensibly.

Sorry,

here is a plain text version:

  app-data      = *(%x00-FF)

  ex-data       = *(%x00-FF)

So for a simple text frame the encoding is more or less as it is today
but with length first

  0x06 0x40 "Hello "
  0x0A 0x40 "Wonderful "
  0x05 0x00 "World"

If the total message length is known when fragmenting, this can be
sent as extension 0x80 on any of the frames :

If we want to add meta-date like a mime type, this can also be added
as an extension :

If we want to add a channel ID (eg 0x1234) to each frame, the 0x83
extension can be used:

  0x1B 0x83 0x02 0x1234 0x84 0x0A "text/plain" 0x80 0x01 0x15 0x40 0x06 "Hello "
  0x0F 0x83 0x02 0x1234 0x40 "Wonderful "
  0x09 0x83 0x02 0x1234 0x00 "World"

A hypothetical compression extension could use 0x8f as the ex-code and
could carry the original frame op-code as the extension payload. So
the original message

could be compressed as:

could be fragmented as:

> I proposed a simple, extensible format a few weeks ago that allows you to include a checksum, frame sequence number etc. and does not break if fragmented.

I completely agree that if such features are included in the base
protocol rather than pushed to extensions, then the extension
mechanism can be very simple.  The issue is that we want to make
extensions that will effect frames, fragmentations, sequencing, flow
control, etc without having explicit support for them in the base
protocol.

cheers

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 20:43:35 -0400

> method if the base protocol's fragmentation is overly constrained.

Finally a sensible reply in this thread!

Exactly what I just wrote:

http://www.ietf.org/mail-archive/web/hybi/current/msg03732.html
http://www.ietf.org/mail-archive/web/hybi/current/msg03730.html

"brittle spaghetti"

>   extension.  The number needn't be finite, or fixed in advance.

Exactly what I wrote in this thread:

http://www.ietf.org/mail-archive/web/hybi/current/msg03729.html
http://www.ietf.org/mail-archive/web/hybi/current/msg03722.html

>   but must pass" ones, and "safe to drop" ones?

Even simpler than that, the lower layer does not need to understand the
payload of the higher layer.  See my links above.

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 27 Aug 2010 01:51:31 +0100

> complicated when it is so simple?

Indeed. This is basic, ancient IETF wisdom.

I think you may be missing a couple of things:

  - For short messages, it seems to be desired to avoid having a separate
    "fragment header" and "message header".  Keep it short, simple.

  - When fragmentation isn't used, it seems to be desired to have a
    simple header which is easy for lesser-skilled programmers to
    understand, and is compatible with the message format of a stream
    which does use fragmentation sometimes.

Squishing them into a single layer may prove to be an "anti-pattern"
when it comes to bugs, if it turns handling fragmentats into a rare 
case, therefore a bug-ridden, often misimplemented or ignored case.

> chosing, it can call send().

That's not a particularly fast way to do it because you're copying
data around more than necessary, and calling twice as many callbacks.
But that's not really important - the implementation can be optimised
by anyone dedicated enough.

What's important is the design is simple enough to understand, and it works.

Shelby I basically agree with your thinking, though with this caveat:

   - If fragmentation support at a receiver is *negotiated*, it is
     better as a type of extension.  Implementations which don't
     handle it don't have to, because they won't agree to it in
     negotiation.

   - If fragmentation support at a receiver is *mandatory*, it
     probably ought to be a cleanly separated two layers, if only to
     ensure all implementations actually handle it.  Then again, it
     wouldn't surprise me to see implementations which blindly assume
     there is a 1:1 message:fragment correspondence in all the traffic
     they care about, if that proves true in practice.

-- Jamie

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 21:02:05 -0400

>  0x08 0x8F 0x01 0x00 0x01 0x1f5bc345

It seems really awkward to add an ex-code for compression, keep the original
opcode, and add a binary opcode.  Instead, it would seem much clearer to
make COMPRESSED_FRAME as the leaf opcode and encode it like:

0x07 0x8F 0x00 0x1f5bc345

What value is gained by having the compression ex-code carry only the
original opcode and then change the opcode to binary?  Of course, this would
need alterations to how you interpret opcodes vs ex-codes.

The fact that the compressed frame opcode contains binary data is of no
extra value to something that doesn't understand the compressed frame
opcode.  Likewise, a multiplexed frame that aggregates several logical
channels can't easily be implemented without making the mux-data frame type
a leaf opcode since there will be multiple logical frames.  For example:

      Opcode=MUX_DATA, LEN=n: payload=[stream_id_1 original_frame_on_stream1
stream_id_2 original_frame_on_stream2 ]

So, I think if we are going to go the approach of extension only via
opcodes, it must be possible to define extension opcodes as well.

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 27 Aug 2010 02:09:13 +0100

Greg Wilkins wrote:
> Sorry, [regarding HTML]

Well, sorry I have multiple day jobs unrelated to hybi, so I keep disappearing.
I'm sure I'm the last person still using text mail anyay :-)
Thanks for bringing the best ideas together and representing them well.

>                 | %x81    ; total message length in chars

One little nit springs to mind: If you count characters, best define
what that means.  UTF-8 code points don't correspond 1:1 with characters.

-- Jamie

---

From: Brian <[email-masked]>
Date: Thu, 26 Aug 2010 18:10:52 -0700

I like where this is going.  :)

The whole fragmentation issue seems to be the point of contention now.
 Can anyone remind me why we decided that we needed it in the base
protocol?  From what I remember, a few people wanted it and it seemed
like an easy thing to throw in at the time.  But now it's seeming less
sensible.

It seems that we have to decide from among one of these options:

A.) Decide that fragmentation exists at the top level of framing and
happens without regard to the data contents of the packet, extensions
and all.  The server/client implementations can process extensions
only after all fragments have been joined to reconstitute the original
message.  The purpose of fragmenting in this way is not apparent to
me...

B.) Decree that frames with extensions cannot be fragmented.

C.) Eliminate fragmentation.  Keep a clean one-to-one-to-one
correspondance between messages, frames, and extension metadata.  This
seems to me like the obvious choice.

Fragmentation is useful for multiplexing so why not move it into a
potential multiplexing extension?  Without muxing, what's the point of
breaking a message into smaller chunks at this level?  The TCP layer
and the layers below that already fragment the data stream for
multiplexing multiple streams on a single connection.

My vote is for C.  Are there other options that I haven't considered?

Brian

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 21:20:28 -0400

> I like where this is going.  :)

This reply shows you do understand the tradeoffs :D

That is big improvement.

> message.  The purpose of fragmenting in this way is not apparent to

Multiplexing:

And I guess to recover failed frames instead failed TCP packers?

> B.) Decree that frames with extensions cannot be fragmented.

Can't never work correctly with multi-plexing:

http://www.ietf.org/mail-archive/web/hybi/current/msg03738.html

> seems to me like the obvious choice.

I was pondering that too:

> breaking a message into smaller chunks at this level?

If you are going to do mux, then you must break at the lowest layer.

You can't break within an extension layer.

So the mux must sit below any other extension.  And that is precisely what
fragmented framing is, it is the base for the mux layer:

>  The TCP layer

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 21:24:52 -0400

> like an easy thing to throw in at the time.  But now it's seeming less

First, it was accepted as a base requirement of the WebSocket protocol.

Second, for multiplexing to be reasonably implemented, it must be able to
fragment frames to fairly serve all channels it is multiplexing.  A number
of people feel that multiplexing must be implementable on WebSocket for it
to scale and will not support it if it does not scale.

> message.  The purpose of fragmenting in this way is not apparent to

I believe it has to work this way for a multiplexor to be able to work
properly.

Note that depending on the extension, you may be able to process parts of
the message before it is complete -- in the example I gave above, if the mux
message were fragmented after the end of original_frame_on_stream1, it would
be able to demux/process that frame while waiting for later fragments.

> potential multiplexing extension?

If you did that, the demuxer would have to buffer all fragments until it
received the last fragment of an original frame before it could send it out
the wire, which would cause buffering problems.  Imagine the muxer receives
a 4G frame on one logical channel, but it doesn't want to block other muxed
streams for that  long so it fragments it.  The physical channel to the
demux carries mux data frames, each of which carry one or more logical
messages, including separate fragments of the 4G frame.  The demuxer can't
write any of the 4G frame's fragments it receives until it reads the last
one.  Now imagine lots of such streams, and the demuxer has a serious
buffering problem.  Perhaps you can work around it depending on how you
implement flow control in a muxed environment, but if the sender isn't aware
of the mux extension then it doesn't seem like it can participate in that
decision.

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 21:41:51 -0400

> Indeed. This is basic, ancient IETF wisdom.

<joke>How did we lose it?</joke> :D

below...

>     "fragment header" and "message header".  Keep it short, simple.

1 extra byte in the data or even a few bits if you want to twiddle them
(you said it was short length) seems hardly worth conflating the layers,
given the TCP overhead is several times that.

>     which does use fragmentation sometimes.

We already agreed in this WG that the amatuer programmer requirement only
applies at the higher scripting layer, not in the implementation of the
frame/fragmentation layer.

> case, therefore a bug-ridden, often misimplemented or ignored case.

I am not advocating we squish the frame header and the message/data header
together.  The opposite, I am stating they must be orthogonal.

> data around more than necessary, and calling twice as many callbacks.

Actually I think not.  I had explained how that would work in post yesterday.

The higher layer reads its header and figures out the incoming size, it
re-allocates its buffer and then uses that buffer for subsequent reads. 
So there is never any shuffling of data, it gets read right into
contigious place.

> by anyone dedicated enough.

Exactly.

> What's important is the design is simple enough to understand, and it

Exactly.

>      handle it don't have to, because they won't agree to it in

I think I was proposing that the fragmentation was independent of the
receiver (higher layer) and instead handled at the lower layer.  Do you
mean something else?

>      ensure all implementations actually handle it.

Yes the framing layer and the data layer must be orthogonal. That is what
I am proposing.  Simple and sweet.

>  Then again, it

Can you explain please?  I am not comprehending your concern.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 11:42:26 +1000

> 0x07 0x8F 0x00 0x1f5bc345

Sure - I think I like your hypothetical compression extension more
than my hypothetical compression extension :)

I picked my compression extension really to show the cost of a adding
an extended boolean flag to a frame is 2 bytes, but then I realized I
needed the original op-code, so may as well put compressed data in
there.

Of course this is a frame that is all extension and no op-code, but
that can be deduced from the lengths.

So however the compression extension is designed, I think this
extension approach can handle it.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 11:51:35 +1000

On 27 August 2010 11:10, Brian <[email-masked]> wrote:
> protocol?

Fragmentation in the core protocol simplifies sending dynamic content
who's length is unknown.

I think there is support for the idea that mandatory handling of
receiving fragmented messages would allow a lot more freedom in
intermediaries, which could fragment as they need to (for their use
cases) and deliver those fragments over the last hop.   If
fragmentation reception is not mandatory, then any large messages will
need to be reassembled before the final hop, potentially having
difficult buffering issues.

> seems to me like the obvious choice.

Maybe, it can be made an extension by design, but I think it's use
would be widespread enough that it would become a defacto mandatory
extension.

Bottom line for me is that I now feel that fragmentation is an issue
about designing the op-code space and not an issue about framing
itself.   I'm happy to have preallocated fragment op-code, but could
also live with negotiated fragment op-codes.

cheers

---

From: "Shelby Moore" <[email-masked]>
Date: Thu, 26 Aug 2010 21:53:34 -0400

> given the TCP overhead is several times that.

Or we can set a minimum size that doesn't get fragmented ever.  The the
higher layer can ask the framing layer to tell it the frame size.  No need
to dedicate extension bit to this.  Just put a MUST NOT fragment below xx
bytes.

That would also address Greg's ping fragmentation concern.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 11:53:50 +1000

John,

actually your compression extension would be better implemented by
having it negotiate to use one of the unallocated op-codes rather than
one of the unallocated ex-codes.

An ex-code is needed only if there is meta-data to be transported with
the actual data. An op-code can be negotiated if there is only a
transformation of the existing data.

cheers

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 22:59:33 -0400

> transformation of the existing data.

Ok, I thought ex-code was explicitly for extensions, as in EXtension code.
 If the whole address space of excodes/opcodes are available for extensions,
and ex-codes are basically prefixes to the leaf opcode, that seems fine
(though I wonder about calling them different things rather than just saying
that some opcodes are prefixes and some are leaves).

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 23:01:21 -0400

On Thu, Aug 26, 2010 at 9:42 PM, Greg Wilkins <[email-masked]> wrote:

> an extended boolean flag to a frame is 2 bytes, ...
>

Right, which means that by having no reserved bits at all (only reserved
opcodes), if we find an extension in the future that would be best served by
a bit per frame it will now take up two bytes.

---

From: Brian <[email-masked]>
Date: Thu, 26 Aug 2010 20:11:28 -0700

> to scale and will not support it if it does not scale.

That's my point.  It's important for multiplexing.  But what *else* is
it important for?  If multiplexing is involved here, it would
necessarily be the "outermost" extension in the onion-style extension
mechanism, and it could very easily handle fragmentation for the
purposes of multiplexing entirely within the extension.  That seems
fairly straightforward.

Also there's all this talk about intermediaries wanting to add
fragmentation themselves to the stream that they're passing through...
I can't for the life of me figure out why an intermediary would want
to fragment?  As far as I'm concerned, intermediaries should be mostly
concerned with forwarding bytes as a layer-3 proxy -- just like
HAProxy does right now with draft 75.  There is literally no case, no
matter what we do, where it's ever appropriate for an intermediary to
mangle the payload of a ws frame.  If the proxy attempts to, say,
redact confidential corporate information or something like that,
there is no scenario in which it won't break the application whose
data it's forwarding.  Companies can use automated systems to do that
with email.  It will not ever be possible to safely do that with
websockets.

> the wire, which would cause buffering problems.

If that's true for the muxer, then it would be true of the base
protocol.  Either way.
If there's fragmentation, the fragments must be reconstituted to
recover the complete original message.

This is a message-oriented protocol after all, not a stream-oriented
one... which is probably mostly because of its origins as a simple way
of sending UTF-8 based messages from the JS API.  Do we want to turn
it into a stream-oriented protocol where messages are expected to be
handled incrementally, or keep it as a message-oriented protocol,
where messages are dealt with in whole?

Imagine the muxer receives
> buffering problem.

If we're thinking of being able to begin processing individual
messages incrementally in the application layer due to fragmentation
-- more stream-oriented -- then there's no reason the same can't hold
true of muxed and fragmented messages.  And there's no reason that the
fragmentation has to happen at the framing layer for that to be the
case.

> Perhaps you can work around it depending on how you

If either end of the connection isn't ware of the mux extension, there
will be no muxing occurring.  It's supposed to be a negotiated
extension.  If it's not negotiated, it won't be used.  All extensions
will be understood by both ends of the connection or they won't be
used.  There should never be a case of either end of the connection
not understanding an extension it encounters.  That would only be an
issue for intermediaries, which must necessarily either drop the
message or forward it wholesale without modification in that case.
And if it drops the frame it will most surely break the application
sending/receiving the data, so it might as well drop the connection
instead.  If there's such an overbearing intermediary, it ought to
participate in the handshake process to make sure neither end offers
up a negotiation for an extension that the intermediary can't
understand.

Brian

---

From: John Tamplin <[email-masked]>
Date: Thu, 26 Aug 2010 23:24:46 -0400

> purposes of multiplexing entirely within the extension.  That seems

As Greg mentioned, generating dynamic content.  If you can't fragment, you
have to buffer everything in memory just so you can give the message size.
 if instead you can send fragments, you allocate a reasonable buffer and
when it fills you write a fragment.

> recover the complete original message.

Only on the ultimate receiver, rather than the intermediaries.  If the
demuxer can write fragments in the base protocol, it just writes them as it
gets each mux-data frame from the muxer.  The muxer simply leaves the bytes
of the large frame unread (it read the frame length and knows it is 4G,
reads the part it wants to write in the next fragment and waits, etc), which
in turn relies on TCP windows to flow-control the sender.

> where messages are dealt with in whole?

Yes, at the JS API level, it has to remain a message-oriented protocol, at
least for the time being.  I would expect more sophisticated servers to
treat the messages themselves as a stream though, just as they do with HTTP.

> up a negotiation for an extension that the intermediary can't

There may be multiple logical connections with different extensions on them.
 This is just a hypothetical scenario, but imagine browsers have a way to
redirect WebSocket traffic to a proxy, similar to HTTP.  The browser doesn't
support muxing, but it connects to the corporate proxy and it sends unmuxed
frames.  The corporate proxy however wants to conserve bandwidth across a
VLAN and better share bandwidth with other connections to other corporate
offices, so it uses the MUX extension.  For intra-corp traffic to a
particular other office, it connects to another mux'ing proxy, which will
then demux the streams to the servers which also arent mux aware.  So you
have n streams coming into the mux where the mux extension is not
negotiated, 1 stream between proxies which are muxed, and then m streams
between the remote proxy and servers.  Neither ultimate endpoint are aware
of the muxing going on and thus can't participate in any mux-related flow
control.

---

From: Douglas Otis <[email-masked]>
Date: Thu, 26 Aug 2010 20:59:06 -0700

On 8/26/10 1:30 AM, Greg Wilkins wrote:
> Similarly I'm having concerns about fragmentation as part of the 
> framing. I'm thinking that fragmentation is an attribute of the 
> payload and not of the frame itself (ie do we really want fragmented 
> ping frames?), so perhaps it should be done as an op code.
>
> So here is an alternate proposal that makes extension and 
> fragmentation all op-code based mechanism, so the basic framing is 
> just length+data.  The new idea is that the data can itself be 
> ex-code+length+data and we can recurse down with onion skins of 
> extensions. So every frame can have multiple extension op codes, each 
> with it's own data.    For those who hate the idea already, please 
> wait until you see the examples below.
>
> For now I have guessed a couple of standard extensions, but those are 
> not really part of the proposal, just there for illustration. Formally 
> the proposal is:
>
>
> For the examples below, I will keep all the lengths < 127, so I can 
> just use a single byte for the length fields. For longer lengths we 
> can use whatever length encoding is agreed.
>
> 0x1B 0x83 0x02 0x1234 0x84 0x0A "text/plain" 0x80 0x01 0x15 0x01 0x06 
> "Hello "
> For this formulation of the extension codes, I've preallocated some 
> ex-codes that might be part of the core. However it would be entirely 
> possible to negotiate all ex-codes and have them dynamically 
> assigned.  So please when considering this proposal don't get hooked 
> up by any extension codes you think we don't need (eg mime-type) and 
> just consider them as a general example of an op-code that needs 
> meta-data associated with a frame.
>
> The minimal overhead for an extension is 2 bytes (ex-code + zero 
> length), which is basically a boolean (the frame has the extension or 
> it doesn't).  If an extension needs a numeric value per frame, then 
> the overhead is 3-10 bytes for 8-64 bits. Extensions needing strings 
> are string length +2 bytes.
>
> I think it captures the benefits of Johns proposal (Eg extension data 
> can be associated with every frame), but gives some additional benefits:
>
>     * less opportunity for extensions to conflict with op-codes
>     * you can't fragment control frames like ping, final etc.
>     * rather than one blob of extension data, there can be multiple
>       tagged extensions per frame.
>     * simple implementations can ignore/forward all extended frames by
>       looking at first code
>     * moderate implementations can scan through the extensions looking
>       for any that they understand or at least know they can ignore. 
>       They can always find the raw frame under the extensions.
>
Greg,

Sorry for not being more fully engaged.  I agree CML is something that 
should not impact framing.

However, looking at the current progress, extensibility and multiplexing 
seems wrong. To avoid head of line blocking, a channel ID should be 
included within object framing, but only a few channels are needed to 
resolve this type of issue.  After all, time-multiplexing a channel is 
also an option when handling many smaller objects, which can then 
utilize a TLV object definition structure.   By being able to "stream" 
objects, a tag definition could indicate whether an object begins with a 
TLV structure that further defines the object.  In this way, 
comprehensive extensions can be defined without altering the framing 
layer or object binding.

In a prior message you indicated a desire to avoid handling error states 
while passing channel objects within the framing layer.  A single 
continuation bit is unable to encode three different states normally 
found when handling fragmented data, such as opening, appending, and 
closing an object.  Errors are typically detected when appending or 
closing objects not opened.  Knowing when a object is opened can also 
locate future object declarations without altering handling at the 
framing layer. However, only indicating an object is continued means 
most errors go undetected prior to an exchange and the error will cascade.

Combined with indeterminate state weaknesses, extensible frame lengths 
make the effect of errors far less predictable.  Explicitly asserting 
the beginning or end of a object, instead of whether an object is being 
continued offers a more robust, and more secure protocol.  After all, 
TCP fails to detect 2% of bus related bit errors.

The data could appear as follows:

frame data = def:1, beg:1, end:1, chan-id:5, (def tlv) app-data (bytes).

channel 0-7 utf-8 only unless modified by TLV
channel 8-30 binary unless modified by TLV
channel 31, framing control (such as ping/pong etc.)

TLV structure type:2, length:6, definition (length is number of 16 bit 
words)

when def = 1  the def TLV prefixes the channel object data.

def tag 0: text (defines byte/char length and encoding)
             1: mime
             2: tdm (time division multiplexing) sequence/sub-channel 
information
             3: tdm followed with def

-Doug

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 27 Aug 2010 13:59:27 +1000

> Ok, I thought ex-code was explicitly for extensions, as in EXtension code.

John

yes sorry - the language is not 100% clear - not enough words in the
english language for all the nuances :(

My intention is that both the undefined op-codes and ex-codes would be
available for negotiation by extensions. The difference being that
ex-codes have a length and extra content (meta-data), where as the op
code is the code that refers to the main payload of the frame (the
frame length minus the extension data).

> that some opcodes are prefixes and some are leaves).

Perhaps the conceptual difference is small and may be we just need a
frame that can carry  1 or more (op-code length data)?
I'll let the conversation settle for a day or so, and then will
consider if recasting it as single op-code space.

> Right, which means that by having no reserved bits at all (only reserved opcodes), if we find an extension in the future that would be best served by a bit per frame it will now take up two bytes.

Indeed, that is the cost of an extended boolean. Perhaps could be made
1 byte if we have a single op-code space.  Reserved bits would help,
but then we would need to negotiate both op-codes and reserved bits -
and I think negotiation is going to be difficult enough to agree on
without have more things to negotiate.

cheers

---

From: John Tamplin <[email-masked]>
Date: Fri, 27 Aug 2010 00:02:57 -0400

> without have more things to negotiate.

I am not sure there is call for negotiating the use of them -- either we
find sufficient value for them to be allocated (and they could be allocated
in different mutually exclusive extensions if that was useful) or we let
them be larger escape sequences like your example.  Trying to negotiate
opcodes/bits seems fraught with peril.

---

From: John Tamplin <[email-masked]>
Date: Fri, 27 Aug 2010 00:07:28 -0400

On Thu, Aug 26, 2010 at 11:59 PM, Douglas Otis <[email-masked]> wrote:

> continuation bit is unable to encode three different states normally found
> when handling fragmented data, such as opening, appending, and closing an
> object.  Errors are typically detected when appending or closing objects not
> opened.  Knowing when a object is opened can also locate future object
> declarations without altering handling at the framing layer. However, only
> indicating an object is continued means most errors go undetected prior to
> an exchange and the error will cascade.
>
> Combined with indeterminate state weaknesses, extensible frame lengths make
> the effect of errors far less predictable.  Explicitly asserting the
> beginning or end of a object, instead of whether an object is being
> continued offers a more robust, and more secure protocol.  After all, TCP
> fails to detect 2% of bus related bit errors.
>

In the VI proposal, there were two bits for fragment support but there
seemed little support for it.  If we can't trust the TCP connection
underneath WebSocket, I think we already have problems -- if the fragment
bit could be corrupted, why not the length?  If out-of-order frames or
frames after a dropped frame can get through TCP, then how could we hope to
do anything without recreating a transport layer on top of it?

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 27 Aug 2010 05:16:15 +0100

> given the TCP overhead is several times that.

You can fit some 10000 messages into a single TCP packet.
The TCP overhead is relatively very low in that scenario.
It's just a remainder not to focus on just one end of the spectrum.

> frame/fragmentation layer.

Sorry, I'm afraid I missed that agreement.  Last time I read much, it
was still being considered.

> together.  The opposite, I am stating they must be orthogonal.

Yes I know.  My response was explaining why that may be good.

> re-allocates its buffer and then uses that buffer for subsequent reads. 

Exactly.  That's not the raw fastest method, which is to read from TCP
(or even to *not* read from TCP, but to look at the kernel skbuffs),
and process messages in place, not copying them to a message-sized
buffer, and certainly not allocating a buffer after seeing message
size.  But it still isn't important.

> receiver (higher layer) and instead handled at the lower layer.  Do you

By "receiver" I mean the component which reads from TCP.  It handles
defragmentation.

I don't think it is agreed yet, whether fragmentation is a negotiated
feature or not.  There are pros and cons either way.

You are free to propose that it be mandatory.  I'm simply saying that
if it's negotiable in order to support *simpler* implementations that
don't do it, then it should get out of the way when it's not used, by
using optional extension codes.

> I am proposing.  Simple and sweet.

Some people prefer that it be simpler.

> Can you explain please?  I am not comprehending your concern.

There is a possibility that some not very good implementations will
emerge which *assume* each fragment contains exactly one message, if
that is what they see happening in practice.  It's easier to code that
way, so some implementations may do it, if it works well enough.

Then, when some new device wants to send a fragmented message, it will
break when it reachs those broken implementations.

If those implementations are very widely deployed before a problem is
discovered, it will prevent fragmentation from being used, effectively
killing the feature.

I don't think that's very likely, but I expect there are protocol
designs which make it more likely, and others which make it less
likely.

Unlike, say, AMQP servers or NFS clients, I expect WebSocket is likely
to have thousands of unfinished-but-good-enough independent
implementations fairly soon after it is standardised - especially
server implementations.

-- Jamie

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 27 Aug 2010 08:36:31 -0400

> It's just a remainder not to focus on just one end of the spectrum.

I had provided an optimization for that, through a standard API, that
won't conflate the frame and data layers:

http://www.ietf.org/mail-archive/web/hybi/current/msg03748.html

I think you probably saw it after you sent this reply.

[snip]

> Yes I know.  My response was explaining why that may be good.

Okay thank you.

> size.  But it still isn't important.

We can optimize for that case without conflating the layers.  Very simply
have a standard API for framing layer, SetContiguiousBufferSize().

Then the framing layer can optimize the reading as it wishes.

> By "receiver" I mean the component which reads from TCP.  It handles

Disagree.  The receiver at the higher data layer determines how data is to
be re-assembled.  The framing layer which is controlling the TCP, only
sees frames with blackbox payloads.

The only case that the framing layer handles the defragmentation (aka
re-assembly) is for the optimization I wrote about above, a proposed
SetContiguiousBufferSize().

> using optional extension codes.

It is not any simpler to make it negotiable.  Making it negotiable will
actually make the design more complex and brittle and hard to implement.

The framing layer sends and receives frames with blackbox payloads.  It
the receiving end of the framing layer has set the maximum size lower than
the current frame, the sender fragments it.  That is so simple.  How is it
making it simpler than that going to help anyone implement?  (I realize
you are not arguing against me, so this is an open challenge for anyone on
the list)

> Some people prefer that it be simpler.

It can't be any simpler.  The next simpler thing is no frames, just use
TCP directly instead.

> way, so some implementations may do it, if it works well enough.

You mean that each frame contains exactly its entire data (no
fragmentation).  Yes I could see a remote chance of that happening if all
data structures end up being very small.

I don't find that to be a good argument for conflating the layers and
making it more complex and brittle as a result of that conflation.

And that is why I am proposing we set a minimum frame size for
fragmentation, in order to make this explicit and it is also a necessary
optimization:

http://www.ietf.org/mail-archive/web/hybi/current/msg03748.html

> designs which make it more likely, and others which make it less

Agreed it is not very likely.  See my solution above and below.

> implementations fairly soon after it is standardised - especially

I think we need a verification suite.  A website that can client can
connect to, which tests their implementation.

Simple solution.

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 27 Aug 2010 09:01:15 -0400

> purposes of multiplexing entirely within the extension.  That seems

The key epipheny is that multiplexing has to go immediately above or below
the framing layer (and preferably immediately above, not below):

http://www.ietf.org/mail-archive/web/hybi/current/msg03768.html

The reason is because the multiplexer MUST NOT be constrained by the data
structures in the extensions (multiplexer must see frames with blackbox
payloads that can be split at any byte):

http://www.ietf.org/mail-archive/web/hybi/current/msg03768.html
http://www.ietf.org/mail-archive/web/hybi/current/msg03769.html

> with email.  It will not ever be possible to safely do that with

Understood, but that is an irrelevant concern if you agree that
multi-plexing has to go below the first layer of extensions.  See above.

> If that's true for the muxer, then it would be true of the base

Afaiu, that is incorrect logic.  If the mux sits on top of the extensions,
then it can not longer break anything up unless it knows the data
structures below it.  I explained that more above.

>  Either way.

Yes but that only happens at ends that process the data layer, other
intermediaries simply deal with frames that have blackbox payloads.  See
the two links I provided above.

> where messages are dealt with in whole?

As I am proposing it should be for optimum orthogonality (simplicity,
robustness, future composibility), the messaging (aka data) layer sits
instead in the frame's blackbox payload.  That messaging layer is indeed a
messaging layer.  The framing layer is a chunked streaming layer, that
could be multiplexed by inserting a mux layer between the framing layer
and the messaging (aka extensions, aka data structures) layer.

> fragmentation has to happen at the framing layer for that to be the

No you misunderstood how the orthogonality is accomplished through
standard APIs.  Please read the links I provided above.

> will be no muxing occurring.

That is not true, because the mux could be happening between ends that are
on either end of those 2 intermediaries.

>  It's supposed to be a negotiated
> extension.

It reasonably can't be.  See the links I provided above.

[snip]

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 27 Aug 2010 09:07:36 -0400

> treat the messages themselves as a stream though, just as they do with

It can easily be used as a streaming protocol, simply by declaring
(opcode, flag, very large data size, etc) it to be so in the data layer. 
Both higher layer (aka application, aka extension) ends then understand it
is.

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 27 Aug 2010 15:40:00 +0100

> actually make the design more complex and brittle and hard to implement.

Not for people who decline the negotiation.  It's trivial to fail to
negotiate (you just do nothing), and then you don't have to handle the
feature.  It does make it more complicated for people who do support
the feature.  That's why there are pros and cons.

Btw, negotiation (of anything) does add complexity but it can reduce
brittleness, depending on how it's done.  Protocols with widespread
buggy implementations and no means of identifying when talking to one
of them are brittle *because* of lacking sensible negotiation.

> making it simpler than that going to help anyone implement?

You are underestimating the hassle of reassembling frames in very
compact script implementations of the entire protocol, compared with
just assuming 1:1 message:fragment and extracting the message.

> connect to, which tests their implementation.

Wrong way around, because the poor quality implementations are more
likely to be servers.

Please do write a good verification suite, it will be most welcome.

Though I'm not sure how to about it, when people write quick-and-dirty
application-specific servers that are only designed to work with their
application running on a generic browser client.  The
application-independent features (fragmentation etc.) need testing,
but testing needs to use the actual application to generate messages
that the server will accept.

It is likely that the work involved in putting a test together will
put less dedicated application writers off bothering.  Imho, this is a
compelling argument for a generic test-assisting proxy whose purpose
is to deliberately mess with the framing to assist people in testing
their server and client code.

-- Jamie

---

From: Patrick McManus <[email-masked]>
Date: Fri, 27 Aug 2010 10:48:39 -0400

Hi Brian,

On Thu, 2010-08-26 at 20:11 -0700, Brian wrote:
> it important for? 

The primary use case of fragments is to simplify the memory or processor
requirements on the sender and improve receiver latency when dynamically
generating large messages. The message can be streamed to the network as
it is created instead of spooling the whole thing to ram in order to
construct the header. This saves significant memory on the sender and
gets the message to the receiver sooner. Win Win.

I think it is useful to note that HTTP has a fragmentation facility
(i.e. "chunking") that is very commonly and usefully used even though
http has no multiplexing capabilities. We have operational experience
with http without chunking (pre 1.1) and it was identified as a
necessary thing to add and subsequently been widely embraced by
implementors.

Fragmentation also allows the transport of unbounded max message sizes.
Even though that isn't interesting to the existing JS API it doesn't
mean it won't be a useful feature.

While I won't speak for the document author(s), I believe those are the
fundamental reasons the group reached consensus on fragmentation as a
requirement.

And I personally think based on what we have learned over time about
connection management in http, websocket multiplexing should be an
important first class feature that is obviously implemented with
fragmentation (among other things). I'm open to supporting a protocol
without it if that is a blocker to websockets standardization, but I'm
more convinced than ever it would be a valuable thing to have in the
base protocol and a missed opportunity to omit it.

>  There is literally no case, no
> data it's forwarding.  

I don't agree. Consider the case of HTTP transport of well defined SOAP
messages which I have some experience with. The message contents are XML
with specific SOAP defined semantic meanings. SOAP on HTTP
intermediaries can:
 1] identify the message contents via HTTP definitions
 2] identify the message contents as SOAP via SOAP/HTTP binding
definitions
 3] manipulate the messages as necessary in the SOAP context. Maybe that
means encryption, redaction, signature verification, routing,
normalization, etc.. but the result is still a message that has semantic
meaning in the application's context.

There is no reason not to build a web-sockets version of that kind of
system. But to do so the intermediary needs to be able to effectively
delimit the message. That is a fragile scenario if the transport
mechanism is negotiating multiplexing in a way that is opaque to the
intermediary - so it is better in my opinion to define something that
fundamental as part of the base transport semantics.

-Patrick

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 27 Aug 2010 11:42:24 -0400

> the feature.  That's why there are pros and cons.

I think you will agree that it not only makes it more complicated/complex
overall for entire parallel system of independent actors, but it will
inject state into the channel that will cause the multi-plexing to become
much more complex with many more failure scenarios (deadlocks, etc), as we
discussed in this prior post:

And for what gain?  See my comments below on that.

> brittleness, depending on how it's done.

That can only be true OVERALL if the negotiation is decreasing OVERALL
statefulness (i.e. increasing referential transparency).  Of course it can
be true locally without decreasing OVERALL statefulness, if local issues
are paramount.  Is your point that some agents can not refragment a simple
frame header that includes nothing but size and blackbox payload??

>  Protocols with widespread

I am asserting now that it is not sensible to negotiate whether an agent
can refragment a simple frame header that includes nothing but size and
blackbox payload.

That is one of the other HUGE advantages of keeping the framing layer as
simple as possible. I know you agree on that point, but still we must nail
down your point above?

> just assuming 1:1 message:fragment and extracting the message.

Okay you are talking about an agent that wants to parse the messaging
layer, because an agent that only wants to refragment and handle the
framing layer, never will have that hassle.

And so therefor, I hereby assert that by having separate layers, you have
by defacto negotiation.

If an implementation can't handle reassembly (defragmentation), then it
can't read the messaging layer.  And that is perfectly okay for
intermediaries that want to be blackboxes.

I can not understand why we should conflate the layers so that messaging
agents don't have to do simple re-assembly?

Could you give me an example of your concern that I can not easily
optimize away?  (I am very good at finding optimizations that eliminate
arguments)

> Wrong way around, because the poor quality implementations are more

Servers can still call out to another website to run the verification suite.

> Please do write a good verification suite, it will be most welcome.

Agreed, thank you.

> application running on a generic browser client.

They will break when they go into the wild of the internet.  Then they
will fix themselves, or be disowned by users.

I find very difficult to accept the argument for amatuer programmers at
the network infrastructure layer.  The economics of the internet insures
they die.

>  The

+1

AGREED!!!

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 27 Aug 2010 12:23:54 -0400

> agents don't have to do simple re-assembly?

The only way you could support an optional framing fragmentation, would be
to not have fragmentation.

It is absolutely impossible to have negotiated fragmentation.  Think it out.

Because even if you defragment before sending to an agent that requests no
fragmentation, that means you kill dynamic content and streaming.  And
there is a performance (potentially deadlock) issue where intermediary is
waiting to receive and resend the whole unfragmented frame to the agent
that won't accept fragmented message frames.

That is enough to prove we can't make it optional.

Additionally, even that agent which demanded unfragmented message frame
may still need to send to another agent that has its maximum frame size
set lower than the unfragmented frame size.  And we already discussed 2
days ago that some agents want to be able to set their maximum frame size.

---

From: "Shelby Moore" <[email-masked]>
Date: Fri, 27 Aug 2010 12:31:07 -0400

Clever solution below...

>>>> the current frame, the sender fragments it.  That is so simple.  How
>>>> is
>>>> it

With respect to framing, we can simply negotiation whether to run
WebSockets messaging layer on framing or directly on TCP.  Those are then
two distinct protocols stacks.  We can do this because the framing and
message layers will not be conflated.

If you are running on framing, then any intermediary which doesn't want to
participate, can decline to connect that protocol.

Then you downgrade to not running on framing (remove the frame headers,
just the messaging layer on top of TCP).

The downgrade does not force you to aggregate the entire message into one
frame.  That is a big advantage over making fragmentation negotiable.