Subject: Re: [hybi] Masked framing VS mask in frame

From: Greg Wilkins <[email-masked]>
Date: Fri, 25 Feb 2011 22:02:46 +1100

So is there any support for changing away from masked framed to masked payloads?
or should I just accept that we will be building masking into framing
for all time?

cheers

On 1 February 2011 08:47, Greg Wilkins <[email-masked]> wrote:
> I think we still have a fundamental divide in opinions between masked framing:
>
>   [ key1][masked frame1] [key2][masked frame 2] ..... [keyN][masked frame N]
>
> vs mask in frame:
>
>   [ key1, masked payload 1 ]  [ key2, masked payload 2]  ...  [keyN,
> masked payload N]
>
>
> I'd like to eventually call for a straw poll on this, but would first
> like to have a thread dedicated to the pros and cons of each.
>
>
>
> IMNSHO:
>
>
> Masked Frames PROS:
>  + all bytes other than the key are masked.  This prevents an attacker
> from using frame fields like the length to send "clear text"
>
> Masked Frame CONS:
>  - Masking is not independent of framing - poor layered protocol design.
>  - all WS handling that needs to be aware of frame boundaries will
> need to implement masking.  This will mean that more infrastructure
> will have masking hard coded into it, thus it will soon become very
> difficult, if not impossible to change framing as it will be baked
> into WS aware intermediaries.
>  - Intermediaries that do not care about content will still need to
> unmask and remask the frames.
>  - the parser/generator code for WS becomes more complex and
> asymmetric because it must implement masking for some WS streams but
> not for others.
>  - unmasking cannot be done as single optimised pass over the data,
> but must be done on-the-fly in the parsing code.
>  - the options for masking algorithms is limited.  For example session
> based keys could not be used in a MUX environment as the session key
> would need to be known to access the extension data to know the stream
> ID, so the session ID could be known so you can't know the session
> key.   While session based masking may not be desirable for other
> reasons, this limitation indicates a fundamental limitation of the
> masked frame approach.
>
>
>
> Mask in Frame PROS:
>  + Masking is independent of framing.   Good layered protocol design.
>  + Intermediaries that do not process the payload will not need to
> implement masking. This saves CPU, memory and also will make it easier
> to change and/or remove masking in future.
>  + WS parsing code is simple and symmetric.
>  + session based masking algorithms can be used if desired
>  + Can utilise the extension data space to carry the key - thus
> testing the implementation of the parsing of extension data length and
> preventing implementations assuming this will be zero.
>
> Mask in Frame CONS:
>  - The frame op-code, flags and length are sent in the clear.  Of
> these, an attacker has reasonable control over the length field, so
> could conceivably send 8 characters including "GET\n" as the length
> (although this would be very difficult to do via the current
> javascript API.  There are also simple heuristic defences for this,
> such as fragmenting any frame that has a length that is all ascii and
> has a CR)
>
>
>
> In summary,   mask in frame is far more flexible, low cost and
> demonstrable better design, at cost of exposing a very very unlikely
> subcase of an already very very unlikely vulnerability to an as of yet
> undiscovered threat, of which an exploitable variation is already in
> the wild using non ws clients.
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 25 Feb 2011 12:11:55 +0100

On 25.02.2011 12:02, Greg Wilkins wrote:
> ...

I would support it, if this helps ;-)

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 12:23:20 +0100

> I would support it, if this helps ;-)

count me in too :-)

Willy

---

From: Andy Green <[email-masked]>
Date: Fri, 25 Feb 2011 11:25:15 +0000

> or should I just accept that we will be building masking into framing

Yes it would be a nice improvement.  It's not deadly if it doesn't 
happen but there are obvious advantages and it's pretty cheap to define 
in the spec.

-Andy

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Fri, 25 Feb 2011 13:10:11 +0100

> or should I just accept that we will be building masking into framing

I do agree with your proposal. Mask in frame is obviously much better.

I've documented it in the wiki:

  http://trac.tools.ietf.org/wg/hybi/trac/wiki/MaskingProposals

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: "Alexander Philippou" <[email-masked]>
Date: Fri, 25 Feb 2011 14:15:53 +0200

> > I would support it, if this helps ;-)

You have my +1 as well.

Alexander

---

From: David Endicott <[email-masked]>
Date: Fri, 25 Feb 2011 09:00:39 -0500

+1 from the lurking contingent.

On Fri, Feb 25, 2011 at 6:02 AM, Greg Wilkins <[email-masked]> wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 09:41:30 -0500

> or should I just accept that we will be building masking into framing

I am willing to support it, but I think there are a few complications
that need to be addressed:
 - you point out attacker control over the length - we would need to
codify exactly what would
   be done about it
 - I would not support use of any reserved bits to indicate the mask
is in the extension data
   portion of the frame - instead, it should simply be defined to be
the first 4 bytes if masking is
   present (ie, client->server frames)
 - it will complicate how mux and other extensions are defined, but I
think we can deal with that
   when we define those extensions

--
John A. Tamplin
Software Engineer (GWT), Google

---

From: Brian McKelvey <[email-masked]>
Date: Fri, 25 Feb 2011 07:55:17 -0800

Yes, absolutely!!  Count me in.

Sent from my iPhone

---

From: Bruce Atherton <[email-masked]>
Date: Fri, 25 Feb 2011 08:23:08 -0800

> or should I just accept that we will be building masking into framing

+1.

---

From: Bruce Atherton <[email-masked]>
Date: Fri, 25 Feb 2011 18:19:38 -0800

Right now there is no way to tell whether a particular Websocket frame 
is travelling from client to server or server to client without knowing 
which end sent the packet and whether that end is acting as the client 
or the server. This complicates decoding as, just given the packet data, 
you can't tell whether a frame key is present and demasking needs to be 
done. Even with human intervention this can get a little confusing given 
that a server can act both as a client and a server on different 
connections.

If the framing is not masked then one solution would be to use one of 
the reserved bits to indicate whether the frame originated at a client 
or not. Would your objection to using a reserved bit to indicate a mask 
in the extension data extend to this use as well? I realize that the 
reserved bits are very limited, but masking and its omission does seem 
rather fundamental to the protocol as it affects the parsing of the 
frame (ie. read 4 extra bytes at the beginnning, or don't).

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 22:22:09 -0500

On Fri, Feb 25, 2011 at 9:19 PM, Bruce Atherton <[email-masked]> wrote:

> Right now there is no way to tell whether a particular Websocket frame is
> travelling from client to server or server to client without knowing which
> end sent the packet and whether that end is acting as the client or the
> server. This complicates decoding as, just given the packet data, you can't
> tell whether a frame key is present and demasking needs to be done. Even
> with human intervention this can get a little confusing given that a server
> can act both as a client and a server on different connections.
>
> If the framing is not masked then one solution would be to use one of the
> reserved bits to indicate whether the frame originated at a client or not.
> Would your objection to using a reserved bit to indicate a mask in the
> extension data extend to this use as well? I realize that the reserved bits
> are very limited, but masking and its omission does seem rather fundamental
> to the protocol as it affects the parsing of the frame (ie. read 4 extra
> bytes at the beginnning, or don't).
>

I agree it is unfortunate that the framing is no longer symmetric now that
we have added client->server masking, but I don't think there is sufficient
value to justify using one of a very few reserved bits to distinguish them.
 There are a number of cases which already require a protocol decoder to
maintain state about which side of the communication is involved:
 - the initial handshake is asymmetric, so you have to know when side
initiated the connection to interpret it
 - if compression is being used, there are separate compression states for
each direction
 - fragmentation requires maintaining state about each side of the
connection separately

There are also other protocols which have asymmetric framing, including HTTP
(though you can get at least the basic structure of the message), and it
doesn't seem to have imposed much burden in that case.

I wouldn't object to a final protocol including it if many others wanted it,
but personally I feel it would be a horrible waste.

---

From: Adam Barth <[email-masked]>
Date: Sat, 26 Feb 2011 12:35:29 -0800

I don't believe that is a wise course of action, for the reasons we've
discussed ad nauseum.

Adam

---

From: Bruce Atherton <[email-masked]>
Date: Sat, 26 Feb 2011 12:51:38 -0800

On 25/02/2011 7:22 PM, John Tamplin wrote:
>
> I agree it is unfortunate that the framing is no longer symmetric

It isn't the asymmetry, it is the fact that there are multiple frame 
types and no way to tell how to parse some of them without additional 
contextual information. Assuming Greg's proposal is successful, a frame 
is one of the following:

[Frame Header][7-bit Length][Payload]
[Frame Header][126][16-bit Length][Payload]
[Frame Header][127][64 bit Length][Payload]
[Frame Header][7-bit Length][Frame Mask][Payload]
[Frame Header][126][16-bit Length][Frame Mask][Payload]
[Frame Header][127][64 bit Length][Frame Mask][Payload]

(or perhaps the frame mask goes before the length field(s))

Of these variations, the only portion of the frame that can't be 
determined for parsing from the bytes that go before is the presence or 
absence of the frame mask. This strikes me as an unnecessary restriction.

> now that we have added client->server masking, but I don't think there 
> is sufficient value to justify using one of a very few reserved bits 
> to distinguish them.  There are a number of cases which already 
> require a protocol decoder to maintain state about which side of the 
> communication is involved:

The issue isn't interpretation. When you are debugging, you supply the 
interpretation. The issue is parsing, and the HTTP portion of the 
handshake can be parsed easily even though the two ends are asymmetric 
in what they send.

>  - if compression is being used, there are separate compression states 
> for each direction

True, which is why people will often turn off compression when 
debugging. Turning off the masking is not an option by design, though. 
The browser vendors would never allow it.

>  - fragmentation requires maintaining state about each side of the 

Not in terms of parsing, it doesn't. It might for parsing the payload, 
of course. Or it might not. People designing the subprotocols will have 
to decide for themselves how debuggable they want their streams to be. 
But Websockets design itself shouldn't rule it out from the start.

>
> There are also other protocols which have asymmetric framing, 
> including HTTP (though you can get at least the basic structure of the 
> message), and it doesn't seem to have imposed much burden in that case.

Again, this is missing the point. Imagine if an HTTP request was ASCII, 
and the response was EBCDIC and there was no way to tell the difference. 
Interpreting the ASCII as EBCDIC or vice versa appeared to be correct 
HTTP but with vastly different interpretations. Then you would have an 
analogous situation.

>
> I wouldn't object to a final protocol including it if many others 
> wanted it, but personally I feel it would be a horrible waste.

And that's the question, isn't it?  I think it only makes sense to have 
information in the frame that allows someone to parse the frame 
correctly given a variety of formats. If you are going to use one of the 
reserved bits for anything, it should be for that. I'm not sure there is 
any other use I can think of that would be more important. But if I'm 
the only one who thinks so, the idea dies here. So what do others think 
about this?

---

From: John Tamplin <[email-masked]>
Date: Sat, 26 Feb 2011 16:10:22 -0500

On Sat, Feb 26, 2011 at 3:51 PM, Bruce Atherton <[email-masked]> wrote:
> It isn't the asymmetry, it is the fact that there are multiple frame types
> and no way to tell how to parse some of them without additional contextual
> information. Assuming Greg's proposal is successful, a frame is one of the
> following:
>
> Of these variations, the only portion of the frame that can't be determined
> for parsing from the bytes that go before is the presence or absence of the
> frame mask. This strikes me as an unnecessary restriction.

The fact that you know you are at the start of a frame header implies
you know about the "bytes that go before" all the way back to the
start of the connection, which will include the state about which side
of the connection you are processing.

> interpretation. The issue is parsing, and the HTTP portion of the handshake
> can be parsed easily even though the two ends are asymmetric in what they
> send.

If we aren't talking about debugging where you might conceivable start
observing the communication after it starts, then I really don't see
the issue.  To interpret the stream properly, you are already going to
have to know whether you are processing the client or server side, and
then it is easy enough to only look for the mask (and apply it) if you
are processing the client side.

>> There are also other protocols which have asymmetric framing, including
>> HTTP (though you can get at least the basic structure of the message), and
>> it doesn't seem to have imposed much burden in that case.
>
> Again, this is missing the point. Imagine if an HTTP request was ASCII, and
> the response was EBCDIC and there was no way to tell the difference.
> Interpreting the ASCII as EBCDIC or vice versa appeared to be correct HTTP
> but with vastly different interpretations. Then you would have an analogous
> situation.

It seems similar to me -- is the first line of an HTTP message a
method/URI or not?  It happens in this case you can guess because you
can't have spaces in headers before the colon, but I don't think
anybody writes parsers that way -- instead, they will either read the
request line outside of the code that parses the rest of the header,
or they will pass in a flag saying whether this is the client or
server side.

This seems very analogous to masking -- if the mask is before the
frame, external code will read it and pass in the mask (and for the
server case just pass in 0 unless they are trying to avoid unnecessary
XORs), or will pass in a flag into the frame parsing routine that says
it is client or server.  Even with the mask inside the frame, I would
expect it to be written the same way -- pass in a flag that says
whether the mask is expected or not.

I guess to summarize what I am saying, the client->server and
server->client are two different framings, and you already have to
know which one you are dealing with.  Sure, they are mostly the same,
which means that with a little bit of effort you can use the same code
to process them, but fundamentally there is no requirement that they
be related at all.

---

From: Bruce Atherton <[email-masked]>
Date: Sat, 26 Feb 2011 16:58:51 -0800

On 26/02/2011 1:10 PM, John Tamplin wrote:
> On Sat, Feb 26, 2011 at 3:51 PM, Bruce Atherton<[email-masked]>  wrote:

Oh, it isn't that bad. Frame headers always start at packet boundaries, 
after all. You may have to test a few packets before finding a frame 
header, and it becomes problematic to find one with big frames, but 
generally I would expect you could try decoding just a few packets and 
see which ones came out with data that looked sensible. The trouble is 
that the number of manual tests is doubled per packet when you have to 
test both for mask keys and without.

> the issue.

Since I *am* talking about that, perhaps the issue is clearer.

> or they will pass in a flag saying whether this is the client or

I'm not talking about a parser inside a client or server. Sure, they 
know what role they are going to play. I'm talking about one inside a 
tool like Wireshark that deciphers the state of all the fields in a 
Websocket packet. Perhaps tokenizer is a closer approximation to what 
I'm describing. Something that takes a bit stream and describes clusters 
of bits as fields in a Websocket frame or as ASCII/UTF-8 characters, 
possibly after applying a demasking if a frame mask is present.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 28 Feb 2011 09:29:42 +1100

> codify exactly what would

My thoughts regarding the length field issue is that we have to decide
simply between  the risk of some kind of cross protocol attack being
carried out in the 4 bytes of the length field vs the CONs listed
about masked frames.

If we seriously think that there is any kind of conceivable  attack
that can be carried out in 4 bytes (and that browsers are able to
generate frames of the required size), then we should stick with
masked frames and accept the cons.

>   present (ie, client->server frames)

See a separate thread I'm about to start about extension data.

>   when we define those extensions

I actually think it simplifies them... but mileage may vary.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 28 Feb 2011 09:39:27 +1100

> See a separate thread I'm about to start about extension data.

Cancel that - I don't think I will open another thread about extension data.

I remain concerned that the framing does not have a way to indicate
the length of the extension data, nor to partition it between
extension.
Essentially this means that there is really no extension data in the
framing and that the only way to extend a frame is to prepend the
extended data to the payload itself - so tools and intermediaries will
not be able to distinguish between extended data and real payload.   I
fear there will be issues because of this, but I don't think I can
provide the clear examples needed to win a debate on this.

Thus keeping the current "extension" data will have to do and Johns
example of how masking could be done is an exemplar of how it would
have to be used.   Both endpoints would need to negotiate the
activation of an extension (or in the case of masking, the lack of
negotiation can be seen to turn on the default masking extension).
The endpoints would then just know that the first 4 bytes of the
payload are the mask to be applied to the remaining payload.

regards

---

From: John Tamplin <[email-masked]>
Date: Sun, 27 Feb 2011 18:06:49 -0500

> provide the clear examples needed to win a debate on this.

The point is if an extension is negotiated, that extension determines how to
know which, if any, bytes at the beginning of the payload are extension data
vs application data.  Some extensions might include length information,
while others might have fixed fields that don't need anything.

Yes, that means that intermediaries that don't understand the negotiated
extensions cannot do anything with the payload of the frame, and that is
documented in the current spec.

This was the compromise that got us agreement on the framing, since
otherwise there seemed to be divergent opinions on how the extension data
should be used.  Basically, we just postponed it until each extension, or we
may come along later and define a standard way for groups of extensions to
identify their data -- ie, "the Foo extension uses the standard tag/value
encoding of extension data, with tags 14-18 as defined below" etc.

---

From: Andy Green <[email-masked]>
Date: Mon, 28 Feb 2011 07:20:50 +0000

> masked frames and accept the cons.

Actually those are good points, although there would be 8 length bytes 
in clear in the protocol, in order to issue a frame with the length you 
chose from Javascript, since you can only send existing Javascript 
objects you're going to actually have to allocate a Javascript object of 
the size you wanted.

If you wanted even a four byte length sequence on the wire, with the 
first byte as 'G', it means you had to allocate over a Javascript object 
over 1GBytes.  Setting the byte before that with b6 set for uppercase 
alphabet is > 256GByte allocation which is unreasonable for a few years.

Whatever else you did like navigate away from the script to abort the 
payload send, it would be prepended by header and unused length bytes like

  04 7F 00 00 00 00 XX XX XX XX [ EE EE... ] MM MM MM MM PP PP PP...

Where XX is the length, MM is the frame mask, EE is the extension data, 
PP is payload.

Personally I think an HTTP parser surviving the preamble is too broken 
to care about, but I see other people don't agree.

To think about controllability of extension data, which would allow 
carrying on the data in the length bytes, leads to your other point 
about if structure of extension data is really OK as it stands, when it 
is implicit from the connection handshake, so if you missed the 
handshake you can't understand where the payload starts and for example, 
all muxed channels on the connection must carry all negotiated 
extensions like compression.

-Andy

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 28 Feb 2011 19:20:53 +1100

On 28 February 2011 10:06, John Tamplin <[email-masked]> wrote:
> should be used.

Well yes that true and why I don't really want to open the discussion
on this point again.

In fact, if I was to argue anything... I'd argue the other way and say
that we should remove extension data from the framing as it is not
really a part of  the framing in any meaningful way that would allow
any code to do anything with.  Nor is there any way we could actually
police that it is used in the way we intend.  If nobody but the
extension knows the boundary between the real payload and the
extension payload, then the extension can put the data anywhere it
likes and nobody would be the wiser.

Instead of extension data, we could instead talk about how extensions
may mutate and/or extend the payload of a frame.   For example may
even be more efficient for an extension in an intermediary to append
rather than prepend data.

regards

---

From: John Tamplin <[email-masked]>
Date: Mon, 28 Feb 2011 09:42:56 -0500

> rather than prepend data.

I think that makes it harder to reason about exactly what happens when
there are multiple extensions - about the only way I could think of is
that each extension is modeled as a function taking the current
payload and mutating it arbitrarily, with the extensions applied in
the order they are listed in the handshake.  However, even that
complicates things -- what about an extension that combines multiple
frames, such as an aggregating MUX?  Trying to define a regular way to
apply arbitrary extensions that may cross message boundaries seems
hard without it excessively hard to describe simpler extensions.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 1 Mar 2011 07:19:52 +1100

On 1 March 2011 01:42, John Tamplin <[email-masked]> wrote:
> the order they are listed in the handshake.

I think we are in agreement here?   I don't think there is an extension data
field, but an extension is a function of payload producing payload.

> However, even that
> frames, such as an aggregating MUX?

That's not that complex:

    extFn(payload0) produced no payload
    extFn(payload1) produced the aggregate payload

> Trying to define a regular way to

I actually don't think the protocol document needs to describe this
extension mechanism.
It only needs to describe a way that the extensions are negotiated and
ordered and then
say that extensions are allowed to mutate and extend payload.

cheers

---

From: Brian McKelvey <[email-masked]>
Date: Mon, 28 Feb 2011 13:36:07 -0800

I agree with this.  At the very least it's nothing short of ridiculous to have an amorphous extensions field without a length indicator sitting before the payload.  At the least we should add a variable length "extension data size" field.

Having the extensions basically work like middleware (similar to the way ruby/rack middleware transforms incoming/outgoing http requests before handing the request to the next middleware function in the stack) I think makes the most sense, and indeed would require very limited explanation in the protocol draft.

How each extension works is up to the extension and it can have free reign to transform the payload however it wants, including splicing out it's extension specific metadata so that the next extension in the stack never even sees it.

All the spec needs to say on the matter is roughly that extensions are applied in the order in which they're negotiated and can transform the message in whatever way they like.

Brian

Sent from my iPhone

---

From: John Tamplin <[email-masked]>
Date: Mon, 28 Feb 2011 17:04:00 -0500

On Mon, Feb 28, 2011 at 4:36 PM, Brian McKelvey <[email-masked]> wrote:
> I agree with this.  At the very least it's nothing short of ridiculous to
> have an amorphous extensions field without a length indicator sitting
> before the payload.  At the least we should add a variable length
> "extension data size" field.

If an extension has a fixed-length data field, requiring a length indicator
is wasteful.  For example, would you want to include a length byte for
the fixed-length masking key?

I think it is better to leave it up to the extension in the base spec, and
then we can argue about a particular extension when we get to it.

---

From: Brian McKelvey <[email-masked]>
Date: Mon, 28 Feb 2011 14:13:40 -0800

But thats the point.  The field itself is specifically *not* fixed length.  An extension may define some fixed length data to add to it, but that's not the same thing.

An intermediary or debugging system should be able to at least find all of the fields defined by the framing without knowing anything about the extensions.

An alternative might be to move it after the mask key and include it in the body size.  The extensions can then transparently splice it into and out of the body before handing it off to the next layer of middleware.  Or for that matter just eliminate the extension data field altogether since it's effectively part of the body at that point.

Brian

Sent from my iPhone

---

From: John Tamplin <[email-masked]>
Date: Mon, 28 Feb 2011 17:18:01 -0500

On Mon, Feb 28, 2011 at 5:13 PM, Brian McKelvey <[email-masked]> wrote:
> An alternative might be to move it after the mask key and include it in the body
> size.  The extensions can then transparently splice it into and out of the body
> before handing it off to the next layer of middleware.  Or for that matter just
> eliminate the extension data field altogether since it's effectively part of the
> body at that point.

So how is that different than what is currently in the spec?

---

From: Brian McKelvey <[email-masked]>
Date: Mon, 28 Feb 2011 16:06:37 -0800

It's exactly what is in the spec now.  In the 06 draft, the mask applies also to the framing so it's not an issue.  Andy Green's previous message in this thread referenced the potential of placing the masking key between the extension data and payload fields:

Where XX is the length, MM is the frame mask, EE is the extension data, PP is payload.

If that were to be the case there wouldn't be a reliable way to find the payload without knowing how all the extensions manipulated the extension data field.

Brian

Sent from my iPhone

---

From: Brian McKelvey <[email-masked]>
Date: Mon, 28 Feb 2011 16:10:51 -0800

I admit I got a little confused about the state of things.. Thanks for the reminder to check on the spec for frame of reference.  :)

Brian

Sent from my iPhone

---

From: John Tamplin <[email-masked]>
Date: Mon, 28 Feb 2011 19:25:56 -0500

> If that were to be the case there wouldn't be a reliable way to find the payload without knowing how all the extensions manipulated the extension data field.

Whether it is before or after the mask it doesn't change that
intermediaries must understand any negotiated extensions to interpret
the payload further.

Personally, it seems dangerous to not mask the extension data, since
you don't know what extensions might carry data controllable by an
attacker.  If we don't mask extension data now, then it seems like
such extensions will reinvent their own masking mechanism, probably
including another key, or just ignore the problem and then we haven't
actually avoided the problem we are trying to avoid.

Regarding whether to specify how an intermediary can parse the payload
without understanding all the extensions, that requires including some
mechanism before we ship the base spec.  The compromise that led to
the current framing was precisely because we couldn't all agree on how
that should be done.  Maybe we can get agreement now, but it seems
better to get the base spec out, and then when we define the first
extension that uses extension data we can argue about it then.

Even if we come up with something like a tag/value approach, I think
we can't ever have intermediaries that understand the payload without
understanding extensions -- for example, what do they do when they see
a reserved opcode after an extension they didn't recognize was
negotiated?

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 1 Mar 2011 12:38:55 +1100

I think that the mask should apply to all data (extension and normal
payload) after the mask itself.

So by default, masking should be the only extension and we will have
frames like:

   084 LL MM MM MM MM MX MX MX .........

LL = length
MM = mask
MX = masked payload

If we negotiate an extension E1, that prepends some data to the
payload, then by default it should apply after the default masking
extension, so the extension order would be mask,E1 and we should get:

   084 LL MM MM MM MM M1 M1 MX MX MX .........

M1 = masked extension 1 data

If we then gave masking an actual extension name, it should be
possible to negotiate extensions with masking order E1,mask and we
would get:

   084 LL E1 E1 MM MM MM MM MX MX MX .........

E1 = extension 1 data

If we added another extension E2 we could have an extension order like
E1,mask,E2 and get:

   084 LL E1 E1 MM MM MM MM M2 M2 MX MX MX .........

M2 = masked extension 2 data

This interpretation would allow John's idea of extensions being
mutation functions on the payload, so this last example the payload
would be:

  frame_payload = extension1( mask ( extension2 ( application_payload )));

Note that there is no real requirement that extension data be
prepended, just that each extension knows how to find it's own data.

cheers

---

From: Brian McKelvey <[email-masked]>
Date: Mon, 28 Feb 2011 19:03:07 -0800

That seems fine to me.  My main concern was to make sure something like a wireshark packet interpreter can find the extension/payload data field, whether or not it can interpret the contents due to unsupported extensions.

Brian

Sent from my iPhone

---

From: Brian McKelvey <[email-masked]>
Date: Mon, 28 Feb 2011 19:05:23 -0800

I like this approach, though john's point about all extension data needing to be masked seems a valid one to me.  So as long as the masking "extension" is non-negotiable and always the end of the extension chain, that's fine.  And at that point, the methodology for applying masking in the same way as extensions are applied becomes an implementation detail.

Brian

Sent from my iPhone

---

From: John Tamplin <[email-masked]>
Date: Mon, 28 Feb 2011 22:06:48 -0500

On Mon, Feb 28, 2011 at 10:03 PM, Brian McKelvey <[email-masked]>wrote:

> That seems fine to me.  My main concern was to make sure something like a
> wireshark packet interpreter can find the extension/payload data field,
> whether or not it can interpret the contents due to unsupported extensions.

The current spec handles that fine -- a Wireshark dissector that didn't
understand the negotiated extensions would know where the combined
extension/application data was, but couldn't interpret it further (not that
it can really do much with the payload anyway, other than UTF8->text).

---

From: Brian McKelvey <[email-masked]>
Date: Mon, 28 Feb 2011 19:13:15 -0800

Yes it does.  I just got worried/confused when I saw andy's example where extension data came before the mask key and payload data, in which case it wouldnt be able to find and apply the mask without knowing how to interpret the extensions.

The current draft's handling is fine.

Brian

Sent from my iPhone

On Feb 28, 2011, at 7:06 PM, John Tamplin <[email-masked]> wrote:

> On Mon, Feb 28, 2011 at 10:03 PM, Brian McKelvey <[email-masked]> wrote:
> The current spec handles that fine -- a Wireshark dissector that didn't understand the negotiated extensions would know where the combined extension/application data was, but couldn't interpret it further (not that it can really do much with the payload anyway, other than UTF8->text).
> 
> -- 
> John A. Tamplin

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 1 Mar 2011 14:18:41 +1100

> I like this approach, though john's point about all extension data needing to be masked seems a valid one to me.  So as long as the masking "extension" is non-negotiable and always the end of the e…

I agree that masking is should be non-negotiable in the browsers, and
probably always the first extension as well.  But I think that for
flexibility, the protocol specification should allow non-browser
clients to negotiate otherwise.

cheers

---

From: Brian <[email-masked]>
Date: Mon, 28 Feb 2011 19:32:56 -0800

To clarify, are you suggesting that masking be allowed to be specified
as an extension during the handshake for the purposes of negotiating
the extension order, with the caveat that it be required?  What
happens if it's not specified by the client?  Where would it appear in
the order in that case?  Or would the connection be rejected?

I suspect you'll encounter the same resistance to that as we've seen
against having the masking be optional, since as John says, we don't
know what user-provided data might show up in the prepended metadata
for a yet-to-be-thought-of extension that could end up on the wire
unmasked.  :-/

Brian

---

From: John Tamplin <[email-masked]>
Date: Mon, 28 Feb 2011 22:37:26 -0500

> the order in that case?  Or would the connection be rejected?

I presume any standards-compliant implementations would reject the
connection, including browsers.

> for a yet-to-be-thought-of extension that could end up on the wire

There was significant objection from Maciej and others to even making
masking listed as a specific extension.  I also think that leaving the
possibility of any extension data unmasked would be frowned upon.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 1 Mar 2011 15:42:22 +1100

> possibility of any extension data unmasked would be frowned upon.

Indeed there were a few vocal opponents.    However, I do not think
they have well communicated the technical case against it.   I think
there were two concerns they expressed:

 1) That it could be turned off to the browsers - which I think is
adequately addressed simply by allowing the browser to make it
mandatory.
 2) That a server could be contacted by a non-WS client that
negotiates a no masking WS connection and then uses that send bytes in
the clear that may poison a cache.    However, I would think that any
non-WS client that was able to negotiate a WS connection would
obviously have pretty good capabilities to send arbitrary data.

On the other hand, there  are some clearly defined benefits of masking
as an extension within the frame:
  + framing symmetry
  + no masking within data centre
  + future agility to change masking.
  + the ability to turn off masking if SSL or similar is used.

In the context of this thread,  the subject of masking and how it
should apply to extension data came up.  My response that if we treat
masking as an extension and if we consider extensions as a change of
functions on the payload - then it is the extension order that gives a
clear indication of what masking applies to and what it does not.

So I know this is an oft repeated topic of conversation, but I do feel
that there is more support for masking as an extension than there is
opposition. There are also clear benefits vs unsubstantiated security
concerns.    I really really really don't want to start the security
debate again, as I think almost all that has been said there has been
said.     But that does not mean we have consensus on this issue, nor
does it mean the status quo should hold.

cheers

---

From: Andy Green <[email-masked]>
Date: Tue, 01 Mar 2011 07:25:33 +0000

On 03/01/2011 12:25 AM, Somebody in the thread at some point said:

Hi -

> you don't know what extensions might carry data controllable by an

Yeah that was the context of the example packet, showing that moving the 
mask to be extension data you could control at most four bytes but then...

''To think about controllability of extension data, which would allow 
carrying on the data in the length bytes, leads to your other point 
about if structure of extension data is really OK as it stands''

> a reserved opcode after an extension they didn't recognize was

What would typically go in this per-packet extension area?  Maybe it 
could be moved into being just normal payload in other packets with 
extended opcodes, eliminating the extension area concept.

-Andy

---

From: John Tamplin <[email-masked]>
Date: Tue, 1 Mar 2011 02:36:11 -0500

On Tue, Mar 1, 2011 at 2:25 AM, Andy Green <[email-masked]> wrote:
> What would typically go in this per-packet extension area?  Maybe it could
> be moved into being just normal payload in other packets with extended
> opcodes, eliminating the extension area concept.

Again, the whole reason we have this framing was that we could get
consensus on it.  We went around and around for months on different
ideas of particular ways to represent extensions, some with tag/value
extension data, some with nested opcodes, etc.  We only made progress
when we defined a bare framing that could be extended later to
implement any of these choices.

Please, let's not revisit the basic framing discussion.

---

From: Andy Green <[email-masked]>
Date: Tue, 01 Mar 2011 07:36:50 +0000

On 03/01/2011 03:03 AM, Somebody in the thread at some point said:

> That seems fine to me.  My main concern was to make sure something
> like a wireshark packet interpreter can find the extension/payload
> data field, whether or not it can interpret the contents due to
> unsupported extensions.

Don't forget a packet snooper is screwed by the mask already in current 
draft, it doesn't know if the packet is client -> server and has a 
4-byte mask prepended, or server -> client and lacks it.

It could be solved by moving the mask down the packet at least a byte or 
two and using a reserved bit in the first header byte to indicate if the 
mask is coming next or not.

-Andy

---

From: Andy Green <[email-masked]>
Date: Tue, 01 Mar 2011 07:42:02 +0000

On 03/01/2011 07:36 AM, Somebody in the thread at some point said:
> On Tue, Mar 1, 2011 at 2:25 AM, Andy Green<[email-masked]>  wrote:

Well, I feel your pain, but it's notable nothing has been suggested to 
need that extension area per-packet and it's legitimate to ask for an 
example of what would need it and if we can get rid of it.  I'm not 
asking out of the blue it is part of looking at moving masking down the 
packet and make sure it can't run on into the undefined extension area 
in a controllable way.

If there are reasonable examples I am happy to accept that and forget 
about it.

-Andy