Subject: Re: [hybi] Comments to draft-ietf-hybi-thewebsocketprotocol-05

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Sat, 19 Feb 2011 18:40:38 -0800

as per the first page of the draft, the proper forum is [email-masked]. reply
inline.

On Sat, Feb 19, 2011 at 11:01 AM, Silvio Ventres
<[email-masked]>wrote:

> Hello, Ian.
>
> Please let know if there exists a forum for discussion of these points.
> At this point, it seems the WebSocket has been designed with very
> vague goals, while the spec draft has been written to encompass
> additional goals, which can only be guessed, and are not explicitly
> specified anywhere, such as prevention of some usage scenarios or
> stream fragmentation.
>
> The comments follow section numbers:
>
> 1.6: quote: "fields starting with |Sec-| cannot be set by an attacker
> from a Web browser, even when using |XMLHttpRequest|."
> "Sec-" are _only_ filtered for XHR as specified in XHR spec, not any
> HTTP specs per se.
>
>
No other browser APIs allow these to be sent, and browser vendors would
consider that to be a bug. XHR is the closest thing to sending an arbitrary
request from a browser, and this forbids Sec- headers.

> 1.6: quote: "fail to establish a connection when data from other
> protocols, especially HTTP, is sent to a WebSocket server, for example
> as might happen if an HTML |form| were submitted to a
>   WebSocket server.  This is primarily achieved by requiring that the
> server prove that it read the handshake"
> What does the "server prove that it read the handshake" mean? How does
> the server "prove" anything? If the server sends any response to the
> HTML <form> request, that's all the response that was expected anyway,
> so who cares about "proof" ?
>

By making a computation involving data in the handshake, the server "proves"
that it actually parsed, at least in some minimal version, the request, as
opposed to returning a canned response. The intent was to ensure the client
is communicating with a WebSocket server, not an arbitrary
server/intermediary/... that may be confused and returning a response
without actually understanding WebSockets.

>
> 1.6: quote: "This protocol is intended to fail to establish a connection
> with
>   servers of pre-existing protocols like SMTP or HTTP, while allowing
>   HTTP servers to opt-in to supporting this protocol if desired.  This
>   is achieved by having a strict and elaborate handshake, and by
>   limiting the data that can be inserted into the connection before the
>   handshake is finished (thus limiting how much the server can be
>   influenced)."
>
> What are the malicious scenarios that you are defending from here ?
> Seems like the "intention to fail" establishing connection is there as
> a "good to have" thing, without elaborating as to why it's a good
> idea.
> Moreover, the "strict and elaborate" handshake does not seem to be
> necessary anymore, as the data frame format differs significantly in
> the latest versions of the draft from plaintext-based SMTP and HTTP,
> and thus WS clients will be incompatible with non-WS services with or
> without the handshake.
>
>
Please see the literally hundreds of emails on this mailing list, especially
those from adam barth, about how intermediaries can be confused by
attacker-controlled data to the effect of poisoning the cache of deployed
intermediaries. This is not something that the WG is going to revisit.

> 4.1: "..masked to avoid confusing network intermediaries, such as
> intercepting proxies."
> 4.2: "MUST NOT make it simple for a server to predict the masking-key
> for a subsequent frame."
> Wait. Can it be decided what the masking-key is there for? Is it to
> avoid confusing proxies? Or making life harder for servers?
> Why would you want to "NOT make it simple for a server to predict the
> masking-key" ?
> The masking key may as well have been chosen statically at compile
> time, or fixed in spec, in same way
> as the "258EAFA5-E914-47DA-95CA-C5AB0DC85B11" nonce was.
> Or just removed completely, as it doesn't serve any clearly defined
> purpose.
>
>
If the next key is predictable, then the server can cause the client to send
data with a predictable value on-the-wire, which is exactly what masking is
designed to prevent. Again, I would encourage you to read the history of
this mailing list for further details on masking.

> 4.2: "..unpredictability of the masking-nonce is essential.."
> What is masking-nonce? Why is unpredictability essential?
>
>
see above

> "..prevent the author of malicious application data from selecting.."
> 4.2: If talking about the masking-key, it's transmitted along with the
> data every time, how does that prevent anyone from decoding the data?
> Sending encoded data + key is same as sending plaintext in the first
> place, why add this complexity?
>

it's not to prevent people from decoding the data. It is to prevent an
attacker from controlling the bytes on the wire and potentially sending data
that could cause an intermediary to misinterpret the data for ill effect.

> While at that, what or who is "author of malicious application data"
> and what is meant by "selecting" ?
>

again, read the mailing list, but in brief, user goes to webpage controlled
by attacker (or an ad supplied by attacker), attacker supplies javascript
that, upon request from the attacker's server, sends data to the attacker's
server. If the attacker can control the format of this data on the wire,
there exists a threat that this data can be used to trick intermediaries for
ill effect.

>
> 4.4:
>  TCP already handles fragmentation. Why add additional layer of
> fragmentation?
>

Because the frame includes the length of the frame, and without
fragmentation one would have to know the length of the message before being
able to send any part of the message. With fragmentation, a server can begin
sending a message whilst the message is being generated.

Again, please see the mailing list archives.

> If the reasoning is "to allow application to use imcomplete received
> data instead of blocking waiting for for completion",
> then the additional fragmentation doesn't add any benefit. Moreover,
> the WS fragments might _not_ align with TCP fragments,
> leading to delays in processing, while WS parser waits for the
> completion of "WS-non-fragmented" data which was fragmented
> by TCP. If the reasoning was, at some point, "to allow connection
> through proxies", then, again, by using CONNECT, the proxy
> doesn't care about the request length.
>
> 5.1.2: "If the user agent already has a WebSocket connection.."
> What is meant by "has a WebSocket connection" ? Is the connection in
> handshake state? In TCP CONNECTING state?
> Seems like it's implied that there is no limit to _established_
> connections, just limit on connections in CONNECTING
> or handshake state. This should be clarified.
>

I will clarify this. It is intended that there only be one connection in
CONNECTING state.

>
> Regarding the handshake and security in general, what is the
> Sec-WebSocket-Key made to prevent?
> Non-WebSocket servers answering WebSocket clients?
>
>
Yes

> It won't work anyway, because of the response requirement to contain
> echoes of custom headers and 101 status code.
>

Getting something to echo a request is not a significant bar to satisfy
various members of the working group. Performing computation on that data
ensures that the endpoint is WS aware, especially given that some of the
data required for the computation is not sent, but instead exists  only in
the protocol spec.

> But, by adding this complexity, you're practically removing embedded
> and low-computing-power devices from being able to
> act as WS servers.
>

This was already discussed ad-nauseum, again I refer you to the archives of
the list. Many of us do care about power, but believe that this is not a
sufficient barrier.

> Section 5.2 actually talks about a server which only purpose is
> processing the complex handshake. Why do you want to
> force people to add such front-end-servers instead of connecting to
> the real ws-server directly?
>
>
In reality, many people in large deployments do have frontends/load
balancers that terminate the connection. The spec does not require this
though.

> 5.2.2.3.3 "The nonce MUST be randomly selected randomly for each
> connection"
> Aside from the double-"randomly", why is the randomness so important here ?
> What is it made to prevent?
>

Double randomly is a typo :) As for why it's important, as I said earlier in
the response, if the server is controlled by an attacker and the server can
predict the next nonce, the server can instruct the client to send data
that, when masked with that predicted nonce, will have an appearance on the
wire that could confuse the intermediary for ill effect.

> Non-WS client getting data from the WS server?
> They wouldn't be able to parse the response anyway. And if they would,
> how is that different from a buggy "WS-conforming" client?
> The keys/nonces don't solve any real problems, they just add
> complexity to the handshake.
>
>
> Thank you for reading.
> Hope the draft can be mended to fix this, and the protocol simplified
> to conform to the goals declared.
>
> --
>  silvio
>

---

From: Andy Green <[email-masked]>
Date: Sun, 20 Feb 2011 05:47:34 +0000

On 02/20/2011 02:40 AM, Somebody in the thread at some point said:

Hi -

> This was already discussed ad-nauseum, again I refer you to the archives
> of the list. Many of us do care about power, but believe that this is
> not a sufficient barrier.

This is a serious issue and in -04 it really was too much for the 
weakest embedded devices to keep up; there was a SHA-1 per frame in that.

However Silvio if you look at -05, although it still breaks zerocopy and 
that's annoying, actually the load is really reduced at both the client 
and server side.  It just boils down to an extra 4 bytes per frame on 
the wire and visiting the buffer with an xor action.  It slows a device 
down and reduces simultaneous peers it can handle, but it probably 
doesn't actually kill it any more unless it was close to the edge of 
being able to cope anyway or an extreme case.  (Further, an embedded 
client can just - illegally - insist to choose 00000000 frame key all 
the time and then just pays for the extra 4 bytes; in my case the client 
is on a local USB Ethernet link and there are no intermediaries).

There was a lot of politics about the munging decision and the 
background was that two major browser vendors had disabled websockets by 
default due to Adam Barth's claims generating "security noise" and 
people like CNET writing articles about "security" issues in websockets. 
  Munging the frame answered those concerns in a way that asking more 
questions about exactly what the Barth had discovered and asking for 
stuff to repeat his nondeterministic and unrepeatable (by others; he 
refused to repeat them himself) experiments or to identify these alleged 
vulnerable proxies (NONE have been identified AFAIK leading me at least 
to suspect they do not exist) did not.

>     complexity to the handshake.

They are solving problems but in the frame key / munging case, that 
problem is not a technical one but had to be addressed anyway (or 
practically, websockets effort would wither up and die via lack of 
browser support).

-Andy

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 20 Feb 2011 08:22:54 +0100

Hi Ian,

> as per the first page of the draft, the proper forum is [email-masked]. reply

I think that the fact that some people don't get the reasons for some
choices when reading the draft means we should add a short FAQ at the
end, basically with Silvio's questions and your responses. It will limit
the risk of misinterpretations, and will ensure that implementors won't
get bored by algorithms they don't care about and try to simplify them.

Regards,
Willy

---

From: Patrick McManus <[email-masked]>
Date: Sun, 20 Feb 2011 08:06:08 -0500

> get bored by algorithms they don't care about and try to simplify them.

please make that a different document if needed and let the protocol
document just describe the protocol as simply as possible and be
published as soon as possible.

---

From: Silvio Ventres <[email-masked]>
Date: Mon, 21 Feb 2011 02:57:55 +0200

> as per the first page of the draft, the proper forum is [email-masked]. reply

As "hybi" doesn't seem likely to stand for any meaningful
working-group or mailing-list name, it's hard to guess that it's a
mailing list intuitively.
A reasonable alias, such as, [email-masked] would do great
here, redirecting to hybi if needed.

> request from a browser, and this forbids Sec- headers.

The point is that the "even when using" should be changed to just
"when using", as the "even" in reality means "only".
Better yet, reference the XHR spec as the source of the protection, as
it's not easily found by google search for, e.g. "Sec- headers".

Regarding general browser behavior, if a browser includes a plugin
that allows user to send requests with custom headers, nothing
precludes it to do so, and allowing Sec- headers to be set is not a
bug in any way. To reiterate, Sec- headers are ONLY disallowed by the
XHR spec, and not any other document.

> without actually understanding WebSockets.

Again, the client is required to verify that the server returns
Sec-Websocket* headers.
If some "arbitrary server/intermediary/..." actually echoes the
Sec-Websocket* headers correctly, and answers with the 101 response
code,
it's fair to assume that it was made deliberately to do that.
What kind of server do you foresee echoing the WS headers correctly,
returning 101 response code and NOT being a WS or WS-compatible
server?

Moreover, why is the reasoning for the goal of non-compliance with
SMTP/HTTP not explained anywhere?
Can understand that you would want to limit WS clients from sending
spam through SMTP, so write it somewhere.
HTTP XHR can theoretically be used to send mail through SMTP. Is it in reality?
Does HTTP include weird "pseudo-security" measures to prevent
"pseudo-risky" scenarios?
The sky is not falling. If you're adding security measures, please
explain what are the scenarios that you are protecting from.

> intermediaries. This is not something that the WG is going to revisit.

Can you please give a short summary regarding the attack scenarios in
question? This summary should probably go either inside the same spec
doc or into a separate rfc to explain the reasoning behind the
protocol.

> this mailing list for further details on masking.

With XHR the server can _already_ force the client to send predictable
value on-the-wire.
Are there any real security problems caused by that? How are/were they
dealt with?
If some intermediaries are broken by such "malicious data" then any
XHR request can be used to break them.
Isn't it a bit too obscure of an attack vector?

>> What is masking-nonce? Why is unpredictability essential?

The "masking-nonce" keyword is not used in the document up till 4.2.
Is that the same as "masking-key" ?

> that could cause an intermediary to misinterpret the data for ill effect.

What kind of effect are you talking about?
To actually _connect_, before receiving any data, a server would have
to support the WS protocol. There isn't even a way to start sending
data unless the server answers the handshake.
Now let's go from the other side, assume the server actually _wants_
to support WS protocol. Now, instead of answering a short _static_
handshake, and then sending and receiving data in the most natural
way, the developer has to code the XOR mechanism and connect a random
number generator. Why? Why the hate for service and client developers?

> there exists a threat that this data can be used to trick intermediaries for

If the attacker wants the user to send some data to the attacker's
server, it can be done with XHR. How did XHR deal with this scenario?
Were there any dire consequences as an outcome of XHR's simplicity?
Moreover, a simple "link" tag, or any "src" attribute can be used to
send arbitrary data.
And these are used in the wild.
It is not the place of WS to fix this problem one-sidedly, if it
actually exists. If the problem can be shown to actually be real, then
the information should be used to discuss the problem with
HTTP/HTML/XHR maintainers so as to create a real solution.

> Again, please see the mailing list archives.

Server cannot begin transmission of the message until the user has
called the send() function.
When user has called the send() function, the length of data to be
sent becomes known.
The message is sent regardless of its length, with the stream being
fragmented/reassembled by TCP.
If the user/server _actually_ wanted to send data in chunks, he would
run send() several times with the chunks to send.
Is WS making the decision about the transmission blocks on behalf of
the user? Why?
What happens when a user wants to send a huge amount of data in one block?
What part of the API supports this fragmentation ability and how is it
exported to the user?

> data required for the computation is not sent, but instead exists  only in

It's not only echoing a request, but also returning the 101 response code.
If you want additional proof that it's a WS server, make server always
return smth like:
"101 This Magic Phrase is here to verify this is an actual WS server"
instead of simple 101 code, although the "101" response code is not
something that most servers answer most clients with, especially while
echoing the Sec-Websocket headers...

The computation part does NOT actually increase security in comparison
with the echo+101 static-handshake part.
If a server does echo+101, it's probably a WS server or something that
wants to look as WS server, in which case it's probably gonna do
whatever else WS servers do, whether there is computation step
involved or not.

> the list. Many of us do care about power, but believe that this is not a

Well, this "barrier" doesn't actually increase security, but adds a
burden on devices implementing the protocol. Is that what is meant by
"sufficient" ?

While at that, why not add a nice "security" fix: make sure
intermediaries don't touch the message in-flight. Hash the total
contents of the message when sending, save it, and request an echo of
that hash in the reply. Compare them when receiving the reply, thus
you will verify the message was not changed. Great feature. Oh, even
better, have the other part always quote the last message received
with messages sent. This way, not only will you be able to verify
whether your packets get delivered correctly, but also see what was
changed in case there was change! Let's go even farther: maybe your
packets have been maliciously delayed, to play part in some kind of
timing attack! Let's have the transmitted packets include the time of
transmission, and reply packets include the time of last packet
reception. This way you'll also be able to prevent replays! Anyone
wants to continue?

> balancers that terminate the connection. The spec does not require this

You are talking about creating a _separate_ server just to take care
of the computational burden of the handshake!
That is a lot of complexity added to servers that are already quite
busy with the load-balancing jobs.
The spec does not require this, but it acknowledges that the burden
might be too much for busy servers, and operators might wish to
offload it somewhere.

> wire that could confuse the intermediary for ill effect.

As mentioned before, the "ill effect" is ill-defined (please excuse the pun).
The same "ill effect" can be caused by using standard HTML with or without XHR.

All in all, with due respect to the ingenuity of solutions, the
security features in the current WS spec don't seem like they were
developed
by people with any real experience in security field. The best thing
to do is remove the features that provide questionable defense. In
case any contributors feel like there are security problems, define
exactly the attack scenarios or problematic cases and analyze them for
impact, and, in case problems are found to be real and fall in scope
of the WS protocol, rather than HTTP or HTML proper, find solutions,
possibly with help of security specialists from the community, or at
least be reviewed by one. Bruce Schneier might be a good candidate.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Mon, 21 Feb 2011 03:16:02 +0200

> This is a serious issue and in -04 it really was too much for the weakest
> embedded devices to keep up; there was a SHA-1 per frame in that.

The tool is great. SHA-1 is a wonderful hash. MD-5 is not any less
wonderful, although less secure.
The SHA-1 was changed to MD-5. Did anyone analyze the impact of that?
If there's no difference whether it's
MD5 or SHA-1, why not use CRC? What is the reason of using a hash in
the first place?

> that's annoying, actually the load is really reduced at both the client and
> server side.  It just boils down to an extra 4 bytes per frame on the wire
> and visiting the buffer with an xor action.  It slows a device down and
> reduces simultaneous peers it can handle, but it probably doesn't actually
> kill it any more unless it was close to the edge of being able to cope
> anyway or an extreme case.  (Further, an embedded client can just -
> illegally - insist to choose 00000000 frame key all the time and then just
> pays for the extra 4 bytes; in my case the client is on a local USB Ethernet
> link and there are no intermediaries).

Exactly the point! Since there's no way to check the actualy client
implementation,
and no penalty for using a static masking-key, many clients will
probably do just that.
Then, what's the point in requiring it? If there's a real reason for
this masking, then a better solution
can and should be found. If there's no reason, the feature should be removed.

> There was a lot of politics about the munging decision and the background
> was that two major browser vendors had disabled websockets by default due to
> Adam Barth's claims generating "security noise" and people like CNET writing
> articles about "security" issues in websockets.  Munging the frame answered
> those concerns in a way that asking more questions about exactly what the
> Barth had discovered and asking for stuff to repeat his nondeterministic and
> unrepeatable (by others; he refused to repeat them himself) experiments or
> to identify these alleged vulnerable proxies (NONE have been identified
> AFAIK leading me at least to suspect they do not exist) did not.

This doesn't seem like a good protocol design practice...

> They are solving problems but in the frame key / munging case, that problem
> is not a technical one but had to be addressed anyway (or practically,
> websockets effort would wither up and die via lack of browser support).

The problem can also be solved by getting a security expert on board.
That should alleviate FUD distributed by security amateurs.
Moreover, masking the contents removes a HUGE benefit of debugability,
being able to see the traffic in plaintext, that part which made HTTP
so successful.

--
 silvio

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Sun, 20 Feb 2011 17:19:33 -0800

Again, please read the archives. I already responded to your email in
reasonable detail, but the truth is that I have little desire to rehash
arguments we have already moved beyond on the list.
>> This was already discussed ad-nauseum, again I refer you to the archives
of
>>> Aside from the double-"randomly", why is the randomness so important
here
>>> ?

---

From: Silvio Ventres <[email-masked]>
Date: Mon, 21 Feb 2011 03:19:58 +0200

> get bored by algorithms they don't care about and try to simplify them.

Exactly!
The security goals in the document are not specified at all.
And, given some of Adam Barth's comments on security may be correct,
the way to deal with those is to distill them into well-defined attack
scenarios,
analyze the impact and scope, possibly find a fix and then add it.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Mon, 21 Feb 2011 03:50:55 +0200

Hello.

Please do not consider this a personal disfavor, as, with all due
respect, your previous reply seemed slightly dismissive, implying that
the questions have been answered in the archives. Please excuse the
reposting of the questions, as this is the only way to start a
discussion.

After thorough reading of the archive, this is a request for comments
regarding the following recommendations and questions to be answered
or commented upon.

1. What attack scenario is the computation part of the handshake? Why
requirement of echo+"101" response is not considered enough to remove
compatibility with non-WS services?
Recommendation: remove the complex computation procedure from the
specification, leaving just the requirement of header-echo and "101"
response, possibly adding a predefined static string to be included in
the "101" response to identify WS server as such, f.e. "101 WS Server
Magic String".

2. Remove misleading "even" word in the section 1.6 of the
specification and add clarification that only XmlHttpRequest
specification actually forbids requests with custom headers starting
with "Sec-".

3. Clarify the reasoning behind removing compatibility with SMTP/HTTP
protocols, as mentioned in section 1.6, and the exact attack scenarios
that the protocol strives to prevent.
Recommendation: if the reasoning cannot be clarified, remove the
incompatibility goal and relevant features from the spec.

4. Clarify the attack scenarios being prevented in section 4.1 and 4.2.
Recommendation: unless the attack scenarios can be shown to have real
impact and being specific to WS as compared to standard HTML link, src
or XmlHttpRequest methods, remove the goal statement in 4.1 and 4.2 as
well as relevant prevention features.
Recommendation: in any case, change or clarify wording of "author of
malicious application data" and "selecting data".

5. Clarify the goal of the masking-key feature and its randomization
procedure and analyze it's impact on protocol debug capability.
Recommendation: unless shown that the masking feature actually
prevents a real attack scenario, remove the feature to simplify
protocol debug and development.

6. Clarify the goal of fragmentation feature and provide a clear use case.
Recommendation: in case no clear benefit to either client or server
performance can be shown, or the benefit is connected to considerable
impact on user control of the transfer timing, the feature should be
removed.

Thank you.

As advised in RFC 2026, section 6.5.1, this email is being CC'd to the
HyBi WG chairs.

--
 silvio

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Sun, 20 Feb 2011 20:57:10 -0800

reply inline

2011/2/20 Silvio Ventres <[email-masked]>

> 1. What attack scenario is the computation part of the handshake?

it was a design goal to ensure that the endpoint actually parsed the
handshake. By performing computation this is demonstrated to a satisfactory
level. Simply regurgitating a fixed response (your magic string) doesn't
show that the endpoint actually parsed anything.

> Why
i will address this

> incompatibility goal and relevant features from the spec.
If merely by sending javascript to a browser (e.g. taking out an ad) I can
get it to open TCP connections to unsuspecting servers, that violates many
assumptions people make about security and network boundaries, and
essentially turns the browser into a proxy by which an attacker can attack a
local network that assumes connections are only being made by local devices,
not on behalf of remote devices. I am happy to add in some text to that
effect, but I thought that was clear already.

> protocol debug and development.
You clearly did not read the archives thoroughly enough :) This was the
source of much debate, and months of impasse. The chairs declared consensus
and we have moved on. Please do read the archives for more detail.

> impact on user control of the transfer timing, the feature should be
Use case:

Client makes a request to server, e.g. "Perform a search for foo". Server
wants to start sending back data before the entire results are available,
e.g. "results 1-2". Server wants to send the results as a single message, to
maintain messaging semantics. By making use of fragmentation, server can
begin sending data before all of the data is available. If server couldn't
use fragmentation, it would have to wait for all the data to be available,
so that its length could be calculated. By sending the data as multiple
fragments, the server can begin sending the data as it is generated,
reducing total latency.

Fragmentation is also necessary for multiplexing, to avoid a large message
totally blocking the channel. Again, I encourage you to read the archives as
this is another well discussed issue upon which consensus has been
declared.

> As advised in RFC 2026, section 6.5.1, this email is being CC'd to the

I am perplexed. "(a) his or her own views have not been
   adequately considered by the Working Group, or (b) the Working Group
   has made an incorrect technical choice which places the quality
   and/or integrity of the Working Group's product(s) in significant
   jeopardy. "

Do you believe your views have not been adequately considered? As I have
said repeatedly, most of what you brought up was discussed to death, was the
cause of months of impasse, and we moved past this and the chairs declared
consensus. Please do read the archives.

Clearly it's not (b) as if you read the archives, you would see that we are
finally making progress, and most of your suggestions have already been
discussed to death and found to be something that would not be implemented,
i.e. your suggestions would place the product in significant jeopardy.

At any rate, have a good weekend, and please do read more of the archives. I
can assure you that it takes me hours to just read the emails from one week,
so I sympathize with the effort it takes, but most of this has already been
discussed to death and consensus declared by the chairs.

-Ian

> >>> data with a predictable value on-the-wire, which is exactly what
> masking
> >>> is
> >>> there exists a threat that this data can be used to trick
> intermediaries
> >>> for
> >>> ill effect.
> >>
> >>> Double randomly is a typo :) As for why it's important, as I said
> earlier
> >>> in
> >> The same "ill effect" can be caused by using standard HTML with or
> without
> >> XHR.
> >>
> >>

---

From: Andy Green <[email-masked]>
Date: Mon, 21 Feb 2011 07:23:12 +0000

On 02/21/2011 01:16 AM, Somebody in the thread at some point said:

Hi -

> MD5 or SHA-1, why not use CRC? What is the reason of using a hash in

For handshake, using SHA-1 is fine even on a very weak box because a 
weak box will very seldom see connections and usually only one or two at 
a time.  A little bit of latency making the connection is OK since it's 
no worse than if the box was geographically distant anyway.  And the 
handshake is doing something useful in terms of giving good confidence 
it's a real websocket server on the other end.

But you didn't see what I was telling you in the quote.  In the 
previous, -04 version of the standard for EVERY FRAME the client and 
server each had to compute a SHA-1, regardless of payload length.  A 
frame can be one byte of actual payload, can come very often, a server 
can see a huge number of them per second, etc.

In -05, the current version, that's gone and replaced with the 
simplified 4-byte frame key / XOR action you saw.  So from that 
perspective, the -05 scheme reflects sanity and moderation.

So although I think the whole munging thing is a farce, it's now so much 
better than what went before continuing to argue against it has to be 
balanced against ending up with no websockets at all due to the other 
points I mentioned.  (And indeed Opera have made it clear they won't 
track any further evolution in websockets and are waiting for it to lay 
its 1.0 egg or die.)

> This doesn't seem like a good protocol design practice...

It was doing OK before Adam Barth's paper, then the process has had to 
deal with what boils down to hysteria, "security posturing" and 
ass-covering.

> being able to see the traffic in plaintext, that part which made HTTP

I don't think it's relevant to the discussion but AIUI Barth does this 
for a living and is then a professional.  There were crypto experts 
involved but since the issue was not bounded by facts and reason, it 
just meant they tended towards the most extreme solutions to solve this 
unbounded problem like AES-based obfuscation.  That has been soundly 
rejected now so we are not in such a bad place.

Munging is a fact tied to bringing Mozilla back on board with 
re-enabling websockets by default (so Mozilla can say "Phew good job 
websockets fixed their problem! hey everyone it's SAFE to turn it back 
on now!") and we can all agree that action seriously impacts the future 
of websockets.  Pragmatically, the -05 weak munging is small enough 
impact it's not worth setting the effort back months or killing it to 
argue to get rid of it now.

-Andy

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 22 Feb 2011 20:21:19 +0100

> being able to see the traffic in plaintext, that part which made HTTP

So right!

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 22 Feb 2011 20:31:25 +0100

> show that the endpoint actually parsed anything.

And what is the benefict of knowing that the server has parsed something?
If the server replies 101 with a "magic string", is it not enough to
ensure that the server wants to behave as a WebSocket server?
Anyone desiring to behave as a websocket server can perform such
computation, so...

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 22 Feb 2011 20:43:49 +0100

On Tue, Feb 22, 2011 at 08:31:25PM +0100, Iñaki Baz Castillo wrote:
> 2011/2/21 Ian Fette (????????????????????????) <[email-masked]>:

Iñaki, please consult the archives. Many persons had different opinions
on various points related to the security around the handshake and the
masking, and reopening this can of worms is something terribly painful
for everyone. We managed to reach a rough consensus, nobody's fighting
anymore so let's not re-discuss the pros and cons of every choice, please.
Also, you may find that when reading everyone's opinion and experience,
you will change your mind multiple times along the learning of the various
cases, and there's nothing wrong with that, so it's an interesting exercise
anyway.

Thank you,
Willy

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Tue, 22 Feb 2011 21:17:06 +0100

> cases, and there's nothing wrong with that, so it's an interesting exercise

Hi, as a suggestion it would be very useful if the draft itself, or a
FAQ section somewhere in a web, would describe the arguments so many
times discussed in the maillist. Checking the maillist archives is not
so easy for people recently interested in WebSocket protocol.

For example, I read the revision 05 of the new draft and I find no
reasons at all for the client-to-server masking, neither for the
complex handshake. Neither I find a simple Changelog section in the
draft so I don't know which exact changes have ocurred in the revision
101 of the draft!

You can redirect people to maillist archives but that will never be a
suitable solution for newcomers. Reasons must be documented, not just
discussed (IMHO). If Websocket becomes a RFC this year, we can not
expect that people reading it two years later would check the maillist
archives to understand the motivations of the protocol.

Please, don't take me wrong ;)
Cheers.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 22 Feb 2011 21:28:37 +0100

> so easy for people recently interested in WebSocket protocol.

I know, and I too asked if we could add a FAQ section to the draft, but
it seems like it's not the best place to put it. Rest assured I'm not
taking you wrong, I'm just trying to ensure we don't discourage our
editor with many "please do this/that" when there's still so much work
in the pipe !

Cheers,
Willy

---

From: Salvatore Loreto <[email-masked]>
Date: Tue, 22 Feb 2011 22:56:21 +0200

On 2/22/11 10:17 PM, Iñaki Baz Castillo wrote:
> 2011/2/22 Willy Tarreau<[email-masked]>:
if we find volunteers to write down a FAQ we can host it on
http://trac.tools.ietf.org/wg/hybi/trac/wiki

people interested in contributing can contact me privately

> draft so I don't know which exact changes have ocurred in the revision

that is a good suggestion,
we will insert a Changelog section from the 06 version on...

cheers
/Sal

-- 
Salvatore Loreto
www.sloreto.com

---

From: Silvio Ventres <[email-masked]>
Date: Wed, 23 Feb 2011 21:35:25 +0200

Hello, Ian, Andy.

It took quite a long time to read the archives more extensively, but
it seems an exhaustive read cannot be done, since there is not
information before hybi-merge in 2009.
In any case, as the Author of the draft, you are supposed to have been
pursuaded by the discussion and be able to answer questions regarding
the draft, instead of saying
"read the archives". The "archives" answer is a dangerous precedent
that creates a huge barrier to discussion entry for anyone not
involved in the discussion from the start.
You wouldn't tell someone that wants to ask for clarification of goals
behind some law to go read the whole history of the law and it's
discussions in the Congress to enter
into discussion about it. You are the unique person that should be
able to clearly summarize why any and all features of the standard
exist in its current iteration. This is
the responsibility of the Author, and it cannot be simply pushed away.

In any case, here is a preliminary analysis of the current state of affairs.
It's a bit of a long read, but is intended to clarify any and all
goals behind the possible and actual protocol implementations.

Please comment when time allows.

========================================================================================
Hypertext Bidirectional protocol state-of-affairs analysis draft
====

Let's start from the beginning so the goals of the protocol can be
clearly mapped out and defined well.

>From the point of view of benevolent application developer, access to
TCP-style send()/recv() functions
is the best possible way to interact with the various non-HTTP data
sources, especially chat-type services.
The Web Browser, then, takes the role of frontend for applications
that have full network access, with
standard limitations such as amount of connections in CONNECTING state etc.

This gives ability for the Web developer to implement innovative apps
such as chats directly connecting
to IRC servers, IM services with direct connection to each IM server -
Aol, GTalk, Jabber, VoIP-support
services directly talking to SIP/TCP peers, and, possibly, using the
new audio/video tags to interact with
media, even P2P and other file transfer protocols. With such
capability, possibilities for innovation are vast.

The problem here is, that the client "Web Surfer" is, mostly, naive
about whom to trust.
He might go to cnn.com to see the news, but also to *warez*.com to
download "free" apps.
He might also visit various blogs which source their ads externally,
and thus, unknowingly visit
google-analytics.com as well as ads.doubleclick.net and others.

Given full unrestricted access to TCP sockets, these data sources
might cause the browser application
to participate in various actions not condoned by the Web Surfer, such
as trying to send spam through SMTP,
or mounting cyber-attacks on various services. These actions in
general will be from now on called "attacks".

Given the empirical effectiveness of the same-origin policy, it should
be clarified that the attacks
are unlikely to target the same subdomain that serves the webpage html source.

Another clarification to be made is that, excluding
security-compromised servers, the malicious scripts
written to mount the attacks are unlikely to come from the actualy
subdomain, as compared to externally
sourced resources, such as ad mechanisms.

So, from the point of view of responsible party, the solutions are as follow:

1. Push responsibility to the user = Web Surfer:
   Disallow socket access unless user has acknowledged such with full
understanding of the consequences.

2. Push responsibility to the web developer:
   Prevent socket access from being initiated from externally-sourced scripts.

3. Push responsibility to the protocol-implementation developer:
   Mangle control and/or data packets to prevent compatibility with
other protocols.

The question then is whether 1+2 are enough to stop most of the
attacks, and if the web developer can be
trusted to issue socket calls. Indeed, there is a mechanism to verify
web developer's trust, which is
widely used in practice, HTTPS certificate verification.

The solution should thus implement one or more of the following precautions:

--
1. Prevent socket-based actions from targeting external hosts

Possible implementation:
   Java/Flash/Silverlight implement this, with external-hosted
crossdomain.xml exception list

Pros:
   This prevents most of the attacks, and is proven to work empirically

Cons:
   This solution prevents most of the services mentioned above,
disallowing external data source interaction

--
2. Prevent socket-based actions from external/untrusted sources from
being executed

Possible implementation:
   This might be implemented as an explicit <script
uses-socket-api=function_name> parameter in the <head>
   section, which would only allow the named function to access the
socket-api. If any function tries to
   use the socket-api, the name will be verified against the name
given in the header.
   Trusted/untrusted sources can be differentiated with HTTPS
certificate verification. Only verified
   hosts will be then able to use the "uses-socket-api" attribute.
This can be combined with white-lists
   maintained by the browser developers, such as the anti-phishing filters.

Pros:
   This allows web developer to protect socket calls from being
executed from unintended sources, such as
   comment fields, ads, analytics scripts and various other resources
loaded from external sources.
   With HTTPS Certificate verification, there is low chance of web
developer authoring malicious scripts.

Cons:
   This still leaves open the ability of the web developer himself to
write scripts that mount attacks,
   although it is mitigated by HTTPS Certificate verification.

--
2a. Provide ability for users to log socket accesses and a central
repository for CA complaints

Possible implementation:
   A button named "Report communications from this website" can be
provided to log access and initiate
   a process to verify claims with the Certificate Authority.

Pros:
   Allows proactive users to take part in removing bad websites from
"trusted" lists.

Cons:
   Mostly-manual process, expensive both for users and report-analysing parties.

--
3. Make user explicitly confirm any/dangerous socket-based action

Possible implementation:
   Push the burden of explanation on the web developer. Have a
"reason="Connecting to IRC server" type
   explanation for each socket call to a different host. Ask user
every time a connection is being made,
   providing explanation. Provide "Allow Always" or "Allow next 60min"
answers in addition to "Allow/Deny".
   Provide a clear "Stop communications" button to the user.

Pros:
   Limits undesired activity by making user explicitly acknowledge it.

Cons:
   Pushes the responsibility from web developer to the Web Surfer, who
might be clueless and press
   "Allow Always" just to not be bothered.

--
4. Use a specific connection-starting handshake, incompatible with any others.

Possible implementation:
   Make sure the data cannot be sent until an elaborate handshake is
completed, can be done with various
   challenges, such as magic strings, crypto-challenges, making sure
only a compatible WS-server will pass.

Pros:
   Limits undesired activity to handshake-implementing hosts, making
sure no non-WS services will be sent
   any user-specified data, limiting the attack vector to the
handshake itself, which is controlled by the
   WS implementation in browser.

Cons:
   The handshake makes development of services interconnecting with
non-HTTP/WS peers impossible in practice.
   Thus, instead of allowing wild innovation, the solution simply
replaces the COMET hacks currently in use.
   Depending on handshake complexity, may add unnecessary
computational burden on WS-compatible services.

--
5. Use a specific data-transfer protocol, incompatible with any others.

Possible implementation:
   Use any randomized obfuscation technique to remove correlation
between the plaintext data transferred and the
   data seen in the packet on the line. Methods such as XOR, crypto
ciphers, permutation tables can be used with
   a key randomized by an entity beyond control of the user api, such
as WS client implementation.

Pros:
   Limits undesired activity to WS-speaking hosts, in similar way to
solution number (4). In comparison,
   does not depend on handshake test, instead making sure that even if
test is passed, the transferred data
   cannot in principle be parsed meaningfully by the non-WS speaking services.

Cons:
   In the same way as number (4), prevents interoperation with non-WS
services thus limiting innovation in practice.
   Depending on obfuscation complexity, puts considerable burden on
WS-speaking hosts.
   Removes ability to debug protocol completely without special
support for protocol-debugging tools, thus
   making it more probable that debugging will not be done correctly
on WS services and clients and ensuring
   implementations with large amounts of bugs.

Please feel free to add to the list of these solutions, as well as
clarifications for the problem definitions above.

----------------

New Protocol proposal
---------------------

>From the standpoint of the above, it seems that certain-level SSL
Certificate-backed "trusted-sources-only" full
socket api can be provided without significant security risks
involved. The responsibility can be clearly mapped
to the Certificate owner, whose real-world details should be on-file
with the issuing CA.
This would allow all kinds of innovative services interoperating with
the multiple modern non-HTTP protocols.
The protocol would implement solutions (2) and (3) with optional (2a)
for user communication with CA's.

Web Sockets analysis based on above
-----------------------------------

(This part is per its current definition = version 5 of draft authored
by Ian Fette.)

The WS protocol implements solutions (4) and (5), although the goals
of the solutions are basically the same.
The reasoning behind (5) is, thus, not to remove interoperability with
non-WS protocols, but to prevent
confusion with legacy HTTP speaking hosts. This can be done by using
CONNECT header, as was shown by Adam Barth.
Thus, solution (5) has no real goal within WS, adding considerable
complexity to the protocol and removing debug
capability.
Thus, it is recommended to remove solution (5) from WS, and use
CONNECT header instead.

It is also hard to imagine a non-WS protocol that would be able to
answer a handshake with a WS-specific magic
string. The argument that a WS service "should prove header parsing"
has no real purpose. Parsing headers to
pass the initial handshake does NOT ensure the parsed information will
be used in further processing. Indeed,
a separate WS-compliant library can be used by most services to pass
the handshake and do whatever internal
processing that was designed by the service author.
Thus, it is recommended to remove the complex solution for (4) and use
a simple magic-string based handshake.

Thoughts on HTTP compatibility/incompatibility goal
---------------------------------------

Regarding HTTP implementation overall, while the goals included
incompatibility with HTTP, for some reason it
was decided to use HTTP-compatible initial handshake. The goal, it
seems, is to allow the HTTP port (80/443)
to be multiplexed between the actual HTTP server and a WS server.
Although the idea is fascinating, if WS is
really a different protocol, why isn't it allocated a new TCP port,
especially if the actual protocol is meant
to be incompatible? The theoretical goal might have been to use
persistent script interpreters to service both
HTTP and WS requests, but in practice, the WS servers are generally
answered by services running separately
from the generic HTTP server parts.

Thoughts on framing in application layer
----------------------------

The current API provides the client script an onmessage() event
callback to read received data.
Thus, let's consider the server sending 1MB of data to the web browser.
To reduce the latency, the server may send the data without waiting
for the whole 1MB buffer to
fill. In that case the client will be notified with several
onmessage() callbacks.

If the client has capability to work with the incomplete data, it will
be able to process the
data provided. In case the client does not have this capability there
is no point for the server
to send the data in parts.

When a client is expecting a large input from the server, it will do
its own input buffering and,
if implemented, will show the user incomplete data to reduce visible
latency. The information
regarding when the data is considered complete is something that
application developer should decide.

If the maximal transfer block is to be limited, then client's API
should be extended to allow
browser to do internal buffering, but the completion control should
not be taken from the application
developer.

==================================================

--
 silvio

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 23 Feb 2011 11:54:27 -0800

On Wed, Feb 23, 2011 at 11:35 AM, Silvio Ventres
<[email-masked]>wrote:

> into discussion about it.

Actually, that's exactly what happens when cases go to court. Some minimally
paid paralegal does a ton of research on the law, its intent as best as it
can be divined, its interpretation via various courts... letters to
congressmen asking them to explain all the history and rationale of a law
from a random lawyer tend not to get answered afaik :)

You are the unique person that should be
> the responsibility of the Author, and it cannot be simply pushed away.
That doesn't scale, sadly. I do agree that we want people to be able to
participate in the discussion, but if a new person shows up every week and
wants to re-open debate on things that took us months to move past, at some
point that's not a scalable solution.

> Please comment when time allows.
To a point. WebSockets is designed to be a web-safe version of TCP. There
are other groups (RTC) looking at more UDP-centric version.

> general will be from now on called "attacks".
These are a subset of the possible attacks, but sure.

> The solution should thus implement one or more of the following
As you note, the cons here are a bit heavy. I also don't want to give raw
socket access, as that opens up all sorts of attack vectors. WebSockets is
designed to be safer than raw sockets, and also easier for a web author to
use (message semantics, as opposed to just "oooh, I can now recv() 20
bytes!")

Again, I don't think we want to expose /raw/ sockets. You can do all sorts
of interesting attacks that way. I may want to let arbitrary origins send me
requests, but not have to worry about intermediaries / my own servers
confusing them for other requests.

and we tend not to favor manual / reactive approaches :)

prompting the user is not going to be accepted by any user agent I can
imagine. "Do you want a pony? Sure, why not!"

what's in -05 is not complex

>   implementations with large amounts of bugs.
it's relatively easy to debug, and/or write a wireshark plugin to help you

> clarifications for the problem definitions above.
we considered SSL-only for a while but the group wanted to be able to use
unencrypted connections. (Long story, but e.g. if schools can't examine the
contents of a stream for porn/etc, you don't get to deploy in the schools.
as an example.)

Also, #3 would never fly with UAs. #2 goes against design goals of allowing
connections from untrusted sources.

> to be incompatible?

Because in practice, if you aren't using port 80 or 443, you cannot traverse
firewalls/proxies and you lose up to a third of all your connections.

> The theoretical goal might have been to use
No, a server may send the message as multiple fragments which the
browser/client will reassemble into a single message, and issue an
onmessage() callback when the full message has been received.

---

From: Silvio Ventres <[email-masked]>
Date: Wed, 23 Feb 2011 22:44:33 +0200

> point that's not a scalable solution.

The request is mostly: "Clarify why this is so" and not really a debate.
The debate start after the clarification ;)

> are other groups (RTC) looking at more UDP-centric version.

WebSockets as defined now is not a "web-safe version of TCP", whatever
that means, it's a custom protocol compatible only with itself, not
unlike FTP or SSH.

[.. some attacks vectors skipped ..]

> These are a subset of the possible attacks, but sure.

Well, if you have examples of other attack vectors, please list them.
Otherwise it seems that the defenses were built "for fun" without a
clear understanding what they're protecting from.

[.. solution disallowing access to external hosts skipped ..]

> use (message semantics, as opposed to just "oooh, I can now recv() 20

Yes, the cons are quite heavy, that's why this solution is not the best one.
On the other hand the "safer than raw sockets" is not exactly true: WS
does provide
generic access to send and receive data so there is no inherent difference here.

[.. solution pushing responsibility to webdev, disallowing external
scripts, skipped ..]

> confusing them for other requests.

What do you mean by "confusing" ? If some script _wants_ to talk to an
IRC server,
that's not a "confusion", that's actually an intent to talk to an IRC server.
If you don't want to worry about it, run scripts only from _trusted_
sources, identified
by Certificates given by a trusted Certificate Authority. Push
responsibility for the
actions to the CA.

[.. CA complaint procedure skipped ..]

> and we tend not to favor manual / reactive approaches :)

Well, this is something that is an optional instrument to help the CA
vetting procedures.
It is not required in any way.

[.. asking user for approval ..]

> imagine. "Do you want a pony? Sure, why not!"

Yes, explicit user acknowledgement is improbable to be implemented safely.

[.. skipped solutions (4) and (5) dealing with custom handshake and
custom protocol-obfuscation ..]

> what's in -05 is not complex

The handshake is quite complex. Instead of a simple MAGIC-STRING, it
uses a computationally intensive algorithm.
Instead of a plaintext data transfer, it uses XOR-encoded data transfer.

First, the solutions have same goals, there's no reason to use both,
one will do.
Second, the handshake solution is fully satisfied with MAGIC-STRING.
The "pseudo-goal" to "prove header-parsing" is unreal: an author that
doesn't want to parse the header, but wants to
seem compatible with WS, will just use an external library to pass handshakes.

> it's relatively easy to debug, and/or write a wireshark plugin to help you

Really?
So you never heard of people using curl, tcpdump or plain telnet/nc to debug ?
The fact is, HTTP is so prevalent today _because_ it is so easy to debug.
This is a _major_ feature of HTTP and should not be underestimated.
Moreover, by removing this feature you make it _harder_ to write
bug-free WS-compatible servers, clients and applications .
And for what ? To use GET instead of CONNECT ?

> contents of a stream for porn/etc, you don't get to deploy in the schools.

The point was not that the socket calls use SSL, but that the
socket-API-using script is downloaded and authenticated via SSL.
And while at that, are you sure that the schools like the idea of
upgrading their filters to support the new _obfuscated_ stream?
The solution can be found if the problem is specified exactly.

[.. question regarding http compatibility/incompatibility ..]

> firewalls/proxies and you lose up to a third of all your connections.

So the protocol might include an optional parameter to connect call:
connect(,,,options="HTTP-compliant") if you want to us it.
It doesn't have to be a mandatory part of the protocol.

> onmessage() callback when the full message has been received.

So you're simply pushing the buffering from server to the client, with
the client never being shown the incomplete data ?
What's the benefit here then ? To reduce server memory usage ?

Then why not leave the fragmentation/assembly to the operating system?
Instead of the e.g. websocket_send() function creating separate
websocket packets, let it create only one packet.
Add websocket_send_initial(), websocket_send_partial() and
websocket_send_final() functions that the server can use.
Then divide a copied websocket_send() function into the send_initial()
and send_final() and make send_partial() point to a simple TCP send().

There's no need for a separate framing procedure. Packet start/end is enough.

--
 silvio

---

From: Andy Green <[email-masked]>
Date: Wed, 23 Feb 2011 20:48:20 +0000

On 02/23/2011 07:54 PM, Somebody in the thread at some point said:

Hi -

>     "read the archives". The "archives" answer is a dangerous precedent

The status here at the moment is there are few pending arguments with 
05, it seems that it's quite close to "not displeasing anyone enough". 
So huge mails reassessing everything from scratch require a heart of oak 
to smile upon.

> onmessage() callback when the full message has been received.

To expand a bit because I didn't get how this worked at first either, 
this is the Continuation opcode / FIN bit scheme.  It's the mechanism 
for chopping a logical message (1MB in your example) into as many 
arbitrarily-sized sequential frames as you like but still be one logical 
message that doesn't complete - or notify Javascript land - until the 
last frame comes with FIN set.

The first, well, you can call it "subframe" must go with BINARY or TEXT 
and if it didn't have FIN set to end the message, subsequent subframes 
must come with CONTINUATION / FIN clear, and the last guy of the message 
is CONTINUATION with FIN set.

The scheme will enable low latency multuplexing and PING / PONG etc 
transparently independent of the size of the logical message flying around.

-Andy

---

From: John Tamplin <[email-masked]>
Date: Wed, 23 Feb 2011 15:53:28 -0500

On Wed, Feb 23, 2011 at 3:48 PM, Andy Green <[email-masked]> wrote:

> To expand a bit because I didn't get how this worked at first either, this
> is the Continuation opcode / FIN bit scheme.  It's the mechanism for
> chopping a logical message (1MB in your example) into as many
> message that doesn't complete - or notify Javascript land - until the last
> frame comes with FIN set.
>
> The first, well, you can call it "subframe" must go with BINARY or TEXT and
> if it didn't have FIN set to end the message, subsequent subframes must come
> with CONTINUATION / FIN clear, and the last guy of the message is
> CONTINUATION with FIN set.
>

More importantly, fragmentation allows a server returning content of unknown
size (such as dynamically generated content or the result of compression)
without having to buffer the entire message.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Andy Green <[email-masked]>
Date: Wed, 23 Feb 2011 20:59:22 +0000

On 02/23/2011 08:53 PM, Somebody in the thread at some point said:

> More importantly, fragmentation allows a server returning content of
> unknown size (such as dynamically generated content or the result of
> compression) without having to buffer the entire message.

How is that going to work in the Browser / Javascript event context 
then?  AIUI you won't get a data ready callback until a frame comes with 
FIN, implying it must buffer the whole message in the browser, is that 
wrong to your understanding?

-Andy

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 23 Feb 2011 13:01:53 -0800

On Wed, Feb 23, 2011 at 12:59 PM, Andy Green <[email-masked]> wrote:

> How is that going to work in the Browser / Javascript event context then?
>  AIUI you won't get a data ready callback until a frame comes with FIN,
> implying it must buffer the whole message in the browser, is that wrong to
> your understanding?
>
> -Andy
>

A browser may buffer in memory, or may spool to disk. In either case, a user
may actually have more memory or options than a server/intermediary that's
handling hundreds of thousands of concurrent connections and just wants to
pass through data as it becomes available. 100MB may be reasonable for a
client to hold in memory but impossible for a heavily loaded front-end
server or proxy, as an example.

-Ian

---

From: John Tamplin <[email-masked]>
Date: Wed, 23 Feb 2011 16:03:02 -0500

> implying it must buffer the whole message in the browser, is that wrong to

Right, I said in the server.  Ie, I can read dynamically generated content
into a fixed-size buffer, and write a fragment as it fills up.  When the
content is complete, I send a final frame.

Given the current API in the browser, yes, the browser will have to buffer
the entire message.  Perhaps in the future there will be a streaming API
available to JS, or perhaps not.

---

From: Andy Green <[email-masked]>
Date: Wed, 23 Feb 2011 21:10:21 +0000

On 02/23/2011 09:03 PM, Somebody in the thread at some point said:
> On Wed, Feb 23, 2011 at 3:59 PM, Andy Green <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     then?  AIUI you won't get a data ready callback until a frame comes
>     with FIN, implying it must buffer the whole message in the browser,
>     is that wrong to your understanding?

 > user may actually have more memory or options than a
 > ...

> Right, I said in the server.  Ie, I can read dynamically generated
> content into a fixed-size buffer, and write a fragment as it fills up.
>   When the content is complete, I send a final frame.

Sorry I missed 'server' when I read it.  Yes that's a nice advantage too.

> Given the current API in the browser, yes, the browser will have to
> buffer the entire message.  Perhaps in the future there will be a
> streaming API available to JS, or perhaps not.

OK we are all straight then, thanks.

-Andy

---

From: Silvio Ventres <[email-masked]>
Date: Wed, 23 Feb 2011 23:11:22 +0200

> CONTINUATION with FIN set.

Thank you for the explanation Andy.
It makes sense indeed.

--
 silvio

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 01:13:21 +0100

> point that's not a scalable solution.

Then just *document* the reasons/arguments/motivations in the draft so
newcomers would not need to re-open old discussions. But currently the
draft gives no arguments at all about several points of the
specification (neither the handshake, framing or client-to-server
masking are argumented in the draft). Such arguments should not exist
in a FAQ, but in the draft itself which should be the only required
piece of information for understanding a specification. Please don't
refer to a FAQ somewhere to complement the information.

A newcomer should be able to come tomorrow to this maillist/group,
read current draft, and understand motivations for current design of
the protocol. He could then suggest changes or improvements without
asking "why this and this and this?".

But if the draft remains as it is now, this is, without describing
motivations/reasons for its design, then the draft is not good and
IMHO shouldn't become an RFC.

Please don't take me wrong, but I'm used to read lots of drafts and
until now I've never found a draft in which a protocol is described in
the fashion "do this and don't ask why".

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 01:26:57 +0100

> How is that going to work in the Browser / Javascript event context then?

Is not the same concept as HTTP chunked transfer? do you mean that a
HTTP client receiving a 200 OK with "Transfer-Encoding: chunked" is
not able to know when the response body terminates?

For sure you are not saying that so I'm missing something :)

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Wed, 23 Feb 2011 19:28:09 -0500

> refer to a FAQ somewhere to complement the information.

As has been discussed in the archives, there has been strong objection to
including rationale in the spec itself.  I think it could be useful to have
a separate document collecting the rationale, though it would take a fair
amount of effort.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 01:41:40 +0100

2011/2/24 John Tamplin <[email-masked]>:
> including rationale in the spec itself.

It's really hard to find such threads in the archive (not enough time
for that), so I just can give my opinion:
A technical specification must include technical rationale. I cannot
imagine all the drafts I've read if rationale was removed from them.
Just my opinion.

> I think it could be useful to have

I would prefer rationale within an unique draft, but a separate one
could be an option. In SIP protocol this is common (a draft giving
just requeriments for a specification). Such draft could be named
"Requeriments for two-way TCP communicatino between a web server and a
remote host", or just "Requeriments for WebSocket protocol" :)

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 02:48:30 +0200

> a separate document collecting the rationale, though it would take a fair

Your reply is a great example of the wrong attitude.

Let's provide a rough translation:

"As has been discussed in the archives" = "Don't really remember why,
and didn't understand the reason well at the time"
"There has been strong objection" = "Someone objected, but not strong
enough to remember why"

The proposition was: "Let's add rationale to the spec".
The answer can argument for or against, providing the last known or
accepted reasoning.
In comparison, an answer saying: "go read archives" doesn't differ
much from "I don't have anything to say but want to write something".

No offense, but, if anyone plans to write any more "go read archives"
messages, please, try assuming the person asking actually _read_ the
archives, and wants an _actual_ answer.

--
 silvio

---

From: John Tamplin <[email-masked]>
Date: Wed, 23 Feb 2011 20:04:50 -0500

> "There has been strong objection" = "Someone objected, but not strong

No, it is "it is in the archives, if you can't be bothered to find it I
can't be bothered to spend my time looking for it for you".

> much from "I don't have anything to say but want to write something".

So, how is "there has been strong objection to including rationale in the
spec itself" not providing the "last known or accepted reasoning"?  I wasn't
one of the ones objecting and therefore do not remember all the details of
the objections, and we discussed it and moved on, accepting the reasons of
those objecting.

> archives, and wants an _actual_ answer.

Since some of the questions that have been asked after having been asked to
read the archives were discussed for *months* in *hundreds* of messages (for
example, the need for masking), I do not believe they actually read the
archives if they still didn't find anything relevant.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 02:30:05 +0100

> archives if they still didn't find anything relevant.

So, do you still think that in order to properly understand the
WebSocket protocol (as defined in last -05 draft) I must read hundreds
of old mails???
If so, why are you writting a draft? why don't you just tell people
"ey, the specification is writen somehow in the maillist archives,
read them".

If a draft is incomplete and doesn't provide all the required
rationale then it's invalid. A specification can reference other
specifications (informative or normative references), but it cannot
referece a maillist archive!

There are two options IMHO:

1) Include *all* the technical rationale within the draft, so someday
it would become a RFC.

2) Create a separate draft "Requeriments for WebSocket protocol" and
reference it from the current draft. Both drafts should become a RFC
(usually at the same time).

Both options would take some time and efforts, of course, the price of
writing a good standard (and a requeriment, is not?).

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 23 Feb 2011 17:35:31 -0800

> "ey, the specification is writen somehow in the maillist archives,
The point being made was that the protocol should be the minimum document
required to understand how to build a client/server for the protocol. If you
want the gory history, read the archives. The rationale and history are not
required to build a complaint piece of software.

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 02:54:40 +0100

> required to build a complaint piece of software.

Really? so what is this draft? a "HOWTO" for developing software?. A
specification describing a protocol must include extensive information
about the reasoning for its design, even more when it aims to become a
standard. You are describing a *protocol*, not a guide for a lazy
implementor. Could you please check *any* other RFC describing any
other protocol?

I understand all of you have spent long time discussing various
aspects of the protocol in the maillist. Conclusions for those
discussions (possible attacks, masking/handshake reasons...) should be
included in the specification, and not just the techniques to
accomplishe with them. If not, this is a HOWTO rather than a protocol
specification.

Best regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 23 Feb 2011 18:02:28 -0800

> implementor. Could you please check *any* other RFC describing any
If RFC2616 had to list out the full history of every debate of every feature
in HTTP it would be unreadable and worthless. The protocol is a guide to
what you must do if you want to create software that can interoperate with
other WS endpoints, in that sense yes, it is something of a howto. You do
not need a transcription of months of arguments that led to what it is. I am
sure that back in the 90s people were debating about whether Accept-Lang:
en, de; q=0.8 was more understandable or less understandable than en;
de,q=0.8, but at the end of the day one won out, and the history doesn't
really matter as if I want it to work, I need to send en,de;q=0.8.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 03:23:35 +0100

> really matter as if I want it to work, I need to send en,de;q=0.8.

You are comparing a format/grammar rule with a security design.

I don't expect (neither I desire) that this draft explains why
Sec-WebSocket-1 is called as it is rather than "Security-WS-A", even
if you all spent 5 weeks discussing about it.
But if the draft defines a exotic handshake it must argue it. If it
requires masking just for client-to-server communication it must argue
it ("avoid confusing network intermediaries, such as intercepting
proxies" is not enough).

Also -05 seems to be HTTP compliant but I still don't know how should
a websocket client react if it receives a 302 when performing the
handshake. Same as a "real" HTTP client? or not? Initially the
protocol was just "similar" to HTTP (but not HTTP) so it's hard to
figure how is currently defined to behave without explicit mention in
the draft.

If the protocol is HTTP compliant, why is the draft full of phrases
like the following?:

   Headers in the handshake are sent by the client in a random order;
   the order is not meaningful.

This is obvious in HTTP! order of different headers never matter, so
why to include it in this draft? Is it not clear that a WebSocket
implementor must know HTTP protocol? Wouldn't be better to remove such
unnecessary information from the draft and, instead, include useful
information (as technical rationale)? maybe you still save space :)

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Brian <[email-masked]>
Date: Wed, 23 Feb 2011 18:32:00 -0800

I can empathize with the rationale for both positions here, but I'm
going to agree with Ian -- the specification needs to be lean and
readily understandable.  If I'm on a mission to build a new
implementation of a client or server, I want to be able to do it as
swiftly as possible without getting mired down in superfluous details
that aren't critical to the implementation.

On the other hand, I also feel that an accompanying document would be
a helpful nicety, because indeed, there are many, many things in the
draft that, while there are very specific and nuanced reasons for
their existence, would make any new implementer scratch his/her head
and grumble "WTF?!"

That kind of experience would cause most implementers to feel angry
and resentful of the protocol and its creators.  It took me months of
reading every single mail sent to this list to stop feeling literally
angry at certain members of the working group and finally understand
where they're coming from and why things were decided one way or
another.  There are still things that I'm not thrilled with, but like
others, happy to accept to move forward with a workable and stable
specification.

The big problem with creating such a document is that it's not
strictly required for the standardization process (though it might
expedite things in the later stages of ratification) and everyone on
this mailing list is extremely busy without much time to dedicate to
such an involved side-project.  I know I don't have that kind of time.
 I think the best bet, as suggested by someone previously, is to do it
as an FAQ/Wiki that everyone can contribute to so that it can get done
without putting a large burden on any one individual.  That way the
information is available without having to scour the mailing list
archives, but it wont clutter up the draft.

For what it's worth, I feel your pain with regard to searching the
archives.  It really is a completely insurmountable task at this
point, since the volume on this list has been, at times, a veritable
deluge of information.  Indeed there have been a few days where it
took me multiple hours to get through all the posts sent on that one
day.  Searching through hundreds of messages per week in the mailing
list archives is *not* an option for newcomers on most of the topics
that they would be curious about.

Brian

---

From: "Pat McManus @Mozilla" <[email-masked]>
Date: Wed, 23 Feb 2011 21:59:23 -0500

_The WebSocket Protocol, a History_  does not require nor benefit from
rough technical consensus. Please don't weigh down the protocol document
with trying to get the WG members to write and edit this text and then
achieve consensus around the extra verbiage. 

One or more separate blogs, wikis, books, faqs, masters thesis, movies,
stage plays, street theater performances, or separate internet-drafts
would be lovely and undoubtedly contribute to the eco-system. They need
not be 100% consistent with each other; the underlying arguments
certainly weren't. I would enjoy seeing what anyone motivated enough to
do the work creates on the topic.

-- 
http://www.getfirefox.com/

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 24 Feb 2011 12:09:04 +0900

Hello Iñaki,

Why don't you (and maybe Silvio) volunteer to write a draft describing 
background/rationale for WebSockets? I'm sure it would be appreciated by 
a lot of people, and you would get lots of support from the WG.

On 2011/02/24 10:54, Iñaki Baz Castillo wrote:
> 2011/2/24 Ian Fette (イアンフェッティ)<[email-masked]>:
> Really? so what is this draft? a "HOWTO" for developing software?.

In essence yes. For software to work, it has to work to spec; why the 
spec is the way it is is mostly irrelevant. Indeed having too much 
rationale in the spec may lead implementers to implement to the 
rationale rather than the actual spec, which leads to interoperability 
problems. (Implementers read the rationale, think "a, I see how it's 
supposed to work", and miss some details.)

> A

There is no requirement anywhere in the IETF process that a spec (or 
specifically a standard) has to include rationale. The same applies to 
W3C specs, and I guess to the specs of many other standards 
organizations. Many specs are successful because they are short and to 
the point, rather than because they include a lot of rationale.

A case in point would be XML. The XML spec contains a very short list of 
goals at the start (http://www.w3.org/TR/xml/#sec-origin-goals), which 
is far away from a rationale. For the rest, it's just facts, no reasons. 
And XML was and continues to be extremely widely used.

During the development of XML (something like 1996-1997), there was a 
very strict "go read the mailing list first" policy. The spec would 
maybe never have been completed if not for that policy; otherwise, the 
WG would have been busy all the time giving intros to newcommers rather 
than finishing their work.

> accomplishe with them. If not, this is a HOWTO rather than a protocol

You are right for security-related stuff, which has to be discussed in 
the security section (although this is focused on possible attacks that 
are still open, rather than on those that the protocol design closes).
For the rest, a separate document is a much better alternative, and I 
hope you can help us produce one.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 05:48:55 +0200

> the objections, and we discussed it and moved on, accepting the reasons of

If you actually _accepted_ the reasons, then you surely remember what
those reasons _were_.
And if you can't remember, it means you didn't really think hard about
them while accepting,
or the reasons didn't really persuade you, or you merely saw someone
else accepting them.

Then, if you aren't the one that accepted or rejected them, don't
remember the reason and
can't be bothered to find it, why did you answer the message
_explicitly_ requesting to quote
the reasoning for the decision?

In any case, if anyone actually knows the main rationale for this
decision, please let know.

--
 silvio

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 24 Feb 2011 14:51:51 +1100

> a separate document collecting the rationale, though it would take a fair

I think that historically the group has had problems regarding such
text - either in the protocol draft, or even the abandoned
requirements draft, because the reasoning text was used to lobby
and/or advocate for one side or another of some very divisive issues.

However, now that we are having some convergence, you'd really hope
that we would all be able to agree on a sentence or two describing why
we are masking (even if you don't agree with the need to mask, you
surely can understand the reasons others do).

So I think that there should be a happy medium between the
specification providing a complete history and reasoning for each
feature, vs providing just a bit of reasoning to motivate a feature.
For example we could just say something like: The data from the client
to the server is masked by the user-agent to protect against HTTP
aware intermediaries mistaking WS content as HTTP requests.

I really don't think it is good that first time readers of the
protocol document have WTF moments that can only be answered by
reading a vast mailing archive.

cheers

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 05:57:19 +0200

> really matter as if I want it to work, I need to send en,de;q=0.8.

With all due respect, Ian, the current questions are not about amount
of spacing in the headers.
They are about the inherent features that have been included in the
standard for some implied and hidden reasons.
Specifically, security features are only added as a reply to some
possible and explicitly defined attack vectors
You don't just add a defense mechanism, burdening the compliant
implementors, without providing rationale as to why it should exist.
Especially, not when you're only _developing_ a standard.

And the point here is that the debate haven't even begun.
All that was asked is to explain the rationale and provide clear and
concise attack vectors or use scenarios which are promoted or which
are prevented by the measures in the protocol.

Who other than you, Ian, is able to better answer the questions
regarding reasoning behind each feature ?
In the end, it was you that heard all the arguments, accepted some and
rejected others, and thus, you were the
party accepting some argued reasoning, and thus should know and
remember that reasoning fairly well.
Why is it so problematic to provide the reasoning ? If you were
actually persuaded, and understand the reasoning
behind all the features, it should not be that hard to recall them and
write a few sentences of clarification.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 06:19:01 +0200

> aware intermediaries mistaking WS content as HTTP requests.

So, to continue with productive discussion (sorry Greg for hijacking
your reply):
Any HTML "src" attribute and "link" tag can be made to query _any_
plaintext of the target server,
and, obviously, XHR methods can also.
This attack vector doesn't seem to be a problem for either.
So why WS takes precautions against this, while losing easy-debug and
low-complexity implementation
ability ? Even Adam Barth's own paper advocated using CONNECT to
overcome this, arguably, non-existent, problem.
Even if anyone would take this attack seriously ignoring the same
_unused_ attack vector within HTML and XHR,
why would one choose to kill debug ability and add considerable
complexity instead of changing header request type ?
The "let's make traffic look random" idea reeks of overkill
pseudo-security features added just "cause they seem cool",
and that don't serve any actual purpose in practice.

This is one of those WTF moments...

--
 silvio

---

From: John Tamplin <[email-masked]>
Date: Wed, 23 Feb 2011 23:34:13 -0500

> This is one of those WTF moments...

Again, this was discussed ad nauseum for months and months, leading to
Mozilla and Opera disabling WebSockets.

XHR can't be used to attack a server that the script wasn't sourced from,
and can't fake HTTP headers.  Adam & Eric demonstrated that some transparent
intermediaries do not understand Connection: Upgrade and therefore can be
tricked into thinking subsequent traffic is the next HTTP message when it
isn't.  That can be used to poison a cache which will then serve  contents
of the attacker's choosing to all clients behind that cache.

For an idea of the implications, consider if the attacker is able to store
JS of their choice in a cache that will be served for
http://www.google.com/ga.js

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 24 Feb 2011 15:35:31 +1100

On 24 February 2011 15:19, Silvio Ventres <[email-masked]> wrote:

> So why WS takes precautions against this,

Because existing HTTP mechanisms do not provide a mechanism to spoof
headers (host header, Sec- headers etc.)
If WS content was ever misinterpreted as a HTTP request, then it would
be more "powerful" than any HTTP request that can be generated by XHR
or src tags.

> while losing easy-debug and
> ability ?

Well there was significant effort put towards arguing for a masking
mechanism that has the entire key within the frame, so that each frame
can be decoded without needing the entire stream.  Do
easy-debugability was a consideration in the spec.  OK it's not as
easy as plain text, but there were compromises to be made.

> overcome this, arguably, non-existent, problem.

Well you have plenty of supporters who agree that this is arguably a
non-existent problem.   But we were substantiate that it is a provably
non-existent problem -  and for better or worse a proof or a
"solution" was need to achieve rough consensus.

> and that don't serve any actual purpose in practice.

I'm not going to say that masking is a perfect or even necessary
technical solution, but it was the solution that came out of the IETF
process for achieving rough consensus.

I still have issues with it (it should only mask the payload so that
framing is in the clear;  it should be an extension so it can be
replaced or disabled if it proves necessary) and I'm prepared to
significantly increase the volume of the archives in support for those
points.      However I have conceded to the outcome of the process
that we will have masking.

However, I repeat that I think we do need to provide at least some
minimal explanation for it.

> This is one of those WTF moments...

welcome to hybi!

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 24 Feb 2011 15:36:58 +1100

> "solution" was need to achieve rough consensus.

let me try that in english:

You have plenty of supporters who agree that this is arguably a
non-existent problem.   But we were unable to substantiate that it is
a provably non-existent problem, so for better or worse a proof or a
"solution" was need to achieve rough consensus.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 24 Feb 2011 13:59:25 +0900

Hello Silvio,

What you are essentially asking for is that some of the people who 
already spent tons of time in this WG to read lots of mail and 
contribute to spend more time explaining things again and again to 
newcomers, just because these newcomers don't want to invest the same 
time (actually, probably quite a bit less, because you can read the 
mailing list backwards and only pick up the main threads,...) as 
everybody who has participated for a while.

I don't think this is very fair at all.

> Especially, not when you're only _developing_ a standard.

Of course you don't. Why do you assume we did? Please go read the archives.

> And the point here is that the debate haven't even begun.

I hope that unless you provide some very new insight, there will be no 
new debate about closed issues. We can't repeat debates every time a new 
member joins the WG.

> are prevented by the measures in the protocol.

Yes, and all you were told was that you should go and read the archives. 
But rather than doing so, you decided to shift the discussion to 
meta-issues such as how much rationale a spec should provide or why an 
editor (who is doing this work as a volunteer) is supposed to provide a 
free question answering service. Please, please stop bothering this WG 
with such discussions, and go do your homework to catch up with the rest 
of the WG. Thanks a lot!

> write a few sentences of clarification.

Sorry, but Ian is the editor. An editor understands a lot of what's 
going into the spec, but on some issues, he may just use the text that 
has been discussed by the group and approved by the chairs. To some 
extent, it would actually be a bad think if he were too involved with 
the pros and cons of each proposal. The WG has had experience with an 
editor who cared so much about some issues that he was unable to follow 
the instructions from the chairs and decided to hand over his role to 
somebody else.

Regards,   Martin.

---

From: Andy Green <[email-masked]>
Date: Thu, 24 Feb 2011 06:36:30 +0000

On 02/24/2011 12:26 AM, Somebody in the thread at some point said:

Hi -

> For sure you are not saying that so I'm missing something :)

It's similar idea, but with chunked transfer the client is getting each 
piece as it comes, so if it was HTML coming like that the browser can 
redraw the layout if it likes with each new chunk coming until the whole 
thing is done.  Or if it was a PNG coming, the client can redraw with 
what it has got as each new chunk comes.

However with websocket messages, it's a different "promise" to the 
Javascript API: the client buffers up each chunk until the entire 
message "finishes" and presents the whole buffer to Javascript once at 
the end.  So the deal is websocket handler in Javascript only sees 
complete messages and it's invisible how many frames were involved on 
the wire or how big they were.

A message "finishes" by having a frame with FIN bit set one way or another.

You can emulate chunked just fine with websockets, but you'd do that by 
streaming and endless bunch of smaller complete messages, each with 
their own FIN frame that spills the message to Javascript, rather than 
delaying the frame with FIN set in a single message.

-Andy

---

From: Andy Green <[email-masked]>
Date: Thu, 24 Feb 2011 06:58:23 +0000

On 02/24/2011 04:34 AM, Somebody in the thread at some point said:
> <[email-masked] <mailto:[email-masked]>> wrote:
>

Well, just a precision from what I could work out watching it, it was 
the claims in Barth's paper, the subsequent media attention that led to 
Mozilla disabling it by default and writing about the nobility of taking 
security seriously.  For Opera, they have written on the list in a 
pretty annoyed tone that they won't track websocket standard any more 
until it finalizes (or dies) so I think that drove that side as much as 
the allegations.

It wasn't the case that Mozilla and Opera followed the "discussion" for 
"months and months" and then after consideration disabled it.  To be 
fair, if the allegations were true, is was better that they didn't wait. 
  But there has been a really strange lack of repeatability for the 
claims, a lack of open-ness for enabling repeatability by others via 
publishing the scripts AFAIK and a lack of identification of a single 
vulnerable intermediary.  Maybe that's because the attack was so 
effective, they just notify vendors in secret, but -->

> XHR can't be used to attack a server that the script wasn't sourced
> from, and can't fake HTTP headers.  Adam & Eric demonstrated that some
> transparent intermediaries do not understand Connection: Upgrade and
> therefore can be tricked into thinking subsequent traffic is the next
> HTTP message when it isn't.  That can be used to poison a cache which
> will then serve  contents of the attacker's choosing to all clients
> behind that cache.
>
> For an idea of the implications, consider if the attacker is able to
> store JS of their choice in a cache that will be served for

... so curious if one believes that, that Chrome notably didn't disable 
websockets support.

Anyway what's done is done.  For sure this is a good example of why 
excessive rationales would cause more harm than good because munging 
frames would need to be rationalized as "Added to give Mozilla an excuse 
to enable us again".

-Andy

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 11:16:53 +0100

> their existence, would make any new implementer scratch his/her head

That is exactly what I felt when reading this draft. So imagine how
many people will feel the same when it becomes a RFC.

> archives, but it wont clutter up the draft.

The FAQ/Wiki is better than nothing (the mail archives in nothing, as
there are 99% of discussions and just 1% of useful conclusions).
Anyhow IMHO a draft with WebSocket Requeriments whould me more
appropriate.

If you want to see a good example:

- RFC 3323: A Privacy Mechanism for the Session Initiation Protocol (SIP).
- RFC 3324: Short Term Requirements for Network Asserted Identity
- RFC 3325: Private Extensions to the Session Initiation Protocol (SIP) for
                   Asserted Identity within Trusted Networks

An implementor would directly read RFC 3325 and "follow instructions".
A reader interested in motivations and technical rationale of the spec
would also read RFC 3323 and 3324 (instead of being redirected to ugly
mail archives full of discussions and useless mails).

> that they would be curious about.

Unfortunatelly it seems that other members here expect that a newcomer
should navigate through hundreds of mails in an old archive ("it's was
hard for me so it must be so hard for you").

Thanks fo your response.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 11:46:08 +0100

2011/2/24 "Martin J. Dürst" <[email-masked]>:
> background/rationale for WebSockets? I'm sure it would be appreciated by a
> lot of people, and you would get lots of support from the WG.

As I'm a newcomer and didn't participate in old discussions in the
maillist, I don't think I'm the best profile to summarize all the
discussions and conclusions which took place in the archives.

I'm interested in WebSocket protocol, I am able to participate in
someway (I'm not a super-expert but I'm used to deal with network
protocols) but for that I require a real document (or whatever useful
thing) in which rationale is described (I mean conclusions not the
whole and long discussions as mail archives provide). I've not enough
time to read all the mail archives, but IMHO it shouldn't be required
at all if conclusions were documented (as they are in so many other
specifications).

> In essence yes. For software to work, it has to work to spec; why the spec
> is the way it is is mostly irrelevant. Indeed having too much rationale in
> the spec may lead implementers to implement to the rationale rather than the
> actual spec, which leads to interoperability problems. (Implementers read
> the rationale, think "a, I see how it's supposed to work", and miss some
> details.)

Ok, I couldl agree, but the rationale should be (IMHO) *documented*
somewhere (mail archives is not an option, it stores full discusions).

> specifically a standard) has to include rationale. The same applies to W3C
> specs, and I guess to the specs of many other standards organizations. Many
> specs are successful because they are short and to the point, rather than
> because they include a lot of rationale.
>
> goals at the start (http://www.w3.org/TR/xml/#sec-origin-goals), which is
> far away from a rationale. For the rest, it's just facts, no reasons. And
> XML was and continues to be extremely widely used.
>
> During the development of XML (something like 1996-1997), there was a very
> strict "go read the mailing list first" policy. The spec would maybe never
> have been completed if not for that policy; otherwise, the WG would have
> been busy all the time giving intros to newcommers rather than finishing
> their work.

Ok, but honestly, the decisions taken on WebSocket protocol are, at
least, debatable (for example the exotic handshake which IMHO provides
nothing better than a simple "magig string" mechanism). When I read
the draft my first time I asked my self "why such unuseful and exotic
handshake? it's just a waste of computation cycles providing no real
security at all". Same with masking, "it is just randomizing the
bytes, no more, it provides nothing useful".

Of course you have debated a lot about it (and it would be insane for
anyone if he must read all your long discussions to understand the
rationale). So instead of leave such conclusions somewhere in the mail
archives, please properly document them. Maybe I'm wrong, but IMHO it
should be a requeriment for a draft to become a standard.

> You are right for security-related stuff, which has to be discussed in the
> security section (although this is focused on possible attacks that are
> still open, rather than on those that the protocol design closes).

True. The Security sections should not explain motivations for the
design of the protocol but instead it should warn about expectable
attacks or security issues given the current design.

> For the rest, a separate document is a much better alternative, and I hope
> you can help us produce one.

So I just need to read hundred of old mails :)

Thanks.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 11:52:46 +0100

> reading a vast mailing archive.

How is possible that other members in this group don't agree with the
above comment? It's just obvious, it just makes all the sense.

Please, don't assume that future readers of this specification will
just be lazy programmers just interested in make it work ASAP in their
software.

Don't take me wrong, but the handshake, masking and some other stuff
in the draft seem ridiculous wihout a good rationale documenting them.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 12:09:25 +0100

2011/2/24 "Martin J. Dürst" <[email-masked]>:
> Hello Silvio,
>
> What you are essentially asking for is that some of the people who already
> spent tons of time in this WG to read lots of mail and contribute to spend
> more time explaining things again and again to newcomers, just because these
> newcomers don't want to invest the same time (actually, probably quite a bit
> less, because you can read the mailing list backwards and only pick up the
> main threads,...) as everybody who has participated for a while.

If rationale adopted in the maillist archives would be *documented*
somewhere (hopefully in a separate draft) there would be no need of
asking/replyin again and again.

> I hope that unless you provide some very new insight, there will be no new
> debate about closed issues. We can't repeat debates every time a new member
> joins the WG.

I don't want to re-open a debate. I just want to understand the
reasons motivating the final design decisions. And nobody is able to
extract such infromation from the mail archives (except people with
too much spare time or paeplo payed for that).

Redirecting people to mail archives is like ensuring that newcomers
will not participate in the group.

> Yes, and all you were told was that you should go and read the archives.

Which is not a valid option IMHO.

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 12:20:27 +0100

2011/2/24 Andy Green <[email-masked]>:
> piece as it comes, so if it was HTML coming like that the browser can redraw
> the layout if it likes with each new chunk coming until the whole thing is
> done.  Or if it was a PNG coming, the client can redraw with what it has got
> as each new chunk comes.
>
> Javascript API: the client buffers up each chunk until the entire message
> "finishes" and presents the whole buffer to Javascript once at the end.  So
> the deal is websocket handler in Javascript only sees complete messages and
> it's invisible how many frames were involved on the wire or how big they
> were.

This could be achieved within the specific protocol carried by
websocket stream. Maybe the JavaScript client could trigger
"on_partial_message(fragment)" if the negociated protocol is known to
carry "fragmentable" information (as images). But this depends on the
carried protocol on top of WS. Such protocol should use something like
"Content-Type" and so.

For example, the API could include a function
"on_partial_receive(fragment)" which would be invoked when partial
data arrives. It's up to the JavaScript code to decide if it does
something or not when such function is called. It could just do
nothing and wait for "on_message(message)" to be triggered (which full
message). In this way, the feature would exist in WS core spec, but
should be accomplished by protocols on top of WS (propietary or
standard protocols, it doesn't matter as it is the JavaScript code
which decides to react or not upon receipt of partial data).

Cheers.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 24 Feb 2011 20:32:29 +0900

I've been taking some note for tracking discussion on HyBi. I've moved it
from intranet to Google Code.
http://code.google.com/p/pywebsocket/wiki/WebSocketProtocolSpec
I can't promise that I'll keep this maintained. But it may be helpful in
some degree.

Takeshi

---

From: Andy Green <[email-masked]>
Date: Thu, 24 Feb 2011 11:49:48 +0000

On 02/24/2011 11:20 AM, Somebody in the thread at some point said:

Hi -

> which decides to react or not upon receipt of partial data).

Sorry, what's the benefit of all that compared to making the choice of 
how to fragment it in the server as allowed already?  It already "exists 
in the core spec" that you can chop your stream up into multiple 
messages if you want to see stuff coming in piece by piece at the client.

If you're not in control of how the server is going to chop it up you 
can't play those games anyway since it may then buffer the whole message 
on you.  If you are in control of how the server chops it up, you don't 
need to play the game suggested.

The existing way is simple on the wire and code, and doesn't impose 
anything on the protocol inside the payload, while allowing for low 
latency interleaving of other streams / control frames, and as John and 
Ian pointed out also allows intermediaries and the server in many cases 
anyway to just have enough buffer for a frame not a message.

It's also a very simple proposition for Javascript guys that all this on 
the wire is something they never have to know, they just see big or 
small "messages" coming, I think that's valuable.

-Andy

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 13:13:18 +0100

2011/2/24 Andy Green <[email-masked]>:
> Sorry, what's the benefit of all that compared to making the choice of how
> to fragment it in the server as allowed already?

If the server decides to split a message into varios fragments, would
the API function "on_message()" be called in the client for each
fragment? If not the client must wait for the entire content (a 3MB
PNG file) to be buffered before start rendering it, which is not cool
taking into account that HTTP/1.1 already allows clients rendering
partial content.

Maybe I'm wrong.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Andy Green <[email-masked]>
Date: Thu, 24 Feb 2011 12:51:05 +0000

On 02/24/2011 12:13 PM, Somebody in the thread at some point said:
> 2011/2/24 Andy Green<[email-masked]>:

No as you expect you won't get any on_message() for the pieces of a 
message as they come, just one for the message.

But, if you like to see the pieces come in, you can arrange that the 
server sends your 3MB PNG data in a bunch of messages, not just one 3MB 
one.  Maybe that's what you were saying about the protocol, you would 
then need to leverage your protocol header to understand how each 
message that turned up fitted into the 3MB whole.  The spec leaves the 
protocol payload totally alone AFAIK.

3MB PNG is not an unrealistic example but I guess many other 
applications for websocket are going to be focused on realtime low 
latency data.  At least if you are using short whole messages there you 
can leverage message boundaries to simplify your streaming protocol, ie, 
always know there's a protocol header at the start of the message rather 
than it being fragmented partway through.

-Andy

---

From: "Joonas Lehtolahti" <[email-masked]>
Date: Thu, 24 Feb 2011 14:56:48 +0200

On Thu, 24 Feb 2011 14:13:18 +0200, Iñaki Baz Castillo <[email-masked]>  
wrote:

>> how

I'll try to put in other words what others have already said eariler in  
this discussion. Note that in the following text I will be using the term  
application to refer to for example a script on a webpage that uses  
WebSockets message API and doesn't have direct contact with the actual  
WebSockets protocol.

To be able to handle partial data, the client application must know what  
kind of data is coming and how to handle it progressively. If it only  
knows what to do with whole messages, it needs to buffer all the parts  
completely, right? Okay, so let's assume the client is expecting a data  
that it allows to be fragmented and can handle each incoming piece  
separately without needing to buffer all the data first. How can the  
client know what data the server is sending? Well, the application has  
some kind of internal protocol it talks inside the WebSockets channel.

Therefore you can do the fragmentation to smaller chunks in this  
application-level protocol. On the client side the onmessage() will be  
called for each full WebSockets message. Each of those full WebSockets  
messages may be chunks of bigger application-logical messages, so the  
application can handle big data coming in smaller parts. But the actual  
WebSockets protocol is still invisible to the application logic, those  
chunks of data which are complete WebSocket messages may be fragmented on  
the WebSockets protocol level too. No onmessage() will be called for those  
WebSockets parts, but your application doesn't need to care about that.

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 15:53:06 +0200

On Thu, Feb 24, 2011 at 6:34 AM, John Tamplin <[email-masked]> wrote:
> On Wed, Feb 23, 2011 at 11:19 PM, Silvio Ventres <[email-masked]>
> wrote:
>>

Adam's paper clearly states that CONNECT method prevents any problems,
while acknowledging that the problem actually affects about 0.001% of userbase.
In any case, it does _not_ give any rationale to the masking procedure, and
does not try to repeat the experiments with and without masking, proving that
the masking is required in practice.
Masking has never been shown as a required solution, just a "nice to have".
It's an overkill that causes direct and dire harm
to the ability of developers to implement, debug and support WS protocol.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 16:00:35 +0200

> be more "powerful" than any HTTP request that can be generated by XHR

Well, if the "Host" header is the problem, why not solve _that_ problem ?
Change it to "Sec-Host-WebSocket:" so no legacy equipment confuses it
with a real "Host:".
Since only WS servers will actually answer WS requests, you don't lose
any functionality here.

That's a better "tradeoff" than masking, isn't it ? Leaves debug
ability and is easy on the cpu.

--
 silvio

---

From: Andy Green <[email-masked]>
Date: Thu, 24 Feb 2011 14:23:04 +0000

> Since only WS servers will actually answer WS requests, you don't lose

No the header won't answer the claimed problem, which is that 
intermediaries, like proxies and caches who do not really understand 
websockets protocol, may be dumb enough to keep parsing the websocket 
traffic as http even when the two endpoints, the client and server, do 
understand and negotiated an upgrade correctly.  Using another magic 
header the alleged intermediary is clueless about won't help that.

Then a bad guy could do what Barth claims his experiment did, convince 
browsers to send websocket content through these alleged intermediaries 
that is just right to perform the proposed bad deed.

Yes, the alleged intermediaries are still sat there as vulnerable as 
ever to the right packets sent another way.  But that "other way" is not 
so bad-guy-friendly as javascript in a browser that is true.

Yes, no intermediary suffering from that has ever been identified AFAIK.

Yes, if it was identified, the vendor would fix it or the thing would be 
replaced with something less broken.

Yes, until there's more evidence you would not be alone reasonably 
taking from that they don't exist and Barth's one-shot experiment 
interpreted misinterpreted the cause of the few results that seemed to 
show hits.

> ability and is easy on the cpu.

Yes if the above is true masking is worthless.

Unfortunately munging is in there for political reasons, not techincal 
reasons, and the political reasons are real enough.

-Andy

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 16:19:54 +0200

> main threads,...) as everybody who has participated for a while.

There is a difference between explaining something you understand and
something you don't.
If you understand a concept and the rationale, it takes 2-3 sentences
to explain it.
Roughly the same amount of sentences as saying "Go read archives".
If you don't understand or don't agree with rationale, it makes you want to
make the question go away, and this is where "go read the archives" comes in.

> Of course you don't. Why do you assume we did? Please go read the archives.

Because the original masking idea came from Adam Barth, and was never
proven by anyone
to be necessary. Indeed, if the "Host:" header is the problem, it can
be changed to
"Sec-Host-WebSocket", if the "GET" is the problem, it can be changed
to "CONNECT".
The experiments were never repeated with those changes, although they
are much easier
to implement than the masking procedure, and have a much smaller
impact on implementation
simplicity and debug-ability of the protocol.

> debate about closed issues. We can't repeat debates every time a new member

When you have a clear requirement for something, you can explain a solution.
For example, since WS is _required_ to coexist on port 80, it currently has the
HTTP-like headers. This is easy to explain and clarify, since it's a solution to
a clearly-defined problem.

The masking procedure is a solution to an ill-defined problem.
Moreover, the problem,
ill-defined as it is, can be solved by other, much less intrusive solutions.
Thus, it is hard to explain and clarify and much easier to say "go
read archives".

> Yes, and all you were told was that you should go and read the archives. But
> rather than doing so, you decided to shift the discussion to meta-issues
> such as how much rationale a spec should provide or why an editor (who is
> doing this work as a volunteer) is supposed to provide a free question
> answering service. Please, please stop bothering this WG with such
> discussions, and go do your homework to catch up with the rest of the WG.
> Thanks a lot!

You have just written 7 lines of text. Did you consider that instead
of fighting, you could
have used the same time to answer the questions asked ?
Since you probably did, the only answer here is that you either don't
understand the
solution rationale yourself, or didn't accept the solution and don't want to be
reminded of that.
In both cases, with all due respect, if you don't have any
constructive feedback other than
"go read the archives", and don't plan to answer any questions
regarding the issues with
the protocol, please feel free to postpone replies to the posts until you do.

> Sorry, but Ian is the editor. An editor understands a lot of what's going
> into the spec, but on some issues, he may just use the text that has been
> discussed by the group and approved by the chairs. To some extent, it would
> actually be a bad think if he were too involved with the pros and cons of
> each proposal. The WG has had experience with an editor who cared so much
> about some issues that he was unable to follow the instructions from the
> chairs and decided to hand over his role to somebody else.

As an editor, Ian MUST understand the rationale and be able to clarify
the solutions.
It doesn't take much time to clarify solutions when a clear
understanding of those
exist. It takes roughly the same amount of time and text as an
elaborate "go read the archives" answer.

--
 silvio

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 25 Feb 2011 01:29:39 +1100

> while acknowledging that the problem actually affects about 0.001% of userbase.

Yes it does, but I don't think he was able to convince the list that
it was actually true.

> Change it to "Sec-Host-WebSocket:" so no legacy equipment confuses it

The host header in WS handshake is not the problem.

The issue is that a hypothetical intermediary might exist that allows
WS to work, but is actually scanning the data stream looking for HTTP
requests.  It skips all the stuff it thinks is garbage and then sees a
spoofed HTTP request in the WS payload, with a spoofed Host header.
It knows nothing about WS, so it would not be affected by anything the
WS spec said about alternate host headers.  This hypothetical
intermediary then skips over all the "rubbish" in the return stream
(eg the handshake responses needed to allow the client to send a
payload in the first place) and eventually sees a HTTP response
provided by the evil/compromised server.   It then puts the content of
that response into a cache for the spoofed host header, and the cache
is then poisoned.

Masking will prevent evil payloads from looking like HTTP requests.
(it will also occasionally make good payloads look like HTTP requests
:(.

But it was the least worst acceptable option for the most people

cheers

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 16:32:50 +0200

> I'll try to put in other words what others have already said eariler in this
> discussion. Note that in the following text I will be using the term
> kind of data is coming and how to handle it progressively. If it only knows
> what to do with whole messages, it needs to buffer all the parts completely,
> right? Okay, so let's assume the client is expecting a data that it allows
> to be fragmented and can handle each incoming piece separately without
> needing to buffer all the data first. How can the client know what data the
> server is sending? Well, the application has some kind of internal protocol
> it talks inside the WebSockets channel.
>

The point here is that the current solution reduces buffering
requirements on the _server_, while making sure the _client_ still has
to buffer the whole message.
As you noted before, if the client wants to use message fragments and
a reduced buffer, the client is free to use a custom "protocol| on top
of WS to do that,
for example, chopping files on the server and sending chopped chunks
in separate WS messages.
This also opens the door for the fragmentation attacks, or simply
server sending fragments indefinitely. Not only will JS client not be
able to get the message or
its fragments, but it will never be able to stop the buffer from
expanding, as the browser does not export any control information in
the JS API.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 16:35:42 +0200

> understand and negotiated an upgrade correctly.  Using another magic header
> the alleged intermediary is clueless about won't help that.

Adam Barth's paper _explicitly_ identifies the "Host:" header as the
part that confuses
the intermediaries. Thus, even if the vulnerabilities shown in the
paper exist in practice,
the use of "Sec-WebSocket-Server:" instead of "Host:" should prevent
any attacks.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 16:43:36 +0200

> that response into a cache for the spoofed host header, and the cache

Another solution may be to forcibly add a static "Host:
nonexistent.websocket" header
to all packets sent using WS. This will make sure even those rare
proxies Adam found
will not be confused anymore.

--
 silvio

---

From: Simone Bordet <[email-masked]>
Date: Thu, 24 Feb 2011 15:45:14 +0100

Hi,

> the use of "Sec-WebSocket-Server:" instead of "Host:" should prevent

How can you prevent that an attacker puts *in the body of a WS
message* some text that is a valid HTTP request with a "Host" header ?
That text could be accidentally parsed and could poison the cache, right ?

Simon
-- 
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 15:51:26 +0100

> That text could be accidentally parsed and could poison the cache, right ?

Please, please, this was discussed to death for maybe 2 months. We're
going backwards, let's not have everyone re-explain why some things
are possible and others are not. What is currently being done is killing
any hope of progress.

Thanks,
Willy

---

From: Dave Cridland <[email-masked]>
Date: Thu, 24 Feb 2011 14:52:04 +0000

On Thu Feb 24 14:23:04 2011, Andy Green wrote:
> Yes, the alleged intermediaries are still sat there as vulnerable  
> as ever to the right packets sent another way.  But that "other  
> way" is not so bad-guy-friendly as javascript in a browser that is  
> true.

You missed that it's something that's easily detectable in a browser  
prior to allowing websockets (or Flash, or Java, from which one can  
also attack the intermediaries in a way proven to actually exist - as  
unfashionable as that may be).

But primarily, you forgot that this is not the argument behind  
masking - the particular security issue with intermediaries that  
you're referring to has a number of solutions, many of which I can  
even describe as "sensible" with a straight face, such as CONNECT,  
testing, etc.

But the argument for masking is that there exists some security hole  
in intermediaries that we haven't thought of, and the best defence  
against this issue is to randomize the output from the browser. This  
is, naturally, a very hard argument to address, even though it  
strikes me as being very similar to the ontological argument for the  
existence of a supreme being.

Opponents of masking have suggested that a security hole might exist  
that we haven't thought of for which random data might be the cause;  
sadly many of us are mired in the kind of logical thinking that  
doesn't lend itself well to arguments of that nature.

In any case, due to this ontological argument, the supreme being  
*and* the web browser vendors have insisted on masking, so the  
opponents have merely tried to do damage limitation; the ship has  
sailed, and while unfortunately it's on a course to la-la-land, the  
best we can do is steer carefully.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 16:53:25 +0200

> That text could be accidentally parsed and could poison the cache, right ?

The intermediary will only parse it if it's in the header of the packet.
Thus, you make sure that the header will not be poisonable, by forcing
specific, static and _invalid_ "Host: invalid.websocket.security" header for
_each_ packet, whether during or after handshake.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 17:02:21 +0200

> But the argument for masking is that there exists some security hole in
> intermediaries that we haven't thought of, and the best defence against this
> issue is to randomize the output from the browser. This is, naturally, a
> very hard argument to address, even though it strikes me as being very
> similar to the ontological argument for the existence of a supreme being.

As understood from the Adam's paper, the argument was that WS is
allowed to send headers in _each_ packet,
and _after_ the handshake, those packets can include stuff that might
confuse intermediaries, such
as sending a request packet, that, coincidentally results in a
separate packet, starting with what looks
like a valid HTTP request, with "Host:" and other headers in place.
This might actually happen, although the WS packets have considerable
binary content before the actual
frame content, a skilled attacker might cause WS to output such a packet.

The problem is thus: WS packets may include malicious HTTP "Host:"
headers that are authored by JS author.

One of the solutions is to mask all content, but this solution is an
overkill, taking debugging and implementation
simplicity with it as collateral damage.

An arguably better solution to this _specific_ problem is to make sure
no malicious "Host:" headers are parsed,
by, f.e. adding "Host: websockets.invalid\r\n\r\n" to each packet
after, and even during handshake.

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 16:04:51 +0100

> _each_ packet, whether during or after handshake.

Please adapt your terminology so as not to confuse anyone. Those are not
"packets" but HTTP request messages. A message is a stream of bytes which
can be made of one or multiple segments, themselves split over multiple
packets in any order. A same packet may contain both the header and body
part of an HTTP message. The packet header is where you find the IP-specific
information.

Such confusions are precisely the reasons why people are talking about
various types of non-existing risks such as what is written above. 

Thanks,
Willy

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 17:11:18 +0200

> part of an HTTP message. The packet header is where you find the IP-specific

What was meant is that for each WS-sourced TCP packet, there should be a
statically added "Host: websocket.invalid\r\n\r\n" header added before
the actual WS frame.
The rationale is that this will be parsed by the intermediaries and
make them ignore
the rest of the message, having found the "Host:" header and the
"\r\n\r\n" separator.

Please comment on this solution when you can.

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 16:16:10 +0100

> frame content, a skilled attacker might cause WS to output such a packet.

It's vaguely what you describe. Adam's paper what much more precise on the
issue. The issue is that if we allow an attacker to completely manipulate
the stream immediately after the handshake *and* a caching intermediary
ignores the 101/upgrade *and* accepts a second request over the same
connection *and* uses the initial connection's destination IP address to
route the traffic but uses the Host header to index the cache entry, *then*
it is possible for an attacker which has control over both the browser and
the server side to forge a valid request and a valid response that will
be cached by the vulnerable intermediary for the requested host. Note
that this is already possible for anyone using telnet, netcat or by simply
setting the attacker's IP address in /etc/hosts on the client. The worries
here are the massive use of browsers to build the attack.

WS framing already takes care of not emitting data that look like valid
HTTP traffic, but there was speculation that we should not completely
eliminate the risk that some intermediaries would skip over unknown bytes
at the beginning of the stream and resync to what looks like a valid HTTP
request.  

As such, it was preferred to mask the data from the browser to the server
with a random, non-controllable key so that the attacker which has control
over the browser and the server at the same time cannot reliably decide
what the output stream will look like, thus making this attack useless.

The risk was extremely low, and the solution is acceptably cheap, so it
looks like it was worth implementing it to get everyone agree on it.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 16:20:43 +0100

> statically added "Host: websocket.invalid\r\n\r\n" header added before

No, please don't write that. Applications, parsers, senders, etc... have no
visibility over *packets*. There is no WS-sourced TCP packets. There's a WS
sourced TCP stream which is composed of any number of packets and the
application has no way of knowing how they are split. You can have multiple
WS frames per TCP segment, as well as having a WS frame span over multiple
TCP segments.

> the rest of the message, having found the "Host:" header and the

Please see how an HTTP parser works and particularly the ones implemented
in caching intermediaries. Those are much more complex than a random string
lookup in random data. HTTP has strong semantics which forces agents to
strictly respect, otherwise they can't keep in sync with the stream and
process more than one request over a connection.

Regards,
Willy

---

From: Dave Cridland <[email-masked]>
Date: Thu, 24 Feb 2011 15:38:59 +0000

> Please comment on this solution when you can.

The problem is that nobody could prove that:

"GET /ga.js HTTP/1.1
Host: www.google.com

"

would not be interpreted as an HTTP request by some intermediary  
somewhere, wherever it might appear in the sequence of octets  
subsequent to a successful websocket handshake, including a sequence  
such as:

"CONNECT non-existent HTTP/1.1
Host: non-existent

GET /ga.js HTTP/1.1
Host: www.google.com

"

You'd be forgiven for thinking this is tremendously unlikely to occur  
in reality, since if it did then intermediaries would be horrendously  
confused by a number of similar cases that undoubtedly exist in the  
wild.

However, this is not the only issue, and not the one that has coerced  
us into masking - the argument is that there exists some *other* -  
possibly unrelated - issue that a malicious script could take  
advantage of, and that by removing the ability for such a script to  
control any of the octets on the wire, this avenue of exploit is  
rendered impossible.

In other words, we have had this very argument, and you'd be advised  
to read the archives to see if this particular solution has been  
raised before advancing it. (Actually, it hasn't, but it demonstrates  
a lack of understanding of the problem as posed anyway).

Dave.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 16:53:11 +0100

2011/2/24 Andy Green <[email-masked]>:
> No as you expect you won't get any on_message() for the pieces of a message
> as they come, just one for the message.
>
> But, if you like to see the pieces come in, you can arrange that the server
> sends your 3MB PNG data in a bunch of messages, not just one 3MB one.  Maybe
> that's what you were saying about the protocol, you would then need to
> leverage your protocol header to understand how each message that turned up
> fitted into the 3MB whole.  The spec leaves the protocol payload totally
> alone AFAIK.

So, the server should then split the protocol message into various WS
messages. For each one the client would invoke on_message() and it's
to the javascript code to render such partial protocol message, am I
right?

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 16:55:14 +0100

> WebSockets parts, but your application doesn't need to care about that.

Clear now. I was suggesting a different approach, but I don't think it
was better than the mechanism already defined by WS.

Thanks a lot.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 17:53:57 +0200

> "CONNECT non-existent HTTP/1.1

Let's first divide the problem into the "actually-possible attack
vector" such as above, and the "improbable" attack vectors such as
ones that you prevent with masking.

Regarding the attack vector you've shown, isn't it possible to execute
it with a javascript-sourced POST method, which, as part of the data,
contains the HTTP request?
For example:

"POST ...
...
<binary>
GET /ga.js HTTP/1.1
Host: www.google.com
"

Then the target webserver can answer with whatever it wants the proxy
to cache as www.google.com/ga.js as it would in the WS case.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 17:55:58 +0200

> to the javascript code to render such partial protocol message, am I

The browser acts as the fragment-assembler, and the onmessage()
returns the complete message as assembled by the browser from the
parts sent by the server.
At least that is the current API.
Thus, seems like, if you've got a 800MB file coming through, the
browser should buffer the whole 800MB without any progress
indication...

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 17:19:26 +0100

> contains the HTTP request?

It is exactly the same, that's why such an attack is non-sense as it
is described here. Simply try to write that in your favorite webmail
interface and check whether it breaks any intermediary. Response: it
wont because HTTP messages are correctly bounded and there is no
ambiguity as to what is part of a header and what is part of a body.

The issue comes with GET+101+immediate request after the first one+
101-unawared intermediary+all the points mentionned in Adam's paper.
Those are addressed with the current draft.

Regards,
Willy

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 18:21:18 +0200

> WebSockets parts, but your application doesn't need to care about that.

So, the goal of the fragmentation, as noted by Ian, is to allow
servers and intermediaries
to support WS without allocating huge buffers.

For intermediaries, the fragmentation only _suggests_ a solution. That
is, without a mandatory
fragment size limit, the application developer can choose it
arbitrarily, or not use the
fragmentation at all. This means the goal for "intermediaries" is not
being addressed.

Many application developers, without real benchmark to understand the
impact of frame size,
will choose frame sizes which will be suboptimal.  This is not a
variable that is best left for the
application developers to decide.

--
 silvio

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 24 Feb 2011 08:24:58 -0800

On Thu, Feb 24, 2011 at 6:19 AM, Silvio Ventres <[email-masked]>wrote:

> already
> new
> But

Yes, I do, and when I tried to give pithy responses at the beginning I got
dragged into an 80+ email thread that shows no sign of ending where people
want to re-open debate on closed issues. I am happy to give brief responses
giving the general reasoning, but if people want to debate the reasoning of
issues we've moved beyond, that is not something I am interested in spending
my time on. I am currently working on getting -06 out the door, and I feel
my time is better spent doing that than continuing to participate in this
thread.

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 18:25:43 +0200

> ambiguity as to what is part of a header and what is part of a body.

So, you agree that the same attack that is being prevented in WS is
easily possible
by simple a simple POST from the same javascript?

If so, the attack will be done whether it's prevented in WS or not, since the
POST mechanism is actually easier to execute. So WS should not try to prevent
it, especially at a cost of significant complexity in debug and implementation.

> Those are addressed with the current draft.

Can you elaborate on that ?

Thank you.

--
 silvio

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 24 Feb 2011 08:27:08 -0800

On Thu, Feb 24, 2011 at 8:21 AM, Silvio Ventres <[email-masked]>wrote:

> knows

Intermediaries are free to refragment a message. If they see a 10MB message
coming in, they may read 1MB and send that off as a fragment.

> application developers to decide.

I expect it's Apache or IIS or whatever your server is that's choosing frame
sizes. They are probably in a reasonable position...

---

From: John Tamplin <[email-masked]>
Date: Thu, 24 Feb 2011 11:28:39 -0500

On Thu, Feb 24, 2011 at 10:38 AM, Dave Cridland <[email-masked]> wrote:

> would not be interpreted as an HTTP request by some intermediary somewhere,
> wherever it might appear in the sequence of octets subsequent to a
> successful websocket handshake, including a sequence such as:
>
> You'd be forgiven for thinking this is tremendously unlikely to occur in
> reality, since if it did then intermediaries would be horrendously confused
> by a number of similar cases that undoubtedly exist in the wild.
>

See, this is where this discussion is opening old wounds and threatening the
very progress we made.  Adam, AFAIK, was happy with a CONNECT based solution
hiding future WebSockets content from the transparent intermediaries.
 However, many other people were not happy with using CONNECT at all, even
in your proposal where it was never intended to be seen in the context of an
HTTP request but only after the upgrade to WebSocket had been completed.
 Since CONNECT was not goiing to achieve consensus, we had to come up with
some other way to avoid the problem, and masking was the way that did
achieve consensus.  As you note below, it also has the benefit of protecting
from other problems we don't know exist, even if that wasn't the reason
behind this.

> However, this is not the only issue, and not the one that has coerced us
> into masking - the argument is that there exists some *other* - possibly
> unrelated - issue that a malicious script could take advantage of, and that
> by removing the ability for such a script to control any of the octets on
> the wire, this avenue of exploit is rendered impossible.
>
> In other words, we have had this very argument, and you'd be advised to
> read the archives to see if this particular solution has been raised before
> advancing it. (Actually, it hasn't, but it demonstrates a lack of
> understanding of the problem as posed anyway).

Please, let's not reopen these discussions that took months to achieve
consensus.

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 18:29:38 +0200

> my time is better spent doing that than continuing to participate in this

Well, the amount of text you wrote above is basically enough to answer
2 or 3 of the
questions asked, for example, the one about the handshake computation.
You mentioned it's there to "prove the server actually parsed the header".

The argument advanced against that was that if a server developer
isn't interested
in parsing the header, he can easily trick you by using an external library to
pass the handshake test. Given that, the solution fails to meet it's
goals and should be removed.

If you have some other rationale on the handshake-computation, or you don't
accept the fact that the server author cannot be coerced into doing
anything he doesn't like,
please reply with your elaboration.

Thank you.

--
 silvio

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 17:32:16 +0100

> Please, let's not reopen these discussions that took months to achieve

Nobody would reopen old discussions if conclusions of such discussions
would be properly documented as in many other specifications.
Note: "mail list archives" is not a documentation backend.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 18:33:23 +0200

> HTTP request but only after the upgrade to WebSocket had been completed.

The way you solve this is by clarifying the goals, hearing each
argument and finding a solution that
satisfies the goals while proving to each arguing party that the
arguments are false.

You don't leave it be when it is clearly an overkill and impacts such
basic parts of the protocol
as debugging ability.

--
 silvio

---

From: John Tamplin <[email-masked]>
Date: Thu, 24 Feb 2011 11:38:47 -0500

> Note: "mail list archives" is not a documentation backend.

Well, the problem is I don't have time to coalesce years of discussions on
this list into bullet points summarizing the rationale for you, and I don't
imagine anyone else has time to do so either.  When I joined the discussion
(admittedly there was far less history at that time), I spent a week reading
all the archives before posting anything.

So, while it would be nice to have such a document, I don't know that there
is sufficient manpower available to make it happen.  The chair has created a
FAQ editable by anyone, so it can be done incrementally, but I still don't
expect the rationale of all the decisions made to make it there anytime
soon.

As usual, the ones who care wind up doing the work.  In this case, since you
appear to be the one that cares the most about having a rationale document,
it seems likely you will have to put forth the effort to collect the data.

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 18:37:48 +0200

> coming in, they may read 1MB and send that off as a fragment.

For that, intermediaries have to actually know about WS. Simple HTTP
intermediaries
won't be able to do that.

> sizes. They are probably in a reasonable position...

The actual WS-talking parts are the only ones that can set WS
fragmentation flags.
Whether Apache or anything else, a plugin will probably be doing the
work. In many
cases the WS servers are custom-written, in python or node.js.
By _not_ mandating the frame size, you are leaving it up to the user
who does not
have enough information for an informed choice. At least provide a suggestion.

--
 silvio

---

From: John Tamplin <[email-masked]>
Date: Thu, 24 Feb 2011 11:41:56 -0500

> basic parts of the protocol

So how do you think we declared consensus on masking?  It was after months
of discussing various alternatives and determining that while nobody liked
the solution (for various reasons, some thought it didn't go far enough and
others thought it went to far), it was the only solution to the problems
that all involved were willing to live with.

As far as debugging ability, XOR'ing the frame is no harder to deal with
than decompressing compressed content (which will hopefully be a
common occurrence on WebSocket connections as well).

---

From: John Tamplin <[email-masked]>
Date: Thu, 24 Feb 2011 11:42:42 -0500

> Those are addressed with the current draft.

The difference is that the intermediaries know about POST and know that the
data framed by the POST method is not HTTP protocol content, but instead
user data.  In the GET+Upgrade case, those intermediaries did not understand
Connection: Upgrade, so they treated the next bytes after the completed GET
message as another HTTP message.  Adam did not show that those
intermediaries would skip past WebSocket framing, but given the lazy HTTP
parsers we have all seen it seems entirely plausible that they would skip
garbage and try and synchronize on what appears to be an HTTP header.  After
all, these are transparent intermediaries, so they can't just terminate the
connection and resync on a fresh connection attempt, nor can they simply
stop working indefinitely because they didn't understand some command.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 17:58:24 +0100

> by simple a simple POST from the same javascript?

No, instead I'm trying to make you understand that it does not more
exist with WS than it does with a simple POST. An intermediary will
never look at random strings. It parses request headers when it
expects request headers and it parses body when it expects body. It
has no reason to confuse both of them.

However in the case of WS, we know that few intermediaries are aware of
the GET+Upgrade handshake, and that some of them will expect a new request
just after the handshake because they'll think there's no body and they
don't know that a tunnel is established. So if you'd immediately send a
new HTTP request-looking sequence of bytes on the wire after the first
one, some of them could mistake it for the HTTP request they're expecting.
But the WS framing prevents that since it contains bytes that can't appear
in an HTTP request. The masking ensures that nobody will control the stream
in case a less regarding one is discovered.

(in fact, the masking key placed before the stream could possibly make a
valid looking request appear, but since that's random, it's not exploitable).

Hoping this helps,
Willy

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 18:56:43 +0200

> stop working indefinitely because they didn't understand some command.

Since you agree that those intermediaries would NOT get confused if
you use POST,
then POST solves the problem posed.

Why not simply use POST and remove the masking ?

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 18:55:23 +0200

> that all involved were willing to live with.

Well, how you arrived to consensus was to do a "straw vote" which
doesn't seem like a real consensus.
What should have been done is to clarify the goal, add subgoals based
on various inputs,
removing irrelevant ones, such as "add security features for fun", and
clarify that there is
no point in making WS _more_ secure than HTTP, since any attacks that
can be done using standard
HTTP (i.e. POST-based proxy-confusion) will be done using it, since
it's more used in the wild.
After removing the irrelevant goals, solutions should be proposed and
weighed along their
impact on the protocol implementation simplicity, debugging ability,
understanding complexity,
expected performance and maintainability. The solution that impacts
the least while satisfying the
goals stated should have been chosen to be represented in the specification.

> common occurrence on WebSocket connections as well).

Do you think HTTP would be so prevalent today if it had been using XOR masking ?
Did you ever write an HTTP server ? Debugged an HTTP application ?
You can always disable compression. Per current spec, masking is mandatory.

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 18:03:03 +0100

On Thu, Feb 24, 2011 at 06:56:43PM +0200, Silvio Ventres wrote:
> you use POST, then POST solves the problem posed.
> 

POST is not bidirectional, otherwise it would not have been needed to
start this WG if the solution was *that* obvious ! The goal of the WG is
to achieve bidirectional communications over HTTP, which currently can't
reliably be achieved with common methods.

Willy

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 24 Feb 2011 09:03:34 -0800

John Tamplin wrote:
> but given the lazy HTTP parsers we have all seen it seems entirely 
> plausible that they would skip garbage and try and synchronize on what 
> appears to be an HTTP header. 

No, that's not plausible at all.

-- Scott

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:06:55 +0200

> has no reason to confuse both of them.

Well, then it seems you don't believe this attack vector has impact at all.

> in case a less regarding one is discovered.

Since from your explanation it seems that the problem lies with the
"GET+Upgrade" headers,
and the fact that intermediaries expect no body, it can be fixed by
either adding some
data after the handshake, or changing the GET+Upgrade to something else.
In any case, masking is too invasive, and the debug ability is _very_ important.
You _do_ want the application developers to be able to debug their applications.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:11:18 +0200

> reliably be achieved with common methods.

You misunderstood. What was meant is to use POST as the method to start
the WS handshake, not to replace WS with POST :)

WS is _not_ HTTP. The only reason, as mentioned by Ian, that "GET"
method is used instead of "WAFFLES", is to be multiplexed through port 80.
So, if using POST method fixes the problems mentioned by Adam Barth,
there's no reason to even consider the masking procedures.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:12:21 +0200

> No, that's not plausible at all.

Well, then, problem solved?
Masking has no real reason to exist ?

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 18:14:33 +0100

> data after the handshake, or changing the GET+Upgrade to something else.

That was already proposed, with random bytes, invalid requests or even
Dave's proposal of a CONNECT request to force the broken ones to fail.
Still those arguments were not convincing enough to reach consensus.

> You _do_ want the application developers to be able to debug their applications.

When debugging, simply force the mask to 0 on the client and you'll see it as
clear text. It's not that hard. And believe, masking it not much invasive
compared to other alternatives that were studied.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 18:17:43 +0100

> there's no reason to even consider the masking procedures.

It does not change anything at all then. Using POST + Upgrade will
still make the intermediary wait for a new request after the response
is seen. Also this causes more issues related to the mixing of POST
with 1xx responses which can be mistaken for 100, causing a real
desynchronization between both ends.

Silvio, I really assure you that it's very painful to re-argument one
after the other many things that were discussed months ago. As Sal said
it, more than 6000 e-mails were exchanged on the subject, so you can
imagine than many ideas were explored, and I'd say tens more than what
we've discussed since you opened this thread.

Thanks,
Willy

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 18:21:05 +0100

>> appears to be an HTTP header.

So imagine this wrong GET request:

  GET /index.html HTTP/1.1\r\n
  Host: 1.2.3.4GET /lala.txt HTTP/1.1\r\n
  Host: attack.domain\r\n\r\n

The first Host header has wrong content. Do you expect that HTTP
parsers out there would discard garbage data until "1.2.3.4" and start
parsing again from "GET /lala.txt"?

I strongly don't think so. I expect most of the HTTP servers would
drop the whole received data after detecting the grammar failure, and
many of them would also terminate the TCP connection.

If a HTTP server starts parsing from "GET /lala.txt" that is a bug and
a vulnerability (probably).

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Joe Hildebrand <[email-masked]>
Date: Thu, 24 Feb 2011 10:21:42 -0700

On 2/24/11 9:29 AM, "Silvio Ventres" <[email-masked]> wrote:

> anything he doesn't like, please reply with your elaboration.

Please keep in mind that standards creation is as much a political process
as a technical one.  Not every decision we make will have an absolutely firm
technical foundation in everyone's mind because of that -- often the best we
can hope for is to compromise on something that nobody hates enough to
filibuster.

Some of the past political compromises have caused hurt feelings from
multiple parties, and what you're asking for is for those feelings to be
replayed in public, which means that the tenuous progress that we've made is
likely to be lost as we devolve (again) into finger-pointing and
name-calling.

We've also had a history of newcomers joining the process late, and making
us rehash *every* decision or disrupting the process with new personality
conflicts.  Because of that history, you're getting more pushback than seems
appropriate to you, but feels fairly well-justified to those that have been
around for a while.

I'll repeat what someone said earlier.  If someone wants to write an
individual informational I-D describing the justifications for everything
that ends up in the protocol, that would be fine.  However, I'm not
interested in that being a working group document at the moment, since there
will be an entire series of meta-arguments about what the justifications
are, and how to word them, all of which will prevent us from getting to a
point of completion on the protocol in a timely manner.  If the I-D ends up
being useful and complete as an individual draft, we can revisit whether to
make it a working group draft later.

-- 
Joe Hildebrand
(as chair)

---

From: John Tamplin <[email-masked]>
Date: Thu, 24 Feb 2011 12:28:07 -0500

> there's no reason to even consider the masking procedures.

POST by itself does not fix the problem, since immediately after the POST
request is complete, those same intermediaries which don't know about
WebSocket will be parsing WebSocket frames as HTTP requests.

It was proposed and rejected to use POST and chunked encoding to send frames
in both directions, since there are known to be intermediaries that de-chunk
all traffic passing through them (and would therefore wait to read the last
chunk before sending any frame on), and because client->server chunking is
hardly used and therefore likely to be as broken as Connection: Upgrade in
those intermediaries.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 18:29:51 +0100

> compared to other alternatives that were studied.

In many cases the developer will need to analyze data coming from
usual web browsers to his WS server, so he cannot change the client
mask.

For example, I work in SIP world, I code SIP servers and I spent long
time every day debugging SIP protocol *in the server*, so I've no
control over the clients (common SIP phones). Same concept applies for
WS application developers.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: John Tamplin <[email-masked]>
Date: Thu, 24 Feb 2011 12:31:46 -0500

> WS application developers.

Then presumably you are using something like Wireshark, and writing a
protocol dissector for WebSockets will be trivial.

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:33:34 +0200

> Still those arguments were not convincing enough to reach consensus.

Well, what exactly was unconvincing ?
Why is it so easy for you to just leave it be and accept a solution that has
so much impact ? Masking breaks debug.

> compared to other alternatives that were studied.

If the browser is the UA that implements WS for you, you can't really
force it to do anything.
Same with Web Services, you don't always have control over the server
configuration.

--
 silvio

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 18:37:06 +0100

> protocol dissector for WebSockets will be trivial.

No. I use basic ngrep in most of the cases. Easy debugging in realime
without having to open a generated file in my local computer, just run
"ngrep --options" in the server and voila!.

SIP is plain text protocol like HTTP. And SIP has become the winning
VoIP protocol since it uses plain text. Other VoIP protocols like H323
are binary protocols and have mostly dissapeared. Take it into account
please.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Joe Hildebrand <[email-masked]>
Date: Thu, 24 Feb 2011 10:37:11 -0700

> so much impact ? Masking breaks debug.

This discussion isn't moving our work forward.  Everyone get your last
points in, because I'm perilously close to asking for it to stop.

-- 
Joe Hildebrand
(as chair)

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:37:20 +0200

> make it a working group draft later.

Arguably, the fact that justifications will be argumented and worded
correctly will
only bring better understanding and faster completion of the main
protocol specification.

--
 sivlio

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 24 Feb 2011 09:39:16 -0800

> Masking has no real reason to exist ?

No, because there may be other target vulnerabilities other than the 
specific intermediate-synchronization one. Possibly even vulnerabilities 
for protocols that haven't been invented yet. The problem is an 
open-ended one, which makes it more complicated.

The difficulty with the hijacking scenario is that the hijacker has an 
amazing amount of control, particularly with something like websockets 
that's more than just a single request/response of HTTP.

Given the hijacking scenario, it's better to solve the problem by 
denying plaintext control over the output bytes, rather than hoping that 
we've covered every present and future case. That might end up being a 
bit of overkill. It might not.

So consider the masking a hijacking-cost of allowing the client to 
execute arbitrary and possibly hostile code.

-- Scott

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 24 Feb 2011 09:41:47 -0800

> points in, because I'm perilously close to asking for it to stop.
FWIW I stopped reading this thread, and the centithread it forked from, a
while ago.

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:41:43 +0200

> hardly used and therefore likely to be as broken as Connection: Upgrade in

Make each separate WS-sourced packet have a header that makes it look like
a completely new POST request (not a chunk). Then the proxies will not find any
malicious headers at the start of the packet, indeed they would assume
it's binary
data and leave it be.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:43:29 +0200

> protocol dissector for WebSockets will be trivial.

There are other tools except WireShark.
For example, tcpdump, netcat/telnet combined with shell scripts, using grep etc.

--
 silvio

---

From: Dave Cridland <[email-masked]>
Date: Thu, 24 Feb 2011 17:47:24 +0000

> malicious headers at the start of the packet, indeed they would  

You are now describing BOSH.

http://xmpp.org/extensions/xep-0124.html

Dave.

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:49:05 +0200

> So consider the masking a hijacking-cost of allowing the client to execute
> arbitrary and possibly hostile code.

The WS protocol doesn't exist in vacuum.
HTTP protocol has similar deficiencies.
Thus, when specifying WS protocol, the authors have to understand that there is
no point in making WS protocol prevent features that are _already_
available in HTTP.
Indeed, if the usage of POST method will bring WS to the same playing
field as HTTP POST,
you don't need to go any further. HTTP is more prevalent than WS, and
thus, any attacker
would choose HTTP over WS to execute the attack. There is absolutely
_no_ point in
adding features that will prevent attacks that will first happen in HTTP anyway.

To reiterate: if HTTP allows a specific unmitigated attack vector,
there's no point in preventing it in WS.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 19:55:20 +0200

> You are now describing BOSH.

No, the connection is still bidirectional and doesn't need multiple streams.
Basically, the same WS protocol, but adding the fake POST headers at
beginning each packet to prevent proxy confusion

--
 silvio

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 24 Feb 2011 10:00:54 -0800

> beginning each packet to prevent proxy confusion
Do recall that WS is not request/response, i.e. a server can send data to a
client not in request to a specific response. That would imply that a client
has a hanging GET waiting for the data, or that there's some other magic
happening to let you do this with a single channel.

Taking a step back, what exactly are you trying to achieve? Your posts thus
far are not adding to anything the WG is going, your proposals wouldn't
really make any material difference in terms of capabilities / efficiency,
and instead appear ill-informed and argumentative, leading to centithreads
that go nowhere and people wasting their time just at the point that the WG
was once again making progress.

---

From: Silvio Ventres <[email-masked]>
Date: Thu, 24 Feb 2011 20:13:51 +0200

> happening to let you do this with a single channel.

Does WS as currently specified use a "hanging GET" to pass through proxies?
Then how is that different for a web developer than XHR ?

The proposal was to add fake POST headers to each packet sent,
such that proxies do not confuse themselves. If the channel
cannot be kept open for bidirectional messages, then why use WS
compared to XHR ?

Ian, can you please let know if you think the problem with proxy
confusion actually exists?
And the scenario which, theoretically, can trigger the confusion ?

> was once again making progress.

The fact that the posts, having got lots of "read the archives"
replies, ignited a discussion
about the general spec-drafting procedures is the consequence of the
people that wrote
massive amounts of messages with nothing but "read the archives"
answers in them.
If you remove those, rest of discussion taking place is quite constructive, and,
hopefully, will bring benefit to the spec in both long and short term

--
 silvio

---

From: Joe Mason <[email-masked]>
Date: Thu, 24 Feb 2011 13:38:51 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Silvio Ventres
> 

No, it really isn't.  It may appear constructive from your point of view because you're getting your questions answered, but the rest of the group is just wasting their time repeating what's already been said.

Joe

---------------------------------------------------------------------
This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 21:56:01 +0100

> a vulnerability (probably).

Even simpler : the grammar is OK. You simply have a first Host header whose
content is "1.2.3.4GET /lala.txt HTTP/1.1" and a second one whose content
is "attack.domain". Both intermediaries and receiver are even free to fold
them in a single one by delimiting them with a coma :

   "1.2.3.4GET /lala.txt HTTP/1.1, attack.domain"

After that, the upper layers may decide whether this Host header is something
valid they want to process or not.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 24 Feb 2011 22:01:04 +0100

> are binary protocols and have mostly dissapeared. Take it into account

Iñaki, I really understand your point. We all would prefer to get plain
text if no risks were identified, many of us would feel more comfortable
with no masking at all, just as well as many of us don't feel comfortable
with so little masking. We have to satisfy everyone's fears.

If you have control over the server, you can easily insert a simple proxy
which will unmask and forward the frames with a zero mask. Ngrep will then
work on the forwarded stream. I agree it is more cumbersome than being able
to grep the live stream but I'm trying to help you find solutions with the
current design, and they don't look too hard.

Regards,
Willy

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Thu, 24 Feb 2011 22:58:03 +0100

> valid they want to process or not.

It was just an example, however I expect that a HTTP intermediary is
clever enough to invalidate current request (as Host header is
obviously wrong). Or maybe they are bery lazy, ok.

Anyhow, no HTTP server/intermediary will ignore wrong bytes until it
gets something like GET/POST (IMHO).

Regards.

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: "Pat McManus @Mozilla" <[email-masked]>
Date: Thu, 24 Feb 2011 17:32:40 -0500

> gets something like GET/POST (IMHO).

consider this: 
http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpTransaction.cpp#683

it is a client doing response parsing, so it is analogous though not a
direct example. The mess of different bugwards compatible exceptions to
strict http framing rules seen there aren't present because we think it
is a good idea, nor are they there out of any form of laziness I am
familiar with. They are there because various http server stacks
actually generate that crap all the time and one of the realistic things
you have to do is try and resync by looking for known tokens. This
happens in all manner of interesting ways usually involving a framing
error followed by a connection reuse (e.g. content-bodies non sensically
added to 304s, content-lengths that aren't udpated in the face of
runtime compression, extra-or-missing crlfs after the closing
chunked-encoding 0-chunk, content-lengths that are short by a few
because they counted the number of utf-8 characters instead of the
number of octets and didn't test with any cases that contained multibyte
chars, etc..). Many of these scenarios can easily apply to responses
too.

The question for hybi is not whether or not mozilla should have done
this (I personally have some reservations), but whether or not it is
reasonable to assume a request parsing stack would see the same
pressures for greasing the skids of interop - and of course they do.
Masking helps cover up that entire class of exposure when running
websockets through http-only intermediaries, which is a good thing. And
a unidirectional xor does it at a remarkably inexpensive cost both
computationally and in terms of complexity.

We've been over all this before - if you aren't going to bring any new
information to the discussion that warrants reconsideration, then please
respect the existing consensus declaration.

---

From: Iñaki Baz Castillo <[email-masked]>
Date: Fri, 25 Feb 2011 00:22:02 +0100

> chars, etc..). Many of these scenarios can easily apply to responses

I like being more strict when parsing, but I understand your points,
all of them got from the real world. Thanks for the explantion.

> respect the existing consensus declaration.

I respect it. I just want to know rationale for current design. The
new content of the wiki has helped me a lot:

http://trac.tools.ietf.org/wg/hybi/trac/wiki/FAQ

-- 
Iñaki Baz Castillo
<[email-masked]>

---

From: Silvio Ventres <[email-masked]>
Date: Fri, 25 Feb 2011 18:03:42 +0200

> http://mxr.mozilla.org/mozilla-central/source/netwerk/protocol/http/nsHttpTransaction.cpp#683

Let's start from the attack vector itself. The one prevented by masking.

Is the subject attack vector possible with POST requests ?

If the answer is YES, than there is no point protecting from it in WS,
as the POST request is much more supported in the wild and can be
used as a medium to mount the attack _right now_.

If the answer is NO, than, as proposed before, the WS can get to the
_same_ level of protection from this attack as HTTP, by making each
request look like an HTTP POST request.

Thus, _regardless_ of the answer, the masking has no reason to exist.

The point is that there is absolutely _no reason_ to make WS more secure
than HTTP, and thus, if an attack vector is found, bringing the protection
to the level of HTTP is enough.

Why sacrifice _actually-useful_ features, such as low computational complexity,
source code simplicity and debug ability for protections from attacks
that, if they
will ever happen, will happen in the _other_ protocol ?

<sarcasm>
Why not add elliptic-curve crypto to the whole stream ? That will make it
look even _more_ random than XOR masking ? Why not add a dependency
that the endpoint take a random number from a known-good PRNG, and
specify that a PRNG with parameters less than X,Y,Z are not compliant ?
Then for each reply, the endpoint would verify the entropy of the stream on
the wire and discard it in case it's not random enough ?
</sarcasm>

--
 silvio

---

From: Brian McKelvey <[email-masked]>
Date: Fri, 25 Feb 2011 08:14:02 -0800

You jest, but we had a very long discussion where some members of the wg strongly advocated full aes encryption for the exact reason of making all the bytes that much more random looking.  We eventually settled on the per-frame XOR as a compromise.

Brian

Sent from my iPhone

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 11:45:12 -0500

> request look like an HTTP POST request.

How does making it look like a POST apply?   WebSockets is a
bidirectional communication, so you can't send individual frames in
POST messages because there is no way for the server to initiate a
response to a POST that hasn't been sent yet.  The alternative, which
I mentioned before, would be to do a POST of unknown length using
chunked encoding, and each frame is sent as a chunk.  The problem is
that many intermediaries do things based on the current usage of POST,
such as not opening the connection to the server until it receives the
complete request which would never happen in the case of WebSockets.

---

From: Silvio Ventres <[email-masked]>
Date: Fri, 25 Feb 2011 19:01:12 +0200

> response to a POST that hasn't been sent yet.  The alternative, which

What is the exact problem that you envision here ?
The client would include POST headers in each request.

If the proxy does not allow server to reply multiple times to the client,
whether you use POST or not is irrelevant.

"POST .... <ws control-start> ... <ws control-end>"

> chunked encoding, and each frame is sent as a chunk.

How about a POST of a known length of the frame, and each frame is
sent as a separate
POST message ?

> The problem is

Well, then make each message a complete request.

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 18:06:46 +0100

> How about a POST of a known length of the frame, and each frame is

Because that does not allow the server to talk without a client request.

I think you really did not get the principle of establishing a bidirectional
communication between the client and the server. It's not WS which makes it
bidirectional, it's the HTTP handshake based on the Upgrade header and the
101+Upgrade in the response. The method is irrelevant. WS is used on top of
that to provide framing and to ensure that HTTP and WS don't accidently mix.

Willy

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 12:06:38 -0500

> The client would include POST headers in each request.

Imagine a chat client using WebSockets.  It connects and authenticates
to a server, then doesn't send anything until the user types a
message.  However, it needs to know when new messages from other users
are sent, so the server sends a message on the connection.  If using
separate POST messages for each frame (which can only be initiated
from the client), how does the server send that message?  The best you
could do is what is currently done without WebSockets, such as hanging
get.  There are a number of problems associated with that, but if that
is sufficient for you then just use that instead of WebSockets --
there is no need to make WebSockets no improvement over hanging gets.

> whether you use POST or not is irrelevant.

WebSockets as currently specified achieves this.

> How about a POST of a known length of the frame, and each frame is

Again, that does not allow server-initiated messages.

---

From: Silvio Ventres <[email-masked]>
Date: Fri, 25 Feb 2011 21:04:02 +0200

> that to provide framing and to ensure that HTTP and WS don't accidently mix.

The bidirectional capability of WS is provided by either the server
being a WS-server
from the start, or by HTTP server forwarding the request to its
WS-compliant part after
sensing WS-specific headers. Whether those headers are HTTP Upgrade or
any others
is irrelevant, as these have to be defined explicitly in any case.

Thus, whether WS uses GET, POST or any other headers is a matter of
specification.

If you're talking about caching intermediaries, if they don't support
WS specifically,
whether you use Upgrade or not won't matter much. HTTP semantics apply.

If the above is not correct, please enlighten.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Fri, 25 Feb 2011 21:07:59 +0200

> from the client), how does the server send that message?  The best you

The POST headers are there to provide POST-like security. The server is a
WS server, not an HTTP server. Thus, it'll send replies whenever it wants to.
And if it's an HTTP server with a separate WS-module, than the message would
be forwarded to that WS-module anyway.

> WebSockets as currently specified achieves this.

Achieves what ? Does WS as currently specified achieve
"one request, multiple replies" over legacy http proxies ?

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 20:10:23 +0100

> is irrelevant, as these have to be defined explicitly in any case.

No, the Upgrade header is the most important part. In HTTP, it means
"please don't use HTTP after that request, and upgrade to the protocol
I'm naming here if you can". HTTP-compliant intermediaries will
automatically switch to a tunnel between the client and the server
when they see status 101 in response to that header. This is what
transforms your HTTP connection into an established TCP connection
that WS can make use of.

> whether you use Upgrade or not won't matter much. HTTP semantics apply.

Yes it does matter because if they follow HTTP semantics, they will
provide you with this tunnel you need between the client and the server.

Please read section 9.8 of the HTTP spec here :

   http://datatracker.ietf.org/doc/draft-ietf-httpbis-p1-messaging/?include_text=1

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 20:15:14 +0100

On Fri, Feb 25, 2011 at 09:07:59PM +0200, Silvio Ventres wrote:
> The POST headers are there to provide POST-like security.

It is very frustrating to see that you still insist on talking about those
supposted "POST headers" which at best don't exist, or at worst don't mean
anything. And I think that all your reasoning is based on what idea you
have about this imaginary entity.

> The server is a
> WS server, not an HTTP server.

Yes it is.

> Thus, it'll send replies whenever it wants to.

Even if it tries to, the reply will not reach the client through any
possible intermediary because that is not compatible with HTTP. Why use
an HTTP handshake if you have the ability to talk whatever protocol you
want ? WS provides you with the protocol you want on top of HTTP precisely
because in most environments, only HTTP can pass through.

> be forwarded to that WS-module anyway.

That's irrelevant.

> "one request, multiple replies" over legacy http proxies ?

Yes. Even "zero request, multiple replies" if you want.

Willy

---

From: Silvio Ventres <[email-masked]>
Date: Fri, 25 Feb 2011 21:30:10 +0200

>From a quick look at squid, as an example of caching proxy, it does not support
HTTP/1.1 Upgrade. The code actually falls through to a break.

Thus, this is not really useful for caching intermediaries.

For HTTP servers, although the Upgrade mechanism is custom-made anyway,
if you want to leave the HTTP GET/Upgrade intact, leave it, but, to get HTTP
level of security regarding the data contents, add fake-POST headers to each
successive message after handshake.

> provide you with this tunnel you need between the client and the server.

Most proxies don't. The minority that do shouldn't be a considered the norm.
Actually, if you can name that minority, that would be useful.

--
 silvio

---

From: Silvio Ventres <[email-masked]>
Date: Fri, 25 Feb 2011 21:41:42 +0200

> have about this imaginary entity.

What is meant by "POST headers" is that you make each successive message
to the HTTP server _look like_ a POST request. If you have a direct connection
or a proxy-provided raw-socket tunnel to the WS server, it doesn't hurt.
If you have a connection over a proxy that might get harmed by data contents
of your message by whatever means (as shown by Adam Barth), then this
request, being considered a POST request by the proxy, will do no more harm
to the proxy than a regular POST request, thus obviating the need for masking.

>> WS server, not an HTTP server.

That's an interesting argument. An HTTP server that is not supporting WS is not
discussed here. If an HTTP server supports WS, and provides WS an Upgrade
tunnel to the WS-implementing module, the POST headers will  be stripped there.
The HTTP server will never see those POST headers as they are part of
the upgraded
protocol.

> because in most environments, only HTTP can pass through.

If Upgrade header guarantees that it will pass through, by all means, use it.

The discussion was about masking. And a solution proposed that instead of
masking, each successive message include a header that will make it look like
a POST request.
In case the initial HTTP/1.1 Upgrade established a tunnel, the POST headers
will simply be ignored by the WS-server.

> Yes. Even "zero request, multiple replies" if you want.

What are those magical proxies that support HTTP/1.1 Upgrade ?

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 20:47:15 +0100

> HTTP/1.1 Upgrade. The code actually falls through to a break.

Indeed, and you'll also find others which break because they don't full
implement the HTTP/1.1 specification. That does not mean they won't be
fixed for that. Also, Squid is mostly used as an explicit proxy so it
generally will not be asked to support Upgrade, it will only receive
normal CONNECT-based proxy requests which do already work. And once again
as already explained, a big part of the handshake work is to ensure that
incompatible intermediaries are quickly detected instead of leaving a
connection hanging.

> successive message after handshake.

You're still talking about those "POST headers". Care to explain them to
us one day ? It's really getting annoying to read about them in all your
mails without having them presented.

> Most proxies don't.

Most proxies don't need to either. Reverse-proxies need that. Load balancers
need that.

> The minority that do shouldn't be a considered the norm.

The norm is defined as the HTTP/1.1 specification (RFC2616). Those who don't
implement it right now have probably not done it because there was no
immediate use. When users ask for fixing, they'll do, and it's quite easy
for many reverse-proxies.

> Actually, if you can name that minority, that would be useful.

At least I know that haproxy does support it if that helps. I don't know
all products, but if you're worried about one you find that does not support
it (and you'll find), please contact its author to kindly ask when support
is planned. It's possible that the author has never heard about Upgrade and
will take some time to discover how it works and what it implies for him.

In my case, adding it to haproxy was just a matter of a few hours, tests
included. And given the reports we seldom get on the list from people
implementing both a client and a server, I guess that's the average to
expect.

Regards,
Willy

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 15:00:59 -0500

> mails without having them presented.

IIUC, he is suggesting changing the client->server WebSocket framing to
something like:

"POST /bogus HTTP/1.1\r\nContent-length: <length>\r\n<other
headers>\r\n\r\n" <websocket payload>

So that a transparent intermediary that doesn't understand GET+Upgrade will
not see any attacker-controlled data as an HTTP request.   While I believe
that will work, I believe the overhead is excessive and would preclude any
advantage WebSockets has over something like BOSH for small messages.  I
also think the tons of repeated content would impede debuggability at least
as much as having to do an XOR.

If I am reading a binary protocol, I am not going to (generally) look at the
bytes on the wire, but rather than decoded protocol output, such as from a
Wireshark dissector.  Even if you assume the payload will all be text
initially, hopefully most implementations will use compression so you still
have to do some decoding, and once you are doing that applying the XOR at
the same time you decompress it is trivial.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 21:03:48 +0100

On Fri, Feb 25, 2011 at 09:41:42PM +0200, Silvio Ventres wrote:
> to the HTTP server _look like_ a POST request.

That's still very vague, you'll agree. Could you please post an example of
what such a request would look like ? It's not clear to me if you mean to
use the POST method or not, if you want to post a body or not, how you would
indicate it (content-length, transfer-encoding), if you intend to follow it
by an Upgrade or not, nor how you expect the request to be processed, or if
you expect both the request and response messages to simultaneously flow or
just one at a time.

The only thing I seem to understand is that with your proposal, a chat
application would require the client to send a full POST request for each
char or line typed, and I still don't get how you'd get the server's
asynchronous response without a hanging request.

So please take the time to describe the request(s) and the client-to-server
and server-to-client flows for the following sequence :

     client                               server
 1)  "Hello" ---->
 2)                                <---- "Hello"
 3)                                <---- "My name is Willy"
 4)  "Wait a minute!" ---->
 5)  "I'm back, my name is Silvio" ---->
 6)                                <---- "Pleased to meet you Silvio."

I don't see how you make that work with your "POST-like" requests. It's
possible with something like BOSH using multiple connections but with
one single connection without an Upgrade I don't see.

> discussed here.

it's been discussed quite a lot and is the reason for the way the handshake
looks like.

> If an HTTP server supports WS, and provides WS an Upgrade
> the upgraded protocol.

I don't get you without a clear example as requested above.

> If Upgrade header guarantees that it will pass through, by all means, use it.

That's the point. It does not guarantee it will pass everywhere, however
when it passes, it has the precise semantics we're looking for.

> will simply be ignored by the WS-server.

Still confused.

Willy

---

From: Silvio Ventres <[email-masked]>
Date: Fri, 25 Feb 2011 22:03:33 +0200

> mails without having them presented.

Well, a simple
"POST /ws HTTP/1.1
Host: ws

"

Is enough. This identifies the request as part of an, albeit broken,
POST request.

> for many reverse-proxies.

Why would you set a dependency between WS and HTTP/1.1 ?
Anyway, as much as this is interesting, it has no impact on the
proposed solution.

The problem that masking is made to prevent is only relevant to the
data content,
thus only the messages _after_ the handshake need to be protected.
Thus, the initial handshake doesn't need to change from GET to POST or
whatever else.

The poposal is for the subsequent messages to include a simple POST header,
such as quoted above, to bring WS to the same security level as HTTP
POST, regarding
harming intermediates.

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 21:06:31 +0100

On Fri, Feb 25, 2011 at 03:00:59PM -0500, John Tamplin wrote:
> that will work,

It does not allow the server to talk without being invited to do so,
which is quite the opposite of the bidirectionality principle.

> I believe the overhead is excessive and would preclude any

I agree with all your points!

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 21:13:01 +0100

> Is enough. This identifies the request as part of an, albeit broken,

This request has no body. The only thing you can send after it is another
request (provided that intermediaries accept it). So it does not provide
any support for sending any byte of data. It's just an empty request waiting
for a next request once the server responds. At best it adds a round-trip
which slows the handshake down, at worst it's rejected.

> Why would you set a dependency between WS and HTTP/1.1 ?

I'm not setting a dependency. WS *is* HTTP/1.1 compliant, which means it can
pass through HTTP-compliant infrastructure, taking benefit of everything that
is already deployed. Some non-HTTP compliant components will probably need to
be fixed on the server-side but that's much less work than starting something
new from scratch.

> Anyway, as much as this is interesting, it has no impact on the

Yes it has because you *have to* talk HTTP/1.1 over the wire.

> POST, regarding harming intermediates.

did not catch that one either :-/

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 21:16:15 +0100

On Fri, Feb 25, 2011 at 03:10:01PM -0500, John Tamplin wrote:
> On Fri, Feb 25, 2011 at 3:06 PM, Willy Tarreau <[email-masked]> wrote:
> 
> No, not actually a POST request, but just changing the framing of
> client->server frames to prevent transparent intermediaries from being
> confused.  The server could still initiate frames as it does in -05.

Do you mean just inserting a fake POST request *after the handshake* and
*before* the framing ? That's been proposed several times (and not only
POST, also other invalid requests as well as Dave Cridland's CONNECT
proposal). That was never broadly accepted and the most scared ones
were not convinced about the inability for intermediaries to skip over
those requests and resync with whatever would look like the next request.

Regards,
Willy

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 15:10:01 -0500

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 15:20:18 -0500

> those requests and resync with whatever would look like the next request.

I believe he means to insert the fake POST request before (or rather as part
of) each client->server WebSocket frame.  Since the POST request would
include the WS frame as part of the POST contents, intermediaries that were
interpreting the POST message as real would skip over any
attacker-controlled bytes.

So, I believe it would solve the problem just fine, but personally I think
the overhead is far too high.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 21:31:37 +0100

> the overhead is far too high.

If you want my opinion, it solves the problem just as fine as not masking
at all, but the reasons why we got masking is that some of us wanted to
ensure that no request could be found wherever in the stream. I don't see
that the fake request as a big overhead, however it just pushes the issue
slightly further. It's supposed to force deeply broken intermediaries to
close, while we have no idea whether they'll simply parse that as a valid
request and go to the next byte to find a request again. It's not easy to
make a request appear as invalid to everyone. But well, let's not reopen
this smelly can of worms...

Anyway, thanks John for having enlightened me on Silvio's goal :-)

Regards,
Willy

---

From: Silvio Ventres <[email-masked]>
Date: Fri, 25 Feb 2011 22:37:26 +0200

> request and go to the next byte to find a request again. It's not easy to

If they find the request again, they could do the same with a regular HTTP POST.
Which means WS is "safe" as it got to the same level of security as HTTP POST.

--
 silvio

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 21:55:05 +0100

> If they find the request again, they could do the same with a regular HTTP POST.

Yes but after a regular HTTP POST, you have a valid HTTP request. Here after
the WS handshake we don't have HTTP anymore. So if a broken intermediary were
to unexpectedly look for a request and would find yours, after processing it
as a valid request, it would expect another one and process the WS stream as
a next request. We don't fix the issue, we're just shifting it by one request.

As Dave Cridland told us, we can't know how many intermediaries would
correctly block on such random requests, the only thing we know from Adam's
experiment is that less than among 50000 CONNECT requests to invalid hosts,
none of them managed to correclty process a second request after this one.
Still that's a small sample and that does not tell us what state the
connection is left in.

Regards,
Willy

---

From: John Tamplin <[email-masked]>
Date: Fri, 25 Feb 2011 15:58:41 -0500

> a next request. We don't fix the issue, we're just shifting it by one

I assume in his proposal after the HTTP POST would be another valid HTTP
request (ie, the HTTP POST for the next frame) since *every* client->server
WebSocket frame has a fake POST request wrapping it.  So, a transparent
intermediary would be no more likely to treat attacker controlled data in
the POST body as an HTTP request than it is today.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 25 Feb 2011 22:35:34 +0100

> the POST body as an HTTP request than it is today.

Ah so the POST request would also contain a content-length to cover the
framed data then ? This is different from what I understood (having a
single fake request after the handshake).

Silvio's explanation was so vague that multiple interpretations were
possible, but I think yours is the right one.

And then I agree with you that it represents a huge overhead ! At
least having a single fake chunked encoded POST with chunk size in
front of each frame would be much cheaper.  Still, those implementations
won't allow a server to respond if the intermediary thinks it's processing
a real POST request. It won't process a new one until it gets a response,
sometimes will not allow the server to respond until everything is sent,
and most of the time will return a "502 bad gateway" when the server
responds with valid WS framing that does not look like HTTP/1.1.

We're better off with trying to make such faulty intermediaries fast
fail so that alternative schemes can be used without noticeable delay.

Regards,
Willy

---

From: Silvio Ventres <[email-masked]>
Date: Sat, 26 Feb 2011 00:39:32 +0200

> responds with valid WS framing that does not look like HTTP/1.1.

If some buggy intermediary would think that it processes a POST request,
and unable to open tunnel for WS during the GET/Upgrade stage,
such an intermediary would probably be incapable of tunelling the WS protocol
in any case, so the earlier such fault is found, the better.

--
 silvio