Subject: Re: [hybi] #1: HTTP Compliance

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 05:10:09 -0400

An HTTP proxy that is compatible with draft/proposal #76 (the contentious
one that passes 8 extra characters after the HTTP header without a
Content-Length header), would need a specific code path for the WebSocket
upgrade request in order to pass 8, and only 8, extra characters
(concensus is that passing an arbitrary content length would be a security
hole).

Isn't it the antithesis of the 80/20 rule and interoperability to require
every HTTP proxy in the world to have specific code path for every
possible future HTTP Upgrade protocol that comes along?

Whereas, if we pass those extra data after the HTTP 101 server response,
then HTTP proxies only need to support a single code path for all future
compliant Upgrade extensions.

In the former case, proxy code has to be changed every time someone
invents a new non-compliant Upgrade extension protocol. In the latter
case, proxy code has to be changed one time to correctly support the
generic Upgrade and never again.

I assume from what I have read on this list since July, that Ian Hickson
felt the higher priority interoperability issue was to minimize false
positives and thus maximize early proxy failure detection, for proxies
that had not been specifically coded to be compatible with WebSockets
specifically. Agreed, but not at the interoperability cost of requiring
every proxy in the world to be recoded with a specific code path for every
new Upgrade extension protocol. And the incompatible 8 extra characters
doesn't gain any significant DETERMINISTIC benefit against false positive
behavior (because networks are not DETERMINISTIC but rather stochastic). 
Am I missing something? Was Ian thinking that the 8 extra characters would
become adopted by other future Upgrade extension protocols and thus
minimize proxy code path changes proliferation? I am just trying to
understand what basis in logic Ian has for saying he is for 80/20 rule and
interoperability?  I assume he geniunely believes that, so I want to know
what I a missing that is inside his head or not evident to me in what I
read so far?

I intend no animosity. I am really just trying to understand, so as to
make sure I haven't missed something.

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 15 Aug 2010 12:07:17 +0200

> possible future HTTP Upgrade protocol that comes along?

it's even worse than that, it means that such a reverse-proxy, when shared
between HTTP and Websocket, will have to accept to blindly forward those 8 
bytes everytime someones *pretends* to talk websocket, without the server's
consent.

Since reverse-proxies are used everywhere (in IDS, HTTP-aware firewalls,
caches, load balancers, etc...), it is not even advisable to have per-host
or per-IP rules considering the large number of hosts that can be installed
behind any of them.

So it is very important that if we go the HTTP way, we respect the protocol's
semantics.

In fact, the protocol can be made HTTP compliant by switching two lines in
the description of the server's behaviour : the server just has to send the
101 *BEFORE* it reads the /key3/ parameter, and we're done.

Regards,
Willy

---

From: gustav trede <[email-masked]>
Date: Sun, 15 Aug 2010 12:24:38 +0200

> 101 *BEFORE* it reads the /key3/ parameter, and we're done.
So how come that several technical unsound ideas that more or less originate
from one person is ending up in the spec proposal in the first place ?.

regards
  gustav

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 15 Aug 2010 12:50:45 +0200

> from one person is ending up in the spec proposal in the first place ?.

You can't blame one single person for proposing something that does not
happen to work in some circumstances. If everything was that simple, the
proposal would already have become a spec. One of the goal of the single
person was to ensure that we'd get quick failures through incompatible
intermediaries. Achieving this means we may play on the border line of
the protocol. As long as no problem is identified, there's nothing wrong.
It's once the problem is identified that we must fix it and get back to
the compliant side of the protocol.

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 06:53:11 -0400

> bytes everytime someones *pretends* to talk websocket, without the

I had read that point in past discussion.  I would be tempted to assume 8
characters is relatively harmless security hole (as compared to passing an
arbitrarily unexpected content length, which is what would be required to
have one code path for all proxies for all possible Upgrade extensions
that expect such special treatment), but I will grant you that even 1
unexpected bit is a potential security hole.

> or per-IP rules considering the large number of hosts that can be

Yeah I know this from experience.  I suppose I save the IP maybe for law
enforcement who want to trace logs back to the client, or for broad-stroke
geographic position.

Proxies are proliferating with wireless too.

It would be an exponentially increasing cost problem of interoperability
if every Upgrade protocol requires a new code path in the proxies.

Let me confess that I have only been studying the entire issue of Comet,
BOSH, WebSocket, and this mailing list, for a several hours.  I came to
this from Wikipedia, because I need to choose an architecture strategy for
a social network I am developing.  So take my comments as one perspective
of an outsider (but an outsider with considerable experience in web
development and successful commercial software both pre-internet and
post).  I am not a networking expert, but I do think a lot about
interoperability in the real world and also in terms of programming
language design.

---

From: gustav trede <[email-masked]>
Date: Sun, 15 Aug 2010 12:55:32 +0200

On 15 August 2010 12:50, Willy Tarreau <[email-masked]> wrote:

> Its not the person, its the process of how ideas get formalized into the
spec i was asking about.
Discussions like the one going on now regarding a serious problem, is
perhaps something that should happen before there is a magical decision to
move an idea into the spec proposal ?.

regards
  gustav

---

From: Daniel Stenberg <[email-masked]>
Date: Sun, 15 Aug 2010 13:08:33 +0200 (CEST)

On Sun, 15 Aug 2010, gustav trede wrote:

> spec i was asking about. Discussions like the one going on now regarding a 
> serious problem, is perhaps something that should happen before there is a 
> magical decision to move an idea into the spec proposal ?.

This particular detail you're asking about was added to the spec before it was 
handed over to IETF to work on, so it is an artifact of the previous process 
not the current one. (And it has been argued about a lot on this list ever 
since.)

The current discussion is a lot about coming up with a (modified) way that can 
get a consensus from this group.

I don't think its very productive to dig a lot more into that, but to look 
forward and suggest what we should do to advance Websockets into a fine spec.

-- 

  / daniel.haxx.se

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 15 Aug 2010 14:26:48 +0200

> unexpected bit is a potential security hole.

Yes, even one byte is enough because it can be used to change the second
request's method. For instance, let's say you have a reverse-proxy somewhere
which analyses POST requests to validate the body, but which ignores all
other methods. You could send it a first request with Upgrade: Websocket,
with /key3/ being "\r\n\r\n\r\nPO", then a second request with a method
called "ST" which it would ignore. The end server would then ignore the
Upgrade request because it doesn't speak WebSocket, then would skip the
empty lines and get a perfectly valid POST request which has not been
checked by the reverse-proxy. This issue causes an infinite amount of
similar class of problems. This is closely related to HTTP request
smuggling.

What is important is that if we make the websocket handshake pass through
HTTP components, it must comply with the protocol. HTTP is clear on that:
the protocol has switched *only once* the 101 has been seen. So a reverse
proxy must not pass those 8 bytes before seeing the 101.

> if every Upgrade protocol requires a new code path in the proxies.

And it would not make sense at all since this would not be HTTP anymore.

> interoperability in the real world and also in terms of programming

You're welcome, anyone with experience or strong in-field requirements can
share some knowledge and help improve the design.

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 15 Aug 2010 14:31:08 +0200

> move an idea into the spec proposal ?.

It has happened, I've personally ben raising several HTTP incompatibilities
since January. Things are slowly making progress but there are security and
performance trade-offs to take into account, which don't make it easy to get
something which satisfies all needs.

What's important is that those issues don't get lost. Sal entered them into
the bug tracker.

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 09:06:56 -0400

> similar class of problems. This is closely related to HTTP request

I felt compelled to add my (what I feel is a more convincing) point to the
mix, because the security argument against the extra 8 bytes appears to be
weaker, because HTTP request smuggling already exists.

http://www.owasp.org/index.php/HTTP_Request_Smuggling

For as long as servers are free to have variable
interpretations/implementations of HTTP (within the broad constrainsts of
the spec), then HTTP request smuggling will exist.  And since I understand
that variable interpretationsinterpretations/implementations is critical
to HTTP robustness, then I conclude that "security" at the *transparent*
proxy is an oxymoron.  A *transparent* proxy should never be used for
mission critical functions, instead only to achieve optional benefits such
as increases in performance due to caching.  An opaque proxy (e.g. the
Apache server handling my server code or the upstream proxy I configure
into my browser) is theoretically deterministic and thus can be configured
to be reliable with respect to the end implementation.

Thus I don't think the security argument against proposal 76 is as
convincing.  Feel free to correct me, as I am only using deductive logic
of what I have read today, and am not speaking as an expert in the field.

> proxy must not pass those 8 bytes before seeing the 101.

Otherwise we defacto change HTTP for every HTTP Upgrade protocal
extension.  We know we are changing HTTP, because the implementing server
or proxy has to implement a new code path for our Upgrade protocal
extension.

> And it would not make sense at all since this would not be HTTP anymore.

Agreed.  Ditto I wrote above.

[snip]

---

From: Julian Reschke <[email-masked]>
Date: Sun, 15 Aug 2010 15:25:14 +0200

On 15.08.2010 15:06, Shelby Moore wrote:
 > ...
 > ...

Request smuggling happens when an HTTP message is invalid (for instance, 
because of duplicate Content-Length headers). There is no requirement 
for "robustness" in the HTTP spec related to this (if you disagree, 
please be more specific...).

As a matter of fact, HTTPbis is going to require that recipients  reject 
messages with this kind of problem; see 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-11.html#rfc.section.3.3.p.7>.

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 15 Aug 2010 15:58:20 +0200

> http://www.owasp.org/index.php/HTTP_Request_Smuggling

It only exists when servers are not cautious enough with dangerous
inputs (eg: conflicting content-lengths). And it is a problem when
two implementations decide differently how to interprete the request
because the spec makes it ambiguous or does not cover the case. That's
been addressed in http-bis, and now that there is some guidance so that
everyone tries to do the same thing, we should not try to bring the
issue back with WebSocket !

> And since I understand
> proxy is an oxymoron.

I don't agree. First we're not talking about "transparent proxies" but
reverse-proxies, that act as the server when seen by the client. There
are many of them around you without you ever noticing. That makes them
"transparent". They're generally quite robust, and most often they bring
the security themselves because their HTTP stack is well polished and
strict enough to protect the servers from issues that could result from
poor implementation in the application.

> A *transparent* proxy should never be used for
> mission critical functions,

It's quite the opposite. The more mission critical, the more of them you
find because they bring protocol validation, security, filtering, load
balancing, surge protection, content switching, etc... basically things
you don't need to show your photos to your family, but that are needed
everywhere applications are setup to serve tens to hundreds of thousands
of concurrent clients.

In fact, the unreliable components are mainly those which work on string
matching in packets. Those do not rely on proxies and don't have a very
robust state machine. They can easily be fooled in many ways with hand
crafted requests. Some of them will already not be able to process the
protocol upgrade and will likely hang or reset the connection.

> of what I have read today, and am not speaking as an expert in the field.

It's not the security argument against draft 76. It's that draft 76 is
not HTTP compliant and breaks with HTTP-compliant reverse proxies. The
only dirty hack that could be imagined to make those reverse-proxies
support it would not be acceptable in those reverse proxies because it's
incompatible with HTTP and it would break their security.

> or proxy has to implement a new code path for our Upgrade protocal

No, as explained above, we must not do that because it opens a new can
of holes. HTTP requires too much interoperability, it must not to be
cross-dressed for every protocol pretending to rely on it.

> Agreed.  Ditto I wrote above.

I think we agree, it's just that you seem to be dismissing the security
impacts of changing the way the HTTP Upgrade mechanism works, and I'd like
to ensure that the issues are clear.

Regards,
Willy

---

From: S Moonesamy <[email-masked]>
Date: Sun, 15 Aug 2010 07:26:47 -0700

At 03:24 15-08-10, gustav trede wrote:
>So how come that several technical unsound ideas that more or less 
>originate from one person is ending up in the spec proposal in the 
>first place ?.

draft-ietf-hybi-thewebsocketprotocol-00 was used as an input document 
for the HyBi WG.  As it is an Internet-Draft, it is work in 
progress.  In other words, it is subject to change.

There will be several revisions of the Internet-Draft before it is 
published as a RFC.  The Intended Status of the 
draft-ietf-hybi-thewebsocketprotocol-00 is Proposed Standard.  The 
Internet-Draft goes through this Working Group Last Call where it has 
to gain the consensus.  It then goes through an IETF-wide Last Call 
where it has to gain the consensus of the IETF Community.  The 
document will also be reviewed by Gen-ART and the Security 
Directorate before the IESG Evaluation.

At 05:27 15-08-10, Shelby Moore wrote:
>Apologies but from the perspective of an outsider, I wasn't clear that all
>of the community had consented to the wisdom and I am hoping to sway many
>of the rest of them, so that the concensus is not oppression (51% forcing
>their will on 49%).  I have no idea what percentage was still on the
>fence, because lurkers aren't counted.

Consensus is not 51% for and 49% against.  "Consensus is a way for 
people to make decisions and generate commitment to those 
decisions.  Coming to decisions by consensus is difficult and 
time-consuming.  It's a much broader and more flexible process than 
simply picking a position and seeing if everybody agrees.  The 
process cannot be reduced to an algorithm; consensus is complex, 
human and messy" {1].

Regards,
S. Moonesamy
HyBi WG Secretary

1. Acknowledgements to Lisa Dusseault

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 10:55:28 -0400

Hi again all, please do not take my reply as combative.  It is late here
and I am trying to wrap this up, and I am sure I have not taken enough
time to compose this reply with the degree of care that I would like.

> <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-11.html#rfc.section.3.3.p.7>.

The HTTP header is exposed to millions of naive programmers through
popular and easy means, e.g. PHP header(), thus user agents that want to
be robust on the WWW have no choice (the choice of failing for popular
content is not a choice). The compatibility modes for HTML rendering in
browsers is another example. A specification is only a request, the actual
standard is what happens in the market place. Given a sufficient
participation (sample), entropy will yield a bell curve of compliance,
with the center of the curve being imperfectly compliant (average IQ of
implementor) and one tail being more so and the other side less so.

Security that depends on specification is not very strong security because
it is impractical to quantify the determinism (you would have to compile
statistics in the market of implementations in all anticipated scenarios,
i.e. massive regression).  Stronger security is a deterministic body of
code that can not be subverted by any anticipated outlier (entropy or
Murphy's Law).  Security is never absolute, because nothing is every 100%
deterministic (Shannon-Nyquist theorem guarantees this because nothing can
be sampled/measured/contemplated with infinite precision or for infinite
time/experience).

On top of this, it is my understanding (incorrect?) that HTTP itself
allows for varying interpretations that might someday prove to be
exploitable. Off top of my head, legacy support for the caching changes
since early version. I suspect there are many such examples that I am not
aware of or haven't been exploited yet. My reasonable suspicion derives
from the differences between what the spec says that user agents are
required to do and what they are allowed to do. And because the various
syntax elements of the specification are composeable in probably way more
permutations that any human has ever wrapped inside their head. I wish I
could present a better example but I am not much of a hacker nor an HTTP
expert.

The following example of a "claimed" mistake in HTML5 (didn't find time to
email the HTML5 list yet) with respect to HTTP caching is an example of
entropy in the market versus the perfection anticipated in the HTTP spec:

https://bugzilla.mozilla.org/show_bug.cgi?id=563439#c8

So I guess what I am saying is that security becomes weaker and weaker
proposition the more elements relied upon are not under your control
and/or reasonably deterministically anticipated.  For example, the more
interfaces you proliferate under your control (that includes permutations
of composeable elements), the weaker your security due the inability to
anticipate all the scenarios that nature will generate.  So I prioritize
security the closer it is the inner core and don't promise security across
the world of proxies, when I am really need to admit I am just hoping.

> everyone tries to do the same thing,

I predict you will find that the exponential function runs away faster
than the perfection can fix itself.  Allocating human capital is very
critical function.  Standards that do it well are popular.  The others die
in the market.  We need to be realistic about the fact of entropy.

> we should not try to bring the

Agreed lets not add exploits, especially when it doesn't gain us anything,
and it has huge implementation cost also.

> reverse-proxies, that act as the server when seen by the client.

I thought transparent meant that the end point user agent was not aware of
them.  That is all I needed for my point.

> There

I can see that, if they have enough experience in the market place to know
all the possible scenarios.  But isn't it a huge effort, because a new
application comes along and breaks them, like WebSocket's proposed
security hole?  My point is that it isn't a very strong argument to argue
that the applications must be careful not to break the security of the
middle ware, when the middleware has to deal with entropy any way.  Again
it about 80/20 rule and allocating human capital efficiently.

Joke: Well I guess it is good for jobs, so is giving people spoons instead
of shovels.

> everywhere applications are setup to serve tens to hundreds of thousands

Won't we find that most all of those are optional and supplemental
technologies that fallback to something other than total failure for the
marjority of the market even when some specification is violated by the
market?

> protocol upgrade and will likely hang or reset the connection.

I don't know much about this but to the extent the middleware is parsing
the application data in transist and acting upon it outside of any
deterministic specification, we are going to have problem with any HTTP
Upgrade extension and I assume that is what the encryption solves?  I was
trying to get into that topic and to compare on tradeoff of using BOSH and
more market standard HTTP.

> incompatible with HTTP and it would break their security.

Agreed. I was just trying to present what I thought was an even stronger
reason. If the HTTP proxies can work around all kinds of market
application errors, they could support WebSocket proposal 76.  The HTTP
proxies that allow widest interoperability on the WWW, thus support HTTP
request smuggling now (they allow the two Content-length headers to pass
through).

In short, security and features and time are in tension against each
other.  There will always be some tradeoff.  At some points along that
solution space, the security argument is weaker than other priorities.  I
felt by presenting the point that did, the case against proposal 76 was
even more slamdunk, in order to persuade other fence sitters.

> cross-dressed for every protocol pretending to rely on it.

Agreed.  That was my point too.

> to ensure that the issues are clear.

I am not totally dismissing the security impacts, I am only trying to
point out what the opposing view might say about the priority of "perfect
security in the real world".  I presented another point that we both agree
on, and I don't the opposing camp can disagree that it is a huge cost. The
security hole is not really adding any significant cost that doesn't
already exist (notwithstanding the hope to fix it in HTTPbis).

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 15 Aug 2010 17:49:10 +0200

> time to compose this reply with the degree of care that I would like.

OK.

> content is not a choice).

We're not talking about failing for popular contents. Quite the opposite in
fact, we're talking about remaining compatible with clean implementations
so that we're not causing them to fail on popular contents.

HTTP is a very widely deployed protocol and despite the numerous
implementations, all of them are very much interoperable. One of the
reasons is that most mistakes related to the messaging semantics cause
the other side either to fall back to the lowest level compatibility
mode (eg: conenction: close) or to fail (400 bad request, 502 bad
gateway, etc...). Thus, while you may find a lot of buggy applications,
they still have their message framing correct (or at least correct
enough to reliably be transported).

> it is impractical to quantify the determinism

No, that's where you are mistaken in my opinion. Security does not depend
on the spec, but the spec dictates what to do to avoid known security issues.
For instance, there have been some add-ons due to the discovery of HTTP
request smuggling attacks (cf Julian's link above).

> (you would have to compile
> i.e. massive regression). 

You have to consider the fact that the 4th RFC is being written, and its main
goal is not to specify how things should be, but what is currently being done
and how to interoperate with existing components. So there is no reason for
regressions, quite the opposite, it could help various existing actors quickly
become compatible with other implementations they're not yet aware of.

> exploitable.

That's what is currently being addressed. If you're aware of some
vulnerabilities in some implementations that are only caused by a
different interpretation of the spec, then you should really join
the http-bis WG and report the issue so that it can be considered.

> My reasonable suspicion derives
> permutations that any human has ever wrapped inside their head.

I really invite you to read the link above. The whole p1-messaging and
p2-semantics are very interesting to read. You'll find how accurate they
are on many aspects, and very little is left to chance.

> than the perfection can fix itself.

There's no seek for perfection here. Just to avoid known bad patterns.

> security hole?

No, applications don't break implementations, so applications even if
poorly designed, don't introduce security holes into those components.
What introduces security holes is poor implementations. Trying to modify
a clean implementation to add support for an incompatible application may
lead to a poor implementation which is sensible to security holes. But no
application will cause security holes by itself.

> My point is that it isn't a very strong argument to argue
> middle ware, when the middleware has to deal with entropy any way.

No, let me insist. The middleware *is not* sensible to such issues and
tries hard not to be. However, if draft-76 were to be ratified, all
implementors of intermediaries would have to choose between being secure
and HTTP compliant, or being compatible with WebSocket. And *that* is not
acceptable.

> marjority of the market even when some specification is violated by the

Interestingly, with highly interoperable protocols such as HTTP, the
market cannot really violate the basic specs, otherwise the components
simply don't work at all and cannot sell. If you're an ISP and you buy
a proxy-cache that does not correctly parse content-length, you will
probably not ask all your clients to switch to a patched browser, nor
will you contact all the sites they're accessing to ask them to change
their implementation. No. You will simply put that buggy component to
the trash and buy another one from a different vendor.

> Upgrade extension and I assume that is what the encryption solves?

it's not the "encryption" by itself, but the fact of relying on something
non-HTTP. But it also comes with many new issues which are already addressed
by HTTP : no more session stickiness, no more accessibility from the inside
of most enterprises, no more virtual hosting, etc... So nothing is completely
white or black.

> application errors, they could support WebSocket proposal 76.

No, because doing this would break all other usages. To give you an idea,
think about a reverse proxy in front of an infrastructure hosting 100000
domains, most of which run only HTTP, and many of which also run WebSocket.
You have to change the way HTTP works on this reverse-proxy just in order 
to support the WebSocket stack on some of them. And with such numbers, it's
not a solution to think about configuring all of them by hand.

> The HTTP

No. Those which allow request smuggling are not the most interoperable
because if they randomly consider one of the two CL headers, half of the
times they'll pick the wrong one. The most interoperable ones are the
ones that fix or reject the wrong messages.

> even more slamdunk, in order to persuade other fence sitters.

Well, once again, my point is that while security around the new protocol
may not be the most important argument, not forcing reliable and secure
components to become unreliable an insecure for all existing applications
is a very important argument.

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 13:46:20 -0400

This is a horrible reply, sorry I am cross-eyed at 1:45am...just let me
punt after this please...

> We're not talking about failing for popular contents.

We would be if duplicate Content-length was a popular content error.  I am
thinking in terms of non-obvious cases that you are not aware of yet. 
"clean" is hindsight not foresight.  Security is a constant battle, it is
never won, nor 100% secure.

What I meant is that the only way to become more secure is to limit
features (so that we limit the variables/permutations of the unknown
attacks and non-obvious non-compliance).  Limiting features is not
popular.

It is important to admit that even 1 security hole is insecure.  Relative
security is hard to quantify, because the next security attack may harm
more than the prior one or the prior 99 of them.  The harm of the security
attack does not diminish with the duration of prior regression.

> Quite the opposite
> in

We don't know about security attacks that haven't happened yet, but are
already available in that "clean" market place.  That very obvious
duplicate Content-length headers is just being dealt with, makes me
reasonably wary of what is lurking.

> implementations, all of them are very much interoperable.

Agreed that is strong point, that widescale use and open source code
trends to illuminate security risks sooner (more regression in the market
place).

Open source helps too because hackers can identify vulnerabilities faster.

So the attacks may become less frequent, but it does not guarantee that
when an attack does eventually happen, that it won't be less harmful.

> One of the

Yeah the most obvious compliance is in center of bell curve.  That was my
point.

> on the spec, but the spec dictates what to do to avoid known security

If security depends on compliance and the the market has non-compliance,
then there is failure of security.  Worse is when we don't realize the
market is non-compliant under after the security breach.  And that will be
for future cases much less obvious than the duplicate Content-length
headers.

> request smuggling attacks (cf Julian's link above).

And for the future attacks discovered some more years later?  All during
that time there will be false sense of security.

> become compatible with other implementations they're not yet aware of.

Great but that doesn't negate my points.

> the http-bis WG and report the issue so that it can be considered.

But that still wouldn't negate my points.  There is always another
unforeseen attack around the corner.

> are on many aspects, and very little is left to chance.

I will try to find time to read it.  Thank you.

> There's no seek for perfection here. Just to avoid known bad patterns.

No argument against that.  I am just saying that avoiding bad SECURITY
patterns when we know from nature there will be some in our design any way
that we are not aware of, may not be the strongest argument in terms of
prioritizing with design.  Because security is a very difficult
proposition especially as we add features in finite time.

Here is the big epipheny.  Features only have to work how we expect them
to for the cases that we expect and are interested in.  But security has
to work 100% of the time for all possible permutations of those
imperfectly design and implemented features, or we can lose everything.

"improving security" is mostly an lie to ourselves.  We think it is
improved, until we suffer a catastrophic failure and realize it isn't. 
Insurance is a big fraud of mankind (and yes this is a technical response,
although I guess to bizarre for most people to agree and I don't really
have time to expound and I think it is not worth it as we have only a
minor disagreement on importance of security, not on the concensus to
require HTTP Compliance):

http://www.marketoracle.co.uk/Article21650.html
http://www.marketoracle.co.uk/Article21864.html#comment93787

> No, applications don't break implementations,

Applications are implementations.

> so applications even if

Of course they do, as they are part of the overall system implementation.

> What introduces security holes is poor implementations.

Agreed.  And specifications can not stop that, because specifications
actually introduce features.

> Trying to modify
> a clean implementation

There is no perfectly clean implementation of anything in the world.

> to add support for an incompatible application may
> lead to a poor implementation which is sensible to security holes.

Agreed, we shouldn't do it, given no overriding priority.  But if there
was sufficient priority, I have argued that the security argument could
possibly be weaker.

> But no

No just as no server implementation will cause security breach by itself.
They interact and errors as side-effects of feature in each interact. 
These interactions present security holes sometimes.

> and HTTP compliant,

You don't know that they are perfectly "secure and HTTP compliant".  You
don't even know all the ways that they are not.  No one can know that.

> or being compatible with WebSocket. And *that* is not
> acceptable.
>
> simply don't work at all and cannot sell.

You don't know that.  It is not "breaking the spec", rather it is doing
things that work within the spec (at least how the market has implemented
the spec) and yet still open security attacks (even ones we are not aware
of yet).

> If you're an ISP and you buy

I am not talking about popular content that fails in the market place.  It
wouldn't be popular.  That would be an oxymoron.  I am contemplating
popular content that has security attacks that you and others are not
aware of.

> of most enterprises, no more virtual hosting, etc... So nothing is

Yeah it really looks like a mess to me.  Seems using a dedicated port
would be a lot cleaner.  Sometimes entropy stacks up against you, and you
need to start clean.  You can only put so many bandaids or spaghetti on a
paradigm before chaos defeats us.  Everything in the universe is decaying,
it is the 1856 law of thermodynamics.

> not a solution to think about configuring all of them by hand.

But if 99,900 of them work fine when passing the 8 extra characters
through, then it was a nice 80/20 hack if there was any benefit to the 8
extra chars (but I agree there isn't any benefit).  And to be honest,
everything we do is a hack of varying degree.  That is why the simplest
designs win, must less collateral damage.

> ones that fix or reject the wrong messages.

If the end server does not fix or reject the message, the most
interoperable proxy in that case of two Content-length would be the one
that acted as it if did not exist, otherwise the client is going to get a
different result when it connects to the end server without going through
your proxy.

> is a very important argument.

Qualifying "important" is the issue we were discussing.  In any case, I
will give you the last word.  Because my replies will be getting too far
off topic and I don't want to burden you all with that.

I do appreciate your debate and we could continue it in private if you
want after you take last public word.

Maybe we can move on to figuring out how to move the WebSockets forward. 
I think we already agree with the concensus on HTTP Compliance.

I have posted too much, exceeded my worthwhile time limit.  I don't think
it is that important to defend my ideas on security, this isn't a security
working group and besides I am not trying to become an expert in security.

I simply want a reliable and easy way to do server push.  I would also
like a reliable and easy P2P channel (but that is off topic too).

Wish it wasn't so hard to achieve.  Entropy is really kicking our butt. 
Maybe we need a new port!!  For P2P too!! Time to break clean.

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 15 Aug 2010 22:40:23 +0200

> punt after this please...

Well, this thread is becoming really too much off-topic for this list
as it's discussing general points about HTTP only. You should probably
joind the http-bis list. Also, you should take some sleep, this is not
a battle, your arguments can wait for the next day.

Still, a few short responses below to address some of your concerns :

> We would be if duplicate Content-length was a popular content error.

I've already seen duplicate CL headers in network captures, but those were
what it means -- duplicate --. I suggested on the http-bis list to relax
the check to allow at least that case (all CL values must be equal), and
some considered it "acceptable". That's what I did for haproxy. Mind you,
after tens of billions of requests passing through, I've yet to see one
which fails ;-) In fact, if some application were to send two distinct
content lengths, there would be no chance for it to work on the Net
because of the variety of client and intermediate behaviours.

> attacks and non-obvious non-compliance).  Limiting features is not

Not always. If you don't provide enough building blocks, people try to
misuse existing ones in a variety of dangerous ways, then it's often
impossible to later implement controls for correctness. I appreciate
it a lot that this WG tries to define a rich set of building blocks
that applications will be able to use in a safe way. It will surely
result in more secure implementations.

> that time there will be false sense of security.

No, you're painting everything in black. HTTP is everywhere but does not do
everything. It's a transport protocol, not much more. Security issues with
HTTP can fall into a small set of categories :
  - make messages appear different to different agents (smuggling, encoding).
    Generally used to bypass filtering.
    => not due to the protocol by itself but to the way it is implemented
       and how corner cases are implemented ;

  - cache corruption => injection of bad data into caches that many people
    will be able to retrieve. Generally caused by bad implementations or
    poor configurations.

  - server information leak => retrieve data from servers or caches that
    should not have been possible with the credentials. Poor implementations
    or configs too (eg: "GET /..", proxy requests, TRACE and OPTIONS request,
    authentication of the connection instead of the request, ...)

  - user information theft => try to trick browsers into believing it's safe
    to send some credentials, cookies or any information, by XSS, CSRF, ...
    Related to a specific usage.

  - mass attacks => use the ability to direct a visitor's browser to any
    victim (CSRF, cross-protocol attacks, redirects, ...). Related to a
    specific usage too.

  - and obviously all types of denial of service that depend on specific
    implementations (eg: preallocation of content-length, improper session
    termination on error at some points, proxies looping over themselves,
    etc...).

I'm not saying that no other categories exist, but that HTTP is low-level
enough to provide limited support for building exploits. And it's not about
a "false sense of security", it's just that unplanned cases can fall in a
known column and be processed in a known manner.

The funny websocket handshake also tries to prevent some low-quality
implementations from falling into some of the categories above.

> proposition especially as we add features in finite time.

Exactly! And that's why I argued that WS should be HTTP compliant instead
of trying to immitate it and cover one by one a few of the identified
security risks.

(...)
> Applications are implementations.

Hey, they're free to break themselves, and after all that happens everyday,
that's called a bug. But I mean they won't break other components'
implementations. That's very important.

> actually introduce features.

Specifications tell implementations how to act on bad inputs and how to
react to unfriendly or unexpected external actions.

> don't even know all the ways that they are not.  No one can know that.

I disagree. While I know that most people don't test equipments before
deploying them, I can assure you that there are some places where every
bit of ARP, IP, TCP and HTTP behaviours are tested before the equipment
can be deployed, and what does not work as expected has to be worked around
or fixed by the vendor. Sometimes it can take more than one year to have
something fully validated. And when you know that your application firewall
is HTTP compliant with the Upgrade header, and you want it to behave that
way because it's safe, you certainly don't want it to suddenly be degraded
because some moron has added support for draft-76 into it !

> the spec) and yet still open security attacks (even ones we are not aware

The only part of the specs that can be silently violated is the control
of inputs. That's why you should run tests. How do your proxies behave
when facing a GET request with a content-length ? How do they behave
when the content-length is between 2^31 and 2^32-1, when it's larger
than 2^32, with multi-line headers, with trailers ? How do they resolve
multiple host headers, how do they resolve IP addresses larger than 2^32 ?
Does your cache change the size of the cached object if it sees a
content-length in the trailers, etc... Once you have established the
set of incompatibilities, it's up to you to decide whether there is any
risk in deploying that component or to try another one. But in this
respect, security and features are not different, you want something
that works. Spec indicate how to correctly implement something to be
safe and reliable, but implementors have the right to be on the unsafe
side. Still that does not compromise other components' own security.

> Seems using a dedicated port would be a lot cleaner.

Will not pass through many enterprise proxies and firewalls, and result
in lower adoption.

> > To give you an idea,
> extra chars (but I agree there isn't any benefit).

But it's not a matter of working fine. In fact, 100% of them will still
work fine *by default*. It's just that once you start weakening your
reverse-proxy to send those 8 bytes, you end up truncating your requests
at a different place on the reverse proxy than on the server, resulting
in the easy ability for attackers to send uncontrolled requests to all
those 100000 domains. And now it's quite hard to tell which of these
100000 domains won't be affected by the unexpectedly transformed requests.

> different result when it connects to the end server without going through

That's precisely why specs are important, and HTTP specs in particular
falls back to the lowest capable. If you can't parse the content-length
header, you have to fall back to the "connection: close" mode and only
have to process one request. This can be announced to the other side by
adding an explicit header, and the other side will not be fooled. That's
how haproxy worked for the first 8 years of its existence, without
particular problems. Once again, the HTTP specs are there to permit all
sorts of behaviours and ensure maximal interoperability.

OK, stopping here now, my code has gotten too much late :-/

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 17:34:22 -0400

Agreed not a battle. Any way my ideas about these more abstract technical
issues do not seem to receive much comprehension, because they fly in
opposition to human nature to want to be in control of something (even if
it is an illusion).

I will reply publicly because I just realized I can condense the security
compartmentalization debate to fundamental theorems of computer science.

> of inputs.

What about the permutations of side-effects of those HTTP requests?  You
can't possibly have run a regression test on every possible permutation in
that external state machine.

If you are saying that security attacks that manifest in the end
application state machines are not introduced by the features of HTTP spec
which enables them, then I disagree with you because of fundamental
theorems of computer science say that compartmentalization is just a
fantasy[2], and aliasing (Security) errors (bugs) get pushed into other
layers.  I explained it further in a rather bizzarre (embarrassing)
diatribe[1].

[1] http://lambda-the-ultimate.org/node/1277#comment-51742

[2] http://www.mail-archive.com/[email-masked]/msg66538.html

Fundamental theorems tell us that any membership rule for a set can not be
an absolute point, thus we should strive for typing and classing
architecture which is the most granular (these are my summaries):

* Russell's Paradox: there is no rule for a set that does not cause it to
contain itself, thus all sets are infinitely recursive.

* Liskov Substition Principle: it is an undecidable problem that subsets
inherit.

* Linsky Referencing: it is undecidable what something is when it is
described or perceived.

* Coase Theorem: there is no external reference point, any such barrier
will fail.

* Godel's Theorem: any formal theory, in which all arithmetic truths can
be proved, is inconsistent.

* 1856 Thermo Law: entire universe (a closed system, i.e. everything)
trends to maximum disorder.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 16 Aug 2010 00:05:30 +0200

> opposition to human nature to want to be in control of something (even if

No, it's just that you're discussing general security considerations on
this list which is specifically about implementing the WebSocket protocol
(whether over HTTP or not). It becomes way off-topic. And we can't discuss
HTTP's reliability and security here. At most we could discuss whether
we agree to use it or not, and how to use it. If we have a problem with
an aspect of HTTP that can become a risk, then it should be addressed on
the http-bis mailing list. HyBi should not try to address HTTP concerns
in its own corner. That's why your ideas don't receive much comprehension
here unfortunately.

Kind regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 18:17:38 -0400

> in its own corner. That's why your ideas don't receive much comprehension

Apologies for the misunderstanding, I meant "have not in past received
much comprehension". Agreed thank you, this is not the proper forum to aim
such off topic security discussion. Concensus on HTTP Upgrade was already
declared. You and I both amicably agree with it.

The new threads on TLS and specific port+tunnel are interesting, amicable
and productive.