Subject: [hybi] New port and Tunneling?

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 16:48:35 -0400

I googled this discussion list archive and did not find much on this.

Seems the great likelihood of messy soft failure and other a priori
unquantifiable complexity with reusing the port 80 or 443+TLS due to the
HTTP legacy on the network and/or complexity issues with HTTP Upgrade or
non-mainstream use of TLS.

Seems what we really need is a hard fail, so we can dynamically and
gracefully downgrade to long polling solutions (Comet/BOSH...).  K.I.S.S.

We are going to have to downgrade whether due to proxy soft failure or new
port tunnel failure, so why chose the higher complexity "root canal"
route?

Am I missing something or is the K.I.S.S.  way to get a hard fail is
tunnel over a new port?  Then we can get P2P WebSockets too (wow)!

http://samy.pl/pwnat/
http://mackys.livejournal.com/896428.html

FAQ at the above link seems to be pretty simple and deterministic in terms
of failure detection.

The man-in-middle vulnerability and the beneficial features of HTTP (e.g.
URL redirection, virtual hosting,...) could still be obtained by
handshaking orthogonally with standard HTTP GET.

Educate me please.  Certainly the solution can't be this simple correct? Why?

(4:45am and 45 years old, good nite)

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 15 Aug 2010 23:03:32 +0200

> Educate me please.  Certainly the solution can't be this simple correct? Why?

The specific port was discussed too. But it does not solve everything.
In your opinion, why do Comet/BOSH work over HTTP ? Precisely because
they can reach almost 100% of the visitors. In many (I mean *many*)
enterprises, here's what you have as options to access the outside :
  - port 80 via a proxy/cache/anti-virus/url-filtering
  - port 443 on a small hand-selected list of sites
  - nothing else

I've heard it's basically the same with mobile gateways at most operators.

And given the number of malware on the net, the trend is certainly not
going backwards. Thus, port 80 with HTTP has a major advantage over the
other alternatives. Right now the success rate of tests appears lower
with HTTP than with TLS, but in my opinion, it will increase once the
protocol is ratified, because proxy vendors will simply add support for
it in their proxies.

If you open a dedicated port, it's very interesting in order to reduce
the number of round trips, but you'll still miss a part of the population.

Also, TLS as proposed by Adam looks very promising. But it still has the
inconvenient that it will never be blindly opened in many enterprises due
to the impossibility to analyse it. And it does not easily allow server
stickiness for a user connecting to the WS port of the HTTP server he was
on.

However, having TLS as a complement of HTTP would be very nice and it
could probably make sense to try it by default.

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 18:01:48 -0400

> The specific port was discussed too. But it does not solve everything.

But no solution will solve everything.

Rather I am arguing that a hard fail at firewall tunnel attempt, is
superior to soft fail at unknown times in the corruption/delay of data by
the proxies or other complexity of trying to shoehorn a bi-directional
channel into a stack that isn't designed for it.

Once the specific port tunnel succeeds, we get rid of all the other
problems.  If it fails, we know it, and can gracefully degrade to the 100%
reachable Comet/BOSH options.

> they can reach almost 100% of the visitors.

Agreed.

> In many (I mean *many*)
> other alternatives.

Did you look at the links I provided?  AFAIU, the firewall/NAT would have
to block traceroute in order to stop this tunneling.  I suppose the
firewall could block the specific port (knowing that traceroute doesn't
normally run on that port)?  But why would the firewall want to
retroactively block our specific port, when malware wouldn't be able to
abuse that port+tunneling and more than malware could abuse port 80,
especially if there was already a rapid adoption of popular applications
that relied on the port staying open?

> Right now the success rate of tests appears lower

See my point above, maybe the part is not larger than what will be missed
with HTTP Upgrade, once the unknown soft fails are experience in the wild.
 There will need to be fallback to Comet/BOSH no matter what we do.  So
why not go for the simplest solution that gives us the most robust
channel?

> to the impossibility to analyse it.

The tunneled specific port can be analyzed.

> And it does not easily allow server

I don't understand.

> could probably make sense to try it by default.

How about TLS as a complement to specific port+tunnel?  Seems like a
better combination.

Hard fail -> TLS or Comet/BOSH (configurable choice, or let application
layer implement)

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 16 Aug 2010 00:19:22 +0200

On Sun, Aug 15, 2010 at 06:01:48PM -0400, Shelby Moore wrote:
> Did you look at the links I provided?

just quickly.

> AFAIU, the firewall/NAT would have
> to block traceroute in order to stop this tunneling.

Well, mind you that in many enterprises, firewalls don't block explicit
things, they're just open for explicit things. In almost all places I've
been, there is zero communication between the inside and the outside that
does not pass via proxies. So if the proxy refuses your request, you can't
pass. Then you can always try to fool the proxy using request smuggling,
but it does not always work ;-)

> I suppose the
> retroactively block our specific port,

It's just that it would never have been explicitly opened.

> when malware wouldn't be able to abuse that port+tunneling and more than
> malware could abuse port 80,

Right now malware already uses many evasion techniques to send information
outside and download updates.

> that relied on the port staying open?

Give me one port that is supposed to remain open and I install a remote
proxy on it in order to bypass all mandatory filtering that makes the
web boring (but safe) to use from the internal network.

> why not go for the simplest solution that gives us the most robust

The time to detect the failure can be huge. Via a proxy it's generally
fast, you get an immediate rejection. But via a blocked firewall, it
takes several seconds before you can decide that the port is closed.

> The tunneled specific port can be analyzed.

yes that's a good point.

> > stickiness for a user connecting to the WS port of the HTTP server he was

On the server side, you generally don't have one single server, in fact
you have a farm of servers. Sometimes just 2, sometimes 10, or even 100.
For every requests that come in, a server is selected to process the
request. Stickiness is what ensures that a user will have all of its
session processed by the same server in the farm. It is very important,
otherwise each request would go to a different server and the server
context would be lost between clicks.

When implementing WebSocket in web applications, you'll most likely
have to share contexts between the web server and the websocket server.
That means that most often you'll want them to run on the same server,
so you want to be able to associate a client's WebSocket connection
request with its HTTP counterpart, in order to direct him to the same
server. With an HTTP handshake, it's quite easy, just send the same
cookies. With a dedicated port, something has to be explicitly designed
in the new protocol to make this possible. TLS has the same issue.

> Hard fail -> TLS or Comet/BOSH (configurable choice, or let application

See above ;-)

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 19:03:05 -0400

> Well, mind you that in many enterprises,

You also wrote that TLS is likely to fail in many enterprises also.

And HTTP Upgrade has soft fail indeterminism "root canal" risk.  I
understand from archives that we really won't know until we put into the
wild, with all the stacks of transparent proxies mucking around in the
bi-directional packets. You fix that by going to TLS, but you just wrote
that TLS is also going to be blocked by many enterprises.

So I don't yet see that valid argument against port+tunnel for enterprises.

> firewalls don't block explicit
> things, they're just open for explicit things.

The links I provided state that firewalls are only typically closed for
incoming, not outgoing and incoming replies to outgoing. Maybe you are
thinking of mostly in the enterprise? The enterprise is much smaller use
case than home/SOHO/asian netcafe, and we don't have a good solution for
the enterprise other than to fall back to Comet/BOSH (standard HTTP to
avoid potential soft fail/false positive proxy hell).

> In almost all places I've

Fall back to Comet/BOSH, which we are going to have to for HTTP Upgrade on
port 80 or TLS on port 443 also.

> It's just that it would never have been explicitly opened.

Ditto above.

> outside and download updates.

The solution to that is run Deepfreeze, Rollback, Baseline Shield, or any
other means of restoring your computer to virgin state on each reboot.  I
am really surprised this isn't built into every operating system by
default.  If you go back to the security debate we just terminated in the
other thread, the use of firewalls is in direct defiance of the laws of
nature and computer science.

What will end up doing is killing all innovation.  It really sucks, and
especially at a time when we need to be expanding P2P in order to create
robust decentralized economic responses to the global socialism that is
going to render all of us penniless.  But P2P can't go any where because
put the security at the perimeter instead of at the ends.  Insane (not
you, the situation).

> web boring (but safe) to use from the internal network.

You might as well just hack the browser, so then you need to close port 80
too.  You see it is insane.  Security (insurance) is a fallacy that
results in abject poverty.  Rather bring the fence down and just reboot
your computer when you click a bad link and your computer starts
misbehaving.  Give the control back to the user.  End this slavery and
restore freedom.

> The time to detect the failure can be huge.

My other post earlier today I suggested to startup Comet/BOSH in parallel
and use them while you are awaiting the timeout, so no delay at all.

> Via a proxy it's generally
> fast, you get an immediate rejection.

I read in the archives of this list thatu can get a soft fail at any time
with corrupted or delayed data.  Isn't there concern about what
transparent proxies will do if look at the bi-directional data?  Okay
maybe TLS is immediate rejection with no potential soft fail/false
positive? But any way I think the startup delay is not an issue if you
start up Comet/BOSH in parallel.

> But via a blocked firewall, it

Agreed, but see my points above.

>> The tunneled specific port can be analyzed.

Thank you.

> in the new protocol to make this possible. TLS has the same issue.

Because the webpage from served from the same server?

Interesting point.

Might be easy to solve with the tunneling algorithm I shared, because you
could have each server emit a different packet signature to which the
client sent the same signature replies.

>> Hard fail -> TLS or Comet/BOSH (configurable choice, or let application

After replying to all your points, I repeat the question.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 16 Aug 2010 01:40:10 +0200

> You also wrote that TLS is likely to fail in many enterprises also.

exact.

> So I don't yet see that valid argument against port+tunnel for enterprises.

Once again, simply because it requires opening an uncontrolled port
to you-dont-know-where. Maybe it will succeed once it has lived 5 years
and vendors start building devices capable of following the framing and
kicking out unauthorized users. But in the mean time it won't get adopted
in enterprises (at least), and probably not at mobile phone operators that
don't want you to connect home via openvpn or SSH.

> thinking of mostly in the enterprise?

yes, that's what I'm saying from the beginning.

> The enterprise is much smaller use

The problem if we ignore the enterprise market is that site implementors
will have to continue to use Comet/BOSH just for this minority of users.
So why would they implement two solutions when the existing one is still
necesarry and already enough for the job ?

> port 80 or TLS on port 443 also.

Here it's different, it's keeping it for a very few years, the time needed
for buggy intermediaries to get fixed in next version. There's a big
difference between fixing a device which is not HTTP-compliant to make
it compliant, and implementing support for a fresh new protocol.

> other means of restoring your computer to virgin state on each reboot.

Fine if you use your PC for games and don't mind experiencing the same
day every day. But when you use it for work, you certainly appreciate
finding your last files when you boot it :-)

> too.

No, because when the browser can only get out via the proxy, it does not
have access to port 80 itself, otherwise it would be pointless to install
filtering devices !

In the past, I used to telnet over mandatory HTTP proxies using POST
requests, but controls have improved and the data does not flow back now
until the request message is sent and the response analysed.

> and use them while you are awaiting the timeout, so no delay at all.

Two competing implementations to maintain for the site owner.

> with corrupted or delayed data.

Yes, with transparent proxies. Explicit proxies have the cleanest method
(CONNECT). Some of them will still block due to port 80 not being allowed
with CONNECT though, which is where Adam's proposal may improve things a
bit. But this is just a matter of configuration. The desire to accept or
block such requests will be equal whatever the port, in both cases it's
uncontrolled bidirectional tunnel.

> Isn't there concern about what

Also it would be nice to avoid starting the two in parallel for every
connection. It will basically double the traffic on some sites, which
is not nice at all, resulting in them not implementing WS at all in the
end. We must keep in mind that the goal of the spec is to propose
something that people want to deploy, otherwise it's a failure.

> Because the webpage from served from the same server?

You want every part of your application to be able to access its context.
If the context is shared, any host can access it. If it's local (more
scalable), then only local servers may access it. So you have to ensure
that all requests from a given user as well as its WS connections can go
to the same server.

> client sent the same signature replies.

yes, possibly, though it still requires developping support for a new
protocol in existing components, while with HTTP at least there's nothing
to do (except sometimes fix a few ones).

> After replying to all your points, I repeat the question.

As I said, none of TLS and your specific port addresses the stickiness
requirements. WS/TLS + Comet/BOSH still requires support for two different
protocols on the server side. If I had the choice, I'd rather go for
WS/TLS with a quick fail to WS/HTTP.

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 19:42:23 -0400

> You might as well just hack the browser, so then you need to close
> port 80 too.

AFAIR, Skype also uses port 80 (remember an issue of it interfering with
XAMPP), so I don't understand what will stop malware from using port 80,
even without hacking the browser?

  Security (insurance) is a fallacy that
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>
>

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 20:24:44 -0400

> to you-dont-know-where.

I did not imply that the enterprise must open it, it can allow it to fail
and WebSocket can fall back to the another option, e.g. Comet/BOSH and
HTTP Upgrade on port 80 (since TLS won't work on Enterprise).

My point is that at least port+tunnel will work on the more widespread use
cases outside the enterprise, probably as often than HTTP Upgrade on port
80 will and with better hard fail and better performance (less quirks
after tunnel succeeds).

> Maybe it will succeed once it has lived 5 years

Afaik, we don't have that problem with USB broadband mobile convergence
here in Philippines.  They even open the Skype ports, probably because
people are not anal about it, they just want to innovate and do more with
the internet (while they blow by us in terms of usage and market
innovation).

> yes, that's what I'm saying from the beginning.

Lets think outside the Enterprise please, where the big prize is. 
Enterprise is going to always be a least common denominator, e.g.
Comet/BOSH perhaps.  Enterprise will die any way (the future is already
moving away from that slavery over here).

> necesarry and already enough for the job ?

Exactly.  We better focus on the useful use cases.  That is what I am been
trying to say from my earlier posts.  We need to focus on where this
advance can actually advance and be adopted in wide scale.  Focus, focus,
focus.   80/20 rule.

But I am not sure my characterization of the market is accurate.  Please
correct me if you any one knows for sure.

> it compliant, and implementing support for a fresh new protocol.

And the difference favors the latter?  The former may mean years of
gradually surfacing soft fail complexity hiccups, or soft TLS issues.  The
former is either hard success or a hard fail and deterministic downgrade
to status quo.

By "soft", I mean the failures will come in the market and at unexpected
junctures, perhaps even at random times and situations. Am I mistaken?

> finding your last files when you boot it :-)

Use an exception drive or folder.

When I installed Baseline Shield, I turned off my firewall and have never
looked back.  Works like a charm.  I can't understand why there isn't a
massive exodus from the current insanity.  Either I am missing something? 
Or others just haven't thought of it?

> have access to port 80 itself, otherwise it would be pointless to install

Okay I understand your point now which is that HTTP is filtered and
another port is not unless the proxy understands the spec.  But SSL is not
filterable because the proxy can't see the unencrypted data, so block all
secure ecommerce and personal privacy?

I think it is pointless to install filtering devices if you depend on them
for security.  Kills anonymity, privacy, thus free speech.  And especially
critical in this epoch ahead of us.

Also I expect that HTTP over port 80 will become so well attacked (from
the application state machines) that will have to close port 80
eventually.

It is the way insurance always ends, complete failure.  Every time in
history.

Rather just give the user a disk image reset button with an exception file
folder.  This is decentralized fencing and thus is free market.  The
centralized fence with no leaks eventually become a solid wall that
nothing can pass through.  Proxies should only be used to add capability,
not restrict it.  The entire model is wrong and we are going to suffer
horribly for it.

> Two competing implementations to maintain for the site owner.

They are going to have to do it any way, no matter what we do.

> uncontrolled bidirectional tunnel.

Thanks for confirming and qualifying.

> connection.

You only need to test failure once per client and server pair, with the
hard fail of port+tunnel.  Whereas with the confirmed soft fail issues
over HTTP ports, you will never know if you have confirmed success that
you can cache.

> It will basically double the traffic on some sites, which
> end.

Not all sites with have such a profile.  And ditto the point above that we
only need to succeed or fail once for port+tunnel.

Offer a configuration option to only start one type, and let the market
decide?

> We must keep in mind that the goal of the spec is to propose

How does this not support my logic so far?

> that all requests from a given user as well as its WS connections can go

See below...

> to do (except sometimes fix a few ones).

"nothing to do" is not the way I would characterize the mess ahead with
soft fail in HTTP.  Consider the actual cost in the market place.

> requirements.

I thought I did above?

> WS/TLS + Comet/BOSH still requires support for two different
> protocols on the server side.

We must have that fallback no matter what we do.  All options forward have
failure cases.

> If I had the choice, I'd rather go for

WS/HTTP can fail at random times after that quick fail, you still need
Comet/BOSH.

What advantages and disadvantages does WS/TLS offer over port+tunnel? 
Ditto WS/HTTP?

Maybe someone can make an ascii table summary.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 16 Aug 2010 10:36:32 +1000

> WS/HTTP can fail at random times after that quick fail, you still need

and Comet/BOSH can fail at random times.
All solutions need to handle the fact that the internet can fail silently.

I do not think it is possible to design a system that will always fast
fail, other than an explicit declaration by a participant that a
protocol is not supported.   It is impossible to design a handshake,
whose success will guarantee the delivery of a subsequent websocket
message .

The only real evidence of a protocol working is the protocol working.
The level of confidence in the protocol will be determined only by
mechanisms built into the protocol: messages, keep-alives, ping/pongs
and any extensions for acknowledgements etc.

There is no silver bullet.

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 20:51:25 -0400

> There is no silver bullet.

Strong point, but won't the WS/port+tunnel (over TCP or equivalent) only
fail-to-deliver and never corrupt, whereas WS/HTTP could corrupt the data
(and will not support some data types at all, need encryption)?

Also WS/HTTP is likely to fail more often during data transfer (after
initial handshake) given reasonable robust networks.

So they are not equivalent failure costs.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 16 Aug 2010 10:55:59 +1000

Shelby,

Maybe I've missed something in recent threads, but how is WS/HTTP more
vulnerable to data corruption?

cheers

---

From: "Shelby Moore" <[email-masked]>
Date: Sun, 15 Aug 2010 21:12:12 -0400

> vulnerable to data corruption?

Perhaps I misunderstood the issue.  Do some proxies monitor and filter the
data, and may make mistakes because they think they are looking at HTTP,
because they didn't properly recognize the Upgrade handshake?  And maybe
other scenarios?  I really don't know.  I don't have enough experience
with proxies.

Also I read that certain data patterns (e.g. unencoded binary) can not
traverse all proxies safely.  I realize perhaps we can avoid these known
patterns, but doesn't that imply there might also be some unknown
collisions?

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 16 Aug 2010 13:09:22 +1000

Shelby,

The concerns I've mostly seen discussed is for an intermediary to pass
the handshakes, but then not forward arbitrary data.

However,I guess it is possible for a really stupid proxy to inject
"HTTP headers" or similar into a websocket stream and corrupt data,
specially if the data looked like a HTTP message.    Perhaps that is
an argument for a simple checksum in the framing mechanism (previously
suggested and rejected because we are not trying to fix problems with
TCP/IP - but this is a different use-case).

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 16 Aug 2010 07:18:31 +0200

Hi Greg,

> TCP/IP - but this is a different use-case).

What could very well happen in fact is that an intermediary does not
know that "101" requires all the chain to switch to tunnel mode, and
that it then considers the first data from the client as a second
request. It will send return "HTTP/1.1 400 Bad Request" in the response,
which does not look like a WS frame at all. That's why I was advocating
(and Jamie too) for the use of "Connection: close" in the request
headers. At least the intermediaries will know that they must close
after the first request is done. That will also improve fast fail on
incompatible components, because we won't have to wait for sending data.

But I'm not worried for the rest. Having random headers inserted in the
101 response is not a problem at all. And I don't see why any data could
get corrupted at all. At worst we could imagine that some IDS/IPS will
sometimes believe they see a known bad signature and reset the
connection as they sometimes do on HTTPS traffic, but there are many
other reasons for connection that we have to deal with.

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 16 Aug 2010 01:22:49 -0400

> TCP/IP - but this is a different use-case).

I am not stating that will happen or at any significant frequency, I am
asking if it could be a problem? Why is the data currently limited to
UTF8?

Until evidence or expert experience is stated otherwise, my "Murphy's Law"
(entropy) intuition is that if we inject clear (not encrypted) text data
into HTTP channels, whose primary reason for existing is to be an
orgy/feast of filtering (refering to the security discussion today between
Willy and myself in this thread and the HTTP Compliance), that does not
look and act like HTTP that those networks have seen, then we are
prefering to risk a can-of-worms (indeterminacy). Although HTTP Upgrade is
part of the specification (since 1.1?), am I correct to understand (from a
comment made in this list's archives) that it has never been successfully
deployed to any significant scale?

And to the extent the above is a real issue, I had mentioned before in
other thread that I would also be ask if there is a possibility that
messages will silently fail to be delivered, so in that case TCP is not
working as expected because it is not a failure-to-deliver to the network
nodes, but rather a failure to forward data by the silent transparent
proxy. So if so, then you need handshaking because the receiver doesn't
know when data was sent. I had suggested that such handshaking could be
accomplished (with least bandwidth wasted) in messages received and in
keep-alive pings. But the more I think about it, if proxies might not
forward data and we want to assume we have the behavior of normal TCP
without proxies, then we will need to handshake on each sent message,
which means latency advantage is destroyed! That is major point, if true.
If proxies can eat the checksums (hashes), TCP efficiency is lost.

This is why I am arguing in this thread for a clean break to a specific
port+tunneling (aka WS/P+T), because it will give us a true TCP
semantics+efficiency (after the tunnel succeeds) and the only issues not
TCP recoverable will occur as a hard fail of the tunnel handshake, which
is cacheable result (time cost) because can only occur once per tuple
(client, server, DCHP lease, etc).

Ian Hickson wrote more than once last year in this list's archives "443 is
the interesting case", so I assume/deduce he had already worked through
the above in his mind, and realized that WS/HTTP is really a can-of-worms
if we can't get a hard fail at handshake for all non-compliant proxies. I
think perhaps this is why he was so focused on minimizing "false
positives" and why he made the comment that the failure of reverse proxies
for proposal 76 was "a success".

So that is why I say our good choices may be between WS/TLS and WS/P+T, so
I proposed a fall back suite: WS/P+T -> WS/TLS -> Comet/BOSH (or leave
this to the application to decide).

I assume there are many details that need to be discussed (or re-iterated
from prior discussion) to get a more objective understanding of this
issue.  I am not satisfied or confident enough to say I have sufficient
understanding of the facts, to argue that what I wrote about is anything
other than a set of concerns and questions.

I am hoping someone can collect and enumerate comparative facts with great
organization skill.

>>>>>> We must have that fallback no matter what we do.  All options
>>>>>> forward
>>>>>> have
>>>>>> failure cases.
>>>>>>

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 16 Aug 2010 15:48:10 +1000

Shelby,

I believe that we need to consider multiple ways to establish
websocket connections: HTTP upgrade, dedicated ports, TLS next
protocol etc.
We should flesh them all out and then we can make the decision if we
recommend support for all, some or just one of them.

With respect to this particular issue, I've no idea about the
frequency of the problem, but having intermediaries rewrite HTTP
requests/responses is not uncommon.  How such an intermediary would
react to websocket traffic is entirely knowable, thus there could be a
concern that such an intermediary may allow websocket traffic to
commence and then interject some data at some point.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 16 Aug 2010 15:51:26 +1000

Willy,

I agree with your analysis of the problem, but I remain hopeful that a
better solution than Connection:close may be found, as this prevents
the HTTP connection being used after a failed upgrade (by a BOSH/comet
fallback).

Having the server send a ws message after the handshake should be a
good way to quickly detect this problem.  Adding a simple checksum to
the framing would also quickly detect most varieties of intermediaries
not acting as pure tunnels.

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 16 Aug 2010 07:55:58 +0200

> 80 will and with better hard fail and better performance (less quirks

Noone said otherwise. But if you're only focusing on the easiest problems
to solve only one part of the problem, then we don't need a new protocol.

> moving away from that slavery over here).

Please, stop using the list to relay your political opinions. Also, such
unthought and childish comments totally discredits your technical analysis.
I'll put that on the lack of sleep.

> >> You might as well just hack the browser, so then you need to close port
> >> 80 too.
> >

No, just leave open a list of known good sites. But if your site uses SSL and
is not in the list, it cannot be reached from those points. There's a reason
Yahoo Mail still supports HTTP. At some places it's the only option.

> for security.

This statement makes no sense at all. You could as well push the reasoning
further "it's pointless to install this or that operating system and
applications because they're the most targetted by malware and using them
will require filtering devices". Then "it's pointless to try to be
interoperable with our customers because that will required an OS which is
targetted by malware".

> Kills anonymity, privacy, thus free speech.  And especially critical in
> this epoch ahead of us.

Once you've said that, it does not change what we have to deal with.

> the application state machines) that will have to close port 80

I don't buy that a second. It's probably the most well mastered protocol
and it has the advantage to support cleaning devices that work at insane
speeds, and thus are very cheap. In fact it's even the opposite. It's so
much exposed and attacked that it quickly had to become robust. It's as
if you said that Ethernet had so many collisions that it would never
succeed... In fact it succeeded precisely because it was designed to work
in harsh environments.

(... rant ...)
> Proxies should only be used to add capability, not restrict it.

Adding the ability to surf on the web without your browser being hijacked
by ads popups and various keyloggers is a capability. Those components are
not there to prevent users from doing something but to prevent their
browsers from doing something on their behalf.

> The entire model is wrong and we are going to suffer horribly for it.

Quite the opposite again. Look around you. People spend their week-ends
reinstalling their PC at home, while they never have to do that in well
controlled environment at work. This security is necessary and it's not
by claiming it's useless that we won't have to deal with it.

> They are going to have to do it any way, no matter what we do.

There's a difference between having to do that just for some time and
having to do that forever. Look, 10 years ago, not everyone was using
virtual hosting because some old browsers were not HTTP/1.1 compliant
and did not always emit the Host header. Now that's universally used.

> How does this not support my logic so far?

If it only works for 80% of the users, while an existing alternative
already exists for 100% of the users, the one with 80% has no added
value and will not be used. The problem is not getting 80% of the users
happy, it's getting 20% of them angry. If your site gets 1 million
unique visitors each day and 200000 of them can't use some of your
site's features, you'd better not have competition in your domain.

> soft fail in HTTP.  Consider the actual cost in the market place.

The cost is unrelated. Some devices will be upgraded in 1-3 years and once
users have complained about lack of support of XXX, this will be part of
the prerequisite features to select a new one.

> > As I said, none of TLS and your specific port addresses the stickiness

Nothing which works right now. It requires to completely implement your
new protocol in server-side load balancers for it to get a chance to be
used first. Building from scratch generally does not work well. Building
from existing blocks works well.

> We must have that fallback no matter what we do.  All options forward have

but not necessarily in the long run.

> What advantages and disadvantages does WS/TLS offer over port+tunnel? 

TLS offers at least confidentiality, integrity, framing and cross-protocol
protection. It has a larger handshake overhead.

> Ditto WS/HTTP?

HTTP relies on existing infrastructure to work immediately and support
scalability. It supports proxying, does not require the visitors to open
a new port. Supports authentication too. And the connection may be reused
by the browser from the HTTP one, reducing the round trips.

It will fail at some places, but :
  - server-side failures caused by lack of support for 101 will be fixed
    as bugfixes in many implementations when admins will start to look at
    competing products ;

  - client-side failures caused by overzealous proxies can quickly be fixed
    by configuration (either open all or just a small list of sites, as in
    the TLS case).

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 16 Aug 2010 01:59:06 -0400

> request. It will send return "HTTP/1.1 400 Bad Request" in the response,

But what if the data (sometimes randomly) looks like a valid HTTP request,
then all sorts of domino-effect random things could happen?  So every time
HTTP is upgraded with new keywords and syntax, WebSocket is broken!

I suspect this is also a security hole of some sort.

> incompatible components, because we won't have to wait for sending data.

Are you saying that the very first data sent will be guaranteed to always
be ill-formed HTTP, and so thus we can detect the "HTTP/1.1 400 Bad
Request" _always_ in a fast fail on incompatible components?  But what
about for proxies that always pass through and only take action on HTTP
they want to filter on?  You are making the assumption that every proxy in
the world will issue "HTTP/1.1 400 Bad Request" for a request it does not
understand.  Does the HTTP spec require proxies to reply or is pass
through allowed by the spec?

> other reasons for connection that we have to deal with.

If my comments above are correct, take that as flashback to our security
discussion that there is no way you can know your implementation is
"clean".  There are always cases unforseen.  The state machine is too
large to load in the human mind.  We discover specific cases as we come
across a reason to think about or experience them.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 16 Aug 2010 08:03:33 +0200

> the HTTP connection being used after a failed upgrade (by a BOSH/comet

To be honnest, I have very little hope of being able to reuse a connection
after a handshake failure, because this failure will not come from the
server itself (otherwise the application would not have proposed WS at all)
but from client-side intermediaries (proxy or transparent proxies). And it's
hard to guess the state the connection will be in.

Now that you're talking about it, I now remember why we suggested the close
on the server side too. Last week I told Ian I forgot the reason. It was
precisely for that : ensure the proxy won't process the next request without
blocking it at the request level. Basically, if the server sends a close but
not the client, then the proxy will not try to process the second request,
even if it does not understand the Upgrade. This will result in a faster
fail.

> good way to quickly detect this problem.

In my opinion, the data at the end of the handshake should be put in a WS
message. In fact it could be a "pong". That would probably mean less work
for the client as all it would have to do is call the same code on the data
block that comes after the 101.

> Adding a simple checksum to

Possibly, but if we're worried about them, then we should as well support
acknowledgements, because when connections are cut on timeout, you never
know what part has correctly been forwarded. But doing all this will result
in a more complex protocol. Probably that the checksum could be added later
as an extension in the frames if needed.

Regards,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 16 Aug 2010 17:43:13 +1000

> hard to guess the state the connection will be in.

I don't think you can assume that just because the javascript is most
likely to connect to the server that served it, that it will know that
server supports websocket or not.  Frameworks and applications may be
deployed in servers independently of the author of the javascript. So
the javascript author does not know if websockets will be supported by
that server unless they try. Conceivably the javascript could be
dynamically generated on the serverside, with websocket support, but
the server will still not know if websockets is supported by the
network infrastructure or not.

Thus it is likely that websocket handshake will be tried and if it is
not supported by the network infrastructure, then the upgrade header
will not be passed on or the server will not send a 101.  This leaves
a perfectly valid HTTP connection that could be used for BOSH/Comet.

> even if it does not understand the Upgrade. This will result in a faster

I'm not following exactly.  However I find it unlikely that an
intermediary that does not correctly handle the connection header for
upgrades would correctly handle it for close.

> block that comes after the 101.

+1

> as an extension in the frames if needed.

If you have ping/pong, then you effectively can have acknowledged delivery.
Send a message, send a ping, when you get the pong you know the
message has been delivered.

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 16 Aug 2010 03:55:33 -0400

> Noone said otherwise.

Was he aware of the new technique (emulating traceroute) in the links I
provided in the first post that started this thread, and could that change
his analysis?

> But if you're only focusing on the easiest problems

I am focusing on finding the solution that will work in the most number of
cases, so we fall back less often.  I don't understand why you say we
don't need a new protocol.

>> Comet/BOSH perhaps.

Perhaps I should have noted that WS/HTTP might be acceptable to the
enterprise, if WS/HTTP can be reliable and efficient.  From my prior 2
posts, I am doubting whether it can be significantly more reliable and
efficent than Comet/BOSH, but still waiting for the conclusion of that
sharing of ideas.

>>  Enterprise will die any way (the future is already
> Please, stop using the list to relay your political opinions.

I have intended no animosity towards you whatsoever.  Also I forgot to
mention in the prior post that I appreciated you explaining the rational
for the Connection: close.  I have had several lapses in the way I have
worded my emails that may have incorrectly sounded like I was admonishing
you and I assure you I was not. I am searching for the facts only. Thank
you so much for sharing your expertise.

LIST FEEL FREE TO IGNORE SOME OF THE FOLLOWING ASBSTRACT PARAGRAPHS, more
focused technical issue debate resumes futher down this post.

I am stating what I understand to be mathematical fact of entropy. I
believe in mathematical truth, which is that entropy is always driving
towards more independent actors at the expense of centralized control
(albeit with exponential local order growth and decay S curves along the
way).

I have 0 political opinions (don't try to get my way by ganging up in a
group). I never vote (authorize my opinion to be used to gang up against
others), and you are correct above that I don't really have a preference
to what this group decides (if we get a new protocol or not), I am only
here to foster information dispersal (as it will help me learn which
architecture is best for my own plans). Ultimately the market will do what
it is going to do.  But that doesn't mean I don't care about all of you
and I don't want to help you get a great spec.  It just means I am not
invested in that outcome (because mathematically I understand that that
group decisions are just local order on the big trend towards disorder).

Philosophy based in math is a form of technical analysis, but it is very
abstract and broad stroke and most people don't agree with the relevance,
until it hits them in face and even then, they don't recognize it as
entropy in action.  Failure to understand the exponential function is why
humans are destined to repeat the same errors over and over again.

> Also, such

You are free to characterize broad mathematical understanding as
unthought, childish, and non-technical. Thank you for expressing your
opinion. It is a free market. I intend you no harm, only sharing what I
believe to be the facts.

>> port

I am free to characterize that mathematically as blocking port 80 for all
free people who won't sign (implicitly) a futures contract, unfreedom, and
slippery slope towards complete failure.  It is not a political analysis,
rather a mathematical one. I already presented the applicable famous
computer language theorems. I doubt it is worthwhile to go deeper into my
reasoning, as most people won't agree and thus it is more efficient to
just let the market prove it. I am not going to try to troll my wall into
getting others to accept my understanding. I am maybe interested to say no
more on that point if I am not challenged to do so (but that could mean
withdrawing from useful debate, so I can not promise).

> This statement makes no sense at all.

Makes sense to me because I understand mathematically that centralized
security is an illusion/oxymoron. It mathematically forces non-use. When
humans can not control something, they make a law and pretend they can,
which results in putting people in jails for that which can not be stopped
because it is mathematically natural. We are already down to the level of
blocking most freedom on port 80 and blocking all other ports and
innovation (at least in the enterprise slavery zone, which mathematically
must decay eventually and probably already has peaked).

> You could as well push the reasoning

That shows you ignored or did not understand what I meant by the "big
epipheny" I mentioned in HTTP Compliance thread.

There is a big difference between adding features and requiring the
natural law to not apply to them.

Rather I have said if stop assuming we can control that which can not be
controlled, and give ourselves a reset button, then that is the most
entropy directed.  Nature will constantly do creative destruction, stand
in its way at your peril.

> Once you've said that, it does not change what we have to deal with.

It does if a significant and exponentially growing portion of the world
does not try to filter the internet.  I will give you an example. 
Westerners are very concerned about maintaining control over their online
accounts. Asians just walk away from an account and create another free
one.  They often have innumerable accounts.  They don't try to hold on or
enforce unnatural order.  That is the fundamental underlying reason why
facebook gets its butt kicked in China.  They don't understand that Asians
have different philosophy on what can and should be controlled and how to
deal with the march of entropy.  And even more so in the tropical islands
as compared to the Taoist cultures, where filipinos do almost everything
without planning and completely haphazardly without any attempt to control
and direct their lives.

> I don't buy that a second.

You just stated the for SSL (privacy and freedom) that is already
effectively the case.  A whitelist is not freedom.  A blacklist is evil
enough.

> It's probably the most well mastered protocol
> speeds, and thus are very cheap.

Great point about efficiency, and I should never argue against efficiency.
But that local efficiency is only efficient for global free market
perspective, in those cases where it is supplementing freedom, not
destroying it. Impinging on the free market causes a mis-allocation of
human capital, which is domino effect of inefficiency (even though might
get a local measure of great efficiency...correlate that to local order S
curves in progression of entropy).

> In fact it's even the opposite. It's so
> much exposed and attacked that it quickly had to become robust.

I differentiate between robustness of the protocol for freedom and the
mis-use of proxies to curtail freedom.  No argument on the former.

> It's as

No argument there, as far as I know Ethernet does not restrict any freedom
of the market.

> by ads popups and various keyloggers is a capability.

But I the user am free to configure that. Do you know how many times I
need to turn that off to get useful work done in my dating site research?

I can not turn off the filtering that my ISP is doing.  When I registered
asiadear.com, they cached the old nameservers for like a week!  Argh!  I
couldn't do anything about it.  I tried hosts.txt but that didn't work.

> Those components are

I told you already 2x that if you install Baseline Shield, you will never
have to reinstall your PC again ($4 per copy if you negotiate hard for
huge site license).  I couldn't believe I had ever lived without it.  And
besides all the mathematical advantages for freedom I already mentioned,
it also travels with me, unlike the enterprise jail.  And I don't have to
block anything on port 80 or any other port!  I AM FREE AT LAST!!!

Try it.  Freedom is quite addictive.

> having to do that forever.

WS/P+T won't be a partial fail forever, as you admitted if it becomes
popular enough, even the enterprises can parse it and allow it.  Or if you
argue that it will still be an open security hole, I still retort with
that centralized model of control is dying.  The world will turn to
freedom of something like Baseline Shield instead where each user controls
their own level of protection and has an instant reset button.

> Look, 10 years ago, not everyone was using

No argument with that.

> value and will not be used.

Unless the 80% one has better features or performance.

Which proposed solution works for 100% of users?  I think none of them. 
And all proposed solutions can converge to 100% over time.  But which
solution gives the most improvement out of the starting gates?  I think
WS/P+T.

> The problem is not getting 80% of the users
> happy, it's getting 20% of them angry.

Agreed.  That is why the 20% must get fall back to Comet/BOSH.

> If your site gets 1 million

Agreed, but this is not an argument against any of my points.

> the prerequisite features to select a new one.

This violates your 20% failure intolerance argument above, because if you
don't fall back, there will be no adoption, and if you do fall back, there
will no (fewer) complaints.

Also the cost is real in any case, we have to weigh it in comparison to
other options we have.

> from existing blocks works well.

Strong point.  The links I provided for WS/P+T were aimed more at two
client computers, not at server farms.  However, your point does not apply
to the applications running off a server that has a dedicated IP address,
which includes my server at pair.com.  And the "100% compatibility will be
reached over time" argument applies in this case too.  We need to weigh
all the tradeoffs and use cases.  No proposed transport is 100% now.  No
proposed transport can't become 100%.  Shouldn't we let the free market
decide unless we have clairvoyant certainty?

I am starting to lean towards a "one size fit all" is not optimum
solution.  We may need alternative transports for the application to
choose.

> but not necessarily in the long run.

For all our proposed transport options.

> protection. It has a larger handshake overhead.

TLS appears to be the more "secure" option, at a handshake, enterprise
blocking, etc costs.  Someone needs to put all these tradeoffs for all
options into a huge table so we and the market can grok this.  Disagree?

> scalability.

We have Comet/BOSH for that already, and we have to compare the tradeoffs
of the advantages versus disadvantages that WS/HTTP introduces.

> It supports proxying,

That is a negative for real-time communications.

> does not require the visitors to open
> a new port.

Neither does the fallback to Comet/BOSH.  How much are we gaining and
losing with WS/HTTP?  I say much better gain to have a real TCP channel
instead of a proxied one.

> Supports authentication too.

TLS will give us that option (but not on enterprise)?  We can add it to
proposed WS/P+T?

Can't we authenticate orthogonally using HTTP and then pass signatures in
WS/P+T?  This is application layer, and should we be more focused first on
transport layer specific tradeoffs?

> And the connection may be reused

I am not so convinced of this will be so in practice and considering
numerous other factors, but I am not an expert in this area.  Will defer
to others on this one...there is  lot of past commentary on the list about
this and afaics questions that claim.

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 16 Aug 2010 07:20:14 -0400

> provided in the first post that started this thread, and could that change

Obviously you meant "no one" not a person with a last name of "Noone"? 
Two entirely different meanings from absence of a single space.

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 16 Aug 2010 08:40:16 -0400

> options into a huge table so we and the market can grok this.  Disagree?

Can you explain how cross-protocol protection is the responsibility of the
protocol being attacked?

I read up on this a little bit[1], and it appears to a security hole in
the browser, which allows the form to be posted to a port other than HTTP?

[1]
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.7301&rep=rep1&type=pdf

Thus I conclude that "cross protocol protection" is not something we
should be weighing in our comparison process.

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 16 Aug 2010 09:09:21 -0400

I want to make the point below that I mean port 443, but I will also
propose one example where enterprises will be forced to whitelist port 80
in future too.  below...

>> No, just leave open a list of known good sites. But if your site uses
>> SSL
>> and

[snip]

> effectively the case.  A whitelist is not freedom.  A blacklist is evil

Above I was referring to https: so port 443, but consider a JavaScript
encryption that using Ajax to pass content that the enterprise can not
monitor.  If I understand Willy's point about TLS not be acceptable to
enterprises and the whitelisting of https:, then wouldn't the same apply
to port 80 if webpages encrypt with JavaScript (or other application layer
scripting)?

Have I not argued convincingly that the end of the road of the current
enterprise security model is that nothing can pass that isn't whitelisted?

Rather consider my suggestion that clients protect themselves with instant
recoverable disk imaging (set it once and forget it, unless you need to
upgrade executables then you grab a new disk image, an automatible process
in the enterprise scale), and use an exception drive or folder for
persistant data.  So every future unknown attack that can occur with the
current security model can also occur in this proposed one.  So what is
loss of security?  We know from open source that exposing ourselves to
more attacks increases the number attacks that are fixed.

I just can not find any rational for the current enterprise security model
of putting a "one size fits all" wall around 100,000 people.

---

From: "Shelby Moore" <[email-masked]>
Date: Mon, 16 Aug 2010 14:18:38 -0400

Some Benefits of port+tunnel (other benefits have already been mentioned)

=====
P2P:

Perhaps a big bonus of port+tunnel, is we could then allow P2P 
connections for WebSockets. That would be a huge, super huge feature.
Success rates are apparently around 69%. Fall back to Comet/BOSH to an
HTTP server for the remaining 31% (similar rates of fall back be necessary
for other proposed transports too).

The Same Origin Policy does (and SHOULD) not apply to cross-site
communication when the HTTP cookies and/or any form of HTTP session
persistance are not involved[1].

[1]
http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy_for_XMLHttpRequest

That may be a controversial statement, but I am able to explain in
exhaustive detail if any one doubts it.  Summary is that since the user
trusts the page he/she is viewing and that is communicating, the user
trusts that viewed page to not load resources which would be malicious,
and thus the only vulnerability that is not the responsibility of the
other site involved in communication, is the one where the other site can
be tricked into doing something on the user's credentials for that other
site.

=====
UDP:

With port+tunnel we could support both UDP and TCP (and expect TCP to work).

If we instead shoehorn/piggyback on HTTP or TLS, we are stuck with TCP
(and in case of HTTP, I have an open question as to whether the proxies
can break the TCP guarantee-to-deliver benefits as compared to UDP[2][3]).

TCP incurs more overhead than UDP and is not often used for optimum
streaming (where latency an dbandwidth are optimized as compared to TCP,
because data loss is acceptable as often unnoticeable hiccups in rendered
frame rate).

[2] http://www.ietf.org/mail-archive/web/hybi/current/msg03261.html
[3] http://www.ietf.org/mail-archive/web/hybi/current/msg03257.html

---

From: Vladimir Katardjiev <[email-masked]>
Date: Mon, 16 Aug 2010 21:15:12 +0200

Sorry for just jumping in; I'll probably drop further down on the list as I manage to catch up.

> be tricked into doing something on the user's credentials for that other

I'm not sure what the point is here. Yes, you are correct in that the SOP no longer applies to WebSockets, but this has been true from the get-go. However, the _origin policy_ still applies very much. Mostly because even if the _user_ might think one thing, neither browsers nor enterprises expect most users to know any better. Since browsers are, more and more, becoming execution environments that automatically run the executable code they download, this is a very important point of restriction. Web pages should not be able to arbitrarily initiate connections anywhere they want, as that opens up malicious attack vectors.
> [3] http://www.ietf.org/mail-archive/web/hybi/current/msg03257.html

Isn't this a more fundamental question on what the requirements of WebSockets itself are, rather than a benefit of port+tunnel (meanwhile, I can't seem to find a concise definition of what you really mean by "port+tunnel")? So far, apart from gaming usages, I don't think TCP was too much of a concern. Meanwhile, the main targeted areas of WebSockets (correct me if this has changed recently) were mostly relying on data integrity. 

I should also note that TCP doesn't guarantee delivery. TCP merely guarantees that the order of data coming out the remote end's pipe is the same order that went in my end. However, if I send "Hello", it is perfectly possible for TCP to present "Hell" to the other side, after which it will close the socket and go sulk in a corner. This is exactly the reason why WebSockets needs orderly close as part of the protocol itself (because otherwise TCP may lose data).

> Can you explain how cross-protocol protection is the responsibility of the

Rather, it's the responsibility of WebSockets to not facilitate attacks on another protocol. Continuing from the origin policy, suppose I, behind my Mighty Firewall open a webpage with a compromised advert (this has happened repeatedly, so we know that even if I trust the page, the page's contents may be compromised). This can now download JavaScript into the page, to start opening WebSocket connections to, say, my mail server (which is firewalled from the internet to prevent attacks). If WebSockets facilitate cross-protocol attacks, I have just been robbed of my data.

Cross-protocol attacks are thus extremely important in the weighing process.

Cheers,
Vladimir

> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.133.7301&rep=rep1&type=pdf

This applies to HTML <form> elements. Given that major browsers now ship with Cross-Origin XMLHttpRequest support, it is actually trivial to send a POST to anywhere in the known universe.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 16 Aug 2010 21:15:48 +0200

Hi Greg,

On Mon, Aug 16, 2010 at 05:43:13PM +1000, Greg Wilkins wrote:
(...)
> a perfectly valid HTTP connection that could be used for BOSH/Comet.

In fact, not from my observations for a simple reason : the intermediaries
I've seen failing on 101 always abruptly closed the connection. But that
does not preclude that the connection will never be usable, just that in
some (most?) situations, the connection will have to be reopened.

> upgrades would correctly handle it for close.

The connection header (I mean "connection: close") has been used for at
least 10 years and is really mandatory to get HTTP messaging right. The
Upgrade header has only been referred to in the RFC2817 and was not as
much used. I *believe* it is also used by some remote desktop protocols
that start as HTTP and switch to their own protocol, though I can't
verify it.

And most of the components I've seen don't support Upgrade yet while
they've been supporting "connection: close" for ages.

(...)
> message has been delivered.

That's a very smart idea. Simple and efficient. Can be used as an explicit
ACK when needed and does not require more round trips at all !

Regards,
Willy

---

From: "Shelby Moore" <[email-masked]>
Date: Tue, 17 Aug 2010 01:17:03 -0400

> Sorry for just jumping in; I'll probably drop further down on the list as

Happy someone replied, to help check if my logic is correct or not. Thank
you. I am still waiting for someone to concur or refute the logical

>> [3] http://www.ietf.org/mail-archive/web/hybi/current/msg03257.html

below...

> I'm not sure what the point is here. Yes, you are correct in that the SOP
> no longer applies to WebSockets, but this has been true from the get-go.

The point is I am explaining why I think same origin policy SHOULD NOT be
applied, if we do not use HTTP requests. SHOULD NOT can even be argued to
be MUST NOT. And if the "P.S." below is acknowledged, then browser MUST
NOT enforce SOP ever (except when domain requests it, see my "new
compromise idea" optimum further down this)!

I am (correctly) assuming that unlike HTTP requests, it is impossible for
a non-HTTP transported WebSockets to pass HTTP user credentials for a
domain to that domain from a page loaded from _another_ domain. I am also
explaining that this security hole of passing user creditials via cookie
or some other form of  HTTP persistance (e.g. PHP sessions which relies on
a cookie) is the only security hole that SHOULD be blocked by SOP (same
origin policy).

=========
P.S. The same origin policy does not even need to be enforced by the
browser, it MUST also be enforced by the target domain (which also gives
additional benefits being much more granular, because then the market can
choose whether to use sites which are poorly programmed...see below on
ramifications...). To accomplish this, the domain MUST never use any form
of HTTP persistance that is automatically loaded from the client to the
server (i.e. cookie). Rather the client page of the domain can load the
cookie in JavaScript and pass it to the server on each request. This
prevents any other page from a foreign domain from invoking the server
(REST) APIs with persistant user credentials in the cookie. For pages that
do not want to use scripting, we need a non-scripting way (META?) to tell
the browser to insert the cookie credentials in each server request and to
enforce SOP for that page.
=========

Back to what I wrote before the P.S., all other of forms of attacks that
involve cross domain, involve the page, that the user is viewing and
trusting, to violate that trust. I am arguing that (see below for more on
this...) allowing user's to choose which sites they trust is more free
market and superior security model and results (tangential point: the
firewall paradigm will lead to not only a loss of innovation, but
centralized whitelisting all of communication and thus complete failure).

Those are not "MUST" that inpringe on any one, they enable a more free
market that will be more robust and more secure. So the market is going to
force those "MUST" in due time via competition. In long run, it matters
not what any standards group decides, because the truth will always win in
the market place (after a peak of an erroreous paradigm and its failure
and collapse). The importance of standards is they can influence (and
hopefully minimize) the amount of human capital mis-allocation that occurs
interim to the truth asserting itself in the market naturally.

> However, the _origin policy_ still applies very much. Mostly because even
> if the _user_ might think one thing, neither browsers nor enterprises
> expect most users to know any better.

And I think they are wrong to do that, and will end up in 100% whitelist
communication failure for that mistake.

Please try to follow my logic very carefully without any bias so we can
think clearly on the epipheny I am claiming, otherwise the epipheny can
fail to register in one's mind.

We must be very careful with the mind's tendency towards sweeping
generalizations that might conflate different issues and thus lead to the
wrong conclusion.  Let's break the issues down to the essential parts and
analyze orthogonally.

The issue here is that user is viewing a page and the user can be trusted
to form his/her own trust judgement for that page, as long as the user can
be sure that the domain is in complete control of itself.  It is critical
that we first agree on the prior sentence.

The security holes are ONLY those that allow another domain to inject into
another domain, without the other domain initiating it.

If a domain loads a script from another domain which then violates the
former domain's expectations and intentions, that is the responsibility of
the former domain. The programmer of the former domain should make sure
he/she is aware of what is being injecting into his/her own domain by
his/her own code. Because the user is going to form a trust judgement
based on that programmer's proficiency. For example, if I am the
programmer of a domain, and I wish to load user created (unknown trust a
priori) scripts, I should load only signed scripts and monitor them for
security holes and revoke their signature if I discover one.

> Since browsers are, more and more,
> becoming execution environments that automatically run the executable code
> they download, this is a very important point of restriction. Web pages
> should not be able to arbitrarily initiate connections anywhere they want,
> as that opens up malicious attack vectors.

Since the browser is such a critical focal point of global innovation, it
is critical that we don't kill it.

I so very strongly disagree. Please read first what I wrote above for the
context. The domain that is loading the scripts needs to be held
responsible by the user (market place).

The only valid argument that can be made against my logic, is that users
will visit new sites that they don't yet have any trust judgement on. So
then we have to decide what kind of society and culture we want. Do we
want a society where we do not trust people we do not know well? The
western culture has changed from when I was young to being significantly
one that is no longer friendly and trusting of people. In 2006, my car
engine blew up in middle of California desert, I walked to a rest stop at
1am and standing under the bright lights near the restroom, I tried to ask
approaching people if I could get a ride to next gas station. I was
consistently told from a distance "get away, I will call police if you
talk or approach me". Western culture has eroded into one where many/some
people have no capacity to trust. Whereas here in Philippines, I ran out
of gas in a typhoon in the province and was able to wake people up at 2am
to help me locate a coke bottle filled with petrol.

The fact is that only about 2.5% of the society has criminal aspirations. 
That does not stop unintended security holes, but clearly the site
deserved the bad reputation is got (instead we punished the user who
helped illuminate the poor proficiency of MySpace's programmers):

http://sonofsamy.wordpress.com/2005/10/04/the-myspace-worm/

I believe in a free market and competition. I believe I can outcompete
MySpace and I don't want the browser forcing me to be dumbed down to
MySpace's lack of proficiency. Let each entity thrive or die in the market
place on their own merits. I think you all believe the same, if you really
think it through carefully.

So we have a choice. Either we can choose to limit innovation or we can
choose to let users learn to be warying of new sites that have no well
formed reputation in the market place.

But then I had a new compromise idea (thank you so much Vladimir for
stimulating this). How about not choosing? How about we let the domain
choose, and then warn the user in the brower when the domain has chosen no
SOP! Yeah that is the optimum solution! So then domains that don't yet
have a well formed reputation can offer two versions of their pages. One
(default) version would be one for new visitors to use, and the other
version would be the one offered to trusting users (experienced with that
domain or the domain's reputation) that tells the browser to turn off SOP
(from which the domain can offer additional features and innovate).  Again
the browser would (by default)warn when the user loads the page that says
"turn off SOP".

And let me re-iterate the point[1] from my prior posts in this thread (and
HTTP Compliance), which I assume was ignored by this list, that any
collateral damage _outside_ the domain affected can be trivially erased by
the user with a simple and instantaneous reset of their client machine,
running a disk image restoration engine such as Baseline Shield with an
exception drive/folder for persistant file changes. And this paradigm
travels with the user, unlike the current proxy firewall paradigm of the
enterprise. And that firewall/filtering paradigm has already nearly closed
(whitelisted) every port except 80 and soon 80 will be forced closed (or
the filtering will have a hole).

[1] http://www.ietf.org/mail-archive/web/hybi/current/msg03277.html

> Rather, it's the responsibility of WebSockets to not facilitate attacks on
> another protocol.

Agreed, but only to NOT facilitate attacks which the user did not
explicitly agree to.

This is because the user of a browser does not expect it to automated
tasks with other protocols.  Whereas a user might authorize their SSH
client to do automated tasks, that would be with the user's explicit
persmission, since the user downloaded, configured, and ran the custom SSH
client.

> Continuing from the origin policy, suppose I, behind my
> Mighty Firewall open a webpage with a compromised advert (this has
> happened repeatedly, so we know that even if I trust the page, the page's
> contents may be compromised).

We are back on the SOP discussion now. Refer back to all my points farther
above. The bad advert is the fault of the page which is loading the
advert, since the user is trusting that page. If we would stop hiding bad
pages behind the browser's SOP guarantee (I mean stop penalizing those
good pages that want to innovate, i.e. stop socialism), then the web could
quickly evolve to one where bad advertising economically
disappears!!!!!!!!!!!!!!!!  Think about that!!!!!!

The problem is we give good pages no way to declare their intention to the
user to not use SOP, so the web is trapped in socialism.

The brower's centralized SOP firewall being forced on all pages that could
be good and innovate is analogous to the socialism and failure that is
being forced by the centralized enterprise firewalls.  Remember the point
I made in prior post that I can't opt out of the centralized proxy (like I
can for a brower popup blocker when I need to access a site I trust that
uses popups) and I can't carry it with me.

I think my points so critically important. But I am not going to wait for
some WG to get it done. I am just going to try to do it in the market and
help the free market to overrun the socialism.

> This can now download JavaScript into the
> page, to start opening WebSocket connections to, say, my mail server
> (which is firewalled from the internet to prevent attacks). If WebSockets
> facilitate cross-protocol attacks, I have just been robbed of my data.
>

I emphatically disagree (no admonition intended, just trying to convince
you), because the point of responsibility must not be centralized (unless
the user has chosen that it be).  Rather we should give users and
programmers the option to opt out of centralized trust paradigms, so that
innovation and free market can proceed, which will in fact be more secure
and more robust. If you need me to explain why, just ask.

The fundamental problem derives apparently from that computer science
people are not expert in macro economists, and more sadly most Ivy League
trained macro economists apparently do not incorporate the math of free
market and the laws of nature.[2] This is not "unthought nor childish", it
is very critical and serious technically.

[2] http://www.ietf.org/mail-archive/web/hybi/current/msg03233.html

> This applies to HTML <form> elements. Given that major browsers now ship
> with Cross-Origin XMLHttpRequest support, it is actually trivial to send a
> POST to anywhere in the known universe.

But no program should be sending HTTP requests to non-HTTP ports.  That is
a bug in the browser.

Is my logic not lucid?

---

From: "Shelby Moore" <[email-masked]>
Date: Tue, 17 Aug 2010 02:44:53 -0400

Again thank you for jumping into discussion on my points.  Below...

> Isn't this a more fundamental question on what the requirements of
> WebSockets itself are, rather than a benefit of port+tunnel

Oh yes very much so. Thanks for realizing.

That is why I am motivated to explain why the conflated _whitelist_
firewall model we have now (SOP, client firewall, and centralized proxy
filter) is not optimum (forces sharing of limitations, eventually
degenerating to whitelist communication entirely or failed limitations
which we mischaracterize as "security holes"), so that I am argue that
WebSockets should support both connections between any clients and servers
on the internet (even if behind a NAT, and remember NAT is not supposed to
allowed by original IPv4 standard, it was a market violation).

Now if you argue that "Web" is HTTP and thus should not support P2P nor
UDP (because HTTP didn't), then that means we can't do bi-directional
push, because HTTP didn't.  So that logic doesn't hold.

We should ("MUST" as long as we understand that standards are optional
suggestions) not put unnecessary chains on the internet, which are not
optimum security nor robustness.  I explained it in my prior post.

> (meanwhile, I
> can't seem to find a concise definition of what you really mean by
> "port+tunnel")?

I simply mean we use a new port for our transport, and we maximize NAT
traversal (STUN, TURN, and new ideas):

http://www.ietf.org/mail-archive/web/hybi/current/msg03231.html

And also just as for WS/HTTP, we might need to do some special things for
server farms (another form of "NAT"):

http://www.ietf.org/mail-archive/web/hybi/current/msg03287.html

> So far, apart from gaming usages, I don't think TCP was
> too much of a concern.

What about streaming media, such as audio and video?

> Meanwhile, the main targeted areas of WebSockets
> (correct me if this has changed recently) were mostly relying on data
> integrity.

We have Comet/BOSH for that.  WebSockets only market justification is
performance (certainly won't be reliability without fall back to
Comet/BOSH).

And I am arguing to nudge the web back towards the correct free market
security model, in which case WebSockets could offer full connectivity
options and that would be a really huge factor influencing adoption (given
that NAT Traversal works 69% of the time and the application can fall back
to TURN for the other 31%).

I doubt very much this WG will bold enough. But I do want to hash out the
logic, because others who are not so timid, can use this information.

I am not arguing that we shouldn't also consider WS/HTTP, WS/TLS, and
other options. Maybe we need to implement more than one.  Let's try to
hash out the facts.

> I should also note that TCP doesn't guarantee delivery.

Thanks for clarification.  What I meant is that TCP guarantees that we
know when something wasn't delivered.  This was a critical point, because
if I am correct that HTTP proxies could eat data, then TCP's timeout from
the perspective of between the final ends will fail (because there was no
timeout, the data was delivered to a silent and transparent proxy).

> TCP merely
> guarantees that the order of data coming out the remote end's pipe is the
> same order that went in my end. However, if I send "Hello", it is
> perfectly possible for TCP to present "Hell" to the other side, after
> which it will close the socket and go sulk in a corner.

Hehe.

It is up to the application protocol to checkum/hash or check expected
lengths in a header, but at least TCP will deliver something or report a
timeout.

> This is exactly
> the reason why WebSockets needs orderly close as part of the protocol
> itself (because otherwise TCP may lose data).

I do not understand, because that is not my area of expertise (and my
brain is tired and I am in a rush to send this and go do other things). If
you are willing to explain, I am willing to read. Maybe most other readers
understand what you mean.

---

From: Vladimir Katardjiev <[email-masked]>
Date: Tue, 17 Aug 2010 10:02:12 +0200

Just running this explanation (since I know it by heart already); I'll get to other points in due time.

On 17 aug 2010, at 08.44, Shelby Moore wrote:
> know when something wasn't delivered.  

No, it does not. You cannot prove the lack of existence of an event; you can only prove the existence of it. In other words, TCP can, in terms of the TCP stack itself, realise that a given packet has been delivered. It cannot realise that a given packet has been lost (it just assumes it is. This is why some packets are delivered twice). However, the TCP stack does not generally expose this functionality to upper layers. Namely, it's generally not possible for us as an application writer to ascertain that any given byte has been delivered. Since this means that even your garden variety TCP connection is not a stable transfer option, every application needs to assume that messages can and will be lost, and thus will have their own ACKs layered on top.
> understand what you mean.

Simply put, a client wishes to shut down. It needs to send a final piece of data to the server, and then close() the WebSocket connection. The client writes the bytes to be sent to the TCP output buffer, but then what? In the regular case, we can't ascertain when this buffer is empty. If we close the socket while the buffer is non-empty, we risk losing the data in it, depending on the implementation.

However, let's assume that the implementation will first send the bytes, wait for ACKs, then send the FIN to close the connection. Since TCP will ACK on /delivered/ and not /read/ bytes, this does not imply the remote application received them yet (it may be processing a different request). When the remote end receives the TCP FIN, it will close down the socket, and elicit EOF on it (because the socket is closed). The remote application can now not retrieve the data that was sent. 

For this reason, all HTTP servers leave TCP connections open for a certain timeout after a successful request has been answered, because otherwise the client might not get the entire page delivered. And for this reason WebSockets has an orderly close mechanism, to ensure that the data has been delivered before the connection is closed.

The reason I bring this up is because when you state that proxies break the guarantee of TCP, I state they don't. TCP never guarantees that data will be delivered. TCP guarantees the following:

Each byte read from the remote endpoint's socket will correspond to the same byte that was written in the local endpoint's socket. The bytes that arrive will arrive in the same order, and with no data corruption.

TCP also provides flow and congestion control, but these are minor in comparison.

What you likely refer to when you believe TCP guarantees delivery is the TCP retransmission mechanic. While a very good mechanic for a live stream, it does not guarantee you'll know where a transmission fails, only that it has failed. It's up to the application layer protocol to discover this failure and handle it. 

With that in mind, I hold that intermediaries can very well provide a similar level of service that plain TCP can. History has also shown that we're more likely to have intermediaries than not to, so I do not buy the argument that we're better off doing our best to tunnel/bypass/protect/disrupt them. Is it a compromise? Yes. But I believe it is a necessary compromise between those who wish to provide a new type of service and those who wish to manage their networks.

Cheers,
Vladimir

---

From: "Shelby Moore" <[email-masked]>
Date: Tue, 17 Aug 2010 04:47:14 -0400

> No, it does not. You cannot prove the lack of existence of an event; you
> can only prove the existence of it. In other words, TCP can, in terms of
> the TCP stack itself, realise that a given packet has been delivered. It
> cannot realise that a given packet has been lost (it just assumes it is.
> This is why some packets are delivered twice).

You have just stated that TCP knows when a packet is delivered, combined
with timeout, TCP knows when something MAY have not been delivered.  Thus
TCP is able to guarantee that a packet was delivered (and it knows when it
MAY not have been). That is very different situation from UDP, or in
comparison between a TCP channel without application layer proxies and the
case where I am worried about a proxy silently eating a packet that TCP
thinks it was delivered.

> However, the TCP stack does
> not generally expose this functionality to upper layers.

I am not well versed on this, but I think you are saying that it is
possible (even if not generally done) for upper layers to distinquish
between whether if a TCP packet was delivered or MAYBE not delivered?

In any case, even when the upper layers can not read each TCP packet
deliver status, I think you are clarifying issues which do not apply the
context of the point I made. The point I am making is that TCP is much
more reliable channel (in terms of delivery) than for example UDP. We have
certain expectations/experience about how much more reliable it typically
is, and that is why we use is for data integrity applications such as
HTTP. My point being that if we lose that reliability in some cases due to
silent-data-eating HTTP proxies, then we no longer have the TCP
reliability across the entirety of the end-to-end client and server.

> Namely, it's
> generally not possible for us as an application writer to ascertain that
> any given byte has been delivered. Since this means that even your garden
> variety TCP connection is not a stable transfer option, every application
> needs to assume that messages can and will be lost, and thus will have
> their own ACKs layered on top.

Understood. Does not apply to the context of my point above. Thank you for
sharing your expertise.

> Simply put, a client wishes to shut down. It needs to send a final piece
> of data to the server, and then close() the WebSocket connection. The
> client writes the bytes to be sent to the TCP output buffer, but then
> what? In the regular case, we can't ascertain when this buffer is empty.
> If we close the socket while the buffer is non-empty, we risk losing the
> data in it, depending on the implementation.
>
> However, let's assume that the implementation will first send the bytes,
> wait for ACKs, then send the FIN to close the connection. Since TCP will
> ACK on /delivered/ and not /read/ bytes, this does not imply the remote
> application received them yet (it may be processing a different request).
> When the remote end receives the TCP FIN, it will close down the socket,
> and elicit EOF on it (because the socket is closed). The remote
> application can now not retrieve the data that was sent.
>
> For this reason, all HTTP servers leave TCP connections open for a certain
> timeout after a successful request has been answered, because otherwise
> the client might not get the entire page delivered. And for this reason
> WebSockets has an orderly close mechanism, to ensure that the data has
> been delivered before the connection is closed.
>
> The reason I bring this up is because when you state that proxies break
> the guarantee of TCP, I state they don't.

Refer to my explanation above.

> TCP never guarantees that data
> will be delivered. TCP guarantees the following:
>
> Each byte read from the remote endpoint's socket will correspond to the
> same byte that was written in the local endpoint's socket. The bytes that
> arrive will arrive in the same order, and with no data corruption.
>
> What you likely refer to when you believe TCP guarantees delivery is the
> TCP retransmission mechanic. While a very good mechanic for a live stream,
> it does not guarantee you'll know where a transmission fails, only that it
> has failed. It's up to the application layer protocol to discover this
> failure and handle it.

Refer to my explanation above.

> With that in mind, I hold that intermediaries can very well provide a
> similar level of service that plain TCP can. History has also shown that
> we're more likely to have intermediaries than not to, so I do not buy the
> argument that we're better off doing our best to
> tunnel/bypass/protect/disrupt them. Is it a compromise? Yes. But I believe
> it is a necessary compromise between those who wish to provide a new type
> of service and those who wish to manage their networks.

So if I understand you correctly, you think that any transport (port+low
level protocol) and high level protocol we use or create will end up
having intermediaries on it, and that it is futile to try to assume that
we won't?

I would not disagree with that (because I believe in free markets), I am
just arguing that the protocol we use should be well formed.  What worries
me so far about WS/HTTP is that using HTTP Upgrade to send bi-directional
data is not well formed with respect to the historic use of HTTP. It is
another form of tunneling through a legacy market installed base. There is
no doubt that specification should allowed it to be well formed, but I am
just worried about the silent failure. Whereas if we start with a new
thing, there won't be that legacy that we are bumping against. And I
worried that we could break somethings that already work in HTTP. Until
further information, I am leaning that I would rather keep it orthogonal.

And note that is not the only benefit I am trying to point out for this
proposed WS/P+T (port+tunnel).

Some of this can be used to argue for WS/TLS as well.