Subject: [hybi] #1: HTTP Compliance

From: "hybi issue tracker" <[email-masked]>
Date: Wed, 12 May 2010 10:37:21 -0000

#1: HTTP Compliance
-------------------------------------------+--------------------------------
 Reporter:  salvatore.loreto@…             |       Owner:     
     Type:  defect                         |      Status:  new
 Priority:  major                          |   Milestone:     
Component:  websocket-requirements         |     Version:     
 Severity:  -                              |    Keywords:     
-------------------------------------------+--------------------------------
 REQ.7 currently states:

      When sharing host and "well known" port with HTTP, the
      WebSocket protocol MUST be HTTP compatible until both ends have
      established the WebSocket protocol.

 at the Anaheim meeting there was some consensus to change this to "HTTP
 compliant".
 However there is a lot of misunderstanding about what compliance to HTTP
 means and implies.
 The Requirement draft should clarify it, as it may have non trivial impact
 on the protocol design.

-- 
Ticket URL: <http://trac.tools.ietf.org/wg/hybi/trac/ticket/1>
hybi <http://tools.ietf.org/hybi/>
The Hypertext-Bidirectional (HyBi) working group will seek
standardization of one approach to maintain bidirectional
communications between the HTTP client, server and intermediate
entities, which will provide more efficiency compared to the current
use of hanging requests.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 17 May 2010 11:04:45 +0200

All,

I know this issues has been somewhat sidetracked by the
discussion about TLS.

However, this particular requirements is conditional for only when
sharing a port with HTTP, so I think it valuable to continue to
try to finalize this requirement.

Currently there is no clear consensus either way, with perhaps
a few more voices against compliance is not required.

For firstly, I'd like to encourage all list lurkers to
speak up and help sway the debate one way or the other.

Also, in an attempt to move the conversation on, I'd like
flip the question around and ask if somebody can clearly
state what is the down side of adopting HTTP compliance
other than it might force a breaking change in the
-76 draft.   Is there some other requirement that can only
be achieved by breaking compliance?

regards

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 17 May 2010 02:29:43 -0700

> try to finalize this requirement.

I think we need to reframe this requirement so that it makes sense even if we change the handshake to be at the TLS level rather than at the HTTP level. Both the current requirement and the new requirement would make no sense if the handshake was done via TLS.

> be achieved by breaking compliance?

Ian has claimed that the extra bytes after the request help WebSocket fail early in the face of unaware proxies, though we do not yet have data showing this to be the case. We do know that without this, often the handshake will appear to succeed but transmitting messages will fail, for the network setups of many users.

I don't think anyone has indicated a practical problem caused by the extra bytes.

If we do get evidence that the extra bytes make the WebSocket protocol work better, and if we do not get evidence that they actually cause a problem for WebSocket servers that are sharing a port with a Web server, then I think we should sacrifice theoretical purity for practical benefit.

That being said, this whole issue is irrelevant if we switch to TLS-only and a handshake based on nextprotoneg (and possibly TLS upgrade followed by nextprotoneg).

We should try to frame the requirements so they do not overly depend on the technical details of the current draft. If a requirement could be completely invalidated by a plausible technical change, then it is framed the wrong way and depends too much on protocol details. It would be an indication that we're trying to move issues that should be technical decisions about the protocol into the requirements document.

Therefore, I think the "HTTP compatible/compliant" requirement should be deleted completely, or rewritten to make sense for any plausible handshake. 

I note that the following requirement would apply reasonably to both the current draft mechanism and the proposed TLS-based handshake (with perhaps added specific mention of HTTPS):

"The WebSocket protocol MUST allow HTTP and WebSocket connections to be served from the same port.  Consideration MUST be given:
         * to provide WebSocket services via modules that plug in to existing web infrastructure.
         * to making it possible and practical to implement standalone implementations of the protocol without requiring a fully conforming HTTP implementation."

and should cover all the practical issues that would actually be faced by servers in either scenario.

Regards
Maciej

---

From: Ian Hickson <[email-masked]>
Date: Mon, 17 May 2010 09:35:57 +0000 (UTC)

On Mon, 17 May 2010, Greg Wilkins wrote:
>
> Also, in an attempt to move the conversation on, I'd like flip the 
> question around and ask if somebody can clearly state what is the down 
> side of adopting HTTP compliance other than it might force a breaking 
> change in the -76 draft.

There is an assumption in this question that I feel would lead us to make 
suboptimal design decisions: it assumes that the bytes being sent from one 
peer to another peer are intrinsically meaningful, as opposed to the 
reality, which is that it's the software that imbues them with meaning 
when interpreting them.

What matters is that we get interoperable behaviour. Everything else is 
secondary, IMHO. We should look at things from the point of view of the 
client and the point of view of the server, not from the point of view of 
the bytes on the wire. This has some implications for our requirements -- 
for example, it means that the server and client don't have to agree about 
what's going on, so long as the result is reliably interoperable. A client 
might think it's talking one protocol, while a server might think it's 
talking another protocol -- so long as the rules for generation on the 
client result in the right result when the server applies the rules for 
consumption, we've achieved the goal of interoperability.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Vladimir Katardjiev <[email-masked]>
Date: Mon, 17 May 2010 11:39:59 +0200

I assume the problem/desire/requirement is to reduce the number of round-trips required for the establishment of a new connection. The "random bytes" are a byproduct of that -- ensure that a WebSocket connection's "open" event is fired when the client knows the connection is actually open, not just when the connection is "maybe" open.

...

Maybe it'd be good to rethink how the entire thing works. So far, the WebSocket handshake itself has been "baked into" whatever previous authentication there is/has been. So... Make a proper WS handshake. It can be asynchronous (a la SPDY), and thus not require an additional RTT. Allow WS data to be sent from the API before open is fired, at the risk of being lost. The current API always risks data being lost anyway, unless some ack protocol is introduced, so this is OK in my view. Fire open after the WS handshake finishes. There's no speed loss for expert programmers who desire the extra ms boost, while the contract of an established WS connection is preserved for amateurs.

Then what's left to define is the protocol switching. In this case, ensure the HTTP upgrade fulfils the requirements on the Upgrade contract in a general way (not just WebSockets). The actual defining is probably not the task of this group, but the requirement most certainly is. It might be something as "simple" as adding a Sec- prefix or something, but just ensure that Upgrade can be used. This paves the way for not just WS using Upgrade, but potentially other protocols as well (as was originally intended).

As for the TLS nextprotoneg, I think making that a requirement for a WebSocket connection always runs on TLS is overkill. If nothing else, it is going to severely delay the number of available implementations until libraries with nextprotoneg appear. Specifically, I wouldn't start implementing a TLS handshake myself, so I'd have to wait for a compliant library. Obviously, I don't like that.

Vladimir

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 17 May 2010 12:12:56 +0200

On 17/05/10 11:29, Maciej Stachowiak wrote:
> Ian has claimed that the extra bytes after the request help WebSocket fail 
> early in the face of unaware proxies, though we do not yet have data showing
> this to be the case. We do know that without this, often the handshake will
> appear to succeed but transmitting messages will fail, for the network
> setups of many users.

I understand that is the case for the extra bytes.   But that is not
what I asked.     The fact that a requirement might be solve with a
non HTTP compliant mechanism does not mean that it can be solved
only by non HTTP compliant mechanisms.

> I don't think anyone has indicated a practical problem caused by the extra bytes.

I do not think there is consensus one way or the other on this. So I
do not think you can yet dismiss the problems raised.

> That being said, this whole issue is irrelevant if we switch to TLS-only and a 
> handshake based on nextprotoneg (and possibly TLS upgrade followed by nextprotoneg).

nextprotoneg is not yet widely available or even accepted as a standard, so while
I agree it is a promising development, unless we are prepared to significantly delay
websockets then I do not see how we can avoid some port sharing.

Note that even with TLS, this requirement is relevant, because without
nextprotoneg the two ends will not be able to agree that the connection
is a websocket connection and the connection will have to be HTTP compliant
over TLS until an upgrade is done.

> Therefore, I think the "HTTP compatible/compliant" requirement should be deleted 
> completely, or rewritten to make sense for any plausible handshake.

I don't see how your arguments to delete the HTTP compliant requirement would not equally
apply to the requirement below.   The only difference being that it weakens
the word "compliant" or "compatible" to just "consideration".

regards

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 17 May 2010 12:23:28 +0200

> reality, which is that it's the software that imbues them with meaning 

What software is required to imbue meaning to these words?

> consumption, we've achieved the goal of interoperability.

The arguments I have made have been entirely from the point of view
of the server and the client.

The server is going to have to send responses and is currently faced
with a difficult (or impossible) choice between sending a HTTP compliant
or a websocket compliant response.

A client is going to see non 101 HTTP responses - no matter if we rule
that is non websocket compliant.

In reality, clients and servers are going to have to talk compliant
HTTP to each other to deal with these cases - regardless of whatever
the protocol spec says.    Servers will send 4xx and 5xx responses
and clients will correctly interpret them.

So you can continue to pretend that these conversations will never
actually happen.... but that will not change the reality that they
will.

regards

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 17 May 2010 03:35:03 -0700

> only by non HTTP compliant mechanisms.

True. If someone comes up with a proposal to solve the same problem in a way that doesn't technically violate HTTP, that would be great, and we can consider it. What if there isn't? Do we give up and leave the protocol less effective than it could be for the sake of purity?

I can see how the idea of HTTP compliance should be a preference, but I don't see how it can be a hard requirement, if it turns out that we can make the protocol work better only by violating.

> do not think you can yet dismiss the problems raised.

I'm not dismissing problems that may be raised, I just haven't heard any description of how this may break a server. Either a pure HTTP server or a joint HTTP/WebSocket server.

You have raised the notion that it could be a problem if WebSocket clients followed up a refused WebSocket upgrade with HTTP requests, however, the client protocol does not allow this currently, so it is not at this time a practical problem.

Does anyone have an example of how this would cause a practical proble for a server?

> websockets then I do not see how we can avoid some port sharing.

TLS + nextprotoneg is a proposal for a specific way to do the port sharing. Tentatively, I think it seems better than the current scheme.

> over TLS until an upgrade is done.

My understanding is that the connection would fail at the TLS level if the server doesn't support nextprotoneg. 

> the word "compliant" or "compatible" to just "consideration".

I believe the requirement below is still relevant, because it focuses on the practical issues - making it possible for a combo server to serve HTTP and WebSocket over a single port. It is agnostic as to whether the negotiation is done at the HTTP level, the TLS level, or something else completely. Whereas "HTTP compliant" assumes that the negotiation will be at the HTTP level.

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Ian Hickson <[email-masked]>
Date: Mon, 17 May 2010 11:20:27 +0000 (UTC)

> or a websocket compliant response.

If the server is a Web Socket server, and it received a request from a 
Web Socket client, then why would it care about HTTP?

If the server is not a Web Socket server, then why would it ever consider 
sending a response back using Web Sockets?

If the server is aware of Web Sockets and HTTP, and gets a request from an 
HTTP client, then why would it respond with Web Sockets?

I really don't understand what the choice is. In every situation I can 
see, there's an unambiguous right response.

> that is non websocket compliant.

I don't know what you mean by "non websocket compliant" here. If the 
server isn't a Web Socket server, it can't possibly violate the Web Socket 
spec, since it isn't a conformance class that the Web Socket spec covers. 
The spec says how the client should handle non-Web Socket responses, it's 
not non-conforming to handle them.

> In reality, clients and servers are going to have to talk compliant HTTP 
> to each other to deal with these cases - regardless of whatever the 
> protocol spec says.

I have no idea what this means.

> Servers will send 4xx and 5xx responses and clients will correctly 
> interpret them.

Web Socket servers will not send 4xx and 5xx responses.

> actually happen.... but that will not change the reality that they will.

Why would they happen?

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 17 May 2010 14:00:49 +0200

> see, there's an unambiguous right response.

This is not an exhaustive list and you left out the situation
that I have been describing in my examples.

   The server is aware of both web sockets and HTTP and
   receives a request from a websocket client that is
   to be rejected.

For one of the many reasons I've outlined in my other email, that
request is handled either by code that does not know about
websockets, or is unable to act as the websocket spec requires
(ie closing connection).

Thus the only possible choice is for the server to send
a compliant HTTP response, which is contrary to the websocket
specification.

> I have no idea what this means.

Websocket clients are going to receive 404's, 503's etc.
They are going to have to handle them sensibly and report
reasonable errors to the user.

There are also arguments that it is desirable to allow
clients to actually interpret some of these responses.
It could for example handle a 401 or a 302.

> > actually happen.... but that will not change the reality that they will.

Please try reading some of the many examples that I have sent in
detail, but in summary:

  + authentication cookie missing
  + wrong URL given to the websocket client.
  + poor deployment (with apache or similar in front or
    in a server that does not support websocket).
  + some other exception made the webapplication fail
    to start, so the container sends a 503
  + some exception in handling the request causes a 500
  + some stupid caching proxy sends a 200 response

Websocket clients are going to receive 404's, 503's etc.
They are going to have to handle them sensibly and report
reasonable errors to the user.

regards

---

From: Julian Reschke <[email-masked]>
Date: Mon, 17 May 2010 14:12:42 +0200

> reasonable errors to the user.

For the record: +1

Best regards, Julian

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 17 May 2010 14:07:41 +0200

On 17/05/10 12:35, Maciej Stachowiak wrote:

> True. If someone comes up with a proposal to solve the same problem 
> in a way that doesn't technically violate HTTP, that would be great,
> and we can consider it.

I have given several such proposals.

The simplest was just to leave out the random bytes after the handshake
request, but to include them after the handshake response.

This is compliant and will still test if intermediaries forward non-HTTP
content.

There is no extra RTT over the current handshake as the client already
cannot send any messages until the bytes after the 101 are received

> I can see how the idea of HTTP compliance should be a preference, but 
> I don't see how it can be a hard requirement, if it turns out that we
> can make the protocol work better only by violating.

When operating in a standards based environment, the assumption should
be that standards are respected.

If there is a reason to violate the self evident truth that messages
should be HTTP on a HTTP connection, then I think those that
want to violate should make the case that violation is needed.

Then effort should be expended to see of the violated standard
can be improved.

Only then should a violation of an established standard be
contemplated... and then it would have to be for a very very
good reason and I'm not sure that an imperfect fast fail
mechanism is such a good reason.

Regardless, it should be the violators obligation to make
their case.

It should not be necessary for those that wish to respect
and existing standard to defend the need to respect it.
Unfortunately this is what is occurring because the
editor has included a mechanism in the draft that
did not have consensus.

regards

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 17 May 2010 14:27:44 +0200

On 17/05/10 12:35, Maciej Stachowiak wrote:

> if WebSocket clients followed up a refused WebSocket upgrade 
> with HTTP requests, however, the client protocol does not
> allow this currently, so it is not at this time a practical problem.

It is a circular argument to say that HTTP non-compliance is not a
problem because the current protocol forbids HTTP compliance.

I'm saying that the protocol should not be allowed
to forbid HTTP compliance.

Furthermore, I've provided many examples where it is impossible
for a server to actually respect any such compliance ban.

Finally I've also provided many examples where compliance
would also be a benefit (for authentication etc.)

regards

---

From: Salvatore Loreto <[email-masked]>
Date: Mon, 17 May 2010 17:02:32 +0300

(as individual)

if something goes over the port 80 (better over the HTTP well know ports)
I do expect the traffic to be conform to HTTP spec
until the HTTP spec agrees that it no long applies;
that means both ends have agreed that it has been upgraded!

/Sal

---

From: Dave Cridland <[email-masked]>
Date: Mon, 17 May 2010 15:34:55 +0100

> reasonable errors to the user.

I entirely and wholeheartedly agree. I think this is so utterly  
self-evident as to warrant no discussion whatsoever, but just to keep  
people happy, I'll continue with some examples of why this  
circumstance might occur.

Currently, in the XMPP world, there is BOSH. BOSH is a comet-like  
mechanism for tunelling XMPP (and other things) through HTTP. The URI  
is sometimes inherent in the client as delivered over HTTP; but it is  
also sometimes hand configured.

It makes sense that if we were to use WebSockets in a similar way,  
then a desktop client might initiate a session with an HTTP service  
without knowing a priori if it's an HTTP or WebSocket service. If the  
URI is wrong, the client will receive an HTTP error. This will have  
to be handled sanely by the client and presented to the user.

If anyone wants to suggest that these clients should be using TCP  
instead, I'll point out in a highly patronizing way that taking XMPP  
- a TCP native protocol - and slinging it over HTTP as BOSH sprang up  
due to this being impractical in some limited networks, so, y'know,  
duh.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Scott Ferguson <[email-masked]>
Date: Mon, 17 May 2010 09:16:04 -0700

> want to violate should make the case that violation is needed.
+1

If you're using the HTTP port, you need to follow the HTTP protocol. If 
you don't like the HTTP protocol, use another port.

> and existing standard to defend the need to respect it.
+1

I'm surprised this is open for debate.

-- Scott

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 17 May 2010 09:32:59 -0700

> cannot send any messages until the bytes after the 101 are received

That's worth testing, but I wouldn't assume it would be equally effective without testing.  

Servers are in general allowed to send a response without any Content-Length header or any other explicit indicator, though it is not allowed after a 1xx response. However, it seems likely to me that many unaware proxies would forward everything the server sends after the response headers, whether or not a response body is actually expected. This would still make it look like the handshake succeed.

It seems like we could determine whether either proposal works through experiments.

So if it your proposal is effective, great. What if it isn't?

(Of course, this is all moot with the TLS-level handshake.)

> editor has included a mechanism in the draft that

To match the above, it sounds like we should frame the requirement as a SHOULD rather than a MUST (i.e. violate only if there is a specific good reason). With that change, and with a change in wording to make clear the requirement should only be considered if the handshake ends up operating at the HTTP level, I think it would be reasonable.

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 17 May 2010 09:34:48 -0700

> to forbid HTTP compliance.

I think it's inaccurate to say that the requirement to close a client connection on a failed HTTP upgrade means that the spec "forbids compliance". It is completely compliant for an HTTP client to close the connection at any time.

> for a server to actually respect any such compliance ban.

The server's behavior isn't relevant to what I said above. It doesn't matter what the server does if the client closes the connection.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 17 May 2010 19:05:50 +0200

> I think it's inaccurate to say that the requirement to close a client connection on a failed HTTP upgrade means that the spec "forbids compliance". It is completely compliant for an HTTP client to…

The websocket specification says that a server may only send a 101 or
close the connection. That is forbidding compliant responses like 503,
401 etc.

The websocket spec is entitled to say that a websocket client should
treat all non-101 responses as failures, but it cannot say that a
server cannot send a non-101 response.

regards

---

From: Daniel Stenberg <[email-masked]>
Date: Mon, 17 May 2010 19:41:42 +0200 (CEST)

On Mon, 17 May 2010, Scott Ferguson wrote:

> If you're using the HTTP port, you need to follow the HTTP protocol. If you 
> don't like the HTTP protocol, use another port.

+1

-- 

  / daniel.haxx.se

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 17 May 2010 18:48:24 +0100

Vladimir Katardjiev wrote:
> I assume the problem/desire/requirement is to reduce the number of
> round-trips required for the establishment of a new connection. The
> "random bytes" are a byproduct of that -- ensure that a WebSocket
> connection's "open" event is fired when the client knows the
> connection is actually open, not just when the connection is "maybe"
> open.

I'm pretty sure the extra random bytes are only to ensure a
non-WebSocket HTTP server will abort the connection, instead of
wrongly processing WebSocket data as a HTTP request.

That's a good goal, but unfortunately it prevents retrying a different
protocol (such as HTTP-based or a different Upgrade protocol), or even
a different WebSocket subprotocol, from being tried.

It doesn't help RTTs.  If anything, the extra random bytes increase
RTTs: When negotation fails, they force the client to use a new
connection for the next attempt.

They also increase RTTs on success, by preventing the pipelining of
initial client messages.

The "open" event happens when the server's negotiation response is
received at the client.  The extra random bytes don't assist that in
any way, that I see.

(You may guess that I don't agree with the extra random bytes.)

> Maybe it'd be good to rethink how the entire thing works. So far,
> the WebSocket handshake itself has been "baked into" whatever
> previous authentication there is/has been. So... Make a proper WS
> handshake. It can be asynchronous (a la SPDY), and thus not require
> an additional RTT. Allow WS data to be sent from the API before open
> is fired, at the risk of being lost. The current API always risks
> data being lost anyway, unless some ack protocol is introduced, so
> this is OK in my view. Fire open after the WS handshake
> finishes. There's no speed loss for expert programmers who desire
> the extra ms boost, while the contract of an established WS
> connection is preserved for amateurs.

All good ideas, imho :-)

Using a "proper WS" handshake was proposed recently by Greg.  Sending
things asynchronously during negotation was proposed by me; and
therefore also allowing send before the API's open completes.

Maybe we need an issue tracker of some kind to focus these ideas! :-)

-- Jamie

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 17 May 2010 21:28:23 +0200

>a compliant HTTP response, which is contrary to the websocket

It seems to me Ian is saying a hybrid HTTP/Websocket server is required
to behave as mandated by the Websocket protocol only when it choses to
accept a Websocket connection request. If it decides to speak only HTTP
to some client, then it needs to conform only to the HTTP specification.
You on the other hand seem to be saying, because it is a hybrid, and be-
cause it is responding to a Websocket connection attempt, the Websocket
requirements do apply to the server.

It does seem clear to me that the Websocket specification cannot mandate
server behavior of any kind before the server decides to accept the re-
quest to switch the protocol to the Websocket protocol, which it would
signal by sending a 101 response as specified in the Websocket specifi-
cation. If it does not do so, the negotiation to switch protocol failed.

>reasonable errors to the user.

(I note that can lead to information disclosure problems.)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 17 May 2010 23:37:31 +0200

* Maciej Stachowiak wrote:
>True. If someone comes up with a proposal to solve the same problem in a
>way that doesn't technically violate HTTP, that would be great, and we
>can consider it. What if there isn't? Do we give up and leave the
>protocol less effective than it could be for the sake of purity?
>
>I can see how the idea of HTTP compliance should be a preference, but I
>don't see how it can be a hard requirement, if it turns out that we can
>make the protocol work better only by violating.

I do not think the requirement here is useful as sending only proper
HTTP messages over what is assumed to be a HTTP connection is a matter
of course and not some self-imposed requirement; there is no need to
spell that out. If you want an option to violate the HTTP protocol if
deemed beneficial, you would have to argue for putting that option into
the document and not have others argue why underlying specifications
should be adhered to.

However, as we have a requirement to be able for HTTP and Websocket
implementations to share a port, the option would be in conflict with
that requirement (consider an intrusion detection system; it receives a
packet for port 80, analyzes it and finds it has a HTTP request followed
by rubbish where another request is expected; it then rejects the whole
thing). Note that Ian's draft actually argues that port sharing is en-
abled by following the HTTP protocol during the handshake.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Ian Hickson <[email-masked]>
Date: Mon, 17 May 2010 23:57:19 +0000 (UTC)

On Mon, 17 May 2010, Greg Wilkins wrote:
> > If the server is not a Web Socket server, then why would it ever 
> > consider sending a response back using Web Sockets?
> > 
> > If the server is aware of Web Sockets and HTTP, and gets a request 
> > from an HTTP client, then why would it respond with Web Sockets?
> 
> This is not an exhaustive list and you left out the situation that I 
> have been describing in my examples.
> 
> request is handled either by code that does not know about websockets, 
> or is unable to act as the websocket spec requires (ie closing 
> connection).
> 
> Thus the only possible choice is for the server to send a compliant HTTP 
> response, which is contrary to the websocket specification.

This is just an HTTP server getting a request it doesn't want to handle -- 
it's perfectly allowed to do whatever it wants from an HTTP point of view, 
and the Web Socket client will disconnect once it sees the handshake is 
not what a Web Socket server would send.

Again, I don't see a problem here.

> reasonable errors to the user.

Clients can do whatever they want in their user interface. Nothing in the 
spec restricts that.

> It could for example handle a 401 or a 302.

Handling redirects is remarkably complex due to the origin model. Handling 
401s used to be in the spec but was removed based on implementation 
feedback. There's no point us requiring stuff that implementors won't 
implement. (It's not really clear what the UI should be for a 401, since 
the user doesn't know he sent a request. Just prompting the user 
unexpectedly isn't an option because it habituates users to behaviour that 
is too easily replicated by phishing scams.

> Please try reading some of the many examples that I have sent in detail, 
> but in summary:
> 

Why wouldn't this be handled either in the subprotocol or (more likely) by 
the JS before it invokes the WebSocket constructor?

>   + some stupid caching proxy sends a 200 response

These all seem like variants on the same basic idea: the script tried to 
connect to the server, but the connection failed. There are many more ways 
the connection can fail:

   + man in the middle proxy doesn't let it go through
   + server is overloaded and times out
   + server is offline
   + user is not connected to the network
   + server's DNS is misconfigured

The list goes on. In all of these cases, the script gets an "error" 
event. We can't report anything else to the script, because that would be 
an information leak (e.g. it would let you study intranet topologies). The 
user agent can tell the user whatever it wants, though, that's fine. 
Developers would probably want to see the exact bytes sent back by the 
server, and user agents can definitely expose that.

On Mon, 17 May 2010, Dave Cridland wrote:
> mechanism for tunelling XMPP (and other things) through HTTP. The URI is 
> sometimes inherent in the client as delivered over HTTP; but it is also 
> sometimes hand configured.
> 
> It makes sense that if we were to use WebSockets in a similar way, then 
> a desktop client might initiate a session with an HTTP service without 
> knowing a priori if it's an HTTP or WebSocket service.

I don't understand how that could happen. Could you elaborate on what kind 
of desktop client you mean? When would you ever connect to a port without 
knowing ahead of time what protocol you want to talk?

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 18 May 2010 01:37:53 +0100

> knowing ahead of time what protocol you want to talk?

For example, a social networking client running in a browser, designed
to fetch feed information from numerous different services from other
providers (imagine Yahoo, Facebook, Twitter etc.), those services
running a variety of feed protocols over WebSocket.

When the user is given a URL for a third party bidirectional service,
the client tries WebSocket subprotocol A first, then WebSocket
subprotocol B, then C, then a HTTP hanging-GET equivalent.

Although you _could_ give the protocol along with the URL to users,
clearly that would be avoided as much as possible.  Less is better.

And anyway, if you did that, it would break when the provider "Yahoo"
switches their service from standardised subprotocol B to A (because
it works better in some way, and which has no visible effect on
Yahoo's own clients).  So the multi-service client needs to try them
in turn, not be told what to use.

-- Jamie

---

From: Ian Hickson <[email-masked]>
Date: Tue, 18 May 2010 02:39:36 +0000 (UTC)

On Tue, 18 May 2010, Jamie Lokier wrote:
> > > a desktop client might initiate a session with an HTTP service 
> > > without knowing a priori if it's an HTTP or WebSocket service.
>

Since a Web Socket URL starts with "ws" and an HTTP URL starts with 
"http", I don't think that's a realistic scenario.

Furthermore, if a script really did do this (e.g. because it was given 
just a domain name, not a URL), then it would use the WebSocket object to 
open the Web Socket connection, and XMLHttpRequest to open the HTTP 
connection, and they would be separate connections -- at no point would 
the client open a connection without knowing whether it's expecting HTTP 
or Web Socket, as Dave suggested.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 18 May 2010 09:48:08 +0200

On 18 May 2010 01:57, Ian Hickson <[email-masked]> wrote:
>
> Again I don't see a problem here
>
Ian,

If you don't see a problem, and believe that combined servers can
respond as HTTP compliant servers, then why is the proposal to change
"compatible" to "compliant" is being opposed so vigorously? If there
is no problem for a combined server staying HTTP compliant if the
handshake is not accepted, then let's just say so.

For the most part, the only change in the specification that would
result from this is perhaps a change in language. So that for example
instead of saying the only *allowed* response code and headers are ...
the spec should say that the only *accepted* response code and headers
are ...

There is a separate debate regarding if the random bytes after the
request are HTTP compliant or not, but that is a different debate to
should we be compliant or not.

It is also a separate debate about what HTTP features a websocket
client *might* choose to respect or be compelled to reject. I do
understand that there may be problems using some HTTP features, but
they should be considered on a case by case basis and not broadly
ruled out by needless restrictions on headers and response codes. For
example:
+ user-agent headers can surely be allowed
+ some Via headers would be great to work out what intermediaries are
supporting websocket
+ I understand the problem of a 401 challenge collecting credentials
from the user, but in many/ most cases the browser will already have
collected credentials and have them available.
+ cross domain redirection might be a problem, but there are many use
cases for same domain redirection.
+ it may be unsafe to expose error details in the javascript API, but
that is an entirely different matter to having a protocol that can
understand errors, even if only to displayed them on the console of
the browser

Note that some client/servers might even choose to support features on
the HTTP handshake that are not even mentioned in any specification,
because it is the intent of HTTP to be openly extensible.

You might be opposed to the usage of all of such features, but please
make your case on the basis of a discussion about the features rather
than opposing the simple change of "compatible" to "compliant"
or the existence of blanket ban on the sending arbitrary headers or of
4xx or 5xx status codes etc.
If you think the handshake should not be openly extensible, then just
make that case, rather than try to force the result by a needlessly
restricted handshake.

regards

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 18 May 2010 13:03:28 +0100

Maciej Stachowiak wrote:
> Ian has claimed that the extra bytes after the request help
> WebSocket fail early in the face of unaware proxies, though we do not
> yet have data showing this to be the case. We do know that without
> this, often the handshake will appear to succeed but transmitting
> messages will fail, for the network setups of many users.
> 

Yes, we have.  It block useful things:

   - Trying multiple WebSocket subprotocols in turn (on the same connection)
   - Trying a HTTP-based protocol if WebSocket is unavailable (ditto)
   - HTTP authentication if the server responds with an authentication request
   - Pipelining WebSocket frames to avoid the negotation RTT delay

Worse, it makes them difficult to add in future, if they are found to be good.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 18 May 2010 13:12:45 +0100

> or Web Socket, as Dave suggested.

Eh?  The client would use a WebSocket object first, but that object wouldn't
know "a priori if it's an HTTP or WebSocket service", to use your words.

That, therefore, is an example of connecting to a service without
knowing a priori what protocol is available.  That what you said you
could not think of why it would be don, isn't it?

Anyway, I think what you're saying is that the WebSocket object and
XMLHttpRequest objects will open separate TCP connections.

I'm arguing that the second object should be able to reuse the
connection resulting from the first one's failure to negotiate on the
same port - and that that's even more valuable when the client needs
to attempt a sequence of WebSocket subprotocols in turn before it can
begin real communication.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 18 May 2010 14:31:55 +0200

On 18 May 2010 14:12, Jamie Lokier <[email-masked]> wrote:
> same port

Exactly!

If the handshake is HTTP compliant, then the connection for a
websocket handshake could be taken from the existing pool of
idle connections to a host.  That would save the time needed to
establish the connection.

If the handshake is HTTP compliant, then a rejected websocket
handshake could put the connection back into the idle pool to be
available for subsequent handshakes or XHRs.

If the handshake is not HTTP compliant, then the connection
needs to be created on every handshake and disposed as
untrusted after every rejection.

regards

---

From: Pieter Hintjens <[email-masked]>
Date: Tue, 18 May 2010 15:29:13 +0200

> reasonable errors to the user.

This.  and This.

-Pieter

---

From: Roberto Peon <[email-masked]>
Date: Tue, 18 May 2010 10:26:01 -0700

> untrusted after every rejection.

Which, to use the technical phrase, sucks.

-=R

---

From: Ian Hickson <[email-masked]>
Date: Tue, 18 May 2010 21:16:24 +0000 (UTC)

On Tue, 18 May 2010, Jamie Lokier wrote:
>
> [The extra bytes in the handshake] block useful things:
> 

This is currently blocked by far more important things, like the way the 
spec says to close the connection if the subprotocol isn't recognised.

If we want to support multiple subprotocols at once, we can do so quite 
easily by just making the subprotocol list be comma-separated. Would this 
be a good idea?

>    - Trying a HTTP-based protocol if WebSocket is unavailable (ditto)

That wouldn't work anyway because the WebSocket object isn't the same 
object as the XMLHttpRequest object and they therefore create entirely 
different connections. In any case, why wouldn't you know that Web Socket 
is available? I don't understand why you would guess that it is and then 
find it isn't. You need a URL to connect to a Web Socket, why would you 
make up a URL without knowing whether it'll work?

>    - HTTP authentication if the server responds with an authentication request

This can be supported entirely in Web Socket if we want to do that. 
Currently, it's not clear that supporting this is even desireable, due to 
the UI issues with doing so.

>    - Pipelining WebSocket frames to avoid the negotation RTT delay

Pipelining WebSocket frames is a security problem for WebSocket API 
clients, so that's not a problem. The random bytes do not stop pipelining 
of frames for dedicated clients where there is no security problem.

On Tue, 18 May 2010, Jamie Lokier wrote:
> > > For example, a social networking client running in a browser, 
> > > designed to fetch feed information from numerous different services 
> > > from other providers (imagine Yahoo, Facebook, Twitter etc.), those 
> > > services running a variety of feed protocols over WebSocket.
> > > 
> > > When the user is given a URL for a third party bidirectional 
> > > service, the client tries WebSocket subprotocol A first, then 
> > > WebSocket subprotocol B, then C, then a HTTP hanging-GET equivalent.
> > 
> > just a domain name, not a URL), then it would use the WebSocket object 
> > to open the Web Socket connection, and XMLHttpRequest to open the HTTP 
> > connection, and they would be separate connections -- at no point 
> > would the client open a connection without knowing whether it's 
> > expecting HTTP or Web Socket, as Dave suggested.
> 
> Eh?  The client would use a WebSocket object first, but that object 
> wouldn't know "a priori if it's an HTTP or WebSocket service", to use 
> your words.

The WebSocket object doesn't support HTTP. It just connects to a WebSocket 
server or fails if the server doesn't send back the right handshake. The 
situation Dave describes doesn't come up in this scenario.

> XMLHttpRequest objects will open separate TCP connections.

Yes.

> same port - and that that's even more valuable when the client needs to 
> attempt a sequence of WebSocket subprotocols in turn before it can begin 
> real communication.

WebSocket and HTTP are different protocols. Reusing the connection makes 
as much sense as reusing the connection of an FTP connection attempt to do 
an HTTP connection attempt.

On Tue, 18 May 2010, Greg Wilkins wrote:
> 
> If the handshake is HTTP compliant, then the connection for a websocket 
> handshake could be taken from the existing pool of idle connections to a 
> host.  That would save the time needed to establish the connection.

The resemblence to HTTP is nothing more than a hack to alolow us to share 
ports in certain advanced scenarios. Most Web Socket servers will know 
nothing about HTTP. Reusing connections is a level of complexity that is 
completely unwarranted and that would only be useful in the rarest of 
cases. It's a proposal that lies on completely the wrong side of the 80/20 
line and would introduce _massive_ complexity for authors, who would have 
no idea why their WebSocket servers were suddenly receiving random HTTP 
requests and vice versa.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 19 May 2010 01:51:04 +0200

> spec says to close the connection if the subprotocol isn't recognised.

That's an entirely circular argument using a prohibition in the spec
to justify the design of the spec.

> different connections.

Again a circular argument, depending on the current implementation of
the spec to justify the design of the spec.

XHR objects are different from image elements and many other js
elements with a src attribute, yet they all manage to share the
existing connection pools in the browsers.

> the UI issues with doing so.

I agree that UI issues will limit the usability of some authentication
mechanisms in the normal browser use case. But there are also many
authentication use-cases that do not need UI or that might have UI
from a special purpose client.

> an HTTP connection attempt.

Why?  Websockets come from browsers that are highly likely to have
just loaded a page from the the server over HTTP using between 2 and 6
HTTP connections, several of which are very likely to be idle once all
the JS and CSS have been loaded. The websocket handshake is able to
upgrade a HTTP connection to a websocket connection, so it makes
perfect sense to be able to use one of the existing idle connections,
rather than suffer a RTT creating a new connection.

> ports in certain advanced scenarios.

Then let's not standardize a hack.  Instead let's standardize an
orderly upgrade  from compliant HTTP to websocket. It will make no
difference to minimal clients.

> Reusing connections is a level of complexity that is

Nobody is saying that connections MUST be recycled by all websocket
clients. There is no complexity cost for minimal clients. But browsers
are not minimal clients and already have connection pools because the
complexity is worth it for the benefits. We should allow them the
option to use them for websocket if there is a benefit.

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 19 May 2010 02:32:38 +0100

> spec says to close the connection if the subprotocol isn't recognised.

Correct, that's also an unnecessary blocker.

> easily by just making the subprotocol list be comma-separated. Would this 

I think it is a good idea, although there is a risk of low-quality
server but significant implementations matching a literal string,
breaking when other values are added to the comma-separate list, and
therefore making it impossible for clients to actually use the
capability.

But assuming the comma-separated list did catch on, a consequence of
that would be the "below the API" part of WebSocket would have a place
to add its own distinct entries to the comma-separated list, for
recognition by the other side as transport option requests (such as
compression, etc.), safe in the knowledge it wouldn't break negotiation.

(Separate headers would be much cleaner for that, though).

> object as the XMLHttpRequest object and they therefore create entirely 

Just being different objects is not technical reason for different connections.
Separate XMLHttpRequest objects have no problem sharing connections.
There is no technical reason why a WebSocket object could not do the
same, just as easily.  Especially in browsers which routinely do this!

> In any case, why wouldn't you know that Web Socket is available? I
> don't understand why you would guess that it is and then find it
> isn't. You need a URL to connect to a Web Socket, why would you make
> up a URL without knowing whether it'll work?

Earlier in this thread, in a reply to you, we already gave an example.
It was the social networking client that talks to numerous de facto
standardised but different WebSocket protocols that the user shouldn't
have to know about.

Basically because autonegotiation is more user-friendly, both in terms
of users having to be given and enter fewer details, and in terms of
telling the user what went wrong if things didn't work out.

As you noted, people don't always pass around URLs - they often pass
around just a domain name, or a truncated URL that doesn't include the
"http:" prefix.

I would agree if it were _complicated_ to do, but we are talking about
really trivial stuff here.  Trying one thing, then trying another, is
already handled by the handshake, and it is utterly trivial to do from
a script.  It's so easy that it's hard to see why any random web
developer wouldn't use it whenever it seemed like a good idea.

It seems obvious to me that countless Javascripts will do exactly that.

The only question, then, is whether the spec forces that to be
unnecessarily inefficient for the sake of a weak security mechanism
which has alternatives.

> the UI issues with doing so.

(On that topic, how do you propose to get through a proxy on port 443
that needs proxy authentication before it passes a CONNECT request?)

> clients, so that's not a problem.

I think that's false.

We have discussed no less than 6 solutions which support RTT-less
pipelining of WebSocket negotiation and initial data, while preventing
them from being misinterpreted by a HTTP server, and in some cases, by
any kind of server (any port) if it can't be tricked by the specified
WebSocket negotiation anyway.

I'm assuming you mean the security problem is a browser client script
opening a WebSocket and attempting to trick an HTTP server or other
kind of server, by sending "raw" requests that could be
misinterpreted, before the server is confirmed as a WebSocket server?

Did you mean something else?  I didn't find an explanation in draft76
of the security issues its mechanisms attempt to prevent; sorry if I
missed it.

> The random bytes do not stop pipelining of frames for dedicated
> clients where there is no security problem.

Dedicated clients that connect to _known_ servers can skip parts of
the WebSocket spec they don't care about anyway. ;-)

Any kind of client connecting to a server it doesn't know will accept
the connection has to follow the commonly agreed rules, which means it
can't pipeline anything until it knows the result of negotation.  But
that's a propery of the current method; it's not intrinsic to
pipelining, as we have already discussed methods without that limitation.

> an HTTP connection attempt.

This is different.  Both WebSockets and HTTP requests are transient in
many cases.  If FTP was also transient, and it shared the same port
with HTTP, and many servers especially those designed for high
performance implemented both together, and many clients implemented
both, then of course it would make sense to _permit_ connection reuse
between them, without _requiring_ any implementation to do so.

There have been many scenarios presented where it shows a tangible
benefit, in mails to you and the list.

The sole effect of mandating that an idle HTTP connection cannot be
subsequently used for a WebSocket is pointless slowness.
It's not as if the WebSocket negotation will confuse an HTTP server any
differently than it will on a new connection.

The case for reusing a failed WebSocket negotation is not as strong as
that, due to the (imho, dubious) security mechanism of the random
bytes.  It is still a pessimisation, though.

There is a much stronger case for reusing a WebSocket connection after
a gracefully close that puts it into some kind of idle state.  That's
because users follow links every few seconds - and therefore WebSocket
scripts in ordinary pages will connect and disconnect from the same
server and port every few seconds in the current model.  Grossly inefficient.
But that's a digression; we can't do advanced stuff if we can't do the
simple stuff.

> completely unwarranted

Nobody is suggesting that clients _must_ reuse connections or that
servers _must_ support them.

> It's a proposal that lies on completely the wrong side of the 80/20 
> line and would introduce _massive_ complexity for authors,

Patently false.  Making it optional for both sides adds *zero*
complexity for authors who don't do it.  I am not seeing how you can
think it makes any difference to them.

Seriously, how do you imagine it affects them?

> who would have no idea why their WebSocket servers were suddenly
> receiving random HTTP requests and vice versa.

That's a function of connecting to the wrong type of server already,
and it's already dealt with by the spec'd negotation, which the wrong
type of server handles already by design.  Nothing new there.

Thanks,
-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 19 May 2010 02:35:35 +0100

Roberto Peon wrote:
> 
>    On Tue, May 18, 2010 at 5:31 AM, Greg Wilkins <[1][email-masked]>
>    wrote:
> 
>    On 18 May 2010 14:12, Jamie Lokier <[2][email-masked]> wrote:

What do you mean?  I don't understand your point at all.

Do you mean that the current design preventing reuse sucks, or that
doing reuse would suck (even though you wouldn't have to if you didn't
want to)?

-- Jamie

---

From: Scott Ferguson <[email-masked]>
Date: Tue, 18 May 2010 18:54:45 -0700

> no idea why their WebSocket servers were suddenly receiving random HTTP 

Huh? HTTP connection pooling isn't hard at all, certainly not 
"massively" complex, and would be invisible to the JavaScript 
programmers and WebSocket service writers. And pooling wouldn't dispatch 
random HTTP requests to the WebSocket resources.

TBH, this paragraph reads as if it was written by someone who doesn't 
understand how a HTTP client or server is written.

-- Scott

---

From: Roberto Peon <[email-masked]>
Date: Tue, 18 May 2010 18:56:09 -0700

> doing reuse would suck (even though you wouldn't have to if you didn't

Heh.

And in case I'm misunderstanding the joke when it was a serious question,
preventing reuse sucks...

-=R