Subject: Re: [hybi] Feedback on thewebsocketprotocol-22

From: Ian Hickson <[email-masked]>
Date: Fri, 4 Sep 2009 05:50:37 +0000 (UTC)

On Sun, 16 Aug 2009, Jamie Lokier wrote:
> > > 
> > > On that note, is the Origin model used for WebSockets the same as 
> > > that in Cross-Origin Resource Sharing, http://www.w3.org/TR/cors/ ?
> > 
> > Assuming you mean this one:
> > 
> >    http://dev.w3.org/2006/waf/access-control/
> > 
> > Then no, not quite.
> 
> What are the logical differences?

I'll let you do your own comparison analysis, but the reason for the 
differences just boil down to the difference in requirements. CORS must 
protect resources on servers that don't support CORS; Web Socket servers 
always know the details of the handshake. CORS deals with individual 
resources on an HTTP server; Web Sockets are a long-lived connection-based 
mechasism. CORS needs to be a declarative system deployable on static 
servers; Web Socket servers can be dynamic and can therefore specify the 
single origin they want to deal with.

> Even though you wouldn't use CORS, the method of doing the pre-flight 
> check (and caching the result) could be much the same.

It is much the same. CORS was used as the basis for the Web Socket 
handshake.

> > > instead of the verbatim byte sequence which looked (to me) rather 
> > > fragile?
> > 
> > Could you elaborate on why it would be fragile?
> 
> Unavoidable proxies can modify it, even in trivial ways like reordering 
> headers or altering the HTTP version number, as well as adding Via 
> (though not all do that).

This will result in the connection failing, which is what we want. If this 
is fragility, then fragility is a feature, IMHO.

> It's quite clear that when going through such a proxy, it's unlikely 
> that the WebSockets protocol would work.  However it seems a poor match 
> to use this as a method of detecting that, because sometimes it would 
> pass the request unchanged.

How else would you detect it?

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 05 Sep 2009 21:56:54 +1000

> is fragility, then fragility is a feature, IMHO.

Ian,

can you cite any examples where different ordering of headers in a
HTTP message (or any other protocol) has caused any problems?

What class of failure do you think this detects/prevents?

If the request is semantically correct, then
it is a valid correct request - regardless of the order of
the headers.

> How else would you detect it?

Upgrade is a hop by hop header.  A proxy that does not understand
websocket is required to not pass on the upgrade header.

A proxy that does understand websocket will pass on the request,
except the requirement that the headers be strictly order and
contain no additional information will break many proxy
implementations - thus the effort required to update them
to support websocket is increased.   Moreover, because ordering
changes can be non deterministic (eg because of hash table
load factors), then a proxy might appear to comply
to strict ordering, but might fail under different
circumstances.   That is fragile!

regards

---

From: Jamie Lokier <[email-masked]>
Date: Sat, 5 Sep 2009 15:05:51 +0100

> it is a valid correct request - regardless of the order of

I think Ian's position is that WebSocket is _not_ HTTP, it just
resembles it in the initial handshake, so if you accidentally send a
WebSocket request to a HTTP server or via a HTTP proxy, he _wants_
that to fail quickly with a detectable error, and using a strict
header sequence increases the chance of that _intended_ error response.

Unfortunately it's not going fail reliably either.  Without testing,
we must assume it will pass through some proxies and some servers, and
won't pass through others, in a rather random way.

So as protocol engineering goes, it's a poor solution because it looks
like it won't achieve it's stated goal of failing reliably either.

I'm happy with the principle that it should fail (even though it's not
the approach I would take, which gracefully transitions to an
HTTP-compatible and proxy-compatible mode).

But imho, it's important that it _does_ fail reliably.  Having it
appear to succeed and then mangle messages or just hang when used from
behind some proxies, would not make for a very good web experience as
there's no fast and easy way for web apps to transition to a different
mechanism in that case.

> websocket is required to not pass on the upgrade header.

Imho that's a better solution for getting proxies to reject what they
don't understand.

"Connection: Upgrade" as well would increase the chance of a proxy
following the rule it is supposed to follow :-) but I don't know if
that's practically needed.

It'll be necessary to test a few proxies and see what they do with
Upgrade, as I guess it hasn't been much deployed to date.  They might
not do the right thing.

Note that Upgrade is not in HTTP/1.0, so a HTTP/1.0 proxy will not
remove it.  That has to be handled too.

I don't know what intercepting proxies that are in use do, especially
concealed ones.

> circumstances.   That is fragile!

I agree.  Since many proxies already implement CONNECT, it should not
be hard to add support for Upgrade as well if they don't already.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 06 Sep 2009 11:42:54 +1000

> header sequence increases the chance of that _intended_ error response.

Jamie (acting as Ian surrogate :),

I'm not sure how strict header sequence helps if a Websocket request
is sent to a HTTP server or via a proxy.   If the proxy or server support
websocket then who cares about the order.  If the proxy or server don't
support websocket, then they are not going to enforce strict ordering.
However, you would expect them to fail due to the upgrade header
not being supported.

If a HTTP request is accidentally sent to a Websocket only server,
then it wont have a the upgrade headers - so the websocket server
can fast fail it on that basis.  Header ordering is not needed.

I cannot imagine a scenario where a HTTP request accidentally
is sent to a websocket server and it accidentally happens to
have the URL of a websocket resource and accidentally happens
to have the upgrade header and accidentally happens to list
upgrade in the connection header.

That is as likely million monkeys on a million keyboards
accidentally writing a websocket client.

cheers

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 7 Sep 2009 13:53:21 +0100

> support websocket, then they are not going to enforce strict ordering.

Fwiw, I agree with you on all points so far.

> However, you would expect them to fail due to the upgrade header

Stupid proxies might just forward the request without obeying the rule
to stript the Upgrade header.  We don't know without testing.  True
HTTP/1.0 proxies will definitely do that.  You should be able to
detect that from the HTTP version of the request, but then again I
have heard rumours of HTTP/1.0 proxies which forward the request
version unchanged.

> can fast fail it on that basis.  Header ordering is not needed.

Not just header ordering, but strict octet sequence!  I agree that
header ordering / strict octets is pointless, and incongruous with how
the rest of HTTP is formulated, and will make some implementations
unnecessarily complicated.

> accidentally writing a websocket client.

No, but can it be done deliberately using XmlHttpRequest in any browser?
If the answer is yes, how much of a security hole is that?

-- Jamie

---

From: Ian Hickson <[email-masked]>
Date: Thu, 17 Sep 2009 10:36:03 +0000 (UTC)

On Thu, 3 Sep 2009, SM wrote:
> At 12:54 25-08-2009, Graham Klyne wrote:
> > re: http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-35
> 
> This document gives a new meaning to work-in-progress. :-)  The I-D is 
> currently at its 36th version.  The draft contains several "_This 
> section is non-normative._".  Is the document incomplete?

It's not incomplete, but it's also not finished, if that makes sense.

> From the Abstract:
> 
>   "It is intended to fail to
>    communicate with servers of pre-existing protocols like SMTP or HTTP,
>    while allowing HTTP servers to opt-in to supporting this protocol if
>    desired."
> 
> That's the only mention of "SMTP" in the draft.  I suggest describing 
> what the draft is about instead using such a sentence.

That is part of what it is about. Could you elaborate on why that sentence 
isn't useful? What else should the abstract say, if not that?

> In the Introduction section:
> 
>   "Basically it is intended to be as close
>    as possible to just exposing raw TCP/IP to script as possible given
>    the constraints of the Web"
> 
> What is the "Web"?

Is this a rhetorical question? Is the paragraph not clear?

> In Section 2:
> 
>   "Requirements phrased in the imperative as part of algorithms (such as
>    "strip any leading space characters" or "return false and abort these
>    steps") are to be interpreted with the meaning of the key word
>    ("must", "should", "may", etc) used in introducing the algorithm."
> 
> This draft uses RFC 2119 key words.

Yes.

> In Section 4.3:
> 
>  "To *fail the Web Socket connection*"
> 
> No comment. :-)

?

> In its present state, this draft would not survive a review.

Could you elaborate?

On Sun, 6 Sep 2009, Graham Klyne wrote:
> 
> I still find the abstract a bit obscure.

Could you elaborate on this? I just tried writing a new abstract and I 
basically ended up with almost exactly what the spec already has.

On Tue, 8 Sep 2009, Graham Klyne wrote:
> > > Also, along the way, it might be worth mentioning efforts like 
> > > cometd/bayeux and BOSH and saying how this proposal is similar 
> > > and/or different.
> > 
> > I'm not really sure what I would say here.
> 
> I think that an informative reference to 
> http://www.ietf.org/id/draft-loreto-http-bidirectional-00.txt and a 
> high-level comparison with the long-poll and streaming approaches 
> described there would do it.  And mention why you are not just using 
> COMET/BAYEUX or BOSH which are mentioned there.  You might even consider 
> contributing a section describing your approach to the authors of that 
> document.

I really don't feel comfortable having a specification compare itself to 
other specifications. I always feel like this is rather presumptuous, and 
would rather readers draw their own conclusions. (HTML5 had a number of 
such sections a few years ago, and since then they've all been removed, 
one by one, as various people either took offence, or asked what the point 
was, or said they disagreed with it, etc.)

I'd rather the spec just stayed focus on what it is rather than on what it 
is not or what else exists.

> > > 6. Provide a protocol state diagram or similar overview.
> > 
> > I'm not sure I understand what that would look like for this protocol.
> > 
> > Do you mean something like this?:
> > 
> >    HANDSHAKE --> FRAME TYPE <-------------------------.
> >                       |                               |
> >                       +---0x00---> DATA --> 0x00 --->-+
> >                       |                               |
> >                       `---0x80---> LENGTH -> DATA -->-'
> 
> Maybe that:  if it's really that simple, then a diagram like this helps to
> emphasize that point.  See also the state diagrams in RFC821.

Ok, I've added a diagram to the intro. Hopefully it'll help someone.

> > > 7. Section 3:  why is this based on [WEBADDRESSES] rather than 
> > > [RFC3986]? I can't offhand tell if [WEBADDRESSES] is acceptable as a 
> > > normative reference.
> > 
> > RFC3986 doesn't define how to handle errors when parsing URIs.
> 
> Hmmm... I think it would still be better to refer to RFC3986 for the 
> base syntax, since many developers are familiar with that, and refer 
> additionally to [WEBADDRESSES] for the error handling material.

I expect the reference to [WEBADDRESSES] to eventually become a reference 
to IRI bis or URI bis, so I haven't made this change. ([WEBADDRESSES] 
isn't actively maintained, but it's the last draft that actually includes 
the relevant information, because when that draft was merged into IRI bis 
recently, the algorithms did not survive in a usable form. I expect that 
this will be addressed in due course and the Web Sockets spec will then be 
able to just refer straight to the new draft.)

> > > I have not read sections 4 and 5 in detail, as they seem to be 
> > > concerned entirely with very low-level and rather wordy protocol 
> > > processing descriptions, but I find myself asking:  why not just 
> > > specify a hand-off extension to HTTP that allows bidirectional 
> > > communication to proceed using a protocol such as BEEP?  I fear 
> > > there's a certain amount of reinvention going on here.  But I could 
> > > be wrong.
> > 
> > BEEP is about six orders of magnitude too complicated for what we're 
> > doing here.
> 
> Yes, I agree:  that was meant to be illustrative rather than a serious 
> suggestion.  But I think it helps your case if you can say a word or two 
> about why already-defined protocols don't suits your needs.

As noted above, I feel uncomfortable doing this in a technical 
specification.

On Tue, 8 Sep 2009, Graham Klyne wrote:
> 
> In this message, I'm going to stand back from the immediate discussion 
> of my review, and try and present some broader points, in decreasing 
> order of metaphysicality...
> 
> 1. Why are we here?
> 
> 2. Provide context for evaluation
> 
> 3. Declarative vs procedural definitions
> 
> ...
> 
> 1. Why are we here?
> 
> What I really mean is:  why are you bringing this proposal to a 
> standards body, specifically the IETF?

Lisa asked me to. This technology was originally part of the WHATWG HTML5 
draft, and when the W3C published it, the IETF asked the W3C if it could 
be an RFC instead, since apparently the W3C and the IETF have some 
agreement that they won't step on each other's toes.

Personally I don't really care what standards organisation has its name at 
the top of the document specifying this technology, what I'm interested in 
is getting interoperable implementations in user agents (in the case of 
this protocol, primarily Web browsers).

> There are a number of possible reasons (and I am really not trying to 
> suggest that any particular one applies to you):
> 
> (a) to gain the benefit of review by experts in the field to make your 
> technical solution as good as we can jointly make it.

I feel that this draft has received ample review, but certainly I am 
always happy to have more review, and including the IETF in this effort 
has brought this work to the attention of a few more people, so that has 
indeed been a benefit.

> (b) to provide others with access to the benefits of a technology you 
> have developed by making it publicly available.

No, the technology already had plenty of visibility with its expected 
implementors at the WHATWG, even before the W3C's involvement.

> (c) to benefit oneself: as a corporation by encoraging others to use a 
> technology base hence expanding the potential scope of one's market; as 
> an individual to enhance one's career prospects through the kudos of 
> association with a widely-used published standard.

No, these aren't relevant reasons for this work or the forums that this 
work has been placed in.

> My point in saying all this is that by being clear of your goals you may 
> find it easier to engage constructively with this group, or elsewhere, 
> than becoming bogged down in sterile discussions.

My goal is to have browser vendors implement this (which some are already 
doing) so that it can be used to write interactive Web apps without having 
to rely on <iframe> and XHR hacks like today.

> 2. Provide context for evaluation
> 
> Recently, I have read through two drafts offered to this group: yours 
> and http://www.ietf.org/id/draft-loreto-http-bidirectional-00.txt.  It 
> has struck me that I was able to read through the latter in about 10 
> minutes and pick out the key points.  Yours required me to spend at 
> least half an hour trying to figure what was the key message, and I 
> didn't attempt to read the detailed protocol descriptions.
> 
> To get effective review, you need to make it easy for others to 
> understand where you are coming from, and how what you are offering fits 
> into a wider context. Also, by thinking carefully about this wider 
> context it is quite possible that you can find partial solutions already 
> exist, which you can adopt and enhance with very small technical 
> additions.  Such an approach is, in my observation, more likely to 
> succeed in the IETF context.  If it's genuinely not desirable to build 
> on existing solutions then providing context will help others to see 
> better why a new solution is required.

Yeah, the draft was lacking in introductory material and consisted 
primarily of the implementation requirements for a while. Hopefully this 
is significantly improved now.

> Related to this is making as much use as possible of well-known base 
> specifications (hence my comments about RFC3986 vs [WEBADDRESSES]).

[WEBADDRESSES] is pretty well known in the expected implementation circles 
(in that it describes what is implemented by most UAs already, unlike the 
URI specs).

> 3. Declarative vs procedural definitions
> 
> I've noticed some discussion about use of procedural vs declarative 
> specifications.  I'd like to weigh in, in favour of a declarative 
> approach.
> 
> There are purely declarative programming languages, in a tradition going 
> back to LISP, that allow one to capture *and, more importantly, execute* 
> specifications written in purely declarative form, and these can be as 
> expressive as conventional languages.  (BNF syntax description is a weak 
> form of such a language, which, as you point out, doesn't allow you to 
> specify error handling very easily).
> 
> For example, when the URI specification is being prepared, I was, in 
> parallel with the drafting work by others, creating a parser 
> implementation and test cases in Haskell.  The parser was written using 
> productions that were almost all directly related to productions and 
> text in the specification.  Based on this, I was able to feed back 
> questions about ambiguities in the specification as it was being 
> developed.
> 
> By comparison, a procedural description based on sequencing and ordering 
> of operations tends to introduce artefacts that are not essential to the 
> specification, and hence make the specification harder for mere people 
> to follow.  I'm not advocating any particular formalism here, just 
> trying to make the case that formal specifications at any desired level 
> of detail are possible without resorting to procedurally expressed 
> algorithms.

I like using declarative approaches where possible. Unfortunately I find 
that such an approach becomes unwieldly very fast. I don't really see to 
make the Web Socket protocol spec more declarative at this point. If you 
have any concrete ideas along these lines, I'm more than happy to examine 
them.

On Sat, 5 Sep 2009, Greg Wilkins wrote:
> >> Unavoidable proxies can modify it, even in trivial ways like 
> >> reordering headers or altering the HTTP version number, as well as 
> >> adding Via (though not all do that).
> > 
> > This will result in the connection failing, which is what we want. If 
> > this is fragility, then fragility is a feature, IMHO.
> 
> can you cite any examples where different ordering of headers in a HTTP 
> message (or any other protocol) has caused any problems?

There are some weird bugs with browsers and servers along these lines, but 
they are not relevant here.

The point is that if there is a proxy that is not Web Socket aware, then 
we would not want to establish a Web Socket connection through it, since 
it would in all likelihood introduce other errors that would be much 
harder to detect. It might also make it possible for attackers to trick 
the proxy into behaving incorrectly (e.g. by sending what appears to be 
HTTP requests but to hosts that are inside an intranet firewall).

> >> that the WebSockets protocol would work.  However it seems a poor 
> >> match to use this as a method of detecting that, because sometimes it 
> >> would pass the request unchanged.
> > 

Yes, that is another way that we hope to prevent connections to proxies 
from working.

> A proxy that does understand websocket will pass on the request, except 
> the requirement that the headers be strictly order and contain no 
> additional information will break many proxy implementations - thus the 
> effort required to update them to support websocket is increased.

A man-in-the-middle proxy that supports Web Socket can do so trivially by 
simply not doing anything at all and sending the exact bytes through 
unchanged, without any further parsing whatsoever. In fact, anything 
_else_ is more likely to introduce bugs than do anything useful, so making 
it harder to parse than just to pass data straight through is beneficial.

> Moreover, because ordering changes can be non deterministic (eg because 
> of hash table load factors), then a proxy might appear to comply to 
> strict ordering, but might fail under different circumstances.  That is 
> fragile!

Yes, that would be unfortunate. Is there any way we could force failure 
even in those cases?

---

From: SM <[email-masked]>
Date: Thu, 17 Sep 2009 14:13:28 -0700

>isn't useful? What else should the abstract say, if not that?

   "The Abstract section should provide a concise and comprehensive
    overview of the purpose and contents of the entire document, to give
    a technically knowledgeable reader a general overview of the function
    of the document."

You are telling me what the specification is not intended to do 
instead of what the document is supposed to be about.

>Is this a rhetorical question? Is the paragraph not clear?

It is not clear as I do not know what you mean by "constraints of the Web".

>Could you elaborate?

I assume that you have already settled on the structure and main 
content as the latest version is 
draft-hixie-thewebsocketprotocol-41.  You are asking the community to 
put in time and effort to comment on your proposal.  The numerous 
iterations do not encourage that.  In my opinion, the draft is not 
ready for community review.  I suggest that you read Graham Klyne's 
message [1] again.  There's also a message from Julian Reschke [2].

Regards,
-sm

1. http://www.ietf.org/mail-archive/web/hybi/current/msg00503.html
2. http://www.ietf.org/mail-archive/web/hybi/current/msg00513.html