Subject: [hybi] How to define the WebSocket protocol (Was: Is there a traffic jam?)

From: Ian Hickson <[email-masked]>
Date: Fri, 5 Jun 2009 00:33:39 +0000 (UTC)

On Tue, 28 Apr 2009, Greg Wilkins wrote:
> Ian Hickson wrote:
> > I'm happy to add more complexity (so long as it is justified) to the 
> > client implementation, with the caveat that it must not also increase 
> > the complexity of the server-side implementations.
> 
> Would it be sufficient to allow optional support of complex features. So 
> a server could implement a either a simple core of websocket or a richer 
> more complex feature set?

No, that's just increasing the complexity but saying it's optional.

> Also, you have argued against additional frame-type that could better 
> support complex behaviour on the grounds that the base frame types can 
> carry that complex protocol with assumptions.  If a mime-encapsulating 
> frame type is not worth a dedicated frame-type byte, then what kinds of 
> extensions do you see additional frame-type being defined for?

I've mentioned some before; streaming video and JSON data are two possible 
examples. Basically anything that requires API-level changes.

> > The client component has to deal with any server, including 
> > non-conforming servers sending back garbage, future servers that 
> > support an extended version of the protocol, etc. Server components 
> > only have to deal with the output possible from their own scripts 
> > driving a conforming implementation of the API. We can leverage that 
> > asymmetry.
> 
> I do not think this is generally true.
> 
> Server that will implement the websocket protocol are more than likely 
> going to be done by entirely different sets of developers than those 
> that write the javascript content that is served from them.

Sure, but they still have to have a common protocol which they are able to 
decide together. If they have a bug, they get to decide which side fixes 
it. This is different than the browser vendor vs server/script authors 
case, where the two groups aren't in any way related.

> More over, the javascript served will probably be from 2, 3 or even more 
> distinct sets of developers as frameworks, mashups and other such things 
> will bring together the work of many into a single page.

In the more advanced cases, maybe. The more advanced the case, the mroe 
professional the programmers involved, the less the complexity matters. We 
have to take into account the non-professional case too, though.

> There is also the likely hood of more and more cross-site support and 
> websockets will have to support cross site in some way, so a server may 
> receive connections from code it didn't actually serve.

But it still gets to define the protocol that it serves.

> Furthermore, intermediary proxy's etc. may alter what is sent/received. 
> Even if the initial proxies/firewalls don't, I'm sure somebody will 
> develop some smart websocket aware proxy that does some
> 
> So Server components also have to deal wit non-conforming clients 
> sending back garbage etc. etc.

Sure, but how it reacts to them doesn't have to be particularly carefully 
designed, since the clients will still be limited. This is still an 
asymmetry compared to the client side case, where there will be thousands 
or millions of different servers, developed completely independently and 
not tested with those clients.

However, at this point I'm not really sure I follow what change you are 
arguing for, so I'm not sure what to tell you.

> > This is why currently the spec for the client side defines the entire 
> > protocol (including binary frames which might be sent by servers in 
> > the future), but the spec for the server side only defines the rules 
> > required for handling content sent by the current client API.
> 
> I think the specification should entirely define the wire protocol 
> without any assumptions about who/what/where is implementing the client 
> or the server.

I disagree. It is imperative that clients handle future servers in a 
predictable and consistent way, but it is not imperative that servers 
handle future clients in a predictable and consistent way, since they will 
only receive data in a form they don't expect if the clients are violating 
a contract with the server.

However, I have made the server-side definition of the protocol more 
well-defined so that while the rules are still asymetric, they are 
better-defined for the server side.

> If we wait for non-js-in-browser clients before fully specifying the 
> server side of the protocol, then we have a chicken and the egg problem.

How so? Could you elaborate on this?

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 05 Jun 2009 20:25:02 -0700

Ian,

> examples. Basically anything that requires API-level changes.

Why doesn't Mime encapsulated data fit that definition?
It would require API changes so that meta data associated with
the data could be sent and received?

> have to take into account the non-professional case too, though.

So why not support optionally richer (but more complex) features
that better support these mashups, but are well within the capabilities
of these superior programmers?

However, I fundamentally reject the premise that a protocol design should be
affected by the quality of programmer who may code it.   What other
equivalent protocol (HTTP, FTP, SS7, TCP, etc) has been deliberate made
less capable so that amateur programmers may be able to attempt implementations?

> arguing for, so I'm not sure what to tell you.

Hmmmm

I have made several clear proposals, which I do feel have been rejected
with little attempt to understand the use-cases that motivate them.

So this stage I don't think it is worthwhile us continuing to bat this
back and forth, as you appear very resistant to consider any changes at
all to your proposal.  Which is a pity as I don't think it is that far
away from something that could accommodate a wider range of concerns.

Instead, I propose to prepare an alternate draft for a WebSocket
transport protocol proposal that will:

 + support the current Websocket API proposal
 + support mime encapsulated data, but with an assumption of UTF-8 datagrams
 + support multiple URLs/Origins per connection
 + be specified in a style more in accordance with common
   protocol practise of being strict in what is generated and
   forgiving in what is accepted.

If anybody would like to collaborate on this alternative draft please
contact me off-list.

> How so? Could you elaborate on this?

Because if the protocol does not support non-js-in-browser clients, then
they will not be implemented because the the protocol does not support
them.  Thus if we wait for them to be implemented before defining the
protocol to support them, we will wait forever.

regards

---

From: Ian Hickson <[email-masked]>
Date: Sat, 6 Jun 2009 03:37:23 +0000 (UTC)

On Fri, 5 Jun 2009, Greg Wilkins wrote:
> >> support complex behaviour on the grounds that the base frame types 
> >> can carry that complex protocol with assumptions.  If a 
> >> mime-encapsulating frame type is not worth a dedicated frame-type 
> >> byte, then what kinds of extensions do you see additional frame-type 
> >> being defined for?
> > 
> > I've mentioned some before; streaming video and JSON data are two 
> > possible examples. Basically anything that requires API-level changes.
> 
> Why doesn't Mime encapsulated data fit that definition? It would require 
> API changes so that meta data associated with the data could be sent and 
> received?

Because, unlike with, say, video streaming, there's really no need to have 
the browsers support this natively -- it can be layered on top of the 
existing API. One of the core principles I've been following in developing 
HTML5 and other APIs for the Web application platform is to make the core 
platform be as light as possible, supporting just the basics, and then let 
library authors layer on top of this what they need.

> >> More over, the javascript served will probably be from 2, 3 or even 
> >> more distinct sets of developers as frameworks, mashups and other 
> >> such things will bring together the work of many into a single page.
> > 
> > In the more advanced cases, maybe. The more advanced the case, the 
> > more professional the programmers involved, the less the complexity 
> > matters. We have to take into account the non-professional case too, 
> > though.
>
> So why not support optionally richer (but more complex) features that 
> better support these mashups, but are well within the capabilities of 
> these superior programmers?

Because in practice there's no such thing as optional features on the Web. 
We'd end up forcing these features on everyone.

> However, I fundamentally reject the premise that a protocol design 
> should be affected by the quality of programmer who may code it.  What 
> other equivalent protocol (HTTP, FTP, SS7, TCP, etc) has been deliberate 
> made less capable so that amateur programmers may be able to attempt 
> implementations?

I don't think the Internet so far have been particularly successful in 
terms of being used by amateurs. HTML and its APIs is an example of a 
language that _has_ been delibrerately made accessible to amateur 
programmers, with an insane level of success. WebSocket is much more like 
HTML than TCP in this sense.

> with little attempt to understand the use-cases that motivate them.

I'm sorry you don't feel that I've made an attempt to understand them; I 
thought I had.

> protocol to support them, we will wait forever.

I don't understand why the server side of this protocol is affected by 
whether we get non-JS-in-browser clients.

In any case, as far as I can tell the server side of the protocol _is_ 
fully specified. The server-side requirements aren't identical to the 
client-side requirements, but that's pretty normal. I don't really 
understand the problem here. Servers aren't ever going to receive packets 
they don't recognise in between packets they do; a packet a server doesn't 
recognise is a fatal error. This is unlike clients, who might well exist 
long after the protocol is upgraded, and thus have to be able to deal with 
any number of weird future servers gracefully.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 05 Jun 2009 20:45:14 -0700

> I'm sorry you don't feel that I've made an attempt to understand them; I 

Well for whatever reason, we are misunderstanding each other.

Perhaps the more precise language of a concrete proposal will
break this cycle of misunderstanding.

I'll try to have something ready in a week or two.

regards

---

From: Pieter Hintjens <[email-masked]>
Date: Sat, 6 Jun 2009 11:05:33 +0200

> Well for whatever reason, we are misunderstanding each other.

Don't worry, guys.  I don't understand either of you. :-)

-Pieter Hintjens