Subject: [hybi] Protocol simplicity and the "amateur programmer" standard

From: Maciej Stachowiak <[email-masked]>
Date: Sun, 25 Jul 2010 17:43:07 -0700

I recently had an IRC discussion with Ian Hickson about the "amateur programmer" standard, and what level of programmer simplicity we should strive for: <http://krijnhoetmer.nl/irc-logs/whatwg/20100725#l-92>.

Personally, I think the "amateur programmer" standard is a misguided goal, for the following reasons:

1) There is no example of any protocol ever that had significant deployment, where even casual programmers of simple programs would regularly implement it from scratch. Even for fairly trivial protocols like IRC, most people use libraries or stock implementations. Given the lack of existing examples of a protocol that amateurs actually implement, it may be that it's fundamentally not possible to design such a thing. It would be foolish to try to do the impossible. And even if it is possible but no one has yet succeeded, it is risky to experiment like this when the stakes are high.

2) I think a large part of the complexity in implementing a protocol truly from scratch (with no help from libraries) is the need to use extremely low-level primitives like Unix system calls. Use of very low-level primitives like socket(), bind(), listen(), accept(), read() and select() is likely beyond the capabilities of a true amateur or casual programmer, even for the most trivial protocol. Even for those who are able to correctly use these kinds of primitives, they likely would prefer to use a library or server framework for a casual project.

3) My supporting evidence for #2 is that in the case of nearly any kind of programming problem, when there is a choice between building from scratch with very low-level primitives and using higher-level APIs, casual programmers almost always choose higher-level APIs. I can't think of any instance where it's common to do otherwise.

Faced with these arguments, Ian proposed an alternate standard of "Keep the protocol as simple as possible to limit the number of bugs that professional programmers make." Ian thinks this is effectively equivalent to the "amateur programmer" standard, while I and likely many others on the list would expect there to be material differences. 

For one thing, I think it will be much easier for us to evaluate complexity against this standard. I don't think we have any true casual programmers on the list. While some active participants do not primarily do programming for a living, everyone here is an outlier in technical expertise, particularly in the ares of networking and Web technology. On the other hand, we have a number of people who work as implementors of network servers, networking libraries, and network-related middleware. It seems more sensible to me to pick a standard that we can directly evaluate based on the experiences of Working Group members. It makes more sense to me to evaluate our own understanding than to try to imagine what things are like for others.

To sum up, would anyone object to the following as a standard for protocol simplicity, in place of the proposed "amateur programmer" standard:

* Keep the protocol as simple as possible to limit the number of bugs that professional programmers make.

Regards,
Maciej

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 26 Jul 2010 11:26:11 +1000

On 26 July 2010 10:43, Maciej Stachowiak <[email-masked]> wrote:
* Keep the protocol as simple as possible to limit the number of bugs that
professional programmers make.

Maciej,

thanks for trying to find a way around this road block.

However, the problem with your revised requirement is that simplicity is not
an requirement in itself.  Simplicity is only a relative measure that can be
used to evaluate solutions other requirements.

When faced with a choice of multiple solutions for a particular requirement,
obviously simplicity is high up there on the list of attributes to consider
when evaluating the alternatives.  However it is not the only attribute and
matters such as performance, extensibility, compatibility, etc. are also
considerations.  In such circumstances, favouring simplicity is pretty self
evident.

My feeling is that what Ian really wants is that if we are faced with a
choice between the simplicity of no-feature vs  a complex non mandatory
featured (eg content negotiation), that we will opt for simplicity.   Ie, he
is after a requirement to try to limit rampant featurism in the protocol.  I
do not disagree.

So I'm wondering if the requirements document should contain text about
minimalism rather than simplicity:

   it is not the intent of this protocol to provide all the features that
could possibly be needed on
   bidirectional web communication.  Rather it is the intent of this
protocol to provide the minimalistic set
   of features required to achieve secure and efficient bidirectional
communication without causing
   unreasonable complexity and transport concerns within the application
layer. All aspects of the
   protocol must be motivated by specific identified requirements.

regards

---

From: Mike Belshe <[email-masked]>
Date: Sun, 25 Jul 2010 21:08:12 -0700

On Sun, Jul 25, 2010 at 5:43 PM, Maciej Stachowiak <[email-masked]> wrote:

>
> I recently had an IRC discussion with Ian Hickson about the "amateur
> programmer" standard, and what level of programmer simplicity we should
> strive for: <http://krijnhoetmer.nl/irc-logs/whatwg/20100725#l-92>.
>
> 1) There is no example of any protocol ever that had significant
> deployment, where even casual programmers of simple programs would regularly
> implement it from scratch. Even for fairly trivial protocols like IRC, most
> people use libraries or stock implementations. Given the lack of existing
> examples of a protocol that amateurs actually implement, it may be that it's
> fundamentally not possible to design such a thing. It would be foolish to
> try to do the impossible. And even if it is possible but no one has yet
> succeeded, it is risky to experiment like this when the stakes are high.
>
> 2) I think a large part of the complexity in implementing a protocol truly
> from scratch (with no help from libraries) is the need to use extremely
> low-level primitives like Unix system calls. Use of very low-level
> primitives like socket(), bind(), listen(), accept(), read() and select() is
> likely beyond the capabilities of a true amateur or casual programmer, even
> for the most trivial protocol. Even for those who are able to correctly use
> these kinds of primitives, they likely would prefer to use a library or
> server framework for a casual project.
>
> 3) My supporting evidence for #2 is that in the case of nearly any kind of
> programming problem, when there is a choice between building from scratch
> with very low-level primitives and using higher-level APIs, casual
> programmers almost always choose higher-level APIs. I can't think of any
> instance where it's common to do otherwise.
>
>
> Faced with these arguments, Ian proposed an alternate standard of "Keep the
> protocol as simple as possible to limit the number of bugs that professional
> programmers make." Ian thinks this is effectively equivalent to the "amateur
> programmer" standard, while I and likely many others on the list would
> expect there to be material differences.
>
> For one thing, I think it will be much easier for us to evaluate complexity
> against this standard. I don't think we have any true casual programmers on
> the list. While some active participants do not primarily do programming for
> a living, everyone here is an outlier in technical expertise, particularly
> in the ares of networking and Web technology. On the other hand, we have a
> number of people who work as implementors of network servers, networking
> libraries, and network-related middleware. It seems more sensible to me to
> pick a standard that we can directly evaluate based on the experiences of
> Working Group members. It makes more sense to me to evaluate our own
> understanding than to try to imagine what things are like for others.
>
> To sum up, would anyone object to the following as a standard for protocol
> simplicity, in place of the proposed "amateur programmer" standard:
>

I find this statement a bit silly, because nobody wants a complex protocol
:-)

We are already striving for the simplest solution to a set of requirements.
 Nail the requirements down and we'll nail a simple solution.

Mike

>
>
> Regards,
> Maciej
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 26 Jul 2010 06:42:40 +0200

> 2) I think a large part of the complexity in implementing a protocol truly from scratch (with no help from libraries) is the need to use extremely low-level primitives like Unix system calls. Use o…

It's even simpler than that, look at current implementations. Users
are trying to get it to work with apache. They're not trying to write
their own server. Not only this is hard, but this is a big security
risk. And it takes a lot of time while slightly modifying an existing
server is a lot faster. The closer the protocol will be to existing
implementations, the faster and easier the adoption.

I like the example about CGI, this is exactly what we should have :
  - a highly interoperable protocol like HTTP on the wire
  - standard, unmodified servers to handle the protocol
  - CGI for the amateurs

The server would simply take care of the handshake, forward every
parsed header into environment variables that the CGI can use and
be fine with this. The way the protocol is currently designed makes
this complicated, it tries to force the amateur to write the server
side software himself by enforcing incompatibilities with standard
protocols.

(...)
> * Keep the protocol as simple as possible to limit the number of bugs that professional programmers make.

In my opinion, it's not even a matter of that level. The protocol
should just have the complexity required to implement its features
and specificities. It should not be over-complicated. What I mean
is that if something can trivially be implemented in other protocols,
there is no reason it can't be here. If something requires strong
security, it should be easy to analyse it and prove it good. We
should not rely on a wide variety of tricks that any programmer,
good or bad, can easily get trapped into, nor make the programmer
write dangerous code for no added value (eg: counting spaces,
computing MD5 of concatenation of various things, etc...). *THIS*
is hard to implement for an amateur and this is error prone for a
confirmed programmer.

Willy

---

From: James Graham <[email-masked]>
Date: Mon, 26 Jul 2010 12:42:55 +0200

> that professional programmers make.

As I previously said, I think that framing the discussions in terms of 
"professional" vs "amateur" is the wrong way to go. After all one can 
find "amateurs" who are much more capable than many professionals. Using 
these terms appears to be divisive; I believe that you would not have 
used the term "professional" in the above description if it were not for 
the fact that Ian had previously used "amateur"; rewriting you principle 
as "Keep the protocol as simple as possible to limit the number of bugs 
that implementors make." is largely equivalent but less pejorative about 
non-professionals.

Having said that, I think a different goal would be a better statement 
of Ian's intent:

"Websockets should be the minimal layer on top of TCP needed to meet the 
additional requirements of browsers"

I think that neatly folds in the "amateur programmer" requirement 
because if one has a minimal layer then it is, by definition, as simple 
as it can be. You might disagree on the basis that you believe that 
inexperienced people don't use low-level primitives for network 
programming. However I would contest that the typical progression for 
someone learning network programming in a language such as Python is:

1) Use the socket module and low level primitives. Implement simple 
applications with no difficult scaling requirements

2) Learn a library that abstracts away the sockets and has nice scaling 
properties (typically at the cost of some complexity of a different 
type) e.g. Twisted or Diesel

3) Author a library that abstracts away the raw sockets

As, admittedly rather weak, evidence for this hypothesis I note that a 
Google search for "python networking" returns — at least for me — [1] as 
the top hit, which is a tutorial for the low-level sockets / select 
functionality. I would expect people new to server implementation 
writing WebSockets applications to follow a similar progression. In this 
regard, I think the comparison to HTTP, where people do typically reuse 
an existing server, is somewhat misleading because applications using 
WebSockets are likely to be more different than applications using HTTP 
because WebSockets applications have to implement their own 
application-level protocol whereas HTTP applications do not.

Even if my hypothesis is faulty and people go straight to phase 2 above, 
designing the protocol as if they went through phase 1 seems harmless, 
whereas if they would naturally go through phase 1, forcing them to go 
straight to phase 2 because of complexity in the protocol seems bad.

That said, I think some complexity in the handshake is more acceptable 
than complexity elsewhere because one can imagine a library that is a 
very thin wrapper over raw sockets, just dealing with the handshake and 
little else. Nevertheless, I think a good yardstick requirement should be:

"it must be possible to efficiently implement WebSockets using only the 
stdlib of today's popular scripting languages without dropping down to 
lower level (C, Java, whatever) code"

It has been argued that an explicit statement of simplicity is not 
needed since simplicity is an obvious requirement that you can optimise 
for once all other requirements are in. I believe this to be untrue. I 
think collecting a laundry list of requirements and then trying to make 
a protocol that satisfies them all will not result in anything simple. 
Instead one must determine which additional requirements one can forego 
in order to preserve simplicity.

Having said all of that, I think the current discussion on this list 
belies a false belief that we are still at a stage where the protocol 
can be trivially changed. Instead we are in a situation where there are 
three known implementations in major browser engines (presto, gecko, 
webkit) and some of those are close to shipping. There are also numerous 
server implementation. At this stage, the cost of substantial changes is 
already looking prohibitive and if Opera/Firefox/Chrome/Safari all ship 
with interoperable support, we can assume that substantial changes to 
the design will become all but impossible since browsers rarely rescind 
functionality that is needed to keep sites working.

Given the current implementation status, I suggest people adopt the 
attitude that changes are difficult and expensive and only the most 
necessary changes should be advocated. If there are people who believe 
that WebSockets has the wrong requirements, or occupies the wrong part 
of design space, it seems more prudent to work on a seperate, parallel, 
proposal.

[1] http://ilab.cs.byu.edu/python/

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 26 Jul 2010 04:07:12 -0700

On Jul 26, 2010, at 3:42 AM, James Graham wrote:

> As I previously said, I think that framing the discussions in terms of "professional" vs "amateur" is the wrong way to go. After all one can find "amateurs" who are much more capable than many professionals. Using these terms appears to be divisive; I believe that you would not have used the term "professional" in the above description if it were not for the fact that Ian had previously used "amateur"; rewriting you principle as "Keep the protocol as simple as possible to limit the number of bugs that implementors make." is largely equivalent but less pejorative about non-professionals.

Actually, I used the word "professional" because that sentence was a literal quote of something Ian said. I don't think "professional" vs. "amateur" is the best distinction to make. There are many expert programmers who work on projects that they are not paid to work on.

> Having said that, I think a different goal would be a better statement of Ian's intent:
> 
> "Websockets should be the minimal layer on top of TCP needed to meet the additional requirements of browsers"
> 
> I think that neatly folds in the "amateur programmer" requirement because if one has a minimal layer then it is, by definition, as simple as it can be. You might disagree on the basis that you believe that inexperienced people don't use low-level primitives for network programming. However I would contest that the typical progression for someone learning network programming in a language such as Python is:
> 
> 1) Use the socket module and low level primitives. Implement simple applications with no difficult scaling requirements
> 
> 2) Learn a library that abstracts away the sockets and has nice scaling properties (typically at the cost of some complexity of a different type) e.g. Twisted or Diesel
> 

I would expect a lot of people bypass all of these steps and go straight to:

4) Use a library, framework, or modular server that someone else has written.

As one anecdotal example, I would cite the fact that I have written or modified a number of scripts that serve dynamically generated HTTP resources in Perl, Python, PHP and Ruby, but I have never done steps 1-3 in any of these programming languages. I have also written an IRC bot purely as a hobby, and in that case too I used someone else's library rather than any of steps 1-3.

> 
> As, admittedly rather weak, evidence for this hypothesis I note that a Google search for "python networking" returns — at least for me — [1] as the top hit, which is a tutorial for the low-level sockets / select functionality. I would expect people new to server implementation writing WebSockets applications to follow a similar progression. In this regard, I think the comparison to HTTP, where people do typically reuse an existing server, is somewhat misleading because applications using WebSockets are likely to be more different than applications using HTTP because WebSockets applications have to implement their own application-level protocol whereas HTTP applications do not.

Are there any network protocols that it's common to implement from scratch (using any of the levels 1-3 you mention above) in production code, rather than using someone else's library? Is there any common application-level protocol (say "QQP") where searching for "Python QQP" will land you at the sockets tutorial rather than a QQP library? That would be more convincing evidence.

> 
> Even if my hypothesis is faulty and people go straight to phase 2 above, designing the protocol as if they went through phase 1 seems harmless, whereas if they would naturally go through phase 1, forcing them to go straight to phase 2 because of complexity in the protocol seems bad.
> 
> That said, I think some complexity in the handshake is more acceptable than complexity elsewhere because one can imagine a library that is a very thin wrapper over raw sockets, just dealing with the handshake and little else. Nevertheless, I think a good yardstick requirement should be:
> 
> "it must be possible to efficiently implement WebSockets using only the stdlib of today's popular scripting languages without dropping down to lower level (C, Java, whatever) code"

Out of the existing requirements, the one that this statement is most likely to conflict with is security, since many proposals for making the handshake secure against cross-protocol attacks involve crypto. I would not consider efficient implementability in scripting languages to be a higher priority than security. I think other proposed features that conflict with the simplicity argument are not especially inefficient to implement.

> 
> It has been argued that an explicit statement of simplicity is not needed since simplicity is an obvious requirement that you can optimise for once all other requirements are in. I believe this to be untrue. I think collecting a laundry list of requirements and then trying to make a protocol that satisfies them all will not result in anything simple. Instead one must determine which additional requirements one can forego in order to preserve simplicity.

I don't think anyone has a problem with simplicity as long as it doesn't trade off against other requirements that they hold to be important.

I do think simplicity is important, but I hold some requirements as important enough to be worth significant amounts of complexity; for example the security requirements.

Others may hold different requirements as important, and I don't think appealing to some standard of simplicity will make anyone conclude that their particular requirement is less important. If we want to limit protocol complexity, we have to prioritize the requirements, or have some good story to tell about how some requirements can be addressed on top of the protocol instead of built-in.

> Having said all of that, I think the current discussion on this list belies a false belief that we are still at a stage where the protocol can be trivially changed. Instead we are in a situation where there are three known implementations in major browser engines (presto, gecko, webkit) and some of those are close to shipping. There are also numerous server implementation. At this stage, the cost of substantial changes is already looking prohibitive and if Opera/Firefox/Chrome/Safari all ship with interoperable support, we can assume that substantial changes to the design will become all but impossible since browsers rarely rescind functionality that is needed to keep sites working.
> 
> Given the current implementation status, I suggest people adopt the attitude that changes are difficult and expensive and only the most necessary changes should be advocated. If there are people who believe that WebSockets has the wrong requirements, or occupies the wrong part of design space, it seems more prudent to work on a seperate, parallel, proposal.

The last version of Safari shipped with a version of WebSocket that is incompatible with the current protocol. So we are open to further changes, if they are valuable. I don't think we are yet at the point where incompatible changes are inordinately expensive. I don't think the protocol has enough real deployment experience for us to lock it down.

Regards,
Maciej

---

From: Julian Reschke <[email-masked]>
Date: Mon, 26 Jul 2010 13:18:40 +0200

On 26.07.2010 13:07, Maciej Stachowiak wrote:
> ...
> ...

Thanks for that clarification, and also +1 to the other things you said.

Best regards, Julian

---

From: James Graham <[email-masked]>
Date: Mon, 26 Jul 2010 13:48:26 +0200

>> "it must be possible to efficiently implement WebSockets using only
>> the stdlib of today's popular scripting languages without dropping
>> down to lower level (C, Java, whatever) code"
>
> Out of the existing requirements, the one that this statement is most
> likely to conflict with is security, since many proposals for making
> the handshake secure against cross-protocol attacks involve crypto. I
> would not consider efficient implementability in scripting languages
> to be a higher priority than security. I think other proposed
> features that conflict with the simplicity argument are not
> especially inefficient to implement.

To be clear here, I mean "efficient in terms of programmer time". That 
is, I don't think it is acceptable to make would-be implementers write 
substantial amounts of code that are not directly related to websockets 
just to write a simple websockets server.

> Others may hold different requirements as important, and I don't
> think appealing to some standard of simplicity will make anyone
> conclude that their particular requirement is less important.

That could be true, but it is sad if so; I would hope people are capable 
of assessing the relative importance, and relative complexity, of 
features that they would like in the protocol.

>> Having said all of that, I think the current discussion on this
>> list belies a false belief that we are still at a stage where the
>> protocol can be trivially changed. Instead we are in a situation
>> where there are three known implementations in major browser
>> engines (presto, gecko, webkit) and some of those are close to
>> shipping. There are also numerous server implementation. At this
>> stage, the cost of substantial changes is already looking
>> prohibitive and if Opera/Firefox/Chrome/Safari all ship with
>> interoperable support, we can assume that substantial changes to
>> the design will become all but impossible since browsers rarely
>> rescind functionality that is needed to keep sites working.
>>
>> necessary changes should be advocated. If there are people who
>> believe that WebSockets has the wrong requirements, or occupies the
>> wrong part of design space, it seems more prudent to work on a
>> seperate, parallel, proposal.
>
> The last version of Safari shipped with a version of WebSocket that
> is incompatible with the current protocol.

I think one browser shipping doesn't create an ecosystem that depends on 
WebSockets not undergoing incompatible changes (although I note that I 
have seen multiple people ask how to make their servers compatible with 
both -75 and -76 handshakes in order to work with both Safari 5 and 
Chrome). I think that many browsers shipping compatible implementations 
is highly likely to create such an ecosystem. Therefore I would say that 
the timescale for substantial changes to websockets is soemthing like 
the release timescale of the next generation of browsers minus the 
stabilization time and the time needed to implement changes. It seems to 
me that that timescale could be rather short.

---

From: Micheil Smith <[email-masked]>
Date: Mon, 26 Jul 2010 21:58:46 +1000

On:

> I think one browser shipping doesn't create an ecosystem that depends on WebSockets not undergoing incompatible changes (although I note that I have seen multiple people ask how to make their servers compatible with both -75 and -76 handshakes in order to work with both Safari 5 and Chrome). I think that many browsers shipping compatible implementations is highly likely to create such an ecosystem. Therefore I would say that the timescale for substantial changes to websockets is soemthing like the release timescale of the next generation of browsers minus the stabilization time and the time needed to implement changes. It seems to me that that timescale could be rather short.

The current way I do it in my project, Node WebSocket Server ( http://github.com/miksago/node-websocket-server ) 
is to switch between draft -75 and -76 based on the existence of the Sec-* headers and Key3. I also have an option 
where by a developer, if they so choose, can force the server to only use one version of the WebSocket protocol, 
and all other requests will be rejected (this is still not implemented quite right, but it's not really a super high priority 
on the project just yet).

Yours,
Micheil Smith
-- 
BrandedCode.com

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 27 Jul 2010 01:10:45 +1000

I do not think the "we've shipped it so you have to rubber stamp it as an
ietf protocol" is an winning argument.

However, we do have a problem with timescales.

I've previously suggested  that we should give up on the attempt to make the
current version an ietf standard and instead just let the WHATWG get on with
deploying something pretty close to what they already have and they can call
it 1.0.

This WG should then be focused on coming up with a 1.1 version of the
protocol that is an ietf standard and does address the wider concerns that
are more of a concern of the IETF WG.

My reasons for this are three fold.  Firstly I do not think that this WG can
produce a reasonable standard in a time frame that will be acceptable to the
browser vendors (nor community expectations).

Secondly, by getting something out that can start getting wider usage, I
believe that will give more background on which to base our discussions.
Unfortunately I expect that this will result in some painful user
experiences as they find out how much work an application has to do to
realistically use raw websockets... but that will at least drive some demand
for the features that have been rejected to date as too complex.

Finally, I think there are some real significant issues that we need time to
ponder, experiment and solve.

I think this two phase approach has precedent in the way HTTP/1.1 was
developed.

regards

---

From: Dave Cridland <[email-masked]>
Date: Mon, 26 Jul 2010 16:16:00 +0100

> I think this two phase approach has precedent in the way HTTP/1.1  

And, unfortunately, the Cookie header.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: James Graham <[email-masked]>
Date: Mon, 26 Jul 2010 17:26:37 +0200

On 07/26/2010 05:10 PM, Greg Wilkins wrote:
>
> I do not think the "we've shipped it so you have to rubber stamp it as
> an ietf protocol" is an winning argument.

To be clear, my argument was intended to focus on market realities. If 
browsers stay on their current course, a version of WebSockets much like 
the -76 draft will, assuming it is not a complete failure, be a de-facto 
part of the web platform within a short timescale, irrespective of what 
the IETF does.

> protocol that is an ietf standard and does address the wider concerns
> that are more of a concern of the IETF WG.

That is fine as long as you have the requirement that 1.1 is backwards 
compatible with 1.0. That means substantial breaking changes to e.g the 
handshake need to happen on the 1.0 timescale.

> believe that will give more background on which to base our
> discussions.  Unfortunately I expect that this will result in some
> painful user experiences as they find out how much work an application
> has to do to realistically use raw websockets... but that will at least
> drive some demand for the features that have been rejected to date as
> too complex.

I agree it makes sense to find out what the actual pain points are 
instead of guessing.

---

From: Adam Barth <[email-masked]>
Date: Mon, 26 Jul 2010 17:29:03 +0200

On Mon, Jul 26, 2010 at 5:26 PM, James Graham <[email-masked]> wrote:
> browsers stay on their current course, a version of WebSockets much like the
> -76 draft will, assuming it is not a complete failure, be a de-facto part of
> the web platform within a short timescale, irrespective of what the IETF
> does.

To be clear: we've had representatives of Apple and Google state
explicitly on this list that they are planning to rev their
implementations of WebSockets.  No one is asking the IETF to rubber
stamp anything.

Adam

---

From: James Graham <[email-masked]>
Date: Mon, 26 Jul 2010 17:42:04 +0200

On 07/26/2010 05:29 PM, Adam Barth wrote:
> On Mon, Jul 26, 2010 at 5:26 PM, James Graham<[email-masked]>  wrote:
> implementations of WebSockets.

Will people still be willing to make breaking changes in, say, a year, 
if it has come to pass that Mozilla, Google, Apple and Opera are all 
shipping something like the current protocol and a non-trivial amount of 
content is using it?

I am *not* saying that change is impossible at this stage. I *am* saying 
that we should be aware of the limited timescale over which changes can 
be made, and act accordingly.

---

From: John Tamplin <[email-masked]>
Date: Mon, 26 Jul 2010 11:46:07 -0400

> handshake need to happen on the 1.0 timescale.

All the more reason that versioning and negotation should be included in the
base release.  I don't know of any widely used protocol which didn't undergo
changes over its lifetime, and it makes sense to incorporate formal support
up front (for example, you shouldn't have to look for the presence of
particular headers to distinguish v75 from v76 in a server).

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Mon, 26 Jul 2010 08:51:08 -0700

On Mon, Jul 26, 2010 at 8:42 AM, James Graham <[email-masked]> wrote:

>>> the
> Will people still be willing to make breaking changes in, say, a year, if
> it has come to pass that Mozilla, Google, Apple and Opera are all shipping
> something like the current protocol and a non-trivial amount of content is
> using it?
>
>
Let's hope this doesn't take us a year.

> that we should be aware of the limited timescale over which changes can be
> made, and act accordingly.
>

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 27 Jul 2010 02:53:13 +1000

> something like the current protocol and a non-trivial amount of content is

Code running in the browsers should be dependent only on the API and the
behavioural contract that it makes.  I think that there is a large amount of
freedom to change the protocol beneath the websocket API and the application
code will hardly notice (except maybe it will work more reliably).

The servers will have to change, but they should also be protecting their
applications with APIs.  Yet another reason that it would be crazy for
applications to develop their own protocol implementations.

regards

---

From: Ian Hickson <[email-masked]>
Date: Tue, 27 Jul 2010 00:56:07 +0000 (UTC)

On Mon, 26 Jul 2010, James Graham wrote:
> 
> I think a different goal would be a better statement of Ian's intent:
> 

Yes, I think that's an accurate description of what I mean.

There are two parts to the protocol: the handshake, and the framing.

The handshake has basically three purposes: 1: allowing the protocol to be 
served by a server that also serves HTTP on the same port; 2: protecting 
existing servers and future WebSocket servers from cross-protocol attacks; 
3: allowing servers to host multiple domains and resources on the same 
IP/port combination, while supporting the browser origin security model.

The framing has one purpose: provide JavaScript applications running in 
Web browsers with a way to send and receive text messages to and from 
WebSocket servers.

There are a number of ways that we can make the protocol also support 
other goals:

 - Compression of frame data
 - Sending and receiving binary data
 - Allowing messages to be annotated with metadata*
 - Allowing multiple JavaScript applications from the same browser to 
   share a connection to a server (multiplexing)*

All of these could be supported by future backwards-compatible changes to 
the protocol, by having the client advertise support for each feature in 
the handshake, and having the server opt-in to using that feature. None of 
these changes require fundamental changes to the handshake.

Two of these, marked with an asterisk above, can also be implemented by 
script; therefore a good way of determining if they are truly needed is to 
deploy the protocol as is, and to see if people work around the lack of 
that feature by implementing it themselves. If they do, then the feature 
is needed and we should add it.

The other two features are clearly needed regardless, and we should add 
them in the near future. (Adding features piecemeal leads to more 
compliant client-side implementations, so we should not add them now.)

Here's a proposed timeline for adding these features:

 - We fix any critical bugs (not feature additions) in the protocol as it 
   stands today (next 4 weeks).

 - We deploy the protocol in four or more major browser vendors (next 4
   months).

 - When binary support in WebGL is stable, we add binary support to XHR, 
   WebSocket, etc (a matter of months?).

 - We look at how the protocol is used (another couple of months).

 - During this time, we build client and server test suites for the 
   protocol.

 - We add built-in support for the features that are needed based on 
   implementation experience (about 6 months from now), including 
   compression, multiplexing, per-frame metadata annotation, etc, as 
   needed.

 - We update the test suites.

I think it would be a mistake to add every feature we can think of right 
away, because doing so would mean that we added features before their need 
was proven, and it would lead to much less conforming client 
implementations. This dynamic has been proven time and time again with Web 
technologies; we should not repeat that mistake.

It would also be advantageous to allow implementors to experiment with 
ideas like multiplexing before we add the feature to the protocol, since 
it would let us use their implementation experience when designing the 
extensions to the protocol. Only by first getting the basic protocol out 
there and then experimenting with it can we really find how these ideas 
work out in real deployments.

(Thanks to Ian Fette for providing a sounding board for some of this.)

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Maciej Stachowiak <[email-masked]>
Date: Mon, 26 Jul 2010 18:00:08 -0700

Ian's suggested timeline below sounds like an excellent way to proceed, in my opinion.

 - Maciej

---

From: Rob Sayre <[email-masked]>
Date: Mon, 26 Jul 2010 20:10:24 -0700

I like this plan as well.

- Rob

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 27 Jul 2010 14:56:27 +1000

Ian,

I'm generally in agreement with the outline of your plan - specifically I
think it
worthwhile to get a minimalistic solution deployed sooner rather than later.

But a couple of points in-line below:

> IP/port combination, while supporting the browser origin security model.

I would add 4: to support  the deployment of future extended versions of the
protocol.

Thus having a mechanism for versions and feature negotiation is key in a 1.0

>   share a connection to a server (multiplexing)*

I would also add (for consideration at least)

      - allowing connections to be annotated with metadata, If we don't,
        developers will just misuse the sub-protocol field for this.
      - chunking of large messages into multiple frames
      - keep alive messages
      - error handling and graceful shutdown (although this is already
somewhat improved).

> is needed and we should add it.

I don't necessarily agree with this evaluation criteria as features like
sharing
connections between tabs/windows is very much more difficult to achieve in
script than in browser.  So I don't think lack of wide implementations in
script
is a good indication of desirability of these features.  But  I do agree
that they
are not necessarily required in 1.0, so long as we can see how they could be
added.

> compliant client-side implementations, so we should not add them now.)

With regards to the support for binary data, I agree that can mostly wait
until
the API is in javascript to handle it.  But I would also like to have one
final discussion
to see if we could agree to support only a single framing type that supports
both
binary and text - rather than enshrining multiple framing types for all
time.
For example,  we could switch to only supporting binary data with the
assumption
that unless otherwise specified the binary content is UTF-8 encoded text.

> technologies; we should not repeat that mistake.

agreed.   But I do think we need to at least discuss some proposals for the
extra features before a 1.0 so that we can test that we have the right
extensibility to be able to add them later without breaking.

---

From: Michael Carter <[email-masked]>
Date: Mon, 26 Jul 2010 22:47:24 -0700

Ian's roadmap looks great, in my opinion.

-Michael Carter

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 27 Jul 2010 07:58:36 +0200

> Ian's suggested timeline below sounds like an excellent way to proceed,  

+1

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 27 Jul 2010 08:13:07 +0200

On Tue, 27 Jul 2010 06:56:27 +0200, Greg Wilkins <[email-masked]> wrote:
> I would add 4: to support  the deployment of future extended versions of  
> the protocol.
>

Feature negotiation is there, by means of new headers. Breaking backwards  
compatibility so that you need to support two or more versions forever  
seems like a bad idea.

> sharing connections between tabs/windows is very much more difficult to  
> achieve in script than in browser.

Seems relatively simple to me using a SharedWorker.

---

From: John Tamplin <[email-masked]>
Date: Tue, 27 Jul 2010 02:40:40 -0400

On Tue, Jul 27, 2010 at 2:13 AM, Anne van Kesteren <[email-masked]> wrote:

> Feature negotiation is there, by means of new headers.

If that is the intention, it would be good to state that in the protocol,
especially since an earlier version specifically outlawed additional headers
and there is confusion over the interpretation of the current version's
stance on additional headers.  Additionally, I think there needs to be room
for expansion of frame types (which is currently there).

Breaking backwards compatibility so that you need to support two or more
> versions forever seems like a bad idea.

The alternative is simply breaking old implementations if there is no
versioning and you find you need to make incompatible changes.  Including
versioning now is small insurance against such a requirement.

I think it is worth considering keep-alives for the initial version -- the
alternative, if implemented at the application level, is a mess of
incompatible implementations that can't be pooled or limited by the browser
(such as on a phone that needs to curtail network activity to conserve
battery) without interfering with legitimate traffic, since the browser
can't identify it.

Another one that I think is impractical to expect JS implementations of is
compression.  What would demonstrate sufficient need for compression support
if it is not reasonable to expect the compression to be implemented in JS -
would demonstrations of improvement of real-world performance of real-world
apps across a compressed WebSocket be sufficient?

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 27 Jul 2010 08:50:18 +0200

On Tue, Jul 27, 2010 at 02:40:40AM -0400, John Tamplin wrote:
> stance on additional headers.

Yes indeed, that should be clarified. And BTW, saying "Upgrade: WebSocket/1.0"
would be *a lot* better than "Upgrade: WebSocket". And BTW, all Upgrade examples
in RFC2616 suggest such a version, so there is nothing unusual in doing that.
It's also what will a server know what it should consider in a handshake.

> versioning now is small insurance against such a requirement.

Right, it costs nothing right now to advertise the version and to define how
each end should react to an unknown version.

> battery) without interfering with legitimate traffic, since the browser

Indeed, and many protocols relying on long-lived sessions provide a ping
function because this is the only way to test the whole chain. Having it
available does not mean browsers will have to implement it, it just means
that when such a ping request is seen, a ping response must be emitted,
which is not something terribly difficult to implement.

Regards,
Willy

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 27 Jul 2010 09:03:11 +0200

On Tue, 27 Jul 2010 08:40:40 +0200, John Tamplin <[email-masked]> wrote:
> especially since an earlier version specifically outlawed additional  
> headers and there is confusion over the interpretation of the current  
> version's
> stance on additional headers.  Additionally, I think there needs to be  
> room for expansion of frame types (which is currently there).

It outlaws clients sending additional fields and requires servers to  
ignore unknown fields. Isn't it perfectly logical that a revision of the  
protocol can then introduce new headers that the client will transmit and  
older servers will ignore while new servers will act on them? This is one  
of the most basic extensibility mechanisms there is. I.e. MUST not  
generate and MUST ignore so that in the future we CAN generate and CAN do  
something...

> versioning now is small insurance against such a requirement.

The solution is to not make breaking changes. Versioning is ignored by  
clients for HTTP, HTML has no versioning by design, CSS has no versioning  
by design, etc. Versioning does not work well on the Web.

http://dbaron.org/log/2007-03#e20070325a explains some of this in more  
detail.

> I think it is worth considering keep-alives for the initial version --  
> the alternative, if implemented at the application level, is a mess of
> incompatible implementations that can't be pooled or limited by the  
> browser (such as on a phone that needs to curtail network activity to  
> conserve
> Another one that I think is impractical to expect JS implementations of  
> is compression.  What would demonstrate sufficient need for compression  
> support if it is not reasonable to expect the compression to be  
> implemented in JS - would demonstrations of improvement of real-world  
> performance of real-world apps across a compressed WebSocket be  
> sufficient?

These are both rather trivial to add (and the need is relatively clear)  
but do not seem critical to get the protocol deployed. Therefore we can  
probably leave them out until we have more implementations of the base  
protocol.

---

From: Ian Hickson <[email-masked]>
Date: Tue, 27 Jul 2010 07:13:58 +0000 (UTC)

On Tue, 27 Jul 2010, Willy Tarreau wrote:
> > If that is the intention, it would be good to state that in the 
> > protocol [...]
> 
> Yes indeed, that should be clarified.

I've added another sentence to the end of the "Design philosophy" section 
to try to explain this in slightly more detail. I don't really know how 
better to clarify it, though. Could you elaborate on exactly what is 
unclear?

> And BTW, saying "Upgrade: WebSocket/1.0" would be *a lot* better than 
> "Upgrade: WebSocket".

Not having a version field is intentional. Versioning is an anti-pattern 
on the Web.

---

From: Maciej Stachowiak <[email-masked]>
Date: Tue, 27 Jul 2010 00:24:35 -0700

On Jul 26, 2010, at 11:40 PM, John Tamplin wrote:

> I would add 4: to support  the deployment of future extended versions of the protocol.
> 
> If that is the intention, it would be good to state that in the protocol, especially since an earlier version specifically outlawed additional headers and there is confusion over the interpretation of the current version's stance on additional headers.  Additionally, I think there needs to be room for expansion of frame types (which is currently there).
> 
> Breaking backwards compatibility so that you need to support two or more versions forever seems like a bad idea.
>  
> The alternative is simply breaking old implementations if there is no versioning and you find you need to make incompatible changes.  Including versioning now is small insurance against such a requirement.

I think feature negotiation covers versioning for everything but the handshake. You can negotiate a whole separate framing scheme if needed, you can negotiate a different close sequence, you can negotiate keepalives. You can even bundle a set of these things into a single feature to negotiate.

For the handshake itself, it's not clear that any versioning scheme can reasonably handle a truly incompatible change. Two of the alternate handshake schemes proposed so far: (a) TLS nextproto; and (b) Adam's pure crypto handshake; are so incompatible that there's no way to indicate them with a version indicator anywhere in the current handshake. So perhaps the onus has to be on future versions to be distinctive enough that it can be clearly identified.

> 
> Another one that I think is impractical to expect JS implementations of is compression.  What would demonstrate sufficient need for compression support if it is not reasonable to expect the compression to be implemented in JS - would demonstrations of improvement of real-world performance of real-world apps across a compressed WebSocket be sufficient?

People have implemented compression in JS, so it should be possible to do at least rough experiments to see if it has benefits in real-world apps.

Regards,
Maciej

---

From: Simone Bordet <[email-masked]>
Date: Tue, 27 Jul 2010 09:53:02 +0200

Hi,

> Not having a version field is intentional. Versioning is an anti-pattern

Could you elaborate on why "Versioning is an anti-pattern on the Web" ?
As already stated by others many -- if not all -- web (and non web)
protocols do have a version "field" that helped their evolution.
Why WebSocket would be damaged by versioning ?

Simon
-- 
http://bordet.blogspot.com
---
Finally, no matter how good the architecture and design are,
to deliver bug-free software with optimal performance and reliability,
the implementation technique must be flawless.   Victoria Livschitz

---

From: Dave Cridland <[email-masked]>
Date: Tue, 27 Jul 2010 09:06:29 +0100

On Tue Jul 27 07:50:18 2010, Willy Tarreau wrote:
> > > On Tue, 27 Jul 2010 06:56:27 +0200, Greg Wilkins  
> <[email-masked]>
> > > wrote:
> > >
> > >> Thus having a mechanism for versions and feature negotiation  
> is key in a
> > >> 1.0
> > >>
> > >
> protocol,
FWIW, I think current thinking has swerved away from versions - if  
you need to actually change version incompatibly, then you may as  
well define a new protocol. If you need to incrementally add  
features, then proper feature negotiation works fine.

The only advantages of bundling such features is where it makes  
little practical sense to have feature B available without feature A,  
and for political/marketing reasons, to encourage deployment of a  
package of features.

The latter can be addressed by defined profiles of features (rather  
like "WiFi" is a package of IEEE 802.11 features, or XEP-0243  
packages up XMPP featuresets).

I'm not against versions, mind, I just think they're generally a last  
resort, and a simple "I support these optional features" statement  
from one or both sides works much more effectively in most cases.

> which is not something terribly difficult to implement.

There's keepalive frames - ie, I send you data, and you throw it on  
the floor - and pings - ie, I send you data, and you send something  
back.

These are two distinct cases, and although in general terms, TCP will  
make both of them a two-way communication, a "real" ping can be used  
for substantial diagnostics, whereas I suspect that the TCP ACK will  
sneak under various thresholds.

I've tried gathering mobile information for XMPP implementors -  
submitted at http://xmpp.org/extensions/inbox/mobile.html - this may  
be somewhat useful for WebSocket folk as well.

Dave.

---

From: "Thomson, Martin" <[email-masked]>
Date: Tue, 27 Jul 2010 16:10:13 +0800

The nice thing is, if we stuff up websockets, we can always:

Upgrade: someotherprotocol

This goes for when we discover that we can't build an extension.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Simone Bordet
> Sent: Tuesday, 27 July 2010 9:53 AM
> To: Ian Hickson
> Cc: [email-masked]
> Subject: Re: [hybi] Adding clarification regarding future revisions
> (Was: Proposed way forward for WebSockets)
> 
> Hi,
>

---

From: gabriel montenegro <[email-masked]>
Date: Tue, 27 Jul 2010 01:17:52 -0700 (PDT)

Phasing out development is good, but I agree with Greg about building in, from 
day one, some way to extend or add other features as the charter is clear that 
this is not just for browsers. Even though it is possible to define future 
headers or using the subprotocol mechanism to define or change behavior to one's 
satisfaction, what is not clear is what would be the status of those specs, nor 
what document or delivery vehicle would be used. That is fine for some 
functionality, but what the charter calls for is for the WG to produce a general 
protocol (not just for browsers) as part of the official WG document. So it's 
not enough to say that one could define future behavior, we have to define a 
complete protocol which in my mind implies binary support at least.

BTW, I like to look at the charter, as that is what guides the WG. There used to 
be a time when charters were
nice, decorative items in the IETF. That changed already several years ago. Now 
the IESG approves charter for a reason.
The IESG will hold the WG accountable for it.  

Gabriel

----- Original Message ----
> From: Anne van Kesteren <[email-masked]>
> To: Ian Hickson <[email-masked]>; Maciej Stachowiak <[email-masked]>
> Cc: [email-masked]
> Sent: Tue, July 27, 2010 7:58:36 AM
> Subject: Re: [hybi] Proposed way forward for WebSockets
> 
> On Tue, 27 Jul 2010 03:00:08 +0200, Maciej Stachowiak <[email-masked]> wrote:
> --Anne van Kesteren

---

From: "Thomson, Martin" <[email-masked]>
Date: Tue, 27 Jul 2010 16:29:05 +0800

There is such a mechanism:

We might want to avoid this, but it's nice to know that the option is there.

> gabriel montenegro
> Sent: Tuesday, 27 July 2010 10:18 AM
> To: Anne van Kesteren; Ian Hickson; Maciej Stachowiak
> Cc: [email-masked]
> > > Ian's suggested timeline below sounds like an excellent way to
> proceed, in my
> >opinion.
> >
> > +1
> >
> >
> > --Anne van Kesteren

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 27 Jul 2010 10:38:09 +0200

Hi Ian,

> better to clarify it, though. Could you elaborate on exactly what is 

I can't find there the requirement to reject handshakes featuring
unknown headers, nor can I find it in -76, though I'm sure I've read
it not long ago, so probably that it was in a slightly earlier version.

Your wording in the Design Philosophy section looks fine an unambiguous
to me.

> Not having a version field is intentional. Versioning is an anti-pattern 

I don't understand what you mean by "anti-pattern". Right now for
instance it's fortunate that we have the distinction between HTTP/1.0
and 1.1 since their semantics have changed (connection: close vs
keep-alive, support for transfer-encoding, etc...).

Without a version advertised in the request, the server won't know
whether the client is supposed to support feature XXX or YYY that
is only part of an evolution.

Regards,
Willy

---

From: gabriel montenegro <[email-masked]>
Date: Tue, 27 Jul 2010 01:40:35 -0700 (PDT)

Yes, and I can show you a proof of concept of using that mechanism to exchange 
XMPP directly after the upgrade.
My point is that whereas delivering a mechanism to exchange XMPP is not in the 
charter (hence it is ok to define
such an alternative, or other such examples, outside of the official WG 
document), what is in the charter is the need 

for a general protocol, so if one required "Upgrade: someotherprotocol" to 
obtain binary capabilties for use, say,
by a non-browser client, then there would be no guarantee of interoperability to 
the same degree that an official
proposed standard RFC from the IETF would promote.

----- Original Message ----
> From: "Thomson, Martin" <[email-masked]>
> To: gabriel montenegro <[email-masked]>; Anne van Kesteren 
><[email-masked]>; Ian Hickson <[email-masked]>; Maciej Stachowiak 
><[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> Sent: Tue, July 27, 2010 10:29:05 AM
> Subject: RE: [hybi] Proposed way forward for WebSockets
>

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 27 Jul 2010 10:45:21 +0200

> from one or both sides works much more effectively in most cases.

I understand your point but I'm not convinced at all. A version is
a short way to advertise a feature set. Once you have to send all
of your requests with "WS-supported-feature: XXX, YYY, ZZZ", then
you'll find it a lot cheaper to define a new version which sets the
most commonly used sets by default so that we don't have to enumerate
them all. For instance, in HTTP, since version 1.1 we don't advertise
support for keep-alive nor do we advertise support for transfer-encoding,
those are covered by default by the version.

> the floor - and pings - ie, I send you data, and you send something  

yes you're perfectly right, I tend to merge them but they're distinct.

> sneak under various thresholds.

And also the application will not be aware of the TCP ACK, it's only
the system. The application only knows it could successfully push the
data to the system's buffers and that the connection was not yet
detected as being broken.

Regards,
Willy

---

From: Adam Barth <[email-masked]>
Date: Tue, 27 Jul 2010 10:49:23 +0200

On Tue, Jul 27, 2010 at 9:03 AM, Anne van Kesteren <[email-masked]> wrote:
> This is one of the
> most basic extensibility mechanisms there is. I.e. MUST not generate and
> MUST ignore so that in the future we CAN generate and CAN do something...

Anne, for whatever reason, folks at the IETF don't seem to understand
this design pattern right away.  It's some sort of cultural mismatch
between browser folks (who using this pattern all the time for
extensibility and robustness) and IETF folks.  In my experience, folks
understand after I explain it to them, but you shouldn't assume
they'll understand it without explanation.

Adam

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 27 Jul 2010 11:01:36 +0200

Hi Ian,

>  - We update the test suites.

I globally agree with this plan, but everything depends on what we
consider a critical bug and if a missing feature can be seen as a
bug (eg: keep-alives or pings), since these *will* cause random
hangs on long-lived connections. The protocol should really define
at least a NOP message that must be ignored by either end.

> technologies; we should not repeat that mistake.

I would see it differently : when the spec makes it impossible to add
new features, there is no way we'll see richer implementations demonstrating
the need for those features. Look at HTTP for instance : it is very
extensible and keep-alive would never have existed if it had not been
permitted by the protocol spec.

Last, there is something I find complicated when reading the spec : it is
defined exactly as code. When you read it, you read code written in english.
It is very hard to implement something correct based on this because you
don't easily know what relates to *your* implementation, nor how your
implementation specific issues should be handled.

Having an additional section which defines the protocol instead of how to
process it would tremendously help.

Regards,
Willy

---

From: Dave Cridland <[email-masked]>
Date: Tue, 27 Jul 2010 10:16:39 +0100

> they'll understand it without explanation.

Us poor stupid IETF folks tend to declare intent - hence we wouldn't  
say that clients MUST NOT send other headers, we'd say that a future  
extension MAY define new headers, and client SHOULD NOT add headers.  
Then anyone reading the specification knows what's going on, instead  
of having to read between the lines.

Also, extended clients remain comformant to the original spec, which  
seems sane.

Dave.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 27 Jul 2010 11:28:43 +0200

> Also, extended clients remain comformant to the original spec, which  

FWIW, we typically evolve the specification with the implementations so  
they stay in sync. Also seems sane.

---

From: Lars Eggert <[email-masked]>
Date: Tue, 27 Jul 2010 12:41:54 +0200

Hi,

> they'll understand it without explanation.

you must be kidding. That's probably the most common protocol design pattern that you'll find in the RFCs.

Lars

---

From: Pieter Hintjens <[email-masked]>
Date: Tue, 27 Jul 2010 14:02:32 +0200

> Not having a version field is intentional. Versioning is an anti-pattern

The... what?

http://en.wikipedia.org/wiki/Anti-pattern is quite an interesting read
and relevant to this list in many ways but does not mention
versioning.

-Pieter

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 27 Jul 2010 23:38:52 +1000

On 27 July 2010 17:03, Anne van Kesteren <[email-masked]> wrote:

> clients for HTTP, HTML has no versioning by design, CSS has no versioning by
> design, etc. Versioning does not work well on the Web.
>

Versions are used extensively in HTTP on the server side, where the HTTP
version sets the default options and assumed behaviour that is then altered
by negotiation.

For example HTTP/1.0 connections are assumed non-persistent unless
negotiated otherwise, while HTTP/1.1 is assumed persistent unless otherwise
declared.

However,  I generally agree that having a good optional feature negotiation
mechanism is probably more important that a version.  However I think that a
version is at least helpful to define the known option space and is not
particularly harmful.

regards

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 27 Jul 2010 16:42:46 +0100

> This goes for when we discover that we can't build an extension.

No, that doesn't work :-(

Think about when you want the browser to use some protocol extension
that is of no concern to the script writer, if and only if the server
supports it

What are you going to put in the Upgrade field that'll work with all
servers?

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 27 Jul 2010 17:08:06 +0100

> Here's a proposed timeline for adding these features:

Hurrah! I'm in broad agreement with this proposal.

> - We deploy the protocol in four or more major browser vendors (next 4

[stuff]

>  compression, multiplexing, per-frame metadata annotation, etc, as 

Critical issue:

After the first deployment wave, how can client WebSocket
implementations request features without breaking the existing
services that *reject* and *close* connections with additional state
in the headers?

The subprotocol field (as is) cannot be used for this by a WebSocket
API implementation because it is not allowed to change what the script
provides.

Imho a robust, extensible negotiation slot must be present early for
this plan to work.  Assuming this is agreeable as a requirement we can
work on making it simple, robust, and secure.

> is needed and we should add it.

I wholeheartedly support the notion that scripts could implement
protocol features that the browser doesn't perform natively - where
possible.

I'd like to remove one of your asterisks:

Useful connection sharing *cannot* be implemented in script:

- How do you share a connection among multiple tabs?
- How do you report the message events back to each tab?
- How do you avoid blocking all tabs communications when a single tab
  has a full event queue and is too busy to empty it?

Browser security makes it somewhere between very difficult and
impossible, depending on what browser quirks you take advantage of.
This is already a problem today, using long-get HTTP.

Connection sharing in a single tab is of course possible, but
that also has limitations.

It is conceivable to script a replacement WebSocket object type which
"emulates" it on top of the real one while sharing conenctions in a
single tab.  It would be good to determine if that's practicable.  I
imagine it'll behave too different from the real one to be a drop-in
replacement because of browser script limitations.

Negotiation faces the same problem as described earlier for the
WebSocket API implementation itself - namely that the subprotocol
field can't be used.

The solution is the same as earlier - with the addition that scripts
need to be able to participate in negotation by requesting features
(and somehow getting involved at the frame type level?) - which means
an additional API function.

> compliant client-side implementations, so we should not add them now.)

I'm not sure what makes compression clearly needed.  This is not an
argument against, just that I seem to be missing something.

I agree that piecemeal features is good idea, and I support that
approach.  The framework does need to be flexible enough to accomodate
piecemeal features for it to work out.  At the moment the handshake
may be too rigid and prevent it working out.

Although I think it's a good plan, don't assume piecemeal features
automatically results in better client-side compliance.  It makes it
simpler to write throwaway implementations.  That is useful, but might
also result in *less* compliance on average with the base protocol due
to quick and dirty implementations being good enough.  We'll see.

I'm not sure what HTTP history has to say about *client*-side
implementations in this regard.  Regarding HTTP servers, history shows
us that the first decade was littered with buggy, non-compliant
servers due to the ease of writing bad but good enough ones.
Compliance increased dramatically since sophisticated fully-featured
servers became more popular.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 27 Jul 2010 17:29:20 +0100

> Not having a version field is intentional. Versioning is an anti-pattern 

That statement is too opaque: it needs a reference or something.

I don't know what Ian means, but it might be this: In the Javascript
browser world, everyone is told to *avoid* browser version checks and
to check for *feature presence* instead.

But here's the thing: *Using* version numbers is the anti-pattern,
not *providing* them.

Browser version checks are a major anti-pattern, especially the awful
ones which do pattern matching on User-Agent, because they prevent new
browsers from changing anything that existing scripts check for.

Feature checks work quite well in browser JavaScript: Often the
presence of an object or attribute is enough.  It's not great though:
Implementations of XmlHttpRequest still vary enormously, for example.
So almost everyone sticks to a de facto common subset that work in the
big browsers, or they combine it with dodgy "existence of feature A
implies feature B works like blah because the big browsers have that
combination of behaviours".  Thus feature checks have problems too.

Then there's HTTP: version numbers in HTTP aren't reliable.  Quite a
few HTTP things advertise HTTP/1.1 but are not 1.1 complient.  Thus
you cannot trust the HTTP version, and if you care and if you can,
look for other hints that say what not to trust.

So neither versions nor detectable feature are reliable or trustable
in a large ecosystem where something deployed and buggy may need to be
supported anyway.

In general if a buggy implementation becomes widespread, you can't
trust what it says whether it's a version or feature string, and you
need whatever clues you can get to detect the buggy thing.

Version numbers, feature lists and user-agent style strings are all
useful clues when you have to cope with buggy things out there.

If there are no clues, that's an anti-pattern.

Concluding:

  - *Using* version numbers and user-agent strings instead of
    explicit feature checks is an anti-pattern because it blocks
    updates to those numbers and strings and forces them to lie.

  - But *not providing them* is an anti-pattern because it
    prevents workarounds and thus blocks deployment of new features.

Neither extreme is effective.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 27 Jul 2010 17:46:31 +0100

> > features, then proper feature negotiation works fine.

Fwiw, I agree.

> > from one or both sides works much more effectively in most cases.

I agree with that too.  It's good to *have* versions and
identification strings available as a last resort, but it's important
to make sure there's a better way for the other end to detect features.

Willy Tarreau wrote:
> them all.

It's not great: The new implementation then doesn't use those features
with older implementations that support the features perfectly well.

> For instance, in HTTP, since version 1.1 we don't advertise

HTTP is an example of it not working very well :-)

Most HTTP/1.1 clients still send a HTTP/1.0 keep-alive extension
header, because saying HTTP/1.1 doesn't help the 1.0 servers.

Moreover, most HTTP/1.1 clients don't use transfer-encoding for the
request, even though it's standard, for a variety of reasons.  They
don't use chunked, and compressed transfer-encoding is rare in a
request.

HTTP/1.1 pipelining is avoided in many cases where it might have been
usable - even with servers that confirm they support HTTP/1.1 in an
earlier response.

All that is primarily because HTTP clients have no way to negotiate
with the server, so they have to stick with a common subset that works
with all servers.

In other words, HTTP negotiation either by feature tokens in headers
or by version number, is quite bad for the client-to-server direction.

It is an open question whether "HTTP/1.2" could ever be deployed or if
it would break.

> yes you're perfectly right, I tend to merge them but they're distinct.

There's a third type: status-keepalive frames which confirm that state
previously sent is still valid for a given time interval.  That's
application layer, but it benefits from aggregation just like other
keepalive types.

> detected as being broken.

Anyway TCP ACK is a false positive if there's a transparent intermediary.
It is not useful.

-- Jamie

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Tue, 27 Jul 2010 10:07:57 -0700

> servers became more popular.

I talked with Hixie about how to achieve sharing of a connection, and while
I believe that it is possible to do it with a Shared Worker (and yes, this
would address sharing across multiple tabs), I think that there are a number
of gotchas. For one thing, there would be no good way to share a worker
across different subdomains - e.g. imagine mail.google.com, www.google.com,
wave.google.com and a few others wanted to share a single websocket so as to
reduce the number of open connections for a given user. Trying to do shared
workers would basically mean using an iframe to google.com + postmessage.
This could have negative impact on memory usage in the case that various
"clients" are consuming messages at drastically different rates (too much
data in flight == big buffers). There's also the consideration of whatever
latency gets added by having to load an iframe. So, I believe that sharing
of a single connection could be done at the application level, but I am not
certain as to the efficiency of such an approach. Especially if we add
things like compression at the protocol level, which I think we should, I
think we should give strong consideration to Multiplexing and how that would
be best achieved while not making things unnecessarily complex.

As for compression, there are a number of cases where today, via using XHR
you get compression, and if you switched to WS you would lose compression
(unless you did it at application level, where again the latency question
would come up). Think e.g. suggest on google.com where you type something
and get back a list of suggested search terms. This is something that
compresses reasonably well and an example where we would not want to lose
compression.

---

From: John Tamplin <[email-masked]>
Date: Tue, 27 Jul 2010 14:06:49 -0400

> argument against, just that I seem to be missing something.

See my earlier post
http://www.ietf.org/mail-archive/web/hybi/current/msg01810.html for
compression results on real-world apps running over WebSocket (v75, though
that would make little difference).

---

From: Dave Cridland <[email-masked]>
Date: Tue, 27 Jul 2010 19:25:24 +0100

On Tue Jul 27 18:07:57 2010, Ian Fette (イアンフェッティ)  
> compresses reasonably well and an example where we would not want  

Compression is an obvious extension to include in the base spec,  
because it's sufficiently straightforward to do and will prove the  
extension mechanism.

I'd also suggest considering an inline TLS switch, since that makes  
deploying TLS-enabled WebSocket applications very much easier - it  
means that the URI scheme 'ws' does not prevent encryption.

Done sufficiently cleverly, this could even be transparent to  
intermediaries aware of WebSockets, but that would need binary  
framing I think. 
Dave
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 28 Jul 2010 15:12:35 +0900

As many others have said, Ian's proposal is a big step in the right 
direction.

One comment on Greg's comments below:

> On 27 July 2010 10:56, Ian Hickson<[email-masked]>  wrote:

> that unless otherwise specified the binary content is UTF-8 encoded text.

In my opinion, to put it mildly, this is a recipie for disaster. Having 
frame types for UTF-8 and binary separately creates a clear situation. 
Something like "UTF-8, but occasionally binary", or whatever in that 
direction, essentially means "any character encoding goes", which means 
that the "UTF-8" part is just meaningless. This means that we are back 
to unidentified byte gunk, where it's totally unclear what each byte 
means in terms of characters.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Joe Hildebrand <[email-masked]>
Date: Wed, 28 Jul 2010 13:54:23 +0200

>     prevents workarounds and thus blocks deployment of new features.

There is precedent for this.  In XMPP, we have a version number, which we
originally set to 1.0 in order to signal support for the standard version,
rather than the pre-standard version.  We defined a version policy, but have
successfully fought off all attempts to change the version number by
providing other feature negotiation mechanisms.  It's still nice to know
it's there if we need it, however.

The text from 3920bis (http://bit.ly/9eX8Bu) could be used as a template.

-- 
Joe Hildebrand
(as individual)

---

From: Dave Cridland <[email-masked]>
Date: Wed, 28 Jul 2010 13:02:22 +0100

> it's there if we need it, however.

Right, the version in XMPP is used as a flag-day, and one of the  
capabilities its presence signals is the ability to negotiate  
features.

Prior versions - ie, no version at all - could not negotiate features.

Dave.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 28 Jul 2010 14:52:59 +0200

Jamie Lokier wrote:
 [...]
> work on making it simple, robust, and secure.
+1.

---

From: "Anne van Kesteren" <[email-masked]>
Date: Thu, 29 Jul 2010 09:35:02 +0200

> services that *reject* and *close* connections with additional state

http://www.whatwg.org/specs/web-apps/current-work/complete/network.html#reading-the-client's-opening-handshake

"Unrecognized fields can be safely ignored, and are probably either the  
result of intermediaries injecting fields unrelated to the operation of  
the WebSocket protocol, or clients that support future versions of the  
protocol offering options that the server doesn't support."

> This is already a problem today, using long-get HTTP.

You can do this via Web Workers. Specifically, a SharedWorker. As for the  
concern raised by Ian Fette, that should be addressed by adding  
cross-origin support to Web Workers.

> to quick and dirty implementations being good enough.  We'll see.

It has worked pretty well with some HTML5 features so far and having to  
implement less will make it easier for a single person to write a test  
suite that covers most everything in not too much time. But yeah, it  
always remains to be seen what the end result will be.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 29 Jul 2010 17:08:17 +0900

> protocol offering options that the server doesn't support."

So, just to take an example, let's say it turned out that browsers sent 
the User-Agent: header field, and some servers actually looked at it.

Would your understanding of extensibility mean that we can explain this 
as a (as of yet) not standardized future version/feature of WebSockets? 
And that if it turned out that this were reasonably prevalent, and 
remotely useful, somebody could write an RFC (as part of this WGs future 
charter or as an individual submission) that would 'legalize' this header.

I think it would help at least my understanding of your extensibility 
model if you could confirm/deny the above.

Regards,    Martin.

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 29 Jul 2010 18:08:36 +1000

On 27 July 2010 19:16, Dave Cridland <[email-masked]> wrote:

> Us poor stupid IETF folks tend to declare intent - hence we wouldn't say
> that clients MUST NOT send other headers, we'd say that a future extension
> MAY define new headers, and client SHOULD NOT add headers. Then anyone
> reading the specification knows what's going on, instead of having to read
> between the lines.
>

Wow!  I think we have all been in violent agreement with each other for the
last few months about header issue.

I'm sorry if I've been a bit thick coming to this conclusion, because the
spec does read clearly in the parts that Anne has highlighted.   But
obviously the cultural mismatch that Adam refers to has caused me to take
many of the statements from Ian and others the wrong way.  This has not been
helped by some implementations that also misinterpreted the spec and
rejected handshakes that have additional headers in them.

The "IETF folks" certainly do understand the design pattern of being
rigorous in what you generate and forgiving in what you parse.  In fact that
has been used as an argument for the changes in the handshake that we have
been advocating (eg arbitrary ordering of headers).

Anyway,  I think we should all take this as a lesson that simple
miscommunication can result in months and months of pointless bickering.
Hopefully with luck there are other issues that we actually agree on, but
are just expressing ourselves differently.

For this particular issue, it would be really great if the spec could
include verbage to the effect that future extension MAY define new headers
and that implementations MUST ignore unknown headers.

regards

---

From: "Anne van Kesteren" <[email-masked]>
Date: Sat, 31 Jul 2010 17:58:24 +0200

> model if you could confirm/deny the above.

Yes, though preferably sending additional headers from the client is  
discussed with other parties beforehand on a relevant forum, such as  
[email-masked].