Subject: [hybi] Trains, planes and web protocols

From: Paul Prescod <[email-masked]>
Date: Sun, 5 Apr 2009 10:00:26 -0700

So let's say that I am controlling my model trains from my web browser using
web sockets. The trains set sends a message to say "the train is approaching
the junction: you have 20 seconds to make a decision or the train will
crash." At that second, the connection is lost for any of a thousand
reasons, from a neighbour picking up her wireless phone to a roommate
tripping over a power cord. The problem lasts for one second, but the
message is lost. How will Web Sockets help the client and server to get back
in sync?

My understanding is that the protocol defers this problem to the application
level, despite the fact that almost every application is susceptible to
inaccuracies caused by intermittent message delivery problems. I believe
that it is unacceptable to solve the simple problems in the protocol and
defer all of the difficult stuff to the application.

I propose that the standard should address issues of reliable message
delivery, or build on a standard that has already done so.

 Paul Prescod

---

From: Paul Prescod <[email-masked]>
Date: Sun, 5 Apr 2009 10:06:58 -0700

Just to clarify: I didn't mean to say that every client and server should
support reliable delivery, but those that need it should have an obvious
path to do so, rather than: "Just build it yourself on the protocol."

On Sun, Apr 5, 2009 at 10:00 AM, Paul Prescod <[email-masked]> wrote:

> So let's say that I am controlling my model trains from my web browser
> using web sockets. The trains set sends a message to say "the train is
> approaching the junction: you have 20 seconds to make a decision or the
> train will crash." At that second, the connection is lost for any of a
> thousand reasons, from a neighbour picking up her wireless phone to a
> roommate tripping over a power cord. The problem lasts for one second, but
> the message is lost. How will Web Sockets help the client and server to get
> back in sync?
>
> My understanding is that the protocol defers this problem to the
> application level, despite the fact that almost every application is
> susceptible to inaccuracies caused by intermittent message delivery
> problems. I believe that it is unacceptable to solve the simple problems in
> the protocol and defer all of the difficult stuff to the application.
>

---

From: Tony Garnock-Jones <[email-masked]>
Date: Sun, 05 Apr 2009 11:54:56 -0700

> delivery, or build on a standard that has already done so.

I disagree. Reliable delivery (for almost every useful definition of the 
term) is an application-level concern. To me there's a natural hierarchy 
here:

   * transport for getting blobs from named node to named node,
     in a best-effort fashion
   * distributed object system (doesn't have to be complex)
   * messaging system (e.g. pubsub, AMQP etc) and other apps

HTTP serves as the transport. URLs name the nodes. The distributed 
object system is where webhooks come in: the POSTs (etc) activate the 
services (and make requests) of the objects.

The messaging system, with notions of trust and 
transfer-of-responsibility, is a missing piece; here's where, for 
example, AMQP's operations could be transplanted into a webhooks-based 
distributed object system.

When it comes to reliability, it is a sad truth that no matter how much 
help you get from the network, it can only ever be the application's 
final responsibility to ensure that it is properly heard by a remote 
application.

HTTP is actually a very nice stab at a best effort blob delivery 
protocol: it provides positive acknowledgement of acceptance of a 
message, and being based around TCP shares all of the nice reliability 
properties of that.

Your example, of an urgent message that must be reliably delivered, 
requires the use of a persistent identifier for the message, with 
deduplication at the receiving application, and an appropriate 
retransmission strategy at the sender until positive acknowledgement is 
received from the receiver.

Regards,
   Tony

---

From: Tony Garnock-Jones <[email-masked]>
Date: Sun, 05 Apr 2009 12:03:39 -0700

On the other hand, let me contradict myself:

> received from the receiver.

We could absolutely build in the persistent identifier, which is 
necessary but not sufficient for exactly-once delivery. Something along 
the lines of Mark Nottingham's POE, 
http://www.mnot.net/drafts/draft-nottingham-http-poe-00.txt, perhaps. [1]

Or: I've been thinking about (ab)use of Content-Location in a symmetric 
(rHTTP-with-enrolment) HTTP network. Using Content-Location to 
essentially POST links to resources is one thing (e.g. twitter- or 
delicious-style sharing links: "Please go read this"), but it has a dual 
role of providing a URL for the POSTed resource. The HTTP spec 
indicates, but stops short of making it clear, that Content-Location 
could be used in an HTTP request message. It could be our persistent 
identifier.

Regards,
   Tony

[1]: In the POE draft (now long expired!) there's mention of the Link 
header, apropos which 
http://www.mnot.net/drafts/draft-nottingham-http-link-header-03.txt. 
Very cool, am wholly in favour, modulo a quibble or two about rel=first.

---

From: Ian Hickson <[email-masked]>
Date: Mon, 6 Apr 2009 08:28:23 +0000 (UTC)

What I'd like to work on is "TCP for the Web".

I think one of the key points is that when adding things to the Web
platform, IMHO we want to add basic building blocks, rather than
comprehensive protocols.

For example, we have HTML, not DocBook. We have a generic scripting 
language, not a domain-specific language. There are libraries like jQuery, 
Prototype, Y!UI, etc, which provide richer environments for authors. In 
recent times we've added generic SQL database support; despite the main 
use case being client-side e-mail storage, we didn't provide, say, local 
IMAP support. We added generic JavaScript background threads; we didn't 
add, say, a high-level database synchronisation mechanism.

The idea is to provide features that allow people to build libraries that 
provide complex features. This is similar to how the POSIX platform 
provides basic building blocks (libc) and then libraries like libssl, 
libz, libcurl, etc, provide the high-level functionality.

So for instance with bidirectional communication, I think what we should 
be aiming for is something on the level of a TCP socket (modulo security 
requirements), not something like XMPP or HTTP. We're not trying to solve 
a specific problem, we're trying to make it possible to solve a broad 
range of specific problems by providing a simple building block: a socket, 
the Web way.

Now, I have no objection to Jabber or other high-level protocols being 
made available to Web pages, if they can be shown to be securable, but 
that's not the kind of problem I'm trying to solve. WebSocket isn't 
intended to compete with that kind of feature.

With that in mind:

On Fri, 3 Apr 2009, Paul Prescod wrote:
> On Fri, Apr 3, 2009 at 2:07 PM, Ian Hickson <[email-masked]> wrote:
> >
> > [IM is] _an_ application, but I don't think it's the biggest. I would 
> > expect games to be bigger, for example, and would expect background 
> > update notificiations in systems like Google Calendar to be even more 
> > common.
> 
> Most real-time social games will have chat embedded or on the page 
> around them.

Sure, but they'll all have subtly different needs. I imagine there will be 
libraries that provide such functionality for many sites, but for example 
while GMail will want to provide chat features with all the users in the 
greater federated Jabber network, a site like casualcollective will want 
to only have chat within the users of the site, and during a game, only 
the users in the game. A game like that from the Rainbow 6 series of games 
will want to distinguish between players who are still alive and players 
who are not; a game from the Quake series of games will not. Some will 
want some kinds of smileys, others might want audio and video. If we 
provide a protocol that is tuned to the purposes of one kind of chat, 
we'll fail to provide the features desired by another site, and sites will 
have to coerce the provided features into doing what they want.

> For example, how do I send a GIF over a Web Socket? HTTP and Jabber have 
> already solved that problem. How do I correlate a response to a message 
> with the original message?

If the solution HTTP has for sending a file is satisfactory, then 
presumably we don't need one for WebSocket. :-)

There are lots of solutions to the problem of sending files to a server, 
or sending messages in a variety of ways correlated with a request, like 
HTTP and Jabber as you mention, but also like FTP, IMAP, POP, BEEP, and a 
whole host of other high-level application protocols.

I'm interested in providing a feature that is the level below that, like 
TCP is to the above.

> > I'm not convinced that it's really a fair comparison, either. rHTTP as 
> > a protocol is remarkably complex, in that it includes by reference the 
> > whole of HTTP, which is a remarkably complex protocol.
> 
> I did say that the NEW specification is very short and simple.

It doesn't really matter how simple the "NEW" specification is, what 
matters is the complexity the author is exposed to.

> > Actually using rHTTP for true full-duplex connections would be much 
> > more complicated, since you'd need to somehow get two separate 
> > connections from the client to the server into the same CGI script.
> 
> I don't know what you mean.
> 
> These services cannot be served with CGI. CGI is built around 
> request-response. Just as PHP scripts do not in general implement HTTP, 
> I wouldn't expect the handlers of these things to implement the 
> protocol.

On the contrary, I expect the wide variety of uses of this feature to lead 
directly to a situation where people write custom implementations all the 
time. That's one the of the assumptions behind the Web Socket design. 
Interoperable heterogeneity is good, it reduces the impact of any single 
security bug, for instance.

> Here's how I expect it would work: there is a service optimized for 
> long-running TCP connections. Maybe it's written in C or Erlang. There 
> are PHP scripts (or whatever) for info coming in from web clients. When 
> PHP wants to tell the universe that "something happens" it puts a 
> message in a queue. Erlang or C delivers it.

That might work for some uses of bi-directional communication, but it 
wouldn't work for all of them. Luckily, this is the kind of thing that is 
easily built on top of the Web Socket protocol.

> Web developers do not want to become protocol handler developers and 
> they should not. The protocol should not be designed with the 
> presumption that they will be protocol developers. In fact: in my more 
> than 10 years of web development experience, I don't recall ever 
> implementing an HTTP client or server.

Certainly it would be reasonable for people to write servers that can be 
extended in that manner, but that doesn't mean we should preclude the 
possibility of people writing their own. Just like how with CGI scripts it 
is common for libraries to be used for interpreting the CGI environment 
variables, but people still sometimes write their own.

> What I have done frequently though, is trace the text flowing on the 
> wire, which is why a text-based protocol is preferable to me than a 
> binary one.

The Web Socket protocol is trivial to understand (0x00 indicates the start 
of a frame, 0xFF the end, and everything in between is ASCII/UTF-8) so I 
really don't think that it will make be hard to debug on the wire.

> > With Web Socket, there is only one underlying TCP connection.
> 
> So what? How does that make my app simpler? Are you expecting PHP 
> programmers to write long-running TCP server apps?

Yes.

> I don't understand why it matters whether it is one connection or two.

Having two connections is IMHO an misuse of the underlying protocol (TCP).

> > - It's not clear that Jabber's protocol supports the opt-in security
> >   model that is desired for an HTML API.
> 
> Not sure what that means.

We need to make sure that whatever we provide is consistent with the 
browser "same-origin" security model. It's not clear to me that Jabber is 
compatible with this model, since it assumes that the user agent is 
acting on behalf of the user.

> > Also, it's not clear to me that streaming XML is a particularly good 
> > solution. Experience with Web authors suggests that XML's draconian 
> > error handling and XML's namespaces are more confusing than 
> > desireable.
> 
> The web developer is just putting messages into a queue. It's the 
> protocol handler (like Apache, ejabberd, postfix) that talks the 
> protocol.
> 
> Can you point to a single protocol deployed on the Internet that is 
> commonly implemented and re-implemented and then re-re-implemented in 
> scripts, rather than being packaged in servers in libraries?

IRC. The multitude of multiplayer game protocols. The variety of protocols 
layered on top of <iframe>+<script>+XHR. The name/value pairs used in form 
data to write applications over HTML+forms.

The whole point here is to enable more of this for Web application 
developers. Right now they have no way to invent their own protocol at the 
TCP level.

On Fri, 3 Apr 2009, Paul Prescod wrote:
>
> By the way, you mentioned calendars: calDAV is the web protocol for 
> manipulating them. It might be a good fit for reverse HTTP.

We're not taking about manipulating calendars, we're talking about the UI 
side of an application talking to its logic backend. Other examples would 
be Google Docs getting a user's file list, or updating a document or 
spreadsheet as it is being edited. AtomPub and CalDAV wouldn't be 
appropriate, because they are generic inter-application protocols, not 
application-specific internal protocols tuned for the needs of those 
specific applications.

On Fri, 3 Apr 2009, Peter Saint-Andre wrote:
> > 
> >  - It's not clear that one can easily upgrade from HTTP to a 
> >    bidirectional Jabber connection.
> 
> Why upgrade from HTTP to XMPP?

Because in some environments HTTP is all we have, so we need a migration 
path from an HTTP connection to the two-way solution.

> Just build XMPP support into the browser, and you can refer to certain 
> resources as XMPP URIs/IRIs and other resources as HTTP URLs. Or the 
> client (browser) and server can negotiate an XMPP channel between 
> themselves for more advanced bidirectional functionality.

As noted earlier, I have no objection to this happening as well. But it 
dosn't solve all the needs. For example it wouldn't be a very efficient 
way of doing the train control I mentioned. It also would be a pretty poor 
transport for live editing updates as seen in Google Docs -- if the user 
hits one key, and so you need to send one byte to the server, a 200% 
overhead (as with Web Sockets) is already pretty extreme. Forcing the 
application author to frame each character in XML elements and namespaces 
and so forth would not be our proudest moment.

> >  - It's not clear that Jabber's protocol is designed to be resistant to 
> >    the attacks we expect (such as connecting to an SMTP server, or an 
> >    existing Jabber server, or whatnot).
> 
> Have you completed a threat analysis for the projected use cases? I 
> don't see anything about that in draft-hixie-thewebsocketprotocol-07 but 
> perhaps it is defined elsewhere.

I don't have anything written down, but the protocol was examined in this
context, yes. I would welcome an independent analysis though.

> > Experience with Web authors suggests that XML's draconian error 
> > handling and XML's namespaces are more confusing than desireable.
> 
> Whose experience? Where is this documented?

For draconican error handling, the disaster that is XML deployment on the 
Web is the primary documentation.

For namespaces, see: http://wiki.whatwg.org/wiki/Namespace_confusion

> The XMPP developer community has produced a large volume of software 
> (see links above) and runs a large network of servers with millions of 
> IM users. So far we have not experienced any significant interop 
> problems.

Sure, but you're all competent. We're talking about letting random authors 
who are barely able to write CGI scripts create new interactive 
applications here. More than 95% of HTML pages are invalid in some way. 
Simplicity is key here.

> > - The protocol should support running over ports 80 and 443,
> >   ideally with the ability to share the port on the server
> >   with an HTTP server.
> 
> Why? Using a different port for a different protocol seems like the 
> proper approach.

Because in practice, there are environments that block all but port 80, 
and applications like GMail won't use solutions that don't work in those 
environments too.

(Note that Web Sockets default to another port; the requirement is only 
that ports 80 and 443 be supported, not required or the default.)

On Sat, 4 Apr 2009, Greg Wilkins wrote:
> 
> If the IETF does come out with a bidirectional HTTP recommendation, it 
> will be a huge once in a decade opportunity to upgrade the protocols 
> supported by the network infrastructure.

Why would it be a once in a decade opportunity? If there is a need, then 
it can be filled. It's not like we can only create new protocols when the 
planets align in some particular way.

> Websockets sentinel UTF-8 datagrams may well be ideally suited for a 
> specific type of application, but can we really say that this will be 
> the only content type that we need to transport asynchronously server to 
> client for the next decade or two?

The protocol is generic enough (it can do text and known-length binary out 
of the box, and is easily extensible to handle multiplexed infinite 
streams, structured data, finite unknown length binary data via chunking, 
etc), and simple enough (it has virtually no features, leaving everything 
up to the application) to be usable for most bi-directional use cases for 
Web applications.

> What if UTF-16 becomes the preferred encoding? or gzipped UTF-16 or some 
> as-of-yet-un-invented compression algorithm specially designed for 
> packets of UTF-16 data holding JSON becomes in common usage.

We can use the known-length encoding mechanism to transport all of those, 
so they would be easy extensions.

> I'm already working with clients that wish to push images.

We can use the known-length encoding mechanism support by the Web Socket 
protocol as designed today to handle any kind of binary data.

> If web socket supports arbitrary binary data, then why is there a need 
> for the sentinel mechanism and special status for UTF-8 data?

There's a need for a separate "this is text" frame than "this is arbitrary 
data" frame because at the API level the only datatype today is text. 
(This is why the binary mode frames are currently discarded -- JS doesn't 
yet support representing the binary data.)

Once you're sending binary data, you want to avoid length markers because 
authors are very much more likely to get measuring UTF-8 data wrong 
compared to just stick an 0xFF byte on the end.

> Note also with websocket binary transfer, I believe it is a limitation 
> that there cannot be meta data (headers) associated with a datagram. 
> Thus information like mime types, char-sets content-encodings and 
> transfer encodings will need to be handled with stateful mechanisms 
> based on assumption, connection and/or previous sent datagrams.

There can be as much metadata as the applicaton layer wants -- just put 
it before the data, using an application-specific format.

This is just like with TCP, there's no room for MIME types. Protocols that 
use TCP, like HTTP, layer MIME types on top. This isn't a failing of TCP.

On Fri, 3 Apr 2009, Alex Russell wrote:
> >
> > I'm not convinced that it's really a fair comparison, either. rHTTP as 
> > a protocol is remarkably complex, in that it includes by reference the 
> > whole of HTTP, which is a remarkably complex protocol.
> 
> That sort of depends on which version you're talking about. HTTP 1.0 is 
> relatively simple, whereas 1.1 is hugely complicated, particularly when 
> it comes to accommodating chunked encoding and pipelining.

HTTP 1.0 is relatively simple compared to 1.1, but it has all kinds of 
weird subtlties, like header continuations, the many verbs (GET POST 
etc) and respone codes (200, 404, etc), that I wouldn't be at all 
surprised to see misimplemented by random authors. Getting a complete 
conforming implementation is non-trivial, even for 1.0.

> If you're looking to have the complexity of 1.0 as a high-water-mark, 
> that would seem perfectly reasonable. People are going to mis-implement 
> whatever you spec anyway.

Granted, but it's actually pretty hard to misimplement WebSockets as 
specced now and still have it work in the common case. (Though I'm sure 
creative Web designers will prove me wrong.)

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Ian Hickson <[email-masked]>
Date: Mon, 6 Apr 2009 20:21:40 +0000 (UTC)

On Mon, 6 Apr 2009, Alex Russell wrote:
> On Mon, Apr 6, 2009 at 1:28 AM, Ian Hickson <[email-masked]> wrote:
> >
> How about TCP, then?

Unfortunately raw TCP has a number of disadvantages:

 - It doesn't have a good story for preventing cross-origin attacks, 
   e.g. where a hostile page contacts an intranet server. (Web Socket 
   enforces cross-origin integrity by requiring the server to acknowledge 
   the origin of the initiating page.)

 - It doesn't have a good story for upgrading from HTTP, which is one of 
   our requirements.

 - It doesn't play well with virtual hosting, which (in combination with 
   the previous two points) make it hard to use securely in shared 
   hosting environments.

 - It doesn't have protection against connecting to existing services like 
   SMTP servers, HTTP servers, IRC servers, etc.

Note that the SVG working group originally considered using raw TCP, and 
the browser vendors pointed to a number of security problems with this 
which resulted in the feature being dropped.

> I'd also like some form of content-neutral, bi-directional byte 
> packaging format that will let me encapsulate message sending at a 
> higher level. Being able to send binary data seems equally important for 
> the long-term applications (think feeding a <video> element streaming 
> content via a websocket).

Web Socket as designed today supports framing binary data.

> I have a hard time with this line of argumentation. The web has nearly 
> always taken flexibility away in return for ease of implementation. 
> Setting limits and optimizing for the 80% case is *good*, and the web is 
> proof.

I agree, though apparently in this case we reach different conclusions 
based on this principle.

> Not having some sort of metadata at the front of a frame really does 
> seem like a serious oversight. If you want to be TCP for the web, at 
> least copy the parts that worked well = )

Could you elaborate on this? What are the features of TCP that we should 
duplicate at the WebSocket layer? (Isn't it enough that they be available 
at the TCP layer?) I don't really understand. What API are you proposing 
be made available?

---

From: Ian Hickson <[email-masked]>
Date: Mon, 6 Apr 2009 23:20:26 +0000 (UTC)

On Mon, 6 Apr 2009, Alex Russell wrote:
> ...and will require senders and receivers to encode/decode (at least) 
> null bytes with some alternate representation that isn't spec'd.

No, not at all. Binary data is sent with a known length declaration, 
there's no reason to escape anything.

> > Could you elaborate on this? What are the features of TCP that we 
> > should duplicate at the WebSocket layer? (Isn't it enough that they be 
> > available at the TCP layer?) I don't really understand. What API are 
> > you proposing be made available?
> 
> Some sort of header that describes a message seems essential. We made 
> the mistake in Bayeux of not including one and I regret it.

I don't understand. Could you elaborate on this?

What would it look like at the API level?

Right now we have:

   socket.postMessage("move:1,2");

Are you saying it should instead be?:

   socket.postMessage("move", "1,2");

...or some such?

I think in the middle-term we will probably want to extend the API to 
support JSON (or some other encoding, e.g. Protocol Buffers) as a message 
type, would that resolve this issue?

> Greg has argued for a content-length descriptor and I can see wanting 
> other kinds of application-level routing information to be made 
> available in a message so that parsing it isn't required to handle/drop 
> it.

I don't understand. Why would a second length field be useful?

> HTTP did a great job here by allowing an arbitrary number of headers for 
> payloads, and we're still reaping the benefits. At least leaving room 
> for a standard-format message header seems useful.

The payload is completely open-ended, the idea is that applications can 
put whatever format they want in there. If they want HTTP-like headers per 
message, then that's quite possible, e.g.:

   socket.postMessage("Action: Move" + crlf +
                      "Player: + player + crlf +
                      crlf +
                      "1,2");

...or whatever. The idea is to leave this up to the author, just like TCP 
leaves the payload completely open to the next layer.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 07 Apr 2009 09:25:34 +1000

>  - It doesn't have a good story for upgrading from HTTP, which is one of 

I think upgrade from HTTP is key to these discussions.
If HTTP is our starting point, then upgrade is the natural path.
So I see two key questions:

1) Must we upgrade? are there any recommendations or simple
extension for HTTP that will make it better suited (eg 1xx
responses with content)?

2) If we upgrade what should we upgrade to: rHTTP, websocket,
Beep, XMPP?

In the second case, perhaps we should not try to pick a winner.
Instead we could be looking at what infrastructural and
security issues exist to prevent upgrade from HTTP to an
arbitrary connection based protocol between any
pair of consenting HTTP client/server.

My understanding of the situation today is that upgrade
frequently works because proxy servers often act as simple
TCP tunnels after checking the headers of the first
request/response.  But there are also proxies that will
prevent an upgrade as they wish to inspect/handler every
request.  Thus horrid hacks such as using CONNECT are
contemplated.

It would be great if we could come up with recommendations
for both questions.  How to make HTTP work better bidirectionally
for the current long-pollers of the world, and how to make
upgrade work more generally so that the alternative protocols
can be deployed and compete for who's best.

cheers

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 6 Apr 2009 18:34:12 -0500

+1

Too much of the discussion on this list so far has been loaded with assumptions about an existing solution.

As Greg points out, maybe this can be solved without blessing any particular solution.  Then we are all free to go our separate ways and build our nasty, complicated, proprietary protocols without stepping on each others toes.

--Martin

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Greg Wilkins
> Sent: Tuesday, 7 April 2009 9:26 AM
> To: Ian Hickson
> Cc: [email-masked]; Alex Russell
> Subject: Re: [hybi] WebSockets feedback (Was: Bayeux / Jetty
> perspective.)
> 
> Ian Hickson wrote:
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

------------------------------------------------------------------------------------------------
This message is for the designated recipient only and may
contain privileged, proprietary, or otherwise private information.  
If you have received it in error, please notify the sender
immediately and delete the original.  Any unauthorized use of
this email is prohibited.
------------------------------------------------------------------------------------------------
[mf2]

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 07 Apr 2009 09:58:11 +1000

> leaves the payload completely open to the next layer.

Ian,

TCP does not leave it up to the author.   TCP has HTTP, FTP, IMAP,
SMTP, etc. as standards that define content and handling.
TCP has headers and ports to help determine which protocol is in use.

I don't believe Websocket is well suited to become the basis
for other protocols.   It lacks the meta data support and the
UTF-8 favouritism is an issue.    EG I could not use the approach
you advocate above to send a gzipped utf-16 encoded body!

Something like BEEP could indeed be used as a basis for other
protocols, so we should ensure that we can upgrade to both websocket
and protocols like BEEP.

cheers

---

From: Jonas Sicking <[email-masked]>
Date: Mon, 6 Apr 2009 17:24:05 -0700

On Mon, Apr 6, 2009 at 4:58 PM, Greg Wilkins <[email-masked]> wrote:
>>> payloads, and we're still reaping the benefits. At least leaving room for a
>>> standard-format message header seems useful.
>>
> UTF-8 favouritism is an issue.

This is an interesting issue indeed. Maybe a tiny amount of metadata
information could be added without adding a significant amount of
overhead.

/ Jonas

---

From: Ian Hickson <[email-masked]>
Date: Tue, 7 Apr 2009 00:27:54 +0000 (UTC)

On Tue, 7 Apr 2009, Greg Wilkins wrote:
> 
> TCP does not leave it up to the author.  TCP has HTTP, FTP, IMAP, SMTP, 
> etc. as standards that define content and handling. TCP has headers and 
> ports to help determine which protocol is in use.

Sure. WebSocket is URI-addressable, which is intended to allow servers to 
distinguish between different services in this way. This leverages the 
ideas behind HTTP's service addressing mechanisms.

> I don't believe Websocket is well suited to become the basis for other 
> protocols.  It lacks the meta data support and the UTF-8 favouritism is 
> an issue.  EG I could not use the approach you advocate above to send a 
> gzipped utf-16 encoded body!

There's no way from JavaScript to get a hold of a gzipped UTF-16 encoded 
blob right now at all. This is why the API is limited to text, and why 
there is UTF-8 favouritism in the protocol. However, once we have binary 
support in JS, I see no reason why we couldn't extend the API to support 
things like this:

   socket.postBinaryMessage(textToUTF8("Filename: " + name + crlf +
                                       "Format: UTF-16" + crlf +
                                       crlf) + 
                            gzip(textToUTF16(data)));

...where textToUTF8(), textToUTF16(), and gzip() are helper methods that 
handle binary data.

The protocol itself already supports binary transport.

> Something like BEEP could indeed be used as a basis for other protocols, 
> so we should ensure that we can upgrade to both websocket and protocols 
> like BEEP.

I agree. WebSocket is not intended to prevent the user of other protocols. 
Right now, HTTP supports Upgrade:ing to any protocol that defines a way to 
handle an HTTP Upgrade, as far as I can tell, so I don't think there is a 
problem here.

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 06 Apr 2009 18:28:48 -0600

On 4/6/09 5:25 PM, Greg Wilkins wrote:
> Ian Hickson wrote:
>>  - It doesn't have a good story for upgrading from HTTP, which is one
>> of    our requirements.
> 

Perhaps it would help to define precisely what we mean by upgrading from
HTTP. Does this mean that an HTTP client is communicating with an HTTP
server (for some kind of functionality), and the two parties agree to
switch to a different protocol (for some other kind of functionality)?

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 06 Apr 2009 18:53:38 -0600

> What I'd like to work on is "TCP for the Web".

That's what we tried to define with BOSH. Whether we succeeded is an
open question.

/psa

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 07 Apr 2009 11:33:15 +1000

> switch to a different protocol (for some other kind of functionality)?

The upgrade mechanism of HTTP/1.1 is a hop-by-hop mechanism,
so it a user-agent, a server and all the intermediary gateways
and proxies support a specific upgrade, then the protocol
used by a connection can be switched to almost any
connection oriented protocol.

The problem is that all proxies and gateways need to
support the upgraded protocol.  So even if we were able to
make all browsers and servers support websocket, this would
not be sufficient as HTTP/1.1 compliant proxy should not forward
upgrade headers!

(Note that some proxies and gateways do work, as they
ignore upgrade headers and after the first HTTP request/response
some act as just TCP/IP tunnels).

So for non-HTTP upgrades to be widely usable, the network
infrastructure will need to be upgraded to support either
specific protocols (eg websocket) or generic bidirectional
protocols.

I'd prefer the second option (not picking a winner) and
it may be as simple as producing a recommendation
of how a proxy/gateway should handle an upgrade request
for an unknown protocol.   For example, perhaps we could
recommended that unknown upgrades are handled by proxy's and
gateways by them becoming non-caching, non-buffering, TCP/IP
relays.  Ie all packets received are sent without caching,
aggregation, alternation etc.  (like CONNECT).

This legitimises the gateways and proxies that already
work and will encourage more to work over time.  It would
reduce (eventually remove) the need for misusing the CONNECT
mechanism (which has security issues as it allows arbitrary
IP's and ports).

If we can get the proxies and gateways of the world to
support such arbitrary upgrades, then websocket, rHTTP,
XMPP, BEEP should all be able to be easily supported
by consenting pairs of clients and servers.

cheers

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 07 Apr 2009 11:39:01 +0900

>> ...or whatever. The idea is to leave this up to the author, just like
>> TCP leaves the payload completely open to the next layer.

It seems indeed that that's what's currently happening in many
long-polling applications. I'm not sure whether that's a good
thing or not.

I think one difference between things such as HTTP/SMTP/... and
long polling is that for the former, the main scenario is that
totally unrelated servers and clients have to interoperate.
That's why there is a lot of metadata specified and transferred
in the protocol.

For long polling, it's currently mostly that the server code
(Perl, PHP, Python, Ruby,...) and the client code
(Javascript) are written by the same person or group.
That's why less metadata may be needed, and when needed
(e.g. a timestamp), it's sent ad-hoc.

But once we have a well-accepted protocol for long polling,
I'm sure there will be more ideas and needs to communicate
between different, independently written client and servers,
and the value of agreed-upon conventions for metadata and
such will increase.

Also, please note that in it's first version (later called
HTTP 0.9), HTTP was in many ways about as simple as the
current WebSockets protocol. But that proved to be too
simple.

On 2009/04/07 8:58, Greg Wilkins wrote:

> for other protocols.

It would definitely be a good idea to think about how it can be
extended on various levels. But maybe Ian already has ideas,
he just don't want to tell us because otherwise, everybody
will start extending willy-nilly.

It would also be a good idea to think about what kinds of protocols
one would like to run on top of a long-poll connection.

> It lacks the meta data support and the

I disagree with "UTF-8 favouritism". 99.9% of programmers
want to get text across when they want to get text across,
they couldn't care less about encoding. Fixing the encoding
in the general case to something that is well known to work
very widely is a very good thing.

Most IETF protocols until a little while ago came with a strong
"ASCII favoritism". The first version(s) of the Web came with
a strong "ISO-8859-1 favoritism". Both are problematic because
they exclude large parts of the world. But both were extremely
helpful in getting interoperability and ease of implementation
for programmers who don't want to deal with character encoding
details. With UTF-8, we get both the benefit of a single encoding
and coverage of the world.

As for UTF-8 being inefficient in some cases, it indeed takes
about 1.5 times the bandwidth for purely East Asian text, about
2 times the bandwidth for Cyrillic, Greek, Arabic, Hebrew, and
about 3 times the bandwidth for Indic scripts and scripts such
as Georgian (and about 4 times the bandwidth for cases such as
Linear B, but I hope not too many are worried about that :-).
But with every bit of framing, XML, JSON, or whatever, which
mostly will be in ASCII, these differences get smaller and
smaller.

Regards,    Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Ian Hickson <[email-masked]>
Date: Tue, 7 Apr 2009 02:48:02 +0000 (UTC)

On Tue, 7 Apr 2009, "Martin J. Dürst" wrote:
> 
> Also, please note that in it's first version (later called HTTP 0.9), 
> HTTP was in many ways about as simple as the current WebSockets 
> protocol. But that proved to be too simple.

The WebSocket protocol is extensible by design. It supports two kinds of 
frames (0xFF-sentinel-delimited and predeclared-length), and there can be 
up to 128 subtypes of each kind. The current version only exposes one of 
these, with a payload defined to be UTF-8.

> It would definitely be a good idea to think about how it can be extended 
> on various levels. But maybe Ian already has ideas, he just don't want 
> to tell us because otherwise, everybody will start extending 
> willy-nilly.

I've already listed a variety of probable extensions:

 - simple binary data (known length arbitrary payloads)

 - chunked binary data (unknown length arbitrary payloads)

 - structured data (e.g. JSON)

 - multiplexed video

...etc. The protocol is designed such that these new frame types can be 
introduced later without breaking compatibility with deployed 
implementations: UAs are required to skip unknown frame types.

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 07 Apr 2009 14:03:36 +1000

Ian,

this is extensibility by the protocol specifiers, so that later versions
of the protocol can introduce new packets without breaking backwards
compatibility.

But HTTP is user extensible.     So if I invent a new content
type, encoding or compression algorithm, I don't need to wait for
HTTP/1.2 to use that.    I don't need to apply to some standards
body to allocate an identifier from a scarce (128) space.

Defining subtypes for  compress-utf8-data,  gzip-utf8-data,
gzip-utf8-data will use up a 128 space very quickly.

Without such subtypes, then you'd have to use one of the
untype binary data types and rely on assumptions of
compression type or stateful exchange of meta data in
prior datagrams.

I'm sorry if I appear to be bashing websocket.  That really is not my
intention (and he who lives in glass houses should not throw
stones).     I'm just saying any general purpose transport needs
to have the flexibility and extensibility of HTTP.

cheers

PS. and I'm also not saying that Ian is proposing websocket
as a general purpose transport.

---

From: Ian Hickson <[email-masked]>
Date: Tue, 7 Apr 2009 04:53:43 +0000 (UTC)

On Tue, 7 Apr 2009, Greg Wilkins wrote:
> > 
> > The WebSocket protocol is extensible by design. It supports two kinds 
> > of frames (0xFF-sentinel-delimited and predeclared-length), and there 
> > can be up to 128 subtypes of each kind. The current version only 
> > exposes one of these, with a payload defined to be UTF-8.
> 

Right.

> But HTTP is user extensible.  So if I invent a new content type, 
> encoding or compression algorithm, I don't need to wait for HTTP/1.2 to 
> use that.  I don't need to apply to some standards body to allocate an 
> identifier from a scarce (128) space.
>
> Defining subtypes for compress-utf8-data, gzip-utf8-data, gzip-utf8-data 
> will use up a 128 space very quickly.

Sure, but that won't happen. The protocol has one purpose: to be the 
backend for an API exposed by browsers to Web page scripts. Web Sockets 
would be a terrible protocol to use for other purposes -- just use TCP!

Since it is highly unlikely that browser vendors would be interested in 
adding hundreds of different ways of encoding data, the protocol is 
unlikely to need many of its frame types. We wouldn't want anyone to be 
extending the protocol without updating the API; that would be missing 
the point, as browsers wouldn't know how to support those types.

Having said that, if we do find we're running out, we can just define that 
types 0x7F and 0xFF will have further information at the start of the 
payload saying what the type is.

---

From: "Sylvain Hellegouarch" <[email-masked]>
Date: Tue, 7 Apr 2009 08:54:56 +0200 (CEST)

> would be a terrible protocol to use for other purposes -- just use TCP!

I think this long exchange comes from different set of expectations. You
initiated this discussion with a few common use cases you consider being
worthy enough of a new, however small, protocol.

Unfortunately different people have different background. As a developer I
find it miserable not to have direct access to TCP sockets from Javascript
so that I could converse with a XMPP server without having to relay on
BOSH. As an infrastructure/system person I would probably see it with a
keen eye that web applications can't open sockets up at will from an
already prone to security issue that is the browser.

In this context I can understand why your draft is so short and that's
probably a good sign because it'll be that easier to see deployed.
Nonetheless this long discussion makes me wonder whether or not having a
new protocol is a better idea than picking one of the existing
alternatives (Comet, BOSH). Wouldn't it be better to focus energy on best
practices when it comes to long polling connections based on what's
working now?

Indeed, it'd be already a good starting point listing the shortcomings of
those existing alternatives. What are the showstoppers? In which use case?

- Sylvain
-- 
Sylvain Hellegouarch
http://www.defuze.org

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 07 Apr 2009 09:17:49 -0600

> those existing alternatives. What are the showstoppers? In which use case?

This is why we need to write an I-D that describes the use cases. :)

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Frank Salim <[email-masked]>
Date: Tue, 7 Apr 2009 10:49:01 -0700

> those existing alternatives. What are the showstoppers? In which use case?

I started filling out the wiki with that information.
http://trac.tools.ietf.org/bof/trac/wiki/HyBi

-Frank

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 08 Apr 2009 10:47:14 +0900

>> practices when it comes to long polling connections based on what's

Writing this up is definitely a good idea. But the things that are 
working now, and the things that people might want to do once long-time 
connections are available might differ considerably.

Regards,   Martin.