Subject: Re: [hybi] WebSockets feedback (Was: Bayeux / Jetty perspective.)

From: Greg Wilkins <[email-masked]>
Date: Fri, 03 Apr 2009 17:19:11 +1100

Ian Hickson wrote:
> On Tue, 31 Mar 2009, Greg Wilkins wrote:
>> The websocket proposal is both good and bad.
>> The Unicode datagram API is probably very good
>> for javascript programming.   But the zero terminated
>> wire protocol is horrible.
> 
> Could you elaborate on this? I'm curious to find out why you think it is 
> horrible.

Because it means that I need to scan data received just to know if I
have all the data I need to proceed.

If I knew a content-length or a chunk length, then I would know
immediately that a read return that I have enough data to process.

With null termination, I have to scan the data looking for nulls (because
I could receive multiple datagrams).   But if I'm scanning the data then
I'd like to be converting the UTF-8 at the same time, but as I might
not have all the data, I could end up with 4 bytes of a 5 byte utf-8
character and so my utf-conversion has to be interrupted and I have
to save the chars that I've already converted and wait for another
packet to come in.

The worst part is that most datagrams will arrive in 1 packet and
most UTF-8 chars are 1 byte, so most of the time this will not be a
problem!   But I still need to write code that will work with data
split between packets or packets with multiple datagrams etc.  So
I expect there will be lots of buggy implementations out there that
will look like they work, but will fail on a nasty network.

>> We see no reason why the wire protocol should deviate so far from normal 
>> HTTP style framing.
> 
> What would you consider normal HTTP style framing here? What would HTTP 
> framing give you that isn't available with the framing currently proposed 
> with Web Sockets?

HTTP framing is:
  + Some CRLF terminated headers containing meta-data
  + A CRLF
  + The content that is delimited either by:
     * a known content length
     * chunks, each with a known content length
     * EOF

We already have the code written and tested to handle this.  This code
is already robust in the face of nasty networks that send us 1 byte per
packet etc. etc.

It supports more or less arbitrary extensible meta data along with
the content, so all the normal mime type mechanisms can be brought
to bear.

> It seems that having a single byte at the start and a single byte at the 
> end is all the framing one needs. The byte at the start gives the kind of 
> frame, and the byte at the end is a byte that isn't valid in UTF-8 so it 
> gets around the need to have a length declaration (such declaration are 
> often the source of buffer overrun bugs).

It is certainly sufficient, but it is not efficient or convenient to
handle.     More over it is new and different without any reason to
be new and different.   If there is some deficiency in HTTP framing?
Is there some reason that you don't want to transport non-utf8 data?
Eg why can't I push an image or a sound bite?

>> More over, normal HTTP framing would allow binary data to be sent to 
>> non-javascript clients!
> 
> Binary data is supported by the WebSocket protocol, it's just not exposed 
> yet. (One of the design principles we have been following in the HTML5 
> space is "baby steps", starting with small APIs and only adding more 
> features once the first version is widely implemented.)
> 
> Specifically, to send binary data using the protocol you would send a 0x80 
> byte, the length (as a packed integer), and then the binary data.

Again, this feels like re-inventing the wheel.   HTTP already supports
binary data with known and unknown content lengths (chunking).

Things like packed integer lengths can come back to haunt you. With
Jetty I have had several users have problems because we initially
had length being an int and we had to convert to using a long.
OK, I can't think of the use-case for pushing such large content,
but then I don't see any reason to pick a transport that has that
limit built in.

HTTP is tried and tested and can transport anything.
I don't want something new, I just want it to go both ways!

cheers

---

From: Ian Hickson <[email-masked]>
Date: Fri, 3 Apr 2009 09:00:09 +0000 (UTC)

On Fri, 3 Apr 2009, Greg Wilkins wrote:
> > >
> > > The websocket proposal is both good and bad. The Unicode datagram 
> > > API is probably very good for javascript programming.  But the zero 
> > > terminated wire protocol is horrible.
> 

Yes, that's the disadvantage of using sentinel markers. The reason for 
this design decision is to reduce the likelihood of buffer overruns on the 
server side. Many people writing Web software are unfortunately not the 
most competent software engineers, and this has to be taken into account 
when writing protocols that they are expected to implement. You are not 
representative of the majority, to say the least. :-)

> With null termination, I have to scan the data looking for nulls 
> (because I could receive multiple datagrams).  But if I'm scanning the 
> data then I'd like to be converting the UTF-8 at the same time, but as I 
> might not have all the data, I could end up with 4 bytes of a 5 byte 
> utf-8 character and so my utf-conversion has to be interrupted and I 
> have to save the chars that I've already converted and wait for another 
> packet to come in.

An interruptible UTF-8 decoder doesn't seem like a huge problem.

> So I expect there will be lots of buggy implementations out there that 

I doubt most implementations would try to UTF-8 decode at the same time as 
looking for the sentinel byte. That's pretty advanced. I expect most would 
just scan for the 0xFF and then ship the whole string in one go to an off- 
the-shelf UTF-8 decoder.

> > > We see no reason why the wire protocol should deviate so far from 
> > > normal HTTP style framing.
> > 
> > What would you consider normal HTTP style framing here? What would 
> > HTTP framing give you that isn't available with the framing currently 
> > proposed with Web Sockets?
> 
> HTTP framing is:

How is the headers part of this not the same as the 0xFF sentinel byte? 
You have to scan for a CR byte just like you would for Web Socket data, 
no? Indeed it seems the Web Socket format is simpler, since there is no 
complication with dealing with lone CR bytes with no matching LF, etc.

> We already have the code written and tested to handle this.

The Web Socket protocol is designed on the assumption that people will be 
writing new code, regardless of how similar we make it to existing 
protocols. This is similar to how many people have written their own CGI 
handling or form handling despite the existence of many libraries.

(I should note that HTTP is by far not the only protocol that could be 
adapted for this purpose, and indeed is far from the most appropriate one 
of the ones I looked at. IPv6 or Protocol Buffers, for example, would 
probably be more appropriate protocols to model for the purpose of 
arbitrary data transfer. Indeed, the protocol's one-byte frame type and 
the handling of binary data are both based on Protocol Buffers.)

> It supports more or less arbitrary extensible meta data along with the 
> content, so all the normal mime type mechanisms can be brought to bear.

I don't see the relevance of this; the idea is not to send MIME data, but 
to send either text or binary data, for subsequent handling by the client 
and server.

> > It seems that having a single byte at the start and a single byte at 
> > the end is all the framing one needs. The byte at the start gives the 
> > kind of frame, and the byte at the end is a byte that isn't valid in 
> > UTF-8 so it gets around the need to have a length declaration (such 
> > declaration are often the source of buffer overrun bugs).
> 
> handle. More over it is new and different without any reason to be new 
> and different.

As noted above, it's not particularly new. The idea of indicating frame 
types with a single byte isn't new, nor are sentinels in strings (C has 
used null-terminated strings for decades), and nor are variable-sized 
integers for encoding lengths. In fact, pretty much every aspect of the 
Web Socket protocol is based on long-established practice.

> If there is some deficiency in HTTP framing?

HTTP framing is woefully inefficient for this purpose. I don't really know 
why we would even look at HTTP for this. If we were to use an existing 
text-based protocol's framing for this, I would imagine XMPP or IRC would 
be better choices. However, I see no reason why we would want to use 
text-based framing here.

> Is there some reason that you don't want to transport non-utf8 data? Eg 
> why can't I push an image or a sound bite?

The protocol does support binary data transfer, as noted earlier. You use 
frame type 0x80, followed by a variable-sized integer, followed by the 
binary data. This is not yet exposed in the API, and servers do not have 
to support it, but the protocol is designed to handle this and assuming 
the Web Socket protocol is adopted, future versions will almost certainly 
expose it. The protocol is also intended to be extensible to handle 
multiplexed streams and other more elaborate data types.

> > Specifically, to send binary data using the protocol you would send a 
> > 0x80 byte, the length (as a packed integer), and then the binary data.
> 

Many protocols support binary data with known and unknown content lengths, 
HTTP is certainly not alone here. :-)

In the case of the Web Socket protocol as designed today, there is trivial 
support for known-length binary data, and the protocol is designed in such 
a way that unknown-length binary data, using a scheme similar to HTTP 
chunking (but encoded in a more efficient maner), could be relatively 
easily supported.

> Jetty I have had several users have problems because we initially had 
> length being an int and we had to convert to using a long.

I used the wrong term when I said "packed integer"; I was referring to 
variable-sized integers, not boundary-aligned integers, my apologies. The 
length encoding in Web Socket supports arbitrarily sized binary blobs, 
there is no upper limit. (Processors are expected to abort the connection 
if their peer tries to send data with an unsupported length.)

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Paul Prescod <[email-masked]>
Date: Fri, 3 Apr 2009 09:21:38 -0700

In general, the benefit of using an existing protocol is that you take
advantage of a lot of infrastructure and in particular of protocols built ON
TOP of that protocol. Maybe that doesn't matter for train control, but it
matters a lot for webchat, which as far as I'm concerned is the single
biggest application of a "server->client" protocol. To me, this strongly
suggests Jabber, but I'll make the case for HTTP because I'm more familiar
with it:

 * new specification is very short and simple (e.g., not much more than [1])

 * client-libraries and intermediaries are more easily adapted (e.g. Curl,
httplib, ...)

 * domain knowledge is easily applied ("it's just like writing a PHP script
except you use Javascript and listen to Firefox rather than Apache")

 * standards built on HTTP will work more easily (cookies, AtomPub, WebDAV,
Subversion, etc.)

 * in general, the information system called "the Web" will have fewer
protocols and will be simpler, which will help it evolve and survive in the
long run. For example, if we add message signing to HTTP then we get it "for
free" in both directions. If we add a reliability protocol to HTTP then we
get it "for free" in both directions. ("for free" from a specification point
of view...from an implementation point of view it is "for cheap" because you
can reuse C code between Firefox and Apache if you like, or IIS and IE,
etc.)

That all said, my current feeling is still that Jabber is the best choice
because it is already the dominant protocol for the asynchronous
bi-directional use cases we're talking about. But I really cannot see the
advantage in inventing a new protocol from scratch. The Web and Internet are
complicated enough as they are without a proliferation of framing protocols.

 Paul Prescod

[1] http://www.ietf.org/internet-drafts/draft-lentczner-rhttp-00.txt

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 03 Apr 2009 09:32:19 -0700

> representative of the majority, to say the least. :-)

Do you mean a buffer overrun in the sense of a security problem? I  
don't see how the protocol framing makes this more or less likely. If  
anything, servers have to be careful to avoid being tricked into  
sending the end marker in the middle of a message. Multipart MIME has  
this problem, and implementing it in the context of form submission  
it's been difficult to get the security details right. I think a  
declared length has better security properties in a network protocol  
than a distinguished end marker.

> An interruptible UTF-8 decoder doesn't seem like a huge problem.

It's doable but you have to preserve state. However, having tricky  
edge cases that normally don't come up is a design problem. It is,  
once again, the sort of thing that tends to lead to security holes.

> complication with dealing with lone CR bytes with no matching LF, etc.

It definitely seems to me that HTTP-style headers are unnecessary and  
needlessly complex here. They may also look confusingly like a  
persistent http connection, which this is not. And headers would be  
pretty high overhead for short text-based messages, which is likely to  
be a common use case here. However framing with a length or known- 
length chunks tends to lead to simpler processing for the recipient.

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 03 Apr 2009 09:36:32 -0700

On Apr 3, 2009, at 9:21 AM, Paul Prescod wrote:

> In general, the benefit of using an existing protocol is that you  
> take advantage of a lot of infrastructure and in particular of  
> protocols built ON TOP of that protocol. Maybe that doesn't matter  
> for train control, but it matters a lot for webchat, which as far as  
> I'm concerned is the single biggest application of a "server- 
> >client" protocol. To me, this strongly suggests Jabber, but I'll  
> make the case for HTTP because I'm more familiar with it:

I think basing this protocol on the surface syntax of HTTP is a bad  
idea. HTTP is a request/response protocol, but the new protocol is  
simply two-way messaging. Messages are not (at least so far as the  
protocol knows) requests, nor are they responses to other messages.  
HTTP has the baked-in idea that messages are either requests or  
responses, that every request gets a response, and that every  
interaction is initiated by the client side. Using a similar framing  
syntax with totally different semantics is unlikely to make it easy to  
adapt existing tools.

Regards,
Maciej

---

From: Paul Prescod <[email-masked]>
Date: Fri, 3 Apr 2009 10:32:11 -0700

As I said, Jabber is probably a better fit. But if it were to use HTTP I
would not suggest to use the "surface syntax". If it were to use HTTP then
it would just add the minimum required to reverse the communications
direction:
 * http://www.ietf.org/internet-drafts/draft-lentczner-rhttp-00.txt

Then you would use one socket for initiator to listener communications and
another for the reverse. Any HTTP client library that is smart about
persistent connections could be quickly adapted to send "server->client"
messages.

But in any case, given that the dominant use case is web chat and Jabber is
the ascendent web chat protocol, I do think that Jabber is a better choice.

On Fri, Apr 3, 2009 at 9:36 AM, Maciej Stachowiak <[email-masked]> wrote:

> I think basing this protocol on the surface syntax of HTTP is a bad idea.
> HTTP is a request/response protocol, but the new protocol is simply two-way
> messaging. Messages are not (at least so far as the protocol knows)
> requests, nor are they responses to other messages. HTTP has the baked-in
> idea that messages are either requests or responses, that every request gets
> a response, and that every interaction is initiated by the client side.
> Using a similar framing syntax with totally different semantics is unlikely
> to make it easy to adapt existing tools.
>
> Regards,
> Maciej
>
>

---

From: Ian Hickson <[email-masked]>
Date: Fri, 3 Apr 2009 21:07:23 +0000 (UTC)

On Fri, 3 Apr 2009, Paul Prescod wrote:
> advantage of a lot of infrastructure and in particular of protocols 
> built ON TOP of that protocol. Maybe that doesn't matter for train 
> control, but it matters a lot for webchat, which as far as I'm concerned 
> is the single biggest application of a "server->client" protocol.

It's _an_ application, but I don't think it's the biggest. I would expect 
games to be bigger, for example, and would expect background update 
notificiations in systems like Google Calendar to be even more common.

But it's hard to say in advance.

> but I'll make the case for HTTP because I'm more familiar with it:
> 

The Web Socket draft is pretty simple.

I'm not convinced that it's really a fair comparison, either. rHTTP as a 
protocol is remarkably complex, in that it includes by reference the whole 
of HTTP, which is a remarkably complex protocol.

>  * client-libraries and intermediaries are more easily adapted (e.g. 
> Curl, httplib, ...)

One advantage of Web Socket is that the protocol is so simple that there's 
no need to adapt an existing library, new code can be easily written to 
support the protocol.

> except you use Javascript and listen to Firefox rather than Apache")

Actually using rHTTP for true full-duplex connections would be much more 
complicated, since you'd need to somehow get two separate connections from 
the client to the server into the same CGI script. With Web Socket, there 
is only one underlying TCP connection.

>  * standards built on HTTP will work more easily (cookies, AtomPub, 
> WebDAV, Subversion, etc.)

I agree entirely that if you want to do Subversion, or AtomPub, that you 
should not try to layer it on top of WebSocket. That isn't the use case 
we're targetting, though. (I don't know anyone who wants their server to 
be able to speak WebDAV to their JavaScript running in a browser.)

> protocols and will be simpler, which will help it evolve and survive in 
> the long run. For example, if we add message signing to HTTP then we get 
> it "for free" in both directions. If we add a reliability protocol to 
> HTTP then we get it "for free" in both directions. ("for free" from a 
> specification point of view...from an implementation point of view it is 
> "for cheap" because you can reuse C code between Firefox and Apache if 
> you like, or IIS and IE, etc.)

I don't think that a small number of very complex technologies is a better 
situation than a large number of simple technologies.

> That all said, my current feeling is still that Jabber is the best 
> choice because it is already the dominant protocol for the asynchronous 
> bi-directional use cases we're talking about. But I really cannot see 
> the advantage in inventing a new protocol from scratch. The Web and 
> Internet are complicated enough as they are without a proliferation of 
> framing protocols.

Jabber doesn't really meet the requirements for which Web Socket is 
designed [1]. Specifically:

 - It's not clear that one can easily upgrade from HTTP to a bidirectional 
   Jabber connection. Jabber over HTTP uses two TCP connections, as I 
   understand it.

 - It's not at all clear that a fully-conforming server-side component for 
   Jabber can be written in a few dozen lines of scripting code.

 - Jabber uses a separate sidechannel for streaming video, if I 
   understand correctly; it's not extensible to support efficient transfer 
   of binary data over the same channel.

 - It's not clear that Jabber's handling of virtual hosting is really 
   compatible with the "same-origin" model.

 - It's not clear that Jabber's protocol is designed to be resistant to 
   the attacks we expect (such as connecting to an SMTP server, or an 
   existing Jabber server, or whatnot).

 - It's not clear that Jabber's protocol supports the opt-in security 
   model that is desired for an HTML API.

Also, it's not clear to me that streaming XML is a particularly good 
solution. Experience with Web authors suggests that XML's draconian error 
handling and XML's namespaces are more confusing than desireable.

[1] http://www.ietf.org/mail-archive/web/hybi/current/msg00007.html

On Fri, 3 Apr 2009, Paul Prescod wrote:
> would not suggest to use the "surface syntax". If it were to use HTTP 
> then it would just add the minimum required to reverse the 
> communications direction:
>

I think rHTTP makes sense for certain use cases, but it doesn't fit the 
requirements that led to Web Socket's development (in particular, as you 
say, you'd need two TCP connections, which is non-trivial to set up in 
most scripting environments shared with a Web server).

On Fri, 3 Apr 2009, Maciej Stachowiak wrote:
> > this design decision is to reduce the likelihood of buffer overruns on 
> > the server side.
> 
> Do you mean a buffer overrun in the sense of a security problem? I don't 
> see how the protocol framing makes this more or less likely.

Not necessarily a security problem. The problem with lengths is that since 
the data is UTF-8, but the length would have to be bytes, you are relying 
on the programmer being able to measure the byte-length of a Unicode 
string, which internally might well be stored as UTF-16 or some other 
encoding, only to be serialised to UTF-8 on output. Sentinel markers are 
significantly easier to deal with from the point of view of an amateur 
script author, IMHO.

> If anything, servers have to be careful to avoid being tricked into 
> this problem, and implementing it in the context of form submission it's 
> been difficult to get the security details right.

The sentinel marker in Web Socket is 0xFF, which is never valid UTF-8. So 
long as the author properly encodes all text to UTF-8 on output, there 
shouldn't be a problem. If they fail to do that, then their system will 
have bigger problems than just being able to have 0xFF bytes inserted at 
random.

> I think a declared length has better security properties in a network 
> protocol than a distinguished end marker.

The Web Socket protocol uses a declared length for the case of binary 
data, where measuring the data length on the server is likely easy.

> It's doable but you have to preserve state. However, having tricky edge 
> cases that normally don't come up is a design problem. It is, once 
> again, the sort of thing that tends to lead to security holes.

I agree, but in practice I don't think most authors are going to decode 
UTF-8 in parallel with looking for the sentinel. They're just going to 
buffer bytes until they have the sentinel, then get their scripting 
language to just treat the bytes as UTF-8 data.

---

From: Paul Prescod <[email-masked]>
Date: Fri, 3 Apr 2009 15:59:37 -0700

On Fri, Apr 3, 2009 at 2:07 PM, Ian Hickson <[email-masked]> wrote:

> > ...

Most real-time social games will have chat embedded or on the page around
them.

> The Web Socket draft is pretty simple.

Yes, because all of the hard stuff is deferred or punted to another layer.

For example, how do I send a GIF over a Web Socket? HTTP and Jabber have
already solved that problem. How do I correlate a response to a message with
the original message?

> of HTTP, which is a remarkably complex protocol.

I did say that the NEW specification is very short and simple.

> no need to adapt an existing library, new code can be easily written to

It's only simple because it doesn't do much yet.

> the client to the server into the same CGI script.

I don't know what you mean.

These services cannot be served with CGI. CGI is built around
request-response. Just as PHP scripts do not in general implement HTTP, I
wouldn't expect the handlers of these things to implement the protocol.

Here's how I expect it would work: there is a service optimized for
long-running TCP connections. Maybe it's written in C or Erlang. There are
PHP scripts (or whatever) for info coming in from web clients. When PHP
wants to tell the universe that "something happens" it puts a message in a
queue. Erlang or C delivers it.

Web developers do not want to become protocol handler developers and they
should not. The protocol should not be designed with the presumption that
they will be protocol developers. In fact: in my more than 10 years of web
development experience, I don't recall ever implementing an HTTP client or
server. What I have done frequently though, is trace the text flowing on the
wire, which is why a text-based protocol is preferable to me than a binary
one.

> With Web Socket, there is only one underlying TCP connection.

So what? How does that make my app simpler? Are you expecting PHP
programmers to write long-running TCP server apps?

> be able to speak WebDAV to their JavaScript running in a browser.)

If a chat transcript or game board is modelled as a resource then AtomPub is
one way you could edit that resource. Nevertheless, I agree that the
extensions built on top of Jabber are more directly appropriate than the
ones built on top of HTTP, due to the histories of those two protocols.

> situation than a large number of simple technologies.

Well we can't really remove the complexity of HTTP from the web or from Web
browsers or Web Servers. It's there already.

>   Jabber connection. Jabber over HTTP uses two TCP connections, as I

I don't understand why it matters whether it is one connection or two.

>   Jabber can be written in a few dozen lines of scripting code.

Most web developers are uninterested in writing protocol-handlers, and
rightly so.

>   of binary data over the same channel.

So? Why is it so important to have a single channel?

>   compatible with the "same-origin" model.

That's just a question of how the connection is set up, not the protocol
running on top of the connection.

>   existing Jabber server, or whatnot).

That's just a question of how the connection is set up, not the protocol
running on top of the connection.

>   model that is desired for an HTML API.

Not sure what that means.

> handling and XML's namespaces are more confusing than desireable.

The web developer is just putting messages into a queue. It's the protocol
handler (like Apache, ejabberd, postfix) that talks the protocol.

Can you point to a single protocol deployed on the Internet that is commonly
implemented and re-implemented and then re-re-implemented in scripts, rather
than being packaged in servers in libraries?

 Paul Prescod

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 03 Apr 2009 17:02:11 -0600

On 4/3/09 3:07 PM, Ian Hickson wrote:
>    Jabber connection. 

Why upgrade from HTTP to XMPP? Just build XMPP support into the browser,
and you can refer to certain resources as XMPP URIs/IRIs and other
resources as HTTP URLs. Or the client (browser) and server can negotiate
an XMPP channel between themselves for more advanced bidirectional
functionality.

> Jabber over HTTP uses two TCP connections, as I 
>    understand it.

That's the HTTP binding for XMPP (BOSH). The native binding is TCP, and
I think that's what browser vendors would support (rather than BOSH,
which is a stop-gap until that happens).

>    Jabber can be written in a few dozen lines of scripting code.

What exactly is the nature of this server-side component? I'd be curious
to know what functionality you expect.

That said, IMHO it should be pretty straightforward to build the
server-side XMPP handling as a shim onto an existing HTTP server,
because there are lots of XMPP servers and libraries available in
multiple languages:

http://xmpp.org/software/servers.shtml

http://xmpp.org/software/libraries.shtml

>    of binary data over the same channel.

XMPP is optimized for streaming XML, in particular small snippets of
XML. It's hard to optimize for lots of small XML snippets *and* for
streaming video in the same protocol. Besides, the IETF has already
defined perfectly good technologies for streaming audio and video, in
the form of RTP. Why re-invent that wheel?

>    compatible with the "same-origin" model.

Could you expand on that?

>    existing Jabber server, or whatnot).

Have you completed a threat analysis for the projected use cases? I
don't see anything about that in draft-hixie-thewebsocketprotocol-07 but
perhaps it is defined elsewhere.

>    model that is desired for an HTML API.

Could you expand on that, too? It's not clear to me what you mean by
opt-in security. XMPP supports TLS for channel encryption and SASL for
authentication. Both of those technologies are well-understood and quite
"pluggable" (different cipher suites for TLS, different mechanisms for
SASL). Aside from channel encryption and authentication, what security
features do you think are desirable?

> solution. 

Solution for what? So far, XMPP has worked quite well for IM, presence,
lightweight middleware, RPC, cloud computing, gaming, and lots of other
use cases. Which use cases do you have in mind?

> Experience with Web authors suggests that XML's draconian error 

Whose experience? Where is this documented? The XMPP developer community
has produced a large volume of software (see links above) and runs a
large network of servers with millions of IM users. So far we have not
experienced any significant interop problems.

> [1] http://www.ietf.org/mail-archive/web/hybi/current/msg00007.html

Quoting therefrom, I'll address what you see as the requirements (though
it's still not clear to me for what applications or use cases these
requirements have been set forth):

> - The ability for Web pages to send arbitrary text to the server
>   from a script in a Web page, in the form of ordered discrete
>   blocks.
>
> - The ability for Web servers to send arbitrary text to a Web page
>   that has established a connection, in the form of ordered discrete
>   blocks.

As far as I can see, something very much like this already exists in
XMPP. I suppose it depends on what you mean by "arbitrary text in the
form of discrete ordered blocks".

> - Text messages should support all of Unicode.

Done in XMPP.

> - The protocol should support running over ports 80 and 443,
>   ideally with the ability to share the port on the server
>   with an HTTP server.

Why? Using a different port for a different protocol seems like the
proper approach.

> - It must be possible to implement a fully-conforming
>   server-side component for this in a few dozen lines of
>   scripting code, in the case where sharing the connection
>   with an HTTP server isn't required.

See above links about the large number of XMPP servers, clients, and
libraries.

> - The must be extensible so that it can be extended to support
>   binary data and structured data in both directions in the
>   future, without breaking deployments from the first generation.

What kind of binary data, and in what volume? We have ways of chunking
binary data over XMPP (without using a side channel), but those methods
are not intended for things like streaming video but instead for smaller
bits of binary such as thumbnails and files.

> - It must not be possible for the mechanism to be used to
>   successfully connect to existing servers and send them
>   arbitrary commands. In particular, SMTP, HTTP, HTTPS, IMAP,
>   POP, and similar protocols must be safe from this.

That's covered in XMPP -- it is a separate protocol run on a separate port.

> - It must be possible to connect to arbitrary remote hosts,
>   but connections to these hosts must only be allowed if the
>   remote host explicitly opts in to allowing that Web page to
>   connect (ideally using the 'origin' convention).

In XMPP we use a "dynamic federation" model, whereby a client connected
to a given server can route messages to other servers or to clients
connected to other servers -- like email, but optimized for real-time
delivery. That dynamic federation model doesn't require the remote
server or client to explicitly opt in to communication from each origin
host or client (although that is one deployment scenario). But perhaps I
misunderstand your idea of the requirements here.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Paul Prescod <[email-masked]>
Date: Fri, 3 Apr 2009 16:02:49 -0700

By the way, you mentioned calendars: calDAV is the web protocol for
manipulating them. It might be a good fit for reverse HTTP.

 Paul Prescod

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 04 Apr 2009 10:03:49 +1100

Ian Hickson wrote:
>> ... Maybe that doesn't matter for train 

Exactly!

If the IETF does come out with a bidirectional HTTP recommendation,
it will be a huge once in a decade opportunity to upgrade the
protocols supported by the network infrastructure.

Websockets sentinel UTF-8 datagrams may well be ideally suited
for a specific type of application, but can we really say that this
will be the only content type that we need to transport asynchronously
server to client for the next decade or two?

What if UTF-16 becomes the preferred encoding? or gzipped UTF-16, or
some as-of-yet-un-invented compression algorithm specially designed
for packets of UTF-16 data holding JSON becomes in common usage.

I'm already working with clients that wish to push images.

> data, where measuring the data length on the server is likely easy.

If web socket supports arbitrary binary data, then why is there
a need for the sentinel mechanism and special status for UTF-8 data?

Why can't there just be one mechanism that does binary content
with an assumption that it will be utf-8 unless otherwise
indicated.

It is already a pain with HTTP that there is both Content-Length
and chunking framing, but at least they are both variations of
a theme.

Note also with websocket binary transfer, I believe it
is a limitation that there cannot be meta data (headers)
associated with a datagram. Thus information like  mime types,
char-sets content-encodings and transfer encodings will need
to be handled with stateful mechanisms based on assumption,
connection and/or previous sent datagrams.

HTTP framing, while not the best framing possible does
contain a lot of derived wisdom and required flexibility.
While there are deficiencies in with HTTP, I'm not sure
that it is the remit of bidirectional HTTP to unilaterally
try to solve them.

Moreover, I don't think that a true solution can be
found by restricting the problem domain (ie utf-8 datagrams)
to one that is more easily solvable.

However, I do see a gap between rHTTP and Websocket.

While rHTTP does build on existing framing (as I advocate),
it is still a request/response paradigm.   What websocket does
support is multiple responses for a single request.

What I'm looking for is the WebSocket API in javascript,
with rHTTP data framing, but the websocket capability to
send multiple content responses to a single request, but
with the rHTTP capability of associating optional meta
data with arbitrary content.

I want the best of both worlds :)

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 04 Apr 2009 10:26:36 +1100

Maciej Stachowiak wrote:

 > needlessly complex here. [...] And headers would be
 > be a common use case here.

While I agree that when you look at HTTP-style headers they
are unnecessarily verbose, redundant and frequently ignored.

But is this actually a problem?  and if so, is it a problem
that should be solved by the bidirectional mechanism in
isolation?

There are many (albeit mostly anecdotal) reasons that I don't
think that HTTP-style framing is really a problem:

  + If datasize was really is an issue, then there is much
    more to be gained by replacing common content formats
    such as XML, JSON, HTML etc.

  + If datasize for really in an issue, then the flexibility
    of HTTP framing has already demonstrated an ability to
    adapt by supporting compression as it evolves (was nothing,
    then compress, now gzip, next ???).  Supporting
    extensible meta-data as HTTP headers do is key to supporting
    such compression.

  + My understanding is that from a network perspective
    there is very little benefit in sending less that an MTU
    of data. 1 byte or 1500 bytes will transit the network
    at the same speed/latency.   So much so, that TCP/IP will
    induce latency in an effort to fill up packets with more
    data

  + Headers are optional. If your content can be interpreted
    with assumed or contextual meta data, then those headers
    don't need to be sent.    If we designed a protocol that
    assumed chunking then a minimal framing could be <10 bytes

  + Headers are optional, yet there does not appear to be
    any significant push to get browsers or servers to reduce
    many of the under utilized headers in normal web browsing.
    How many millions of times per second is
      Accept-Language: en-us,en;q=0.5
    sent and ignored?   If we want to start byte stuffing
    in internet protocols, there there is a lot of low
    hanging fruit we can get before taking out a CRLF that
    would allow us to have optional meta data.

So I do agree that HTTP-style, humanly readable protocols are
inefficient and represent some overhead.   But this has been
an overhead worth paying for FTP, HTTP, SMTP, POP, IMAP, etc.

One day, this style of protocol will be replaced by
something better.  But is this the day? Is bidirectional
HTTP the use-case that necessitates the divergence from
decades of experience?

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 04 Apr 2009 11:16:27 +1100

> an XMPP channel between themselves for more advanced bidirectional

Peter,

if opening arbitrary connections for arbitrary protocols was always an
option, then we would not be having this list.

I wish the XMPP's of the world luck in their efforts to establish
world domination (like HTTP has) and thus to convince all the
firewall/proxy's to support XMPP connections.  I'm not being
sarcastic here, as eventually some protocol will have a sufficiently
compelling story that it will become widely supported.

But isn't it the focus of this list to deal with the situation
where HTTP is the only available protocol.

What I'd like to see from the IETF are recommendations about
how to better use HTTP as it exists today for bidirectional
purposes: eg.

   + should long-polling clients be exempt from the 2 connection
     recommendation of RFC2616? if so why
   + should we be recommending that HTTP clients allow some
     application control of pipeling etc.
   + can we agree a standard header to identify long poll
     requests.

Such recommendations could start having an immediate impact.

Then with a longer term focus, we can consider things such
as websocket, rHTTP or 1xx responses with content  as
incremental improvements to HTTP.    But it will still
be HTTP until such time as others come up with something
better than we all adopt.

cheers

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Sat, 04 Apr 2009 10:28:24 +0200

> HTTP the use-case that necessitates the divergence from
Interestingly there is even a protocol that claims it'd be the perfect 
ground for higher level protocols: BEEP: http://beepcore.org/

- Sylvain

---

From: Paul Prescod <[email-masked]>
Date: Sun, 5 Apr 2009 09:21:38 -0700

Like Roy, I think it's kind of counter-productive that everyone comes to the
list with their own set of requirements and goals.

Do we agree that:

 a) the list should have a wiki?

 b) the requirements should be documented on the wiki?

If so, does anyone have a preference for wiki platform? Choosing a bikeshed
colour is actually marginally better than going in circles.

I'd start the conversation by suggesting Google Code, which combines wiki
with

 * issue tracker (for unanswered questions)
 * version control for more formal specs
 * repository for diagrams or code samples

 Paul Prescod

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sun, 5 Apr 2009 12:35:51 -0700

On Apr 5, 2009, at 9:21 AM, Paul Prescod wrote:

> Like Roy, I think it's kind of counter-productive that everyone  
> comes to the list with their own set of requirements and goals.
>
> Do we agree that:
>
> If so, does anyone have a preference for wiki platform? Choosing a  
> bikeshed colour is actually marginally better than going in circles.
>
> I'd start the conversation by suggesting Google Code, which  
> combines wiki with
>

The IETF already has those things.  As I said before,

    http://trac.tools.ietf.org/bof/trac/wiki/HyBi

is good enough for now.  The rest can happen if a WG is justified.

....Roy

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 06 Apr 2009 22:19:12 -0600

> Such recommendations could start having an immediate impact.

Agreed. Those would be part of a "best practices" I-D.

Perhaps it's time to start writing I-Ds that capture some of the
conversations on this list?

Peter

-- 
Peter Saint-Andre
https://stpeter.im/