Subject: Re: [hybi] WS framing alternative

From: Ian Hickson <[email-masked]>
Date: Sun, 1 Nov 2009 20:08:19 +0000 (UTC)

On Thu, 29 Oct 2009, SM wrote:
> At 21:22 29-10-2009, Ian Hickson wrote:
> > What is the process for proceeding in the IETF when people have 
> > fundamentally different and mutually exclusive opinions?
> 
> The group seeks rough consensus.  I guess that you could think of it as 
> the view that reconciles the various opinions.  That does not require 
> everyone to agree; although that's better.  It's not a hard rule; 
> there's no voting.  It's up to the Working Group Chair to evaluate the 
> consensus.

I guess we'd better get a chair. :-)

On Fri, 30 Oct 2009, Pieter Hintjens wrote:
> 
> Technical designs should not be opinion-driven at all.  There are
> optimal (provably optimal) solutions for any well-defined problem.  If
> two people do not agree on such solutions then either they are solving
> different problems (this is most common), or they are not aiming for
> optimal solutions but instead familiar and/or pre-existing solutions
> (this is also common).  Or they are just incompetent with delusions of
> competence (perhaps most common).

I believe the problem we have here is that we don't agree on what problem 
we are solving, though apparently Greg disagrees with me on that as well.

> A better way to resolve the conflicts of interest that many of us bring 
> to such discussions is to create competition between solutions, i.e. 
> multiple protocols that implementors can choose.  Over time that will 
> drive the solutions towards an optimum.

I agree that that is the best option. I thought that's what we were doing 
with WebSocket, BWTP, and other proposals, but advocates of solutions 
other than WebSocket have not been making much progress as far as I can 
tell. I personally would very much encourage such competition. I think 
it's very healthy and as you say, will drive towards an optimum solution.

On Fri, 30 Oct 2009, SM wrote:
> 
> Having multiple protocols for the same problem space doesn't favor 
> interoperability.  One or more of the protocols will take over which is 
> like "let the market decide".

Indeed, that's another way to put it.

On Sat, 31 Oct 2009, Greg Wilkins wrote:
>
> After initial failures to get my concerns addressed in WS, I did decide 
> to propose alternative solutions. BWTP mk I and BWTP mk II resulted.
> 
> But now it appears that a working group cannot be formed to consider 
> multiple solutions and that it will be free to consider any solution so 
> long as it is websocket.

I strongly object to us forming a working group that only considers one 
solution. So on this at least I strongly agree with Greg that we should 
support other efforts including BWTP. Why would we disallow other 
solutions than WebSocket?

On Fri, 30 Oct 2009, Thomson, Martin wrote:
> 
> Thankfully, we aren't arguing in a vacuum.  I'm happy to cede to the 
> will of a majority on the issue, just as I hope that you are.  And 
> deployment counts for a lot.  Until I feel that I'm arguing without 
> support, I will continue to argue for a text-based protocol with 
> (marginally) greater provision for extension than currently exists.

Then I shall similarly continue to argue for what I perceive to be the 
simpler and safer solution. :-)

On Fri, 30 Oct 2009, Greg Wilkins wrote:
> Ian Hickson wrote:
> > I'm proposing making the protocol so simple that dedicated frameworks 
> > are superfluous.
> 
> You are proposing a revolution in server infrastructure.

I don't think server infrastructure is anywhere near as homogeneous as 
that would suggest. There are some environments I'm familiar with -- both 
small (e.g. on a shared Web host) and large (e.g. in a dedicated 
datacenter) -- where deploying experimental WebSocket servers has not been 
a revolution at all, but has fit right in. I am sure you are right that in 
certain environments it would be a revolution, and I'm sure that there 
will be many cases in between.

The revolution, however, if any, is a one-time cost. We must ensure, 
therefore, that anything we do to defray this one-time cost is not going 
to cause long-term costs to be higher instead.

If you have specific proposals that would help with avoiding a revolution, 
then I'm happy to hear them, but we need to make sure they don't 
complicate the protocol, or make it harder to deploy WebSocket in 
environments where it's easy already, since if we do then we're just 
moving a one-time cost from one subset of today's authors to a permanent 
cost for all authors going forward.

> There is just no way that any of the organizations that I work with are 
> going to allow a server side application developer access to a raw 
> socket exposed on the internet!

WebSocket doesn't require that you do that. You don't need a MIME envelope 
mechanism to abstract out WebSocket so that libraries can be written that 
just receive an already-parsed packet. Nor do you need server-client 
negotiation for the app to inform the library what size packets to expect.

> No matter how much you think it a bad idea, there will be frameworks 
> that implement websocket on the server side - and they will have little 
> or no idea of the type of application that might be run on top of them.

That depends entirely on how the frameworks are written. There's no reason 
that the framework couldn't provide a framework to implement custom 
protocols such that the framework does all the heavy lifting and thus 
_does_ know what the protocol is. 

> >> If a ws client connects to a ws server, and neither knows if the 
> >> other supports some layered protocol - how do they discover that?
> > 
> > If a TCP client connects to a TCP server, and neither knows if the 
> > other supports some layered protocol - how do they discover that?
> > 
> > The answer is the same.
> 
> No it's not.
> 
> You run an *extensible* protocol like HTTP over TCP, so that new 
> protocols can be layered on top without breaking clients, servers or 
> intermediaries.

So where does this leave IMAP, SSH, Telnet, FTP, Gopher, and SMB?

> > That is, I do not see how our goals could ever be met by the same 
> > protocol, if I understand your goals correctly.
> 
> I don't see how allocating 1 bit of 7 unused can render the protocol 
> unsuitable for your goals.
> 
> You are being a tad melodramatic I think.

I really don't think adding a bit would do what you think it would. Unless 
we also complicate the API, there'd be no way to use this bit anyway. I 
really don't understand the scenario in which you think this would help.

My goal is to have the protocol be as simple as possible. Adding things to 
the protocol makes it less simple. That's how it renders the protocol 
unsuitable to my goals.

> >> I'm asking for 1 bit to be allocated to indicate that a frame 
> >> contains mime-encoded data.  That is mostly sufficient for layered 
> >> protocols to be built on.
>
> I would reuse the text and binary framing mechanism.

Why are frame types 0x00 and 0x80 not good enough then?

> but how do you know if the other end supports mime encoding at all. you 
> end up needing to do all sorts of discovery and other complexities.

You do it in the exact same way that you know that the other side supports 
IMAP. Or HTTP. Or SSH. Or SMB. Or Jabber.

You know because _that's why you're connecting_.

Why would you ever connect to a server that supported a protocol, if you 
weren't expecting to speak that protocol?

I really don't understand the use case that leads you to this line of 
argumentation.

> Browser vendors and server vendors might want to collaborate on a 
> multiplexing protocol built on top ws.  They could use the the mime 
> frames to do this - but they would be able to continue to work with 
> browsers and servers that did not implement the extension - as they 
> would ignore meta data they didn't understand that was used to try to 
> negotiate a multi-channel connection.

If a protocol layered on top of WebSocket wanted to support multiplexing, 
it could just go ahead and do that as part of its protocol.

If, on the other hand, there was a need for a protocol that was a layer 
over TCP like WebSocket but which had built-in multiplexing (for whatever 
reason), then you could do that just by inventing a new protocol, and 
using that. WebMultiplexingProtocol, or something. You would want to 
change the whole way the handshaking is done if you did this.

> The extra capabilities could be used by the browser, intermediaries 
> and/or servers.  They need not be exposed to application developers, you 
> could even argue that they shouldn't be exposed to application 
> developers.

The browsers aren't doing anything other than what the scripts want them 
to do. The servers have no reason to support anything the clients aren't 
going to use, which is what the scripts are going to do.

I really, truly do not understand how you expect this feature to be used. 
You need to explain a concrete use case, with actual examples of actual 
user applications and actual on-the-wire traffic explaining to me what you 
expect to have happen, because I have completely failed to understand the 
more abstract use cases you have given so far.

> > I think overgeneralisation is one of the biggest mistakes one can make 
> > in protocol and language design. My experience has led me to be a firm 
> > believer in picking specific problems and addressing them directly, 
> > rather than trying to solve everything at once (especially when one 
> > might not know all the problems one is trying to solve -- a common 
> > problem when discussing extensibility mechanisms).
> 
> You have already generalized the protocol by allowing for other frame 
> types.

How is that generalisation? It's addressing two very specific needs: 
sending text, and sending binary data (files) in v2.

> You obviously want to control how people extend WS and will only 
> allocate new frame byte values to what you think is worthy cause.

The extension mechanism is intended for providing new features in the 
future, specifically sending binary files, and sending binary streams 
(e.g. video). It's not a generic extension mechanism for protocol users.

> If there was a meta data frames, then anybody could extend the protocol 
> without the need to ask permission ( even if it was to support evil 
> monolithic servers and their unholy load balancing companions).

Extending the protocol in this manner is actively harmful, since it 
reduces the likelihood of interoperability between clients and servers.

If you want to implement new features, you do it at the application layer, 
like *all* features.

This is exactly like TCP. People don't extend TCP to add their features. 
We didn't extend TCP to add MIME encoding. We didn't extend TCP to add URI 
addressing. We didn't extend TCP to add caching or intermediary support. 
We layered a protocol on _top_ of TCP, namely HTTP.

The same applies to WebSocket.

> What authority do you plan to manage the allocation of frame types into 
> the future?  IANA?

The same authority as any other changes to the protocol. The IETF and 
WHATWG specification update processes.

> >> So browsers could add on their origin security model for example.
> > 
> > Security is not a feature you bolt on. It's an architecture that has 
> > to be designed in from the beginning.
> 
> RUBBISH!

...

> If this was the case we would have to put the same origin policy into 
> TCP... no wait into IP... no wait into ethernet..

TCP bakes its security in as part of its handshake. It's an intrinsic part 
of the TCP protocol. It wouldn't make sense to have origin security at the 
TCP level, since that's an application concern (relative to TCP).

> You can design a good protocol for the security needs of
> browser that is built on a more general transport protocol.

That is exactly what WebSocket is. It's a protocol for the security needs 
of browsers built on a more general transport protocol called TCP.

> > Could I ask you to more concretely describe your problem so that I 
> > could more concretely (maybe with actual code examples) describe the 
> > alternative solution that I am describing?
> 
> sticky load balancer with SSL offload that works with any ws app.
> 
> multiplexing protocol implemented in browser/server (not in 
> application).
> 
> orderly closing a ws connection implemented in an application 
> independent way.

If you could provide concrete solid examples, that would be very helpful. 
What would you need to do a "sticky load balancer with SSL offload" other 
than what TCP gives you already? What is the use case for the browser 
implementing multiplexing rather than a SharedWorker from script? (I don't 
buy that authors are going to intentionally DDOS themselves, so if they 
need multiplexing, it seems very likely that they will implement it.) Why 
does TCP's socket closing semantics not address your needs?

On Fri, 30 Oct 2009, Jamie Lokier wrote:
> > 
> > sticky load balancer with SSL offload that works with any ws app.
> 
> This is the key thing, I believe.
> 
> WebSockets forces all negotiation of extensions to be done by the 
> application.
>
> This prevents automatic extensions being done below the application, so 
> "works with any WS app" is prevented by the current proposal.
> 
> Here's an example: Just like HTTP, there could be value in an extension 
> which compresses some frames.  If it worked well, that'd be a good thing 
> to do _transparently_ in the browser+server, without awareness by the WS 
> application.

Sure. We could add compression in a future version of WebSocket.

We would do it as follows:

   Client->Server handshake would include a new header, 
   WebSocket-Compression: gzip

   Server-Client handshake would include the same new header, 
   WebSocket-Compression: gzip

   If the client and the server both sent the header, then instead of 
   using 0x00, the client would use 0x81, and send text data gzip 
   compressed, and instead of using 0x80, the client would use 0x82, and 
   send the binary data gzip compressed.

   The server could use either 0x00 or 0x81 to send text frames, and could 
   use either 0x80 or 0x82 to send binary frames; 0x81 and 0x82 would be 
   gzip compressed.

This would be a sensible thing to support in a future version if we decide 
that implementation experience shows compression to be highly desireable. 
(It's not obviously desireable, or I'd have added it already. Reasons why 
we might avoid it are that compression is not always useful for short 
strings, which are probably going to be most common; and that it's 
possible for compression to be introduced at the TLS level, thus making 
the protocol's use of compression unnecessary.)

> But you can't provide a transparent compression extension, because the 
> browser has no way to signal to the server "I handle this extension".

Sure it can. The handshake is completely extensible.

> Given that the only thing blocking this is the client's inability to 
> signal the server that it supports an extension, and that is so easy and 
> natural to do by allowing clients to add headers to the initial request, 
> I think that should be at least seriously considered.  Not dismissed 
> summarily.

We absolutely would not want clients and servers doing compression without 
a spec defining how it works. Said spec can allow sending headers to opt- 
in to compression, as well as defining which frames to use. It would be 
actively harmful for the WebSocket spec _without_ compression to say that 
clients could add arbitrary headers: interoperability would be a nightmare 
where implementations have to reverse-engineer each other to work out what 
they need to support.

> > multiplexing protocol implemented in browser/server (not in application).
> 
> That's similar to the compression example above.

If we want multiplexing that is transparent to scripts (as opposed to 
layered on top of WebSocket, which seems to make more sense to me), then 
we need a whole new protocol; WebSocket just isn't cut out to do built-in 
multiplexing. (It handles higher-level multiplexing fine.) We would quite 
easily be able to do this again by having a handshake that looks like 
WebSocket v1, but which the server and client can recognise as being just 
a dance to opt into v2.

For example, we could have the handshake be:

        GET /foo/bar HTTP/1.1
        Upgrade: WebSocket
        Connection: Upgrade
        Host: example.com
        Origin: http://example.com
        WebSocket-Upgrade: Multiplexing

Current-version servers would treat that as a regular v1 connection, and 
wouldn't opt in, but v2 servers could response with:

        HTTP/1.1 101 Web Socket Protocol Handshake
        Upgrade: WebSocket
        Connection: Upgrade
        WebSocket-Origin: http://example.com
        WebSocket-Location: ws://example.com
        WebSocket-Upgrade: Multiplexing

...where the Location is intentionally a mismatch so that new servers can 
be written that old UAs automatically drop connections with, and then the 
client could jump straight into our new multiplexing version of this, 
e.g.:

        Open Channel for /foo/bar

...or whatever the protocol is.

It's quite easy to make extensions to WebSocket that are backwards- and 
forwards-compatible. It just needs a revision of the protocol spec to 
define the new feature so that all UAs can be updated to support it.

We should, however, keep such features to a minimum, so as to keep the 
protocol from ballooning into something only experts can implement.

> > orderly closing a ws connection implemented in an application 
> > independent way.
> 
> Orderly close is more important than people realise.
> 
> A major problem with HTTP pipelining is you can't use it for
> non-idempotent requests because the server may close the connection
> any time, without implying an error condition.  That's one of the many
> reasons HTTP pipelining often cannot be used, and that makes HTTP in
> general slower and less network efficient than it could be.
> 
> The same applies to other request-response based protocols, such as
> many of those expected to run over WebSocket.

WebSocket handles this in the same way TCP does: application-layer 
acknowledgements. It guarantees ordering, but nothing beyond that. 
(There's not much more we _can_ do, in practice, given the realities of 
non-instantaneous communication and the unreliability of hardware links.)

> Another problem with closing is that thing not mentioned in the HTTP 
> RFCs, but essential: lingering half-closed sockets.  If a server sends a 
> response followed by closing the socket fully, some clients sometimes 
> don't receive the whole response because of the way TCP and pipelined 
> requests clash.
> 
> Both of these are why orderly close is important - more important than 
> it appears.  Because it makes it safe to do other things.
> 
> I expect "pipelining" in the meaning used by HTTP, that is sending 
> multiple messages before getting a response to each one, to be quite 
> prevalant with protocols running over WebSocket.
> 
> That makes orderly close important for WebSocket, to avoid fragile 
> applications.  Hopefully I've illustrated the kinds of problems to 
> expect.  Unfortunately it's all a bit arcane.
> 
> There's nothing wrong with doing it in the application.  But I don't 
> have high expectation of it being done correctly in those "we don't need 
> no steenkin' framework" 100 line Perl scripts by inexperienced authors.

Indeed. Do you have a suggestion for improving this at the protocol level 
without dramatically increasing the complexity of the protocol? i.e. is 
there something we can add to the spec that _would_ get this done 
correctly in the framework-less case you mention?

(There's no point adding something if it only helps the experienced 
programmers, since they can already solve it with a simple protocol.)

On Fri, 30 Oct 2009, Jamie Lokier wrote:
> 
> Inexperienced authors, especially those writing 100 lines of Perl, will 
> send ISO-8859-1 or other text which occasionally contains 0xff bytes in 
> the middle.

That's possible, but I think that will be rarer than one might expect, 
because naively copying the incoming bytes and then later outputting them 
verbatim will mean UTF-8 is used. Problems could occur when an author 
takes text from another source (e.g. a database) and outputs it in the 
wrong encoding. Is this likely to happen more or less often than not? If 
it's more often, then regardless of the length vs sentinel issue, the 
encoding problems are likely to be visible relatively quickly, which would 
take care of the problem. If we used a length marker in that case, though, 
the people correctly using UTF-8 would be more at risk of buffer overruns 
in incoming text, which would be harder to spot and would be a security 
error, which is worse than the aforementioned data corruption error.

> Even experience authors will make that mistake sometimes.  What do you
> think will happen when someone does something like this:
> 
>     - Read list of filenames in a directory.  They're UTF-8 (assumed),
>       or the author is unfamiliar with character encodings and everything
>       works fine in their ASCII development environment.
> 
>     - Concatenate the list with newlines, as people do.
> 
>     - Send the result as a frame.
> 
> Or this:
> 
>     - Read lines from a text file, which is in UTF-8 encoding.
> 
>     - Send each line as a frame.
> 
>     - (Oops, one of the text files you gave me had an 0xff byte in it.)
> 
> Result: Because of assumptions, 0xff bytes will be sent occasionally
> in the middle of a frame.  Everything afterwards will break, but it'll
> be rare enough that the author doesn't notice.  For the same reason
> you've explained authors get lengths wrong.

The difference is that this error will be immediately obvious (the 
connection will basically stop working, since most "frames" past this 
point would be dropped). The error the current spec is avoiding is much 
more insidious (buffer overrun risk on the server).

On Sat, 31 Oct 2009, Greg Wilkins wrote:
> 
> The sentinel approach also opens an easy attack vector.  If user data is 
> sent, then tricking a poor implementation into sending an OxFF will 
> allow packet insertion.

Yes, if a WebSocket server is taking data in one encoding (e.g. 1252) and 
outputting it verbatim on the wire, then one could inject any number of 
packets. I've added a paragraph about this to the Security Considerations 
section.

> This is similar to CRLFCRLF insertion attacks that can happen if user 
> data is set unfiltered into a HTTP header and/or cookie.

Indeed.

> length framing avoids this vulnerability.

Indeed.

> Note also that my proposal for a meta-data bit would allow headers to be 
> sent in one length frame and data in another, so the CRLFCRLF sentinel 
> would not be needed and that vulnerability would also be avoided.

Sure, but you would instead have the character/byte length discrepancy 
problem, which can become a code injection error.

On Fri, 30 Oct 2009, Jason Duell wrote:
>
> From my reading of the last few days' posts, it does seem like focusing 
> on use cases might be the best way to move the conversation forward.  
> The more concrete the use cases, the better.

Agreed.

On Fri, 30 Oct 2009, Ted Goddard wrote:
> 
> Without multiplexing, I expect we will see each Ajax component in each 
> page opening its own socket, and multiple pages open to the same server 
> is very common (users often forget that they already have a window open 
> in their browser) so a hundred connections to a single server would not 
> be unexpected.

This is only the case if the person hosting the widget doesn't mind this 
setup. If the server wants to reduce the number of connections to one per 
user, then SharedWorker support allows the client-side to share a single 
connection. A server doesn't have to accept connections from origins that 
are ignoring its policy -- just like it can drop connections from people 
trying to perform DOS attacks.

On Sat, 31 Oct 2009, Jamie Lokier wrote:
>
> (I'll add that deliberately opening multiple tabs to the same server is 
> quite common too.  Like, when you want to read 10 different articles 
> from the same news/blog site, many people open 10 tabs by right-clicking 
> from the headline page, let the pages load in the background, and work 
> their way through them)
> 
> Sharing WebSockets between components on different pages is difficult 
> for the simple reason that browser tabs are quite independent.  (But I 
> haven't seen anyone advocate that they should use separate HTTP 
> connections...)

SharedWorkers really let you share sockets pretty easily, assuming you 
have the server-side support for multiplexing over WebSocket.

> On a single page, Ajax component authors _could_ cooperate so their 
> components share connections, but how likely is that?

That depends on the site. How likely is it that they will cooperate to not 
overwrite each other's cookies, to not overwrite each other's global 
variables, to not use too much CPU, to not overload the server by not 
doing too much XHR, etc?

If we are talking about a server that is also serving the widgets, then 
that's a policy issue. If a site wants to DOS itself, there's plenty of 
ways to do it, regardless of what we do.

If we are talking about multiple widgets on a third-party site, then this 
is what Terms of Service are for. You require that the third-party 
components use a particular library, and you block origins that don't.

> Even with good intentions, it's technically *quite hard* to share 
> connections between independent components on a single page.

Nah, SharedWorkers makes it pretty easy.

> That's because one component sending a large message can block progress 
> of all other components on the page, if they share.

That's entirely up to the designer of the multiplexing part of the 
protocol. There's no reason the substrate couldn't define a maximum 
message size, such that larger messages have to be split up.

> And one component taking a long time to accept messages can block all 
> the others even if it's a short message.

That's the case even with multiple WebSocket connections, as far as I can 
tell.

> That is why I advocate a slightly more advanced multiplexing, akin to 
> interleaved HTTP chunks, where no component can block the progress of 
> any others, and chunks can be interleaved in a fair, orderly and 
> network-efficient manner.

Exactly how the multiplexing works is up to the author, but certainly I 
agree that a system like that is amongst the best ones. The great thing 
about leaving this to the application layer is that the application can 
pick the most appropriate scheme for the protocol being used.

On Sat, 31 Oct 2009, Greg Wilkins wrote:
> Jason Duell wrote:
> >
> > Is there any particular reason one couldn't do sticky load-balancing 
> > just by remembering the TCP IP/port four-tuple for a given web socket?
> 
> IP stickiness is a reasonable solution when cookies are not available, 
> but it is far from perfect.  Every user from AOL will look to be coming 
> from just a few IP addresses - which can and do frequently change for 
> any given user.

You can add cookies to a WebSocket connection; you just have to set it 
from HTTP first. Then, they will be included in the handshake.

You can also just add cookies to your application-layer protocol, so that 
the first message you send declares the machine you want to be connected 
to, if you have one, or that you want the server to give you a cookie.

> The primary problem that I would like to see addressed is the issue of 
> idle shutdown of connections.  Currently the client, server or 
> intermediary may decide a connection is idle too long and close it.  
> But there are no common timeouts, no way to discover these timeouts and 
> not way to negotiate these timeouts.

This is all stuff that can be added at the application layer relatively 
easily.

> Thus clients will often select and idle connection and try to use it 
> only to discover that it has been closed.  Lack of orderly shutdown 
> means that the client has no idea if the messages sent have been 
> delivered or not.

Even with orderly shutdown, you'll still have cases of the connection 
being killed unexpectedly.

> Websocket will make this a server problem as well, because servers can 
> now try to use a connection that has been idle closed by an intermediary 
> or client.

Only non-conforming clients would do this, but even without such bugs, 
connections might still be dropped by a network error.

> Having an orderly close will allow implementations to distinguish 
> between a failure (for which there is nothing much that can be done 
> short of a reliable protocol on top) and a routine idle shutdown, which 
> would now be able to be handled without any message loss.

Indeed. Protocols should add message acknowledgement on top of WebSocket 
to handle this if they need it.

Note that not all cases need this; sometimes the data is transient and 
missed messages are not a big deal. For example, a stock ticker system 
might not care about missed updates.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Sylvain Hellegouarch <[email-masked]>
Date: Sun, 01 Nov 2009 23:16:20 +0100

>> The more concrete the use cases, the better.

Well this was the idea already many months ago if I recall correctly :)

I guess concrete use cases are just the various protocols you mentioned 
already: IMAP, XMPP, FTP,etc. Those are not only concrete but well 
specified and implemented. Was there real interests to actually mock 
them up using a WebSocket transport so far?

- Sylvain

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 02 Nov 2009 12:19:42 +1100

> protocol from ballooning into something only experts can implement.

So I think this is our FUNDAMENTAL point of disagreement.

You are arguing about making ws not user extensible
because you believe that it is easier to rev the spec
and update "all UAs".     It is an impossible task
to upgrade "all UAs" and even if it was, you are looking
at 3 to 5 year cycles to get any significant market
penetration.

How is a web ecosystem that is bristling with thousands of open
and/or proprietary extensions is going to be happy with a
protocol that can only be extended by new versions of the base
document - how is that layered?

You honestly believe that a process that can't even agree
on length-vs-sentinel framing is going to be able to centrally
coordinate all extensions to websocket in a timely manner?

> we are solving, though apparently Greg disagrees with me on that as well.

I disagree with you that I disagree with you on this :)

You and I are truly trying to solve different problems.

I believe our major differences are on requirements - mostly on
the requirement to allow transparent extension.

I think we also disagree on what extensions are necessary,
which your appear to believe should be centrally coordinated.
I believe that extensions need to be allowed to be created
and evolve in a Darwinian ecosystem outside of a committee system.

We also significantly disagree on a lot of technical detail
of how to do framing etc.   But that is significantly less important
and we really should try to refrain from debating that blow by
blow until the broad requirements are roughly agreed.

>> What I'm currently trying to compromise on is:
>>
>>
>>    |\/\/\/\/\/\/\/\/\/\/\/\/\/\/|
>>    | Application-level protocol | user application features
>>    +----------------------------+
>>    | Optional transport features| Multiplexing, compression, etc.
>>    +----------------------------+
>>    | Websocket protocol         | Two way comms, browser security model
>>    +----------------------------+
>>    | TCP                        | reliability, connection wrapper
>>    +----------------------------+
>>    | IP                         | packet abstraction, routing
>>    +----------------------------+
>>    | Ethernet                   | link-level routing
>>    |/\/\/\/\/\/\/\/\/\/\/\/\/\/\|
> 
> That's what I'm advocating. I guess our disagreement is that I think the 
> optional transport features should be in a separate spec layered on top of 
> WebSocket, just like BEEP and XMPP aren't in the raw TCP spec.

I agree totally that optional transport features layered on top of
WS should be specified in a separate spec.

( I could argue for some features (eg multi-plexing) to be non
optional - but that's a different debate).

What I'd like to see in the base ws spec are the ground rules for
most extensions - ie a meta data mechanism, so that extension can be done
easily, with negotiation and without breaking non conforming
implementations.

If there was a standard meta data mechanism in WS, then I would rewrite
BWTP protosal to be layered on top of WS.  It currently uses text based
length encoded frames - but it could easily be transported over WS
framing. It could become the basis for mutliplexing and fragmentation
extensions - but only if standardized meta data is supported.

> multiplexing. (It handles higher-level multiplexing fine.)

Why can't a multiplexing protocol be both transparent to scripts
AND layered on top of WebSocket protocol?.

If I need a who new protocol for multiplexing, do I need
a whole new protocol for compression?    Then what do I do
if I want compression AND multiplexing - another new protocol?

> (e.g. video). It's not a generic extension mechanism for protocol users.

OK - the type frame byte might not be a generic extension
mechanism for protocol users.   But that does not mean there
is not a need for one.

This is why I'm advocating for a standard meta data
transport (but not standard meta data).  Something along
the lines  of the base protocol allocating:

  0x01 be a UTF-8 frame of mime headers
  0x81 be a binary frame of mime encoded content (headers CRLFCRLF body)

These would be know by all implementations.  Simple implementations
that don't want to know about extensions can ignore them (as they
have to do now anyway).   Implementations that have extensions will
know the content-type of meta data in these frames and can inspect looking
for extension that they support.

The actual semantics of these frames would not be defined in the
base websocket protocol and would be a matter for the extensions.

> Why are frame types 0x00 and 0x80 not good enough then?

Because they are already allocated for data transport
and thus are unsuitable for meta-data.

If we put extensions in those frames, we will break
implementations that don't support them.

If we want to use multiple extensions (eg multiplexing,
compression and SSL offload), then working out
who an 0x00 message is for is really difficult because
we don't know what the content-type(s) might be so
we can't introspect it.

> [...]

The problem with what you say are the "we could add", "future version"
and "if we decide" parts.

Netscape didn't have to wait for any "we" to formulate HTTP/1.1 so that
persistent connections could be supported.   They came up with HTTP/1.0
keep-alive as a defacto standard that worked because HTTP supports
arbitrary meta-data that can be ignored.

I don't want to have to wait for whoever "we" is, to deign to consider
my particular pain point.     It might not be globally applicable.
It might be very niche and apply only to a small community of users.

But they should not be forced to give up on interoperability
to implement their own extensions.

> the protocol's use of compression unnecessary.)

It might not be desirable to you... but your mileage may vary!

Maybe my messages are not all that short.
Maybe my messages are short but really easy to compress.
Maybe I have really fast CPUs and a really slow network?
Maybe I have different sized messages and want to compress
some but not others?

one-size-fits-all protocols are not going to work!

I really really really REALLY don't want to have to
work on an internet that everytime i want to do something
new, innovative or just plain pragmatic for private reasons,
that I have to confronted with endless "why do you want to

> going to use, which is what the scripts are going to do.

Because the protocol might not be a server protocol, it might
be an intermediary protocol that is setup by your own local
infrastructure provider, by your ISP or by the deployers of
the server side (who may differ from the developers of the
server side).

Load balancing, compression, aggregation, SSL offload,
authentication etc. are all things that can be injected
between a client and server without the application authors
being involved.

> than what TCP gives you already?

SSL offload removed CPU/network load from application servers.

But the offloader still needs to communicate meta data about
the SSL session to the server.

During a connection, the server might need to request that the
offloader  upgrade the connection (get a new key or request
client certificates).

Some server code will know about the offloaders existence
and extension protocols, but it also needs to be able to
work iwth code that does not and just thinks it is getting
a normal none-SSL connection.

Sticky load balancing is very important for mid-sized
deployments where an organization has a handful of stateful
nodes supplying a service (larger deployments tend to be
forced to be truly stateless and smaller deployments have
only 1 node).

If a client has multiple connections or renews a connection,
it is highly desirable (sometimes necessary) that all the
connections terminate on the same node.    Strange gateways
(eg AOL) mean that 2 connections from the same client can
be seen to come from different IPs   (note that
multiplexing would significantly reduce the need for
sticky load balancing!).

Sticky load balancing needs to be able to set state on
the client - typically cookies.   To facilitate load
migration and cluster maintenance, an ongoing meta-data
conversation is necessary between balancer and client
to manipulate this state (eg send a meta data frame
saying that the server has migrated your state to
a different node and please update the cookie (or whatever)
accordingly).

> What is the use case for the browser
> implementing multiplexing rather than a SharedWorker from script?

Because different frames might be implemented by different
developers using different frameworks and libraries, with out
any knowledge of the each other.  Thus they can't cooperate
on a shared worker.

Because code from multiple origins might accessing the
same server and security will be hard to enforce in
script in a shared worker.

Because it is not in the applications interest to share
connections - thus they have very little interest in doing so.

It is in the server/network vendors interest that multiplexing
is used and it is there job to convince the browser vendors
to implement that below the level of the application.

I'd like to see this built into base websocket protocol,
but if a reasonable extension path is provided, then I think
a layered multiplexing specification would have some chance
of successfully attracting support from browser vendors.

> multiplexing. (It handles higher-level multiplexing fine.)

Why can't a multiplexing protocol be both transparent to scripts
AND layered on top of WebSocket protocol?.

If I need a who new protocol for multiplexing, do I need
a whole new protocol for compression?    Then what do I do
if I want compression AND multiplexing - another new protocol?

> more abstract use cases you have given so far.

All I'm doing is pointing out use-cases that exist today for
HTTP.    I'm not the expert on exactly what they require or
what meta-data they exchange - but I do have experience with
working with many of them, so I at least know of their existence
and what is involved to interoperate.

I have listed many many use-cases that exist today that
exchange meta data between clients, servers and intermediaries
in a mostly transparent fashion.

If you don't understand these existing use-cases then perhaps
you should familiarize yourself with these common elements
of internet infrastructure before finalizing your protocol?

> Sure it can. The handshake is completely extensible.

Is it?
It allows extra headers for authentication information or cookies
to be sent - but under very strict ordering conditions and with
limitations on standard HTTP header format.    This is hardly saying
that arbitrary headers for arbitrary extensions might be sent.

I do note that set-cookie is supported in the handshake response
(despite your earlier comment that this would be supported under no
circumstances). This is a start.

But other headers are ignored.    I guess this would allow a browser
to ignore the ignore and do an extension, but it would not allow
an application layer to do an extension (as you are advocating).

Even if the handshake is extensible - problems persist.   Say
I want to write an extension to send Content-Type meta data in
a mime-header frame before every data frame.  I could self
allocate 0x7f for that frame.    But this might clash with
somebody elses self allocated frame type, or break an intermediary
that had ignored my extra headers. etc. etc.

> clients could add arbitrary headers

Now you are contradicting yourself.    How is the header
"completely extensible" if you think it harmful if
"clients could add arbitrary headers" ?

> interoperability would be a nightmare 

who needs to reverse-engineer what?  Whose talking about secret
non-interoperable extensions?

The experience with HTTP is that people do know how to add optional
features that can be harmlessly ignored for backward compatibility.

Experience with attempts to centrally control innovations is that
they fail.

cheers

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 2 Nov 2009 00:07:02 -0800

> arbitrary meta-data that can be ignored.

FTR, Netscape had very little to do with formulating keep-alive.

There were three research projects that showed potential value
in persistent connections.  There was also some list discussion
on how best to do that, culminating in at least three different
proposals that I presented at the IETF's HTTP BOF in December 1994.
None of those proposals were accepted.  The keep-alive header
came out of a compromise that Henrik and I came up with after the
BOF session, in response to the BOF comments, which I later
wrote-up as a set of notes and distributed to a few of the
primary HTTP implementors at the time, including the folks at
Netscape who had been more interested in a batch method (MGET)
but were eventually convinced to implement keep-alive along
the same lines as Apache httpd and W3C libwww.  After we had
demonstrated interoperability, the mechanism was added to a
late draft of HTTP/1.0 and eventually replaced by HTTP/1.1's
persistent connections (triggered by the version change).

...

Although I agree with most of Greg's design points (most of which
were included in the design of waka back in 2001), I still
see no reason for the IETF to standardize anything in the realm
of a hybi protocol at this time.  Simply put, there is not enough
agreement yet on what the goals should be, let alone the standard
way everyone should implement them.

I look at Websockets and what I see is a very limited mechanism to
tunnel arbitrary protocols through an HTTP-established connection,
in spite of the fact that arbitrary protocols are neither secure
nor architecturally scalable on the open Internet.  This is just
a slight twist on the old schemes of using CORBA or Java RMI that
Netscape tried to introduce and ultimately failed, and the DCOM
over HTTP crap that Microsoft tried to introduce and ultimately
failed.  Those schemes failed because they were architecturally
and socially incompatible with organizational trust boundaries
and the anarchic scalability required on the Web, not because the
protocols lacked some secret sauce.

I see no point in trying to convince Ian that he is on the
wrong track.  As long as Websockets isn't wedged into a more
established spec (like HTML), then it can succeed or fail on
its own terms.  The only risk to me is that Apache may have to
introduce code specifically blocking such requests being passed
through the server's extensibility interfaces (API, CGI, etc.),
because there is no effective difference between Ian's requirements
and an extended denial-of-service attack.

Likewise, I look at BWTP and I see it trying to adopt all of HTTP's
weaknesses regarding syntax and none of its strengths regarding
separation of concerns and simplicity.  I don't know if that is
fatal to the idea or not, since AFAICT the only use cases for hybi
that are not unacceptable, either socially or in terms of scale,
are already adequately covered by SMTP.

My advice is: stop trying to convince each other that your ideals,
use cases, and design theories are even remotely similar.  They
are not even comparable.  They do not belong in the same WG.
Neither one is suitable for standardization at this time.
They should demonstrate their viability first without being
designed in committee, just like all of the other successful
IETF protocols.

I think it is a mistake for hybi to continue as if there were
something to standardize here.  Waka could do all of this stuff
(in addition to replacing normal HTTP communication), but
I wouldn't dream of bringing waka to an IETF working group
without a half dozen working implementations and a very firm
grasp of why it should be standardized.  It is just such a
ridiculous waste of time to try to get consensus amongst
people who aren't implementing the same type of system,
let alone the same exact protocol.

A protocol is ready to be standardized when there is
no longer any need to explain what it is used for and why
people should want to implement it.  Standardization is only
needed when independent implementations already exist but
differ in their observed behavior.

....Roy

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 02 Nov 2009 20:34:22 +1100

Roy T. Fielding wrote:

> My advice is: stop trying to convince each other ...

Advice taken.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 02 Nov 2009 20:08:46 +0900

Hello Roy,

You say that waka was already capable of all of this back in 2001. Can 
you give us some pointers?

What I found is 
http://gbiv.com/protocols/waka/200211_fielding_apachecon.ppt, which is 
quite interesting, and says (p. 19):

waka
* Has not yet been fully specified
* Has not yet been implemented
* Has not yet been deployed

* Will eventually be proposed as ASF project
* Will eventually be submitted to IETF
* Will have its progress tracked:
   http://www.apache.org/~fielding/waka/

The later currently gives a 404.

Regards,   Martin.

On 2009/11/02 17:07, Roy T. Fielding wrote:
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 2 Nov 2009 17:02:12 +0100

> Neither one is suitable for standardization at this time.

Nicely put.

Premature standardization is just cause for argument and BWTP vs.
WebSocket is a false choice based (afaics from this long and
interesting thread) on the fear that there is space for only one of
these in the IETF stack.

It's implementations that drive adoption, and standardization comes
after implementation and adoption, not before.  (At least in this
non-consortium process.)

We are arriving at consensus, which is that both WebSocket and BWTP
(and other designs) are important tools for understanding the problem
space.  One does not choose between tools, one collects as many as
possible.

-Pieter

---

From: "Roy T. Fielding" <[email-masked]>
Date: Mon, 2 Nov 2009 10:26:59 -0800

On Nov 2, 2009, at 3:08 AM, Martin J. Dürst wrote:

> Hello Roy,
>
> You say that waka was already capable of all of this back in 2001.  
> Can you give us some pointers?

I said that the design included all of that -- length-delimited
framing, tokenized constants, bidirectional requests, and
interleaved messages (channel-based multiplexing).

> What I found is http://gbiv.com/protocols/waka/200211_fielding_apachecon.ppt 
> , which is quite interesting, and says (p. 19):
>
> waka

Yep, all of those are still true and equally unfinished.

>  http://www.apache.org/~fielding/waka/

Er, completely forgot about that one.

....Roy

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 03 Nov 2009 08:39:33 +1100

Pieter Hintjens wrote:

> space.  

Pieter,

I really like the phrase "a tool for understanding the design space",
and believe that it entirely applies to BWTP.

However, I don't think there is a consensus that the same applies
to websocket.   Unless understanding tools include something that
get's deployed in the wild, causes all sorts of problems and then
makes people realize the errors once it is too late to take it back.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Tue, 03 Nov 2009 08:52:32 +1100

Roy et al,

about the one thing that we do have a rough consensus on here is
that there is a need for framing.

The Websocket protocol is little more than framing - which is why
there is a belief that it could/should form the basis of other
use-cases discussed here that also need framing.
But unfortunately it is rather inflexible and special case.

BWTP has more flexible framing - but is text based and has the
same problems as HTTP with regards to CRLF sentinels and verbose
tokens.

It would be really instructive if you could make more of waka's
design available....  if only the framing and meta data
parts.

cheers

---

From: Pieter Hintjens <[email-masked]>
Date: Tue, 3 Nov 2009 12:02:27 +0100

> makes people realize the errors once it is too late to take it back.

Yes, that's a common enough failure pattern.  We have no control over
how others view and develop their protocols and anyone trying to
standardize at the same time as they experiment and learn will just
end up paying more for their lesson.

I'd reinforce Roy's assertion that what counts is implementation and
the quality of the specification, and that standardization comes when
the experimentation and learning is over.

The core issue here is, IMO, status and territory and there is an
apparent conflict over positioning in the IETF of a single protocol to
accomplish something... unstated... but which somehow makes HTTP work
better.  BWTP solves one set of problems: asynchronous high-volume
delivery as an upgrade of HTTP.  WebSocket solves another set, which
I'm not clear on but which clearly are not reconcilable.

We can set aside the status/territory discussions and just consider
how to ensure that BWTP works well.  WebSocket is the problem of its
designers, who seem to have strong notions of what they need.

Which means a clear spec (that we have), a cheap way to access it and
improve it (which we have) and a small but growing community of
developers and implementors.  We can assume that a well-written and
widely-used spec will get formal adoption when that makes sense.

I'd like to see this WebSocket vs. BWTP discussion end, and the energy
going into prototype code, use case documentation, and community.

-Pieter

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 04 Nov 2009 10:10:19 +1100

> going into prototype code, use case documentation, and community.

Definitely.

But just for the record (and in regard to your point about "alternative" in
other thread), I don't think I've ever thought of this as "WebSocket vs BWTP"

BWTP is currently a thought experiment with lots of possible out comes:

 a) ideas from BWTP (or elsewhere) get incorporated into websocket.
 b) BWTP (or something like it) get's built on top of websocket
 c) BWTP (or something like it) evolves into a real protocol solving
    different problems than websocket.
 d) Some other effort proves to be a better vehicle for a)-c) (maybe waka?)
 e) There is no significant interest and BWTP fades away.

I guess I've been hammering on a) a bit too much - my apologies.

cheers

---

From: Pieter Hintjens <[email-masked]>
Date: Wed, 4 Nov 2009 11:58:14 +0100

On Wed, Nov 4, 2009 at 12:10 AM, Greg Wilkins <[email-masked]> wrote:

> BWTP is currently a thought experiment with lots of possible outcomes...

Let's assume the desirable outcome is (c), a real protocol usefully
solving real problems.

What I've missed for both BWTP and for WebSocket are clear statements
of those problems.  Undocumented assumptions.  Greg, let's collect
this for BWTP to start with: a solid list of the problems we are
aiming to solve.  Then, we can examine alternative solutions, and that
gives us the argumentation to justify BWTP, and for implementors to
understand the cost/benefit tradeoffs that went into every decision.

If it's any help this is how I designed AMQP,
http://www.openamq.org/doc:amqp-background.  That document never went
into the final spec but it was essential to position the protocol
properly, and to defend design decisions afterwards.

(Sadly with AMQP we lacked any kind of diverse community and so I made
some _terrible_ design decisions, such as encoding low-volume commands
in binary, but at least they were coherently argued terrible design
decisions.)

Cheers,
Pieter

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 07 Nov 2009 08:18:19 +0900

> understand the cost/benefit tradeoffs that went into every decision.

I started penning some BWTP requirements, and it grew a little large
and became somewhat of a manifesto.

So I'll post it in it's own thread.

cheers