Subject: [hybi] Frame size

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 16 Apr 2010 15:22:43 +0800

Responding to specific concerns on framing seems not especially productive.

Proposal:

 - Frame size is indicated up front.
 - Frames are binary.
 - Frame size be strictly limited (2 octets should suffice).
 - Sub-protocol needed if messages are larger than the max frame.

Justification:

Most use cases are for low latency bi-directional communications.  These don't need large messages.

HTTP is (much) better suited to file transfers in the server->client direction and despite some historical shortcomings in the other direction, still arguably superior (chunked encoding, etc...).

Limiting message size helps keep simple implementations simple; but isn't a burden on implementations that have more complex requirements.

Binary frames isolate user-space (the payload) from the framing layer.  We had this discussion in the meeting (c.f. Pete Resnick's comments).  Thus, 
 - binary frames are easier to process (intermediaries don't have to inspect every octet)
 - kinder on implementations (framing component does not depend on UTF-8 component)
 - isolates framing layer from the bugs of higher layers

Shortcomings:

Greg did have some use cases where larger messages were desired.  A trivial sub-protocol that consists of a continuation marker would solve this.

IF used for UTF-8 AND implementer counts characters instead of octets THEN framing doesn't work.  (One solution to this problem is to start a frame with a known sequence of octets, so that this can be detected.  Another "solution" is to not worry about it.)

--Martin

---

From: Justin Erenkrantz <[email-masked]>
Date: Fri, 16 Apr 2010 00:43:25 -0700

>  - Frame size be strictly limited (2 octets should suffice).

To be clear, this would be 16-bits for the size or a max of 65,535
bytes in a single frame, correct?

(I'm assuming that you don't mean "2 octets" as being that's all you
can send in one frame.  *grin*)

>  - Sub-protocol needed if messages are larger than the max frame.

One other note from Ireland is that when httpd and tomcat devs talked
about small frame sizes, most people thought that frame sizes in this
range would be artificially limiting - a few folks talked about Comet
apps they've seen transferring large XML files.  Silly perhaps, but
that's what the app writers did...trying to get them to change is
pointless.

IMO, going to 32 bits or even 64 bits simply isn't that much of a
difference - and may actually be easier if it means removing a
sub-protocol or extension - the phrase "premature optimization" was
bandied about in keeping the frame size arbitrarily low.  =)

Anyway, if we flesh out a way up-front to do "jumbo" frames (via an
extension), I think that'd be fine, but I'd be leery of throwing too
much into extensions.

Other than that caveat, I'm okay with what you propose.

...snip...
> IF used for UTF-8 AND implementer counts characters instead of octets THEN framing doesn't work.  (One solution to this problem is to start a frame with a known sequence of octets, so that this can…

+1 to "not worry about it".  -- justin

---

From: Roberto Peon <[email-masked]>
Date: Fri, 16 Apr 2010 00:53:41 -0700

> >  - Frame size be strictly limited (2 octets should suffice).

yes, yes, maybe.
If this happened, I'd think that this was a much better protocol  :)
I don't think you waste much by using 32 bits for framesize, but *shrug*

One other thing which exists in SPDY that would make me happy if existed in
this protocol:

version ID (so that any proxy, etc, can ignore drop, etc frames)

-=R

> > IF used for UTF-8 AND implementer counts characters instead of octets
> THEN framing doesn't work.  (One solution to this problem is to start a
> frame with a known sequence of octets, so that this can be detected.
>  Another "solution" is to not worry about it.)
>
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 16 Apr 2010 04:34:43 -0700

> extension), I think that'd be fine, but I'd be leery of throwing too
Not exactly, because if you want large frames, you need chunking. 
Changing the frame size from 2 octets to 4 octets is not sufficient to 
handle large frames.

The two viable options to handle large frames are putting chunking into 
the WebSockets protocol (like HTTP), or punting to the sub-protocol as 
Martin suggested.

-- Scott

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 16 Apr 2010 14:38:09 +0200

I thought one of the key arguments for a limited frame size,
is that we could avoid the need to consider end points that
could not store an entire frame in a buffer at once.

So 2 byte's can still result in a 64k maximum frame size that
some devices might baulk at... but not many.

However, if frame sizes were 4 bytes or more, then we have the
issue that all end points will have a restriction of frame size
that can be held in a buffer that is smaller than the maximal.
We then need error handling for when a frame that is too large
is offered etc.

So if we are going to limit it, then is should be small
enough so we can say that all websocket endpoints must
be able to accept a minimal frame.

In fact, I would argue for a 16k max frame size with 2
bits given over to help with partitioning and aggregation,
but with no requirement that an endpoint can handle
aggregation and partitioning (unless they indicate they
can in the handshake).

cheers

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 16 Apr 2010 08:11:40 -0700

>  - Sub-protocol needed if messages are larger than the max frame.
+1 as a good starting point for the framing. Details can be worked out 
from this proposal.

-- Scott

---

From: Roberto Peon <[email-masked]>
Date: Fri, 16 Apr 2010 09:49:46 -0700

> could not store an entire frame in a buffer at once.

Why would you need to do that?

> some devices might baulk at... but not many.

"not many" probably isn't good enough. For the protocol to be a success it
needs to work reliably. For clients/servers, that implies that they should
be able to start processing before the entire frame has been received. For
proxies you just stream the data.. unes you're doing multiplexing for
people, in which case you have other concerns.

-=R

---

From: Ian Hickson <[email-masked]>
Date: Fri, 16 Apr 2010 19:52:30 +0000 (UTC)

>  - Frame size is indicated up front.

This seems incompatible with the requirement that we not expose amateur 
programmers to the complications of measuring UTF-8 strings.

>  - Frames are binary.

Not sure what this means in this context, unless you mean the length is in 
bytes and not in characters; if you mean the latter then I agree that any 
length in the protocol should be in bytes.

>  - Sub-protocol needed if messages are larger than the max frame.

This seems like it would make the API highly unpredictable. For example, 
it would mean that you could transmit 65,536 characters if they were the 
character "X", but only 21,845 if they were the character U+263A. That 
would be very confusing. Similarly, if we add compression, it would mean 
that how many bytes you could pass in the send() method would depend on 
how well it compressed. This would result in a very bad programming 
experience, especially for anyone new to this kind of thing. (I would 
expect most subprotocols to be ridiculously primitive; asking amateurs to 
figure out chunking mechanics is non-trivial, especially if they have to 
work out where to split a UTF-16 string as exposed in JS, before it is 
converted to UTF-8 and compressed, which is what matters for deciding what 
will fit in the frame.)

>  - isolates framing layer from the bugs of higher layers

If we were designing a protocol for experts, I would completely agree. 

> THEN framing doesn't work. [...] One solution to this problem is to 
> start a frame with a known sequence of octets, so that this can be 
> detected.

That's an interesting approach, but I worry that anything hardcoded like 
this would be handled in ways that defeat the purpose. This is why, for 
instance, the proposed handshake uses unpredictable keys to force the 
server to prove it read the handshake -- otherwise, servers written by 
amateurs might not actually read the handshake but just send back the 
right response, simply assuming the handshake came from a Web Socket 
server. If it weren't for targetting amateur programmers, the handshake 
could be a heck of a lot shorter; we could even do away with the 
requirement for a round trip and for the client to test the handshake, 
since we could assume the server would only accept connections that were 
trusted.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 16 Apr 2010 20:26:54 -0700

> length in the protocol should be in bytes.
Unless the spec gets an editor who understands the basics of wire 
protocols, discussion is becoming pointless.

Or one capable of basic programming, really. "The complications of 
measuring UTF-8 length"? "chunking mechanics [are] non-trivial?"

-- Scott

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 17 Apr 2010 09:18:29 +0200

> programmers to the complications of measuring UTF-8 strings.

I'm not this is an agreed requirement.

At the 77 meeting, we asked you to put this into the form of some
proposed text for the requirements document.  We can then discuss
and if there is consensus it can be added as a requirement.

However, I maintain my belief that amateur programmers will
most likely be using frameworks, libraries and servers that
implement the protocol for them.

> character "X", but only 21,845 if they were the character U+263A.

Frames are entirely different entities to messages.
The application API deals with message - which if we have a fixed
frame size, may be fragmented and reassembled.

If we did agree on a small fixed frame size, the fragmentation
and aggregation would be an obvious extension.  We may or may not
decide that all clients/servers should implement it.   I think
optional would be best, as browsers are capable enough to implement
the option, but small special purpose clients/servers could be developed
without that complication.

> That 

The bad programming experience is a result of directly exposing a
protocol entity (frame of binary data) to the application programmer.

The solution is to abstract the application programmer away from
the protocol. They send/receive arbitrary length messages in characters,
the protocol then arranges to encode those as 1 or more binary frames,
with or without compression, with or without aggregation etc.

At the other end, the character message is reassembled (or streamed
to disk if it is large etc).

> If we were designing a protocol for experts, I would completely agree. 

I believe the vast majority of people here agree with the view that
the target audience for websocket is Browser developers and Server Developers.
ie expert developers.

I know you do not agree with this, but I do not believe that you have made
your case that amateur programmers are the prime target for this protocol.

Have a look at http://en.wikipedia.org/wiki/Web_Sockets - there are
plenty of implementations listed there, which are written by expert
programmers.

Again, if you believe that targeting amateur programmers is a requirement
for this working group, can you propose some text to that effect
to add to the requirements document.   If you make your case, then
it can be included.

regards

---

From: Justin Erenkrantz <[email-masked]>
Date: Sat, 17 Apr 2010 00:31:10 -0700

> the target audience for websocket is Browser developers and Server Developers.

I know I'm just piling on here, but I will add that I believe that the
JavaScript APIs should certainly be aimed towards casual developers,
but that's not a legitimate reason to dumb down the wire protocol to
the point of gross inefficiency.  I do believe that over time the
server developers will converge on something resembling a common
pattern - it will take time, but I think it is achievable.  Yet, that
convergence will only happen if sites adopt Web Socket.

It is more important to me to have a system that is more efficient
than the status quo than it is for someone who has never seen an IETF
RFC or a wire protocol in their life to deploy a half-backed implement
in a weekend.  I would rather that individual spend the weekend
writing their application and not getting bogged down in protocol
minutiae.  Furthermore, if Web Socket fails to improve the best that
what we can offer now, this is all just a pointless exercise anyway.
-- justin

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 17 Apr 2010 10:13:21 +0200

Roberto Peon wrote:
> 
> 
> On Fri, Apr 16, 2010 at 5:38 AM, Greg Wilkins <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> 
> 

At the 77 meeting, I raised the point that currently error
handling in the protocol is not sufficient to deal with
implementations that impose a maximum frame size.  The spec
currently says the end point should just close the socket
if it receives a frame that is too large.

One option is to negotiate maximum frame size, but that
appears to go against the no negotiation style of this
protocol.

The suggestion was then made that we pick a small fixed
maximum frame size, that all implementations would be
required to handle.

Large messages would then be fragmented and assembled
from 1 or more frames.    This does not solve the problem
of end points that cannot handle large message, but it
does make the problem not a transport problem.  Higher
layers are better able to handle large messages, as
they can stream to disk, reject with errors, parse and
handle as a stream, etc etc.

> "not many" probably isn't good enough. For the protocol to be a success
> it needs to work reliably. For clients/servers, that implies that they
> should be able to start processing before the entire frame has been
> received. For proxies you just stream the data.. unes you're doing
> multiplexing for people, in which case you have other concerns.

That is why my suggestion is to go for a 16k max frame size
that all implementation would have to be able to handle.

This frees up 2 bits that can be used to coordinate
the fragmentation and assembly.

The alternative approach is to not have fragmentation,
but to allow the max  frame size be of arbitrary (or very large)
length so it can always handle a message or arbitrary (or very
large) length.  Then have reasonable error handling to cope with
too large messages/frames.

The problem with this approach, is the sending of a too large
to handle frame, breaks the ability to communicate bidirectionally.

Imagine if both client and server both start sending too large
to handle frames at the same time.   Neither will be able to
send too-large errors because they are sending large frames
and to their knowledge the other side is handling them.

Fragmentation of large messages into frames gives an
endpoint the ability to terminate the transmission of
a too-large-message or even to interleave control frames
containing errors.

regards

---

From: Scott Ferguson <[email-masked]>
Date: Sat, 17 Apr 2010 10:45:03 -0700

> aggregation and partitioning (unless they indicate they
16k is fine, but why are two bits needed instead of one? One bit is 
needed for final vs non-final frame in a message, but I don't see how 
the second bit would be used. (Although, hmm, are you thinking in a 
multiplexing context where the second bit could mean "use the same 
channel"? Or messaging "use the same sender/recipient"? That would be a 
useful capability.)

I'd think basic chunking would be a requirement like HTTP/1.1, not a 
negotiated capability, to support messages over 16k and also to the 
reduce the round trip latency as discussed in the other thread.

For example, a high performance client might pipeline the first request 
frame(s) with the HTTP header since, in reality, it expects the server 
to understand the requested protocol, but it would need some 
fundamental, non-negotiated capabilities like fragmenting for the 
pipelining to work.

-- Scott

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 18 Apr 2010 19:03:04 +0200

Scott Ferguson wrote:
> 16k is fine, but why are two bits needed instead of one? 

1 bit is fine for pure fragmentation.

I suggested 2 bits because of previous discussions on
the list about a bit indicating if an intermediary
can buffer a frame or not.  This gives the ability
to have streaming frames plus other frame types
(see BWTP post for an example of using the 2 bits).

However we could get the streaming by just having
a single bit for fragmentation and the spec can say
that all intermediaries must forward all frames immediately.

So I think for simplicity's sake, a 16k frame size
with 1 bit for fragmentation and 1 bit reserved for future
use would work fine.  (or 32k and no reserved bit).

> reduce the round trip latency as discussed in the other thread.

I agree.

If we consider the complexities of error handling for unlimited
sized frames - then I think the simpler solution is to have
unlimited sized messages that are sent in 1 or more limited
sized frames.

Perhaps that reserved bit can then be used for control/error
frames ie if the unlimited message is too large a control/error
frame can be sent to say enough already!

cheers

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 07:01:06 +0800

> server. 

You could complicate this if you like.  Perhaps each "fixed" sequence is based on the output of a pseudorandom sequence, the seed to which is assigned in the handshake.  I'd like to better understand what this is buying though.*

--Martin

* ...just as I would like to understand the same for the handshake.

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 07:07:50 +0800

> can send in one frame.  *grin*)

Oh no, 2 octets is enough - it's twice as many as we currently use as the base unit for our other protocols.

Think of this as Web 2.0, in that the basic unit is 2.0 times the size of the previous version.
 
> a few folks talked about Comet

On reflection, it really doesn't matter what the maximum allowable size is in the protocol.  I originally thought 16 bits would be great because that would be closer to what actual application limits would be - but then that's a little silly too.  If there is going to be a limit - I want that limit to be flexible too.  16 bits is just as arbitrary.  And, if folks want to shift big files this way, let's not be so hard on them.

Incidentally, you can get continuation frames for free: if any frame is full, the next one is a continuation.  If you happen to have a frame that is exactly 65535 octets long, then the next one can be exactly 0 octets long.

--Martin

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 00:32:07 +0100

> that all intermediaries must forward all frames immediately.

That needs clarification.

What does "forward all frames immediately" mean?

If you receive the first _half_ of a frame 10 seconds before the
second half (due to a single dropped packet on a mobile link), do you
have to forward the first half as-is, or is it ok to wait for the second half?

Streaming apps will be penalised if the intermediary applies the
second rule, and message-oriented apps will be penalised if the
intermediary applies the first rule.

One of the timing/latency problems with fixed frame boundaries is they
will tend not to match TCP segment boundaries, making the above
situation very common (but usually much less than 10 seconds).

If intermediaries are permitted to change frame boundaries, as they
are with HTTP chunking, there is much more scope for adapting to TCP
and also fairer multiplexing etc.

I'm thinking if the spec says all frames must be forwarded
immediately, actual implementations will parse the frame headers to
guess when to delay for a short time.

A little bit of information to guide that may be helpful, but I don't
have hard numbers at this time.

I don't think we should prescribe forwarding strategies in a "tight"
way, but we do need to prescribe something minimal.  We should be
aware that immediate forwarding of every byte is _never_ implemented
(there is always a buffering & delaying heuristic at some layer), so
we should recognise that fact, and provide enough flexibility and
guidance so that intermediaries can be developed which makes good
decisions for all styles of application protocols.  We don't have to
specify how exactly.

-- Jamie

---

From: Mike Belshe <[email-masked]>
Date: Sun, 18 Apr 2010 16:35:07 -0700

> handle as a stream, etc etc.

It seems to me that we're over engineering here for what the websockets API
is trying to do.

I disagree that 32bit lengths means that you need to support chunking.  The
application can decide when it wants to chunk or not.  Also, why does a
buffer need to be contained in RAM?  Just because we have a long length
doesn't mean that the payload can't stream.  Lots of protocols work this way
today, like HTTP.  And what happens if you stream a 10GB message in a single
web page?  The browser just hangs up the phone at some point.  There is no
graceful error.  While this is a theoretical problem (which Greg raised), it
is not a practical one.

Unfortunately, once you decide that 32bit lengths are not good enough,
you're running into lots of subproblems:  fragmentation/reassembly, max
frame size, etc.  I think we're just overthinknig the needs to the
WebSockets API.

I propose we make lengths be 32bits and drop fragmentation/reassembly from
this protocol.

Mike

> that all implementation would have to be able to handle.

16KB is small today.  It seems incredibly small for the year 2020.

> the fragmentation and assembly.

*bits*?  Sure, we don't want to be wasteful, but with average internet
speeds of 3.9Mbps in the US *today*, are you sure you've got the right
optimization here? :-)

I'd take the simplicity of nixing fragmentation/reassembly and suck it up on
the extra bits for a longer length field.

Mike

---

From: Mike Belshe <[email-masked]>
Date: Sun, 18 Apr 2010 16:37:55 -0700

As an anecdote:  every single place that we thought we should use a 16bit
length in SPDY has come back to haunt us and will ultimately get changed to
32bits.  When thinking of protocols to use now and also in 2020, we really
should not limit anything to 16bits unless it is in some critically
bandwidth-sensitive part of the protocol.  Remember that bandwidth is cheap
(and growing fast!)  Round Trip Times are expensive (and not shrinking!)

Mike

On Sun, Apr 18, 2010 at 4:35 PM, Mike Belshe <[email-masked]> wrote:

> I'd take the simplicity of nixing fragmentation/reassembly and suck it up
> on the extra bits for a longer length field.
>
> Mike
>
>
>
>
>>
>>
>>

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 00:08:53 +0000 (UTC)

On Sun, 18 Apr 2010, Mike Belshe wrote:
>
> As an anecdote:  every single place that we thought we should use a 
> 16bit length in SPDY has come back to haunt us and will ultimately get 
> changed to 32bits.  When thinking of protocols to use now and also in 
> 2020, we really should not limit anything to 16bits unless it is in some 
> critically bandwidth-sensitive part of the protocol.  Remember that 
> bandwidth is cheap (and growing fast!)  Round Trip Times are expensive 
> (and not shrinking!)

N years from now (for some value of N) people will be saying the same 
about 32 bits. That's why I used a packed integer instead of a fixed-width 
integer to specify lengths.

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 00:14:14 +0000 (UTC)

On Mon, 19 Apr 2010, Thomson, Martin wrote:
> > >
> > > IF used for UTF-8 AND implementer counts characters instead of 
> > > octets THEN framing doesn't work. [...] One solution to this problem 
> > > is to start a frame with a known sequence of octets, so that this 
> > > can be detected.
> > 
> > That's an interesting approach, but I worry that anything hardcoded 
> > like this would be handled in ways that defeat the purpose.
> 
> You could complicate this if you like.  Perhaps each "fixed" sequence is 
> based on the output of a pseudorandom sequence, the seed to which is 
> assigned in the handshake.

Do you have any concrete suggestions for how we could do this? I'm not 
sure I follow.

> > This is why, for instance, the proposed handshake uses unpredictable 
> > keys to force the server to prove it read the handshake -- otherwise, 
> > servers written by amateurs might not actually read the handshake but 
> > just send back the right response, simply assuming the handshake came 
> > from a Web Socket server.
>
> [...] I'd like to better understand what this is buying though.*

Assuming you are asking for a longer rationale for the current handshake, 
please search for "the rationale" in this message for an explanation:

   http://www.ietf.org/mail-archive/web/hybi/current/msg01607.html

---

From: Mike Belshe <[email-masked]>
Date: Sun, 18 Apr 2010 17:18:20 -0700

> integer to specify lengths.

Of course, agree.  And despite where 32bits has failed us in history
previously, I still think this is far enough out that we'll be able to
upgrade by then.  If you think N is sufficiently small for 32bits, we should
use 64bits, would that make N sufficiently far away? :-)

Mike

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 00:23:09 +0000 (UTC)

On Sun, 18 Apr 2010, Mike Belshe wrote:
> > about 32 bits. That's why I used a packed integer instead of a 
> > fixed-width integer to specify lengths.
> 
> upgrade by then.

Sounds like Y2K reasoning. :-)

> If you think N is sufficiently small for 32bits, we should use 64bits, 
> would that make N sufficiently far away? :-)

Reserving more than four bytes every time just seems like a lot of wasted 
bytes. Not that they're expensive or anything, but most of these frames 
are going to be only a few bytes, so it just seems silly to be padding 
everything full of zeros.

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 01:24:52 +0100

> appears to go against the no negotiation style of this

The current protocol does have negotation, between the application
endpoints.  They can choose not to send messages larger than what they
agreed, implicitly or explicitly.

The current protocol does not have frames and transport-level
fragmentation, so I'm not sure how the negotation in the current
protocol is relevant :-)

This is an argument for transport-level negotiation, even if just at
the beginning, independent of application negotiation.

> maximum frame size, that all implementations would be

Although maximum frame size may be an issue, setting a maximum (or
negotiating one) really doesn't help: Messages can be of arbitrary
length, and are assembled at the receiver from fragments by simply
appending the fragment data.

I can't imaging an implementation benefiting significantly from a
fragment size limit but no message size limit.  After all, it can just
append the bytes to the message as it receives them.

Now a maximum *message* size might be interesting.

That has the same problem at the next level: The WebSocket API,
anyway, requires received messages to be *queued* as events to be
processed.

So even if you limit the message size, you can receive any number of
messages into the queue and overflow memory that way.

It is not made clear, but I presume that web browsers in this
condition will either abort the connection or stop reading and thereby
use TCP flow control to block the sender.

That's quite an unfortunate condition to get into, because it breaks
"graceful close" and other kinds of control messages.

So even limiting fragment size and message size does not prevent
awkward "too much" conditions from occurring.

(I'll obliquely mention flow control as a possible answer...)

> handle as a stream, etc etc.

I am confused: Since assembly of messages from frames is a transport
problem, what part of transport does limiting just the frame size (and
not the message size) solve?

The only component not participating in fragmentation and assembly may
be intermediaries, but there is no benefit to them *always* buffering
entire frames before forwarding; that would only add unwanted latency.

> too large messages/frames.

Don't we need the error handling for too large messages anyway?

> a too-large-message or even to interleave control frames

That should really be about all types of error and also non-error
"abort this message" states, which includes graceful close.  Not just
too-large errors.

If end A sends a graceful close, and end B has just started sending a
large frame, end B cannot confirm the graceful close until it finishes
sending the frame.  That's not too bad because B still sees the
graceful close request and knows that the frame it is sending will be
(or should be!) ignored...

On that note, there should be a frame type which signals "the message
this is part of is aborted / has an error", quite distinct from
"message end".

An alternative encoding: Message end doesn't necessarily have to use
one of the length bits.  It's enough to say that a special frame type,
or a length field of 0xffff (for example) is used to mean "end of
message".  Since there ought to be a way to signal "abort message"
anyway, that'd be quite natural.  (HTTP chunking is like this).

> that all implementation would have to be able to handle.

16kb may seem like a good compromise now, but even today that's small
on some networks.  The lifetime of this protocol is the next 10-20
years: We can expect 10Tbit/s networks to be common by the end of
that.  A 16kb message will take just 10ns to transmit.  And then there
are virtual machines: Increasingly, browser applications talk to
corporate servers over virtual networks (no physical link), which are
very fast and like large packets.

Since I don't see any benefit in limiting the frame size for buffering
purposes (endpoint receivers can just append the bytes to the message
however it's received, and intermediaries shouldn't buffer whole large
frames), and there are performance and even memory-saving
optimisations (Google sendfile) to transmitting larger ones, I think
we shouldn't limit the future in that way.

A good place for frame headers for large messages is one per TCP/IP
packet, at the beginning.  Only sophisticated transmitters are likely
to be able to place them in that way, but if they can, why not.

Since the maximum TCP segment size is just under 64k, 64k works for that.

(I think we shouldn't design the protocol to only work well with
TCP/IP either.  Someone, somewhere will shoehorn it over a fast pipe
with bigger packets than IP.  But that's ok as long as framing is a
transport-level concern and fragmentation/reassembly are allowed at
each hop: non-TCP/IP components can modify the framing scheme.)

Anyway, premature optimisation and all.  We should aim to get the
broad principles and message/frames types right rather than worry
about the details of exactly how many bits go where and how the bytes
are laid out... before deciding those other things.

-- Jamie

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 08:40:01 +0800

> Do you have any concrete suggestions for how we could do this? I'm not

Handshake contains: WebSocket-FrameSeed: <seed>

The peer that sends this header uses the number as follows:

  var PRNG prng = new PRNG(seed);
  WS.prototype.sendFrame = function (data) {
    this.buffer.send(prng.getNextBits(WS.FRAME_MARKER_LEN), WS.FRAME_MARKER_LEN);
 
    this.buffer.send(WS.BINARY_FRAME, WS.FRAME_TYPE_LEN);

    var binary = UTF8.encode(data);
    this.buffer.send(binary.length, WS.FRAME_LENGTH_LEN);
    this.buffer.send(binary, binary.length);
 }

A receiver does the same, but uses the PRNG.getNextBits() to check that the frame was correctly terminated:

  if (this.buffer.read(WS.FRAME_MARKER_LEN) != prng.getNextBits(WS.FRAME_MARKER_LEN)) {
    // ERROR: drop connection, throw exception, etc...
  }  

>    http://www.ietf.org/mail-archive/web/hybi/current/msg01607.html

You have an HTTP client that is being duped into contacting a WebSockets server.  I can understand that far, but I don't see what the attacker gains.  The scenario needs more details.

Invent a scenario for me.  How does this allow the attacker to gain the nuclear launch codes?

If this is as simple as a programmer forgetting the distinction between /Upgrade: foo/ and /^Upgrade: foo/, then we're back to disagreeing on the same old and tired topic.

--Martin

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 01:51:02 +0100

Mike Belshe wrote:
>    2020, we really should not limit anything to 16bits unless it is in
>    some critically bandwidth-sensitive part of the protocol.  Remember
>    that bandwidth is cheap (and growing fast!)  Round Trip Times are
>    expensive (and not shrinking!)

I agree.

>    It seems to me that we're over engineering here for what the
>    websockets API is trying to do.
>    I disagree that 32bit lengths means that you need to support chunking.
>     The application can decide when it wants to chunk or not.

I think the point of chunks are

  - to permit the transport implementation to make network-aware
    decisions (such as aligning with TCP segments)

  - to permit intermediaries to re-chunk when that is advantageous
    (such as aligning with the next hop TCP segments)

  - to carry information for smarter buffering decisions at intermediaries
    (to reduce latency - eager forwarding of every byte when received
    is not ideal) (and I know you care about latency)

  - as a realistic basis for multiplexing, even if that's
    an extension not in the base protocol

  - to leave room for transport control messages inserted by the
    transport (such as graceful close and transport error indicators),
    so that applications don't have implement them

>    Also, why does a buffer need to be contained in RAM?  Just
>    because we have a long length doesn't mean that the payload can't
>    stream.  Lots of protocols work this way today, like HTTP.

WebSocket API spec says that messages must be completely collected
before they are signalled as DOM events.  They cannot be streamed in
that API.

About buffering *frames* as opposed to messages, I agree.  There is no
reason that frames need to be buffered whole before assembling them
into messages or forwarding them to the next hop.  Intermediaries
should not buffer whole frames of arbitrary length, they should
forward them, and probably change the frame boundaries at the moment
of forwarding to avoid blocking control messages later.

>     And what happens if you
>    stream a 10GB message in a single web page?  The browser just hangs up
>    the phone at some point.  There is no graceful error.

Why?  There is no reason why the browser cannot stop reading, and send
the graceful error message...

And that is important: Look at it the other way (although either way
around this can occur).  You're sending a 10GB message to the server
in something akin to HTTP PUT (but over this protocol), and that
particular server is being shut down, times out or whatever.  If the
server just hangs up, the browser doesn't know if it's safe to
retransmit that POST request, so it must report an error.  But if the
server sends a graceful "sorry I'm closing" message, the browser can
retry automatically (like it does for HTTP GETs).

That is currently why pipelined HTTP is so severely broken and close
to unusable...  Let's not repeat that debacle.

>    While this is a theoretical problem (which Greg raised), it is
>    not a practical one.  Unfortunately, once you decide that 32bit
>    lengths are not good enough, you're running into lots of
>    subproblems: fragmentation/reassembly, max frame size, etc.  I
>    think we're just overthinknig the needs to the WebSockets API.  I
>    propose we make lengths be 32bits and drop
>    fragmentation/reassembly from this protocol.

So you're committing to being unable to send 5GB messages?

That seems inconsistent with the mention of future networks.  Today, a
5GB message takes just 5 seconds on a decent 10gig ethernet.  That is
well within the usability range of WebSocket and browser applications,
although typically there isn't enough RAM and processing isn't fast
enough to do something useful inside a browser application with that
kind of data.

In 2020, a 5GB message will take a fraction of a second to transmit,
<1% of RAM, and the CPU will be able to process all of it quickly
enough to do useful things, such as updating a 3D model in Javascript.
I see no benefit to forbidding their use while permitting 3GB messages.

-- Jamie

---

From: Mike Belshe <[email-masked]>
Date: Sun, 18 Apr 2010 17:51:50 -0700

> Sounds like Y2K reasoning. :-)

It's not at all the same.  As you stipulated below, we're talking about the
edge case of super large messages (greater than 4GB!).  Even in the future,
it will be an edge case, and it has an easy workaround (multiple frames).
 Y2K was the common case; once N years elapsed, everyone had the problem in
the common case.  Lastly, it should be noted that everyone *did* upgrade for
Y2K and it wasn't the end of the world.  A better example might be the IPv4
to IPv6 migration...  It should have been the end of the world, except, it
wasn't. :-)

Fixed lengths has a lot of benefits - including for the 'ease of
programming'.

> everything full of zeros.

I wasn't proposing it seriously.

Mike

Mike

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 01:52:15 +0100

Thomson, Martin wrote:
> Incidentally, you can get continuation frames for free: if any frame
> is full, the next one is a continuation.  If you happen to have a
. frame that is exactly 65535 octets long, then the next one can be
> exactly 0 octets long.

Alas, there are very good reasons to send non-final frames that aren't
the full length.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 02:01:36 +0100

> that's what the app writers did...trying to get them to change is

Silly, few?  Most AJAX requesting code I've seen fetches a bit of HTML
to replace in an existing page, such as replacing a DIV or table row,
or the main area in a page (i.e. the bit which isn't border buttons
etc.)  That's easily a megabyte of response in many web pages...
It's hard to see why Comet would avoid that pattern.

Every so often, someone says that big requests should not be using
WebSocket but should go out of line, using HTTP.

But if you can send a large blob of text over WebSocket, why would
anyone *not* do that?  The effects of receiving a short update message
over WebSocket and then issuing a long HTTP request are twofold:

   - More complicated application code (unnecessary)
   - Higher latency (extra round trip + possible DNS, TCP, SSL setup)

It follows that sending large messages over WebSocket is perfectly
natural and might be best, if the alternative is to get the same large
message over HTTP using XmlHttpRequest.

-- Jamie

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 09:06:36 +0800

> Alas, there are very good reasons to send non-final frames that aren't

From my perspective, the only very good reason you stop sending one thing is because you want to send something else (i.e. multiplexing).

Aligning with segments isn't a good argument for me.  How the WS layer is aware of layers below it and the way that it interacts with those is orthogonal to frame size.  TCP provides an abstraction of a stream.  If a particular implementation wants to optimize, it can, but that shouldn't affect layers above those key abstraction points.

Addressing shortcomings in the abstraction (graceful close, etc...) is another thing.  It's reasonable to say that the TCP abstraction does not provide graceful close, so providing it at a higher layer is OK.

--Martin

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 03:01:27 +0100

> From my perspective, the only very good reason you stop sending one thing is because you want to send something else (i.e. multiplexing).
[...]
>
> Addressing shortcomings in the abstraction (graceful close, etc...)
> is another thing.  It's reasonable to say that the TCP abstraction
> does not provide graceful close, so providing it at a higher layer is
> OK.

If you want to send a graceful close response when you have only sent
half a message, you cannot do it unless the half message was sent as
a non-final, non-full-length frame.

That's why it's useful.

It's not because you want to send something else _now_, it's so you
have the _possibility_ to send something else later.  Because you
can't abort a half-sent frame later.  (You would need TCP urgent data
or something, and that's not feasible to use.)

> Aligning with segments isn't a good argument for me.  How the WS
> layer is aware of layers below it and the way that it interacts with
> those is orthogonal to frame size.  TCP provides an abstraction of a
> stream.  If a particular implementation wants to optimize, it can,
> but that shouldn't affect layers above those key abstraction points.

Feel free to ignore segment alignment.  It's just an implementation
tunable, not a requirement and doesn't need to appear in any
specification.

The frame size is part of the WS transport layer, and is closely
related to buffering and forwarding decisions.  (Note that eager
sending is sub-optimal for latency, in case you're curious.)

Those decisions have little or nothing to do with TCP segments, but
the timing of forwarding decisions cause certain kinds of alignment
anyway.  The same outcome occurs with other types of pipe abstraction.

You said "it shouldn't affect layers above those key abstraction
points".  Note that frame size is *not* present in the layer above WS
buffering and forwarding decisions.  (If it is, it is misdesigned.)

You might be thinking of the message size, which is.

-- Jamie

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 10:11:35 +0800

You make a good case... for SCTP :p

--Martin

> -----Original Message-----
> From: Jamie Lokier [mailto:[email-masked]]
> Sent: Monday, 19 April 2010 12:01 PM
> To: Thomson, Martin
> Cc: Justin Erenkrantz; Hybi
> Subject: Re: [hybi] Frame size
> 
> Thomson, Martin wrote:
> > Jamie writes:
> > From my perspective, the only very good reason you stop sending one
> thing is because you want to send something else (i.e. multiplexing).
> >
> [...]
> >

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 02:40:19 +0000 (UTC)

On Mon, 19 Apr 2010, Thomson, Martin wrote:
> > >
> > > You could complicate this if you like.  Perhaps each "fixed" 
> > > sequence is based on the output of a pseudorandom sequence, the seed 
> > > to which is assigned in the handshake.
> > 

I don't understand how this solves the problem of people mismeasuring 
strings. (I don't really follow the pseudocode above, either.)

My concern is that people will write this:

   send(WS.BINARY_FRAME);
   send(encodeLength(data.length));
   send(data);

...and their system will automatically convert the string "data" to UTF-8.

(As opposed to doing:

   var binary = UTF8.encode(data);
   send(WS.BINARY_FRAME);
   send(encodeLength(binary.length));
   send(binary);

...or the equivalent.)

> > Assuming you are asking for a longer rationale for the current 
> > handshake, please search for "the rationale" in this message for an 
> > explanation:
> > 
> You have an HTTP client that is being duped into contacting a WebSockets 
> server.  I can understand that far, but I don't see what the attacker 
> gains.  The scenario needs more details.
> 

There's no way to extract data in such a scenario, as far as I am aware, 
unless the subprotocol supports a way to trigger data to be sent out of 
band. To use your analogy, the risk is not getting nuclear launch codes, 
it's just launching the nukes.

Suppose an author wrote a Web Socket server that launches nukes upon 
request. The server is in on a secure intranet, and all users are trusted, 
so there's no authentication required. The server therefore can ignore the 
handshake, and just needs to scan for frames that have the request.

There's no way for an attacker on hostile.example.com to open a Web Socket 
connection to nukes.intranet.example, since the browser would read the 
handshake from nukes.intranet.example before allowing a frame to be sent, 
and the handshake says that only the origin nukes.intranet.example is 
allowed. However, hostile.example.com can instead do an XHR or form 
submission to nukes.intranet.example, with the payload being carefully 
crafted to contain just a frame causing a nuke. The server would receive 
it, ignore the (wrong) handshake, see the frame, and set off the nukes. A 
worker browsing the Web with a computer on that intranet who happens to go 
to hostile.example.com could, without their knowledge, fire the nukes.

The new handshake works around this (as described in the message above) by 
requiring that the server prove that it checked the handshake. In the 
scenario above, the server would fail to find the right information to 
create that proof; there is a much better chance that the author would 
then bail rather than continue trying to listen for further frames.

(I considered making it even more secure by saying that the first byte of 
the MD5 sum that the author sends back in the handshake must be xored with 
the frame type byte, since then there's at best a 1/256 chance that the 
attacker will have a successful attack even if the server doesn't bail 
upon failing to find all the right bits, but that isn't a good tradeoff -- 
it makes debugging _much_ harder while making the protocol not that much 
more secure -- a few hundred attempts isn't many.)

> If this is as simple as a programmer forgetting the distinction between 
> /Upgrade: foo/ and /^Upgrade: foo/, then we're back to disagreeing on 
> the same old and tired topic.

Yes. The protocol would be very different if I had designed it for experts 
only. As I've said before, if it is the consensus of this working group 
that we should not design a protocol for amateurs, then we should start 
over. I have no objection to this working group designing a protocol for 
experts only; personally, though, I am interested in writing a protocol 
for amateurs. I don't mind if we do this here or elsewhere. I would, 
however, like to know whether that is an acceptable work product for this 
working group, because arguing about whether it is or not is not getting 
us anywhere and is apparently not something we can decide objectively.

---

From: Mike Belshe <[email-masked]>
Date: Sun, 18 Apr 2010 20:05:07 -0700

> >    expensive (and not shrinking!)

Thanks for your comments; we're probably in a rathole.  But I will reply
anyway.

>    decisions (such as aligning with TCP segments)

TCP is a stream, so you must mean IP segments.  But chunking doesn't impact
the ability to align with packets or ensure efficient use of them.  I must
be missing a key point here?

>    (such as aligning with the next hop TCP segments)

This would mean that applications cannot rely on chunking to be preserved
end-to-end.  Is that really what you want?  Why?  HTTP chunking cannot be
tweaked by intermediaries either.  Well, I suppose an intermediary could
try, but they could easily make mistakes, and it is unclear if there is any
benefit?

>    is not ideal) (and I know you care about latency)

It's unclear to me what value intermediaries are adding here.  What feature
is this?

>    an extension not in the base protocol

If we want to add multiplexing, there is more than just chunking involved.

>    so that applications don't have implement them

The protocol has implicit chunks - frames.  Having a second level of
chunking has little value here.  Or at least I haven't heard the use case.

Applications that want to use chunking can do so already by just using
smaller frames.

> before they are signalled as DOM events.  They cannot be streamed in

Good point.  But that is still implementation on top of the protocol, and
the two can be different.  The WebSocket API also doesn't define a maximum
string size.  But obviously, each implementation will have one.

> retry automatically (like it does for HTTP GETs).

I agree that this is a problem - but note that it exists regardless of the
size of the message.  Even if your frame length is 10 bytes, you have this
problem.

> to unusable...  Let's not repeat that debacle.

I agree that pipelining is broken; you're implying that somehow dealing with
frame length would fix it - but I know you don't really believe that to be
true.  The reasons it never got deployed are not this :-)

> So you're committing to being unable to send 5GB messages?

No - just that they have to be split up at the application layer.

> enough to do something useful inside a browser application with that

I don't think we should design for the edge case.  The edge case can be
handled at the app layer.

> I see no benefit to forbidding their use while permitting 3GB messages.

Agree, but I didn't impose a limit.  Just a frame size.  UDP has a maximum
frame size of 16bits (64KB).  Are you suggesting that you can't send more
than 64KB over UDP? :-)

Overall, I'm talking about simplicity.  A 32bit fixed length is simple and
sufficient for purposes today and tomorrow.  It doesn't inhibit the edge
case, it merely makes it so that those wishing to support the edge case have
to do extra work, rather than the common case doing extra work.  And, it
allows us to simplify by not needing complex support of
fragmentation/reassembly.  I actually don't mind Ian's packed integer either
- it also has the property that we don't need fragmentation.  But I like
fixed length best.

Mike

---

From: "Anne van Kesteren" <[email-masked]>
Date: Mon, 19 Apr 2010 12:18:45 +0900

> Fixed lengths has a lot of benefits - including for the 'ease of

The implementation could still use fixed lengths and disallow anything  
over e.g. 32bits or 16bits. There's no reason we should limit the protocol  
by current hardware limits.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 11:40:43 +0800

> strings. (I don't really follow the pseudocode above, either.)

It does this (or significantly reduces the changes of this) by marking the start of a frame with a known sequence.  This sequence could be fixed, or it could be derived from a known sequence.  If this sequence isn't found where it should be, there is an error.

(BTW, I'm not advocating either approach, and I'll point out that the PRNG is most likely less useful and more complicated than a well-chosen fixed value.)

> ...and their system will automatically convert the string "data" to

I know.  Untyped and loosely typed languages have that problem.

--Martin

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 12:02:15 +0800

> > Invent a scenario for me.  How does this allow the attacker to gain
> the
> > nuclear launch codes?
> 

Thank you.  I think that I understand your scenario a little better.

> then bail rather than continue trying to listen for further frames.

Post mortem detection of the problem doesn't solve anything.  If the browser has sent the XHR blindly, the damage is done by the time that it learns that the server is a dumb WS implementation.  You can't go and stuff the missiles back in their silos.

The thing is, this sort of indication (i.e. that the server is a WS server) is made clearer by the presence of the "Upgrade: " header in the 101 response (or the presence of the response itself).

If the server only looks for certain markers, then I imagine that an SMTP message could trigger launch in the same fashion.  It doesn't stop the attack.

What you are trying here is analogous to a qualification exam for server implementers.  If they can pass this test, then you assume that they're smart enough not to make other silly mistakes.

That requires a little too much suspension of disbelief for me.  Your proverbial server implementer will do whatever is necessary to get their app working.  I put it to you that they will pass your intelligence test, and fail in other more spectacular ways.  No matter how complicated you make the test.

It's quite possible that any viable solution to that problem would be more trouble than it's worth.  (you know, there might be a business in that...)

> Yes. 

No comment.

--Martin

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 04:04:00 +0000 (UTC)

On Mon, 19 Apr 2010, Thomson, Martin wrote:
> It does this (or significantly reduces the changes of this) by marking 
> the start of a frame with a known sequence.  This sequence could be 
> fixed, or it could be derived from a known sequence.  If this sequence 
> isn't found where it should be, there is an error.

Oh, I see. I think the behaviour of the protocol in the case of a length 
mistake today is pretty fatal already -- no need to add more catching 
code. (You get a bunch of invalid frames.) The best a client could do is 
abort anyway, and that doesn't seem especially graceful. (Ignoring the 
unknown frames is what the protocol does now.)

I thought your proposal was something that would reduce the chance of the 
error being made in the first place. That would be a good way of removing 
the need for sentinel-based framing.

---

From: Scott Ferguson <[email-masked]>
Date: Sun, 18 Apr 2010 21:06:14 -0700

> chunking has little value here.  Or at least I haven't heard the use case.
Same as HTTP/1.1 chunking: when the length of dynamically produced 
content cannot be calculated beforehand.

Unless the frames can be assembled into messages, it's not implicit 
chunking.
> Applications that want to use chunking can do so already by just using 
I don't see how that makes any sense.

Here's a basic, typical application. Suppose messages are XMPP packets 
(ours are binary HMTP, but the idea's the same). The message is 
identical to the entire packet.  The sender don't know how big the 
packet will be until serialization completes, but receivers may need to 
know the message/packet end before parsing.

For sanity and performance, we use a fixed-size output buffer for each 
packet. Each frame is a fragment of the XMPP packet and the message only 
ends with the final frame. We don't know the length of the entire packet 
until the entire thing is serialized. So the protocol needs multiple 
frames forming a message.

This is a very basic pattern. There's nothing unusual about it and I 
don't see how smaller frames addresses the issue, because it's the 
message boundary that's important. If you're not marking message 
boundaries, you may as well eliminate the frames entirely and just have 
raw TCP.

16 vs 32 vs variable-length integer encoding isn't hugely important. 
Chunking is important.
> Overall, I'm talking about simplicity.  A 32bit fixed length is simple 
> and sufficient for purposes today and tomorrow.
HTTP's fixed Content-Length is not sufficient for the same reasons.

-- Scott

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 04:18:28 +0000 (UTC)

On Mon, 19 Apr 2010, Thomson, Martin wrote:
> > 
> > The new handshake works around this (as described in the message 
> > above) by requiring that the server prove that it checked the 
> > handshake. In the scenario above, the server would fail to find the 
> > right information to create that proof; there is a much better chance 
> > that the author would then bail rather than continue trying to listen 
> > for further frames.
> 
> Post mortem detection of the problem doesn't solve anything.  If the 
> browser has sent the XHR blindly, the damage is done by the time that it 
> learns that the server is a dumb WS implementation.  You can't go and 
> stuff the missiles back in their silos.

The idea is not to make the client able to do anything, the idea is to 
make the server more likely to fail to get to the point where it is 
accepting frames without having checked that it's a Web Socket client.

> If the server only looks for certain markers, then I imagine that an 
> SMTP message could trigger launch in the same fashion.  It doesn't stop 
> the attack.

I don't see how you could cause any privileged machine in this scenario to 
connect to the Web Socket server using SMTP. Could you elaborate on that?

> What you are trying here is analogous to a qualification exam for server 
> implementers.  If they can pass this test, then you assume that they're 
> smart enough not to make other silly mistakes.

Not really. I have no doubt that most amateur programmers would have no 
more trouble with the handshake than anything else (it's pretty trivial, 
after all). However, to do the handshake correctly they need to collect 
the information. If they fail to collect the information, they can't send 
back the handshake. What I'm assuming is that if they can't send back the 
handshake, they won't just send back nothing and then look for frames. To 
help with this, in the case where they don't have the information, they'll 
likely also have to deal with a division by zero, which in many cases will 
effectively mean they ignore the frames (as their program will have 
crashed or at least failed in some non-trivial way).

It's naturally possible for a programmer to fail to collect all the 
information, catch the division by zero, and _still_ go out of their way 
to look for frames and thus be vulnerable. There's only so much we can do.

> That requires a little too much suspension of disbelief for me.

It would be good to test the protocol to find out how often it is 
correctly implemented. Once their are some browser implementations of this 
handshake, I would like to see how a group of programmers from various 
backgrounds who have not been participating in this working group manage; 
this would give us real world data pointing out problem areas, whether 
there are areas prone to security problem that we had not considered, etc.

> It's quite possible that any viable solution to that problem would be 
> more trouble than it's worth.  (you know, there might be a business in 
> that...)

Not sure what you mean.

---

From: Mike Belshe <[email-masked]>
Date: Sun, 18 Apr 2010 21:27:16 -0700

On Sun, Apr 18, 2010 at 9:06 PM, Scott Ferguson <[email-masked]> wrote:

> Same as HTTP/1.1 chunking: when the length of dynamically produced content
> cannot be calculated beforehand.
>

Sorry - we're thinking of two different things for chunking.  I was thinking
of chunking in terms of enabling multiplexing; but maybe I was off from the
original poster.  Sorry if that is the case.

As for the example you pose - of course this is important.  But isn't this
up to the user of the websocket (e.g. the application)?  I don't think the
websocket protocol needs to do anything for this - as chunking for two
different applications is likely to be very different.

Mike

> (ours are binary HMTP, but the idea's the same). The message is identical to
> the entire packet.  The sender don't know how big the packet will be until
> serialization completes, but receivers may need to know the message/packet
> end before parsing.
>
> until the entire thing is serialized. So the protocol needs multiple frames
> forming a message.
>
> This is a very basic pattern. There's nothing unusual about it and I don't
> see how smaller frames addresses the issue, because it's the message
> boundary that's important. If you're not marking message boundaries, you may
> as well eliminate the frames entirely and just have raw TCP.
>
>> sufficient for purposes today and tomorrow.
>>

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 13:16:42 +0800

> accepting frames without having checked that it's a Web Socket client.

That's the tension.  The lazy server implementer does the minimum that they believe they can get away with.  If they have to interoperate with browsers, they will do the minimum to interoperate with browsers.

That leaves a lot of room for error - and you know what, that's probably OK.  The only one who suffers is the server implementer.  They are the ones being duped.

The WS-aware browser works it out pretty quickly; the WS-ignorant browser probably chokes on the Upgrade header in the response, and if not then, they will eventually fail.

> connect to the Web Socket server using SMTP. Could you elaborate on

The point being that if a server only searches for certain byte sequences, those byte sequences can be embedded in any protocol you like.  Nothing you do as a specification writer helps the server that is able to mistake another TCP-borne protocol for WS.

The mechanism doesn't stop an attack.  It only provides a higher bar to entry for server developers.  There is already a means by which the server tells that the request is not a WebSocket request: checking for the presence of the correct Upgrade header.

Sure, the handshake provides incentive for server developers to do the right thing... inasmuch as it is probably easier then to do the right thing than the wrong thing once the handshake is implemented.  On the other hand, it doesn't provide any real assurances to either party.

We still have to take it on faith that the server implementation isn't brain-dead.  But that's possible without the mechanism.

> > more trouble than it's worth.  (you know, there might be a business
> > in that...)
> 
> Not sure what you mean.

Sorry, just me being obtuse:

  More trouble than it's worth == more round trips and latency, more complicated mechanisms.

--Martin

---

From: Scott Ferguson <[email-masked]>
Date: Sun, 18 Apr 2010 22:16:26 -0700

Mike Belshe wrote:
>
>
> On Sun, Apr 18, 2010 at 9:06 PM, Scott Ferguson <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Mike Belshe wrote:
>
>
> Sorry - we're thinking of two different things for chunking.  I was 
> thinking of chunking in terms of enabling multiplexing; but maybe I 
> was off from the original poster.  Sorry if that is the case.
Gotcha. I'm pretty sure both ideas are going on at the same time :).
>
> As for the example you pose - of course this is important.  But isn't 
> this up to the user of the websocket (e.g. the application)?  I don't 
> think the websocket protocol needs to do anything for this - as 
> chunking for two different applications is likely to be very different.
I think chunking for many/most applications will be similar because it's 
a pretty generic need, like HTTP's chunking is.  In the case of HTTP, 
many developers don't need to be aware that their output is getting 
chunked; it just works. I'd expect the same to be true for the simple 
frame/message model. Assuming it does cover the common cases, it's a 
useful abstraction to give users.

To tie this into the KISS requirement, it's a way to keep the 
client/server APIs and application code simple for the simple cases. 
Applications don't need to know about frames/chunks, just messages.

-- Scott

> Mike
>  
>
>
>     Here's a basic, typical application. Suppose messages are XMPP
>     packets (ours are binary HMTP, but the idea's the same). The
>     message is identical to the entire packet.  The sender don't know
>     how big the packet will be until serialization completes, but
>     receivers may need to know the message/packet end before parsing.
>
>     For sanity and performance, we use a fixed-size output buffer for
>     each packet. Each frame is a fragment of the XMPP packet and the
>     message only ends with the final frame. We don't know the length
>     of the entire packet until the entire thing is serialized. So the
>     protocol needs multiple frames forming a message.
>
>     This is a very basic pattern. There's nothing unusual about it and
>     I don't see how smaller frames addresses the issue, because it's
>     the message boundary that's important. If you're not marking
>     message boundaries, you may as well eliminate the frames entirely
>     and just have raw TCP.
>
>     16 vs 32 vs variable-length integer encoding isn't hugely
>     important. Chunking is important.
>
>         Overall, I'm talking about simplicity.  A 32bit fixed length
>         is simple and sufficient for purposes today and tomorrow.
>

---

From: Mike Belshe <[email-masked]>
Date: Sun, 18 Apr 2010 23:06:11 -0700

On Sun, Apr 18, 2010 at 10:16 PM, Scott Ferguson <[email-masked]> wrote:

> Mike Belshe wrote:
>
>>
>>
>> On Sun, Apr 18, 2010 at 9:06 PM, Scott Ferguson <[email-masked] <mailto:
>> [email-masked]>> wrote:
>>
>>    Mike Belshe wrote:
>>
>>
>> thinking of chunking in terms of enabling multiplexing; but maybe I was off
>> from the original poster.  Sorry if that is the case.
>>
> I think chunking for many/most applications will be similar because it's a
> pretty generic need, like HTTP's chunking is.  In the case of HTTP, many
> developers don't need to be aware that their output is getting chunked; it
> just works. I'd expect the same to be true for the simple frame/message
> model. Assuming it does cover the common cases, it's a useful abstraction to
> give users.
>
>
So, given the WebSockets API, how would chunks be used?  The application
calls send(DOMString).  There is no need for chunking here, because the full
string (and it's length) is already available.  Or did I miss something?

> To tie this into the KISS requirement, it's a way to keep the client/server
> APIs and application code simple for the simple cases. Applications don't
> need to know about frames/chunks, just messages.
>

KISS, I think, would have solve the problem in the simplest way to support
the WebSockets API.

Are there really other protocols to be solved with the same websockets
framework?  (if so, let's list them!)  I think we're all in disagreement
because we haven't agreed on what problem we're solving.  If we agreed that
the problem we're solving is providing a protocol to support a
WebSockets-like API to web developers, we'd likely be done.  Another option
would be to say we don't like the current WebSockets API proposal, and offer
up an alternative API which we're trying to support (maybe one which does
streaming or needs chunking?). But I haven't seen that either.

Our charter is fairly clear, I think: "*The Hypertext-Bidirectional (HyBi)
working group will seek standardization of one approach to maintain
bidirectional communications between the HTTP client, server and
intermediate entities, which will provide more efficiency compared to the
current use of hanging requests.*"  Further, "*In particular, the working
group has liaised with the W3C WebApps working group around the WebSockets
protocol and the need to support the WebSocket API..*."  I realize there is
some flexibility in the charter for a more generic protocol, but there is no
guidance as to what other problems we're trying to solve.  I think this is
where each of us carries our own opinions to the table.

My biggest fear is that a generic protocol will take a very high burden of
proof for implementation and deployment because it will offer so many
features and that this time will only delay solving the one problem we do
know of (fix hanging GET) while not really solving any other specific
problem either.  By contrast, if we stay focused on the problem at hand,
coming to agreement on the WebSockets protocol seems quite achievable.

I'm relatively new to this group and may be completely off base - so my
apologies if I am :-)  But I was at the IETF mtg on the topic, and it didn't
seem that we were  in agreement on what we're doing there either...

Mike

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 06:54:46 +0000 (UTC)

On Mon, 19 Apr 2010, Thomson, Martin wrote:
> That's the tension.  The lazy server implementer does the minimum that 
> they believe they can get away with.  If they have to interoperate with 
> browsers, they will do the minimum to interoperate with browsers.
> 
> That leaves a lot of room for error - and you know what, that's probably 
> OK.  The only one who suffers is the server implementer.  They are the 
> ones being duped.

Also the user.

I think that both the implementor getting duped and the user getting duped 
are problems that we should do our best to help with. I do not want to be 
responsible for developing a protocol where half the deployments are 
vulnerable to attack, even if those bugs are technically flaws in the 
implementations and not in the protocol.

> The WS-aware browser works it out pretty quickly; the WS-ignorant 
> browser probably chokes on the Upgrade header in the response, and if 
> not then, they will eventually fail.

Not sure what this means. What's the browser got to do with it? In this 
attack scenario, the browser thinks it's talking HTTP (indeed it could 
be a legacy browser and thus not know about Web Sockets at all), and the 
response is hidden from the user, so regardless of what the browser thinks 
might be going on, the user is never informed.

> > > SMTP message could trigger launch in the same fashion.  It doesn't 
> > > stop the attack.
> > 
> > I don't see how you could cause any privileged machine in this 
> > scenario to connect to the Web Socket server using SMTP. Could you 
> > elaborate on that?
> 
> The point being that if a server only searches for certain byte 
> sequences, those byte sequences can be embedded in any protocol you 
> like.  Nothing you do as a specification writer helps the server that is 
> able to mistake another TCP-borne protocol for WS.
> 
> The mechanism doesn't stop an attack.  It only provides a higher bar to 
> entry for server developers.

Could you describe an attack that uses SMTP? I honest can't see a concrete 
way to abuse this mechanism with SMTP.

If there's a way to cause a machine with elevated privileges to send a 
payload that is sufficiently like Web Socket to a Web Socket port on an 
arbitrary machine, then that's a huge problem. Is there such a mechanism, 
other than XHR or form submission triggered by a hortile page viewed in an 
Web browser?

> Sure, the handshake provides incentive for server developers to do the 
> right thing... inasmuch as it is probably easier then to do the right 
> thing than the wrong thing once the handshake is implemented.

Right, that's the idea.

> On the other hand, it doesn't provide any real assurances to either 
> party.

If there's a way to make it even better, I'm all ears.

> We still have to take it on faith that the server implementation isn't 
> brain-dead.  But that's possible without the mechanism.

Unless we make things very easy to get right, authors will get it wrong, 
often in spectacularly bad ways. We've seen this again and again, with XSS 
all over the Web, with XSRF all over the Web, with information leakage 
through <script src=""> and JSON, etc. We have no reason to believe that 
Web Socket implementors are going to be any more competent (unless we make 
the protocol so complicated they can't begin to implement it, like HTTP, 
but then we run into the problems I described recently [1]).

[1] http://www.ietf.org/mail-archive/web/hybi/current/msg01691.html

> > > It's quite possible that any viable solution to that problem would 
> > > be more trouble than it's worth.  (you know, there might be a 
> > > business in that...)
> > 
> > Not sure what you mean.
> 

It was the "there might be a business in that" part that confused me more.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 19 Apr 2010 10:14:26 +0200

Mike Belshe wrote:
> thinking of chunking in terms of enabling multiplexing; but maybe I was
> off from the original poster.  Sorry if that is the case.

Mike,

I was not proposing Fragmentation for multiplexing.   I was
suggesting it for error handling (which I guess can be considered
multiplexing control frames).

The scenario that I'm concerned about is that we need
a way to handle too-large-handle messages.

Currently the specification says that the endpoint simply
closes the connection.  But that is unsatisfactory as the
reason for the close is not known and the too-large message
will just be resent.

So we need some way of sending an error status back, saying
that the message being sent is too-large.
But if our framing allows arbitrary large frames, then
we conditions in which there is no way of sending any error
state.

If both endpoints start sending a too-large-to-handle
message at the same time, then both endpoints will
want to send a too-large error frame.   But neither
will be able to because they are in the process of
sending a large frame.

This is not an insurmountable problem, and I'm sure
might be able to invent ways to abort frames and send
error state etc.  However, I believe that such error
handling in an unlimited frame-size environment will
be significantly more complex that a fragmentation
mechanism.

Fragmentation does not avoid the need for error
handling for too-large-to-handle messages.  But
by removing the possibility of too-large-to-handle
frames, it means we are more likely to be able to
simply send the error messages.

regards.

PS> Another alternative is to mandate a negotiation
of maximum message size in the handshake.
Or somebody can come up with a good simple design for
error handling with large (or unlimited) frames.

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 11:21:12 +0100

Mike Belshe wrote:
>    TCP is a stream, so you must mean IP segments.

No, IP has packets; TCP has segments :-)

The segments aren't application visible except in the *timing*, but
there are still performance benefits to an application choosing how to
submit its bytes with segment awareness - primarily to avoid sending
partial segments, which TCP then reacts by adding more delays...  But
we don't need to go into details here.  It's just a tweak to reduce
latency in some cases with sporadic messages - an implementation detail.

>     But chunking doesn't
>    impact the ability to align with packets or ensure efficient use of
>    them.  I must be missing a key point here?

Two things: Firstly, if the proxy forwards partially received chunks
without rechunking, then it cannot send any control messages (such as
graceful close or anything else) until it receives all of the rest of
the data.  That means it cannot signal network errors or other
conditions - the only option it has is to abruptly close the
connection - which the receiver can't reliably distinguish from
other network errors.

This means when you do forward partially received chunk data, its
important to rechunk (unless you'll never have control messages).

(To avoid introducing long, cumulative delays, and infinite memory
requirements, forwarding partial chunks is a good idea, rather than
buffering them whole.)

Secondly, the relationship with packets is that, as a performance
optimisation, you might choose to forward all the data you have
received so far up to a multiple of the estimated next-hop segment
size, and keep the remainder for a short time until you receive more
data.  Remember that's just a performance tweak and needn't be
specified anywhere.  But it's good to make it possible.

>    This would mean that applications cannot rely on chunking to be
>    preserved end-to-end.  Is that really what you want?  Why?  HTTP
>    chunking cannot be tweaked by intermediaries either.  Well, I suppose
>    an intermediary could try, but they could easily make mistakes, and it
>    is unclear if there is any benefit?

Ah...  HTTP chunking *is* altered by intermediaries :-)  They are a
hop-by-hop mechanism, and for that matter, some hops don't support
chunks at all, so they must be added by the intermediaries.

Any application which depends on HTTP chunks being preserved is as
broken as an application which hopes that TCP write boundaries are
preserved when reading...  (And I've seen plenty of the latter which
break as soon as they leave the shop they were developed in.)

>    It's unclear to me what value intermediaries are adding here.  What
>    feature is this?

To avoid introducing long delays (which add up over multiple proxies),
forwarding partial chunks is a good idea, rather than buffering them whole.

But to send whole TCP segments when possible (for both bandwidth and
latency reasons), its a good idea *not* to forward every byte as soon
as its received, and instead wait a short time to receive more data.
Or even wait forever, if you're sure that a small partial message is of
no interest to the receiver.

(Whether small partial messages are of interest to the receiver does
rather depend on the application.  For WebSocket API clients they
don't care because the API won't deliver them, but for, say, a byte
streaming application over WebSocket, partial message delivery reduces
the application-visible byte latency)

>    If we want to add multiplexing, there is more than just chunking

There certainly is! :-)

>    Applications that want to use chunking can do so already by just using

Ah, no, the point of chunks is to be *not visible* to the
applications.  The same way as they aren't visible in HTTP chunking.

They're to let the WS implementation make smart decisions for the
network they are on, for the sort of reasons we've covered.  Chunking
is a generic and simple method which enables a lot of things.

The application visible function are called messages.

>    Good point.  But that is still implementation on top of the protocol,
>    and the two can be different.  The WebSocket API also doesn't define a
>    maximum string size.  But obviously, each implementation will have
>    one.

I think you might want to discuss that with Ian Hickson.

>    I agree that this is a problem - but note that it exists regardless of
>    the size of the message.  Even if your frame length is 10 bytes, you
>    have this problem.

Exactly, hence rechunking solves the graceful error problem (as well
as other control messages).  You still get ungraceful errors, but they
are considered a "fault", as opposed to graceful errors in which
applications can make safe assumptions, such as the knowing that a
message definitely was not received at the other end, and the safety
of retrying an application request.

>    I agree that pipelining is broken; you're implying that somehow
>    dealing with frame length would fix it - but I know you don't really
>    believe that to be true.  The reasons it never got deployed are not
>    this :-)

I do belive that to be part of the problem.

Sure, there are a bunch of implementation and compatibility problems
which also get in the way :-)

But the inability of the client to distinguish between "network
dropped the connection" and "server gracefully timed out the
connection" is a fundamental *design* error: It makes it technically
impossible to safely retry non-idempotent requests on a pipelined
connection where no actual network errors have occurred.  A normal
situation where you do want retrying.

Even if there were no interop difficulties, HTTP pipelining is broken
by design in that respect.

Frame length doesn't fix this.  Graceful close control messages (if
done correctly) *do* solve it.

>    I don't think we should design for the edge case.  The edge case can
>    be handled at the app layer.

Edge case?
>    Agree, but I didn't impose a limit.  Just a frame size.  UDP has a
>    maximum frame size of 16bits (64KB).  Are you suggesting that you
>    can't send more than 64KB over UDP? :-)

Are you suggesting applications split their ages into 64kb pieces and
reassemble them at the receiver?

When talking about chunks, we're talking about applications *not* having
to deal with that - keep it in the WS implementation.

You seem to be proposing to make it more complicated for application
authors, and for no benefit that I can see.  (End to end fixed-size
chunking - what benefit does that provide?)

>    and sufficient for purposes today and tomorrow.  It doesn't inhibit
>    the edge case, it merely makes it so that those wishing to support the
>    edge case have to do extra work, rather than the common case doing
>    extra work.

It's only an edge case because you've made it one.

3GB *isn't* an edge case in your picture, but 5GB is.

So in that picture, applications will be written which work fine, and
then some poor user will get upset because the application has a 4GB
file size limit that the app designer didn't think about or care to solve.

I really think by 2020, 5GB strings will be quite common in
applications, including Javascript browser applications.

It's already common to transfer multi-megabyte strings over AJAX requests.

Why have yet another unnecessary hurdle for users and app designers?

Note: There is no problem with having a 32-bit fixed limit on *chunk*
length, invisible to applications.  That doesn't limit anything, it's
just a transport detail and makes life easier there.

Imho, an application message being broken into multiple 32-bit fixed
length chunks is looking good at the moment.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 11:28:47 +0100

> of maximum message size in the handshake.

TCP does something similar with MSS.  In practice, there are routers
which break some TCP ports(!) by fudging their own broken MSS
negotiation into the middle.  If routers can break TCP, I am sure
proxies attempting to fudge their own message size limit into the
middle of a WebSocket negotation is asking for trouble.

Let's try to keep chunks and messages independent in such a way that
proxies only care about chunks as much as they possibly can.
(I.e. WS-aware firewalls will probably still care about message
structure.)

> error handling with large (or unlimited) frames.

You could send TCP URG data on a direct connection, and this is one of
it's most useful uses - avoiding the need to split data streams into
lots of prefixed chunks :-)

But TCP URG isn't handled by most proxies even including "transparent"
(intercepting) TCP relays, so it's a total non-starter.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 11:56:48 +0100

Mike Belshe wrote:
>    On Sun, Apr 18, 2010 at 10:16 PM, Scott Ferguson <[1][email-masked]>
>    wrote:
>      I think chunking for many/most applications will be similar because
>      it's a pretty generic need, like HTTP's chunking is.  In the case
>      of HTTP, many developers don't need to be aware that their output
>      is getting chunked; it just works. I'd expect the same to be true
>      for the simple frame/message model. Assuming it does cover the
>      common cases, it's a useful abstraction to give users.
> 
>    So, given the WebSockets API, how would chunks be used?  The
>    application calls send(DOMString).  There is no need for chunking
>    here, because the full string (and it's length) is already available.
>     Or did I miss something?

The chunking is for the benefit of being able to splice in things like
graceful close, some kinds of error reponse, and any other control
messages as Greg described.

There are other uses too.  Let's use an analogy.  From HTTP (RFC2616):

    8.2.2 Monitoring Connections for Error Status Messages

	  An HTTP/1.1 (or later) client sending a message-body SHOULD monitor
	  the network connection for an error status while it is transmitting
	  the request. If the client sees an error status, it SHOULD
	  immediately cease transmitting the body. If the body is being sent
	  using a "chunked" encoding (section 3.6), a zero length chunk and
	  empty trailer MAY be used to prematurely mark the end of the message.
	  If the body was preceded by a Content-Length header, the client MUST
	  close the connection.

Note the part where a request can be aborted early and the connection
reused _only_ if it was encoded with chunks.  Analogously, it's not
hard to imagine adding an abortMessage() function to some future
version of the WebSocket API, so that applications can abort sending a
large message when their other side has signalled that the message is
not needed.

>      client/server APIs and application code simple for the simple
>      cases. Applications don't need to know about frames/chunks, just
>      messages.
> 
>    KISS, I think, would have solve the problem in the simplest way to
>    support the WebSockets API.
>    framework?  (if so, let's list them!)  I think we're all in
>    disagreement because we haven't agreed on what problem we're solving.
>     If we agreed that the problem we're solving is providing a protocol
>    to support a WebSockets-like API to web developers, we'd likely be
>    done.  Another option would be to say we don't like the current
>    WebSockets API proposal, and offer up an alternative API which we're
>    trying to support (maybe one which does streaming or needs chunking?).
>    But I haven't seen that either.

You seem to keep mixing chunks with the API.  The point of chunks is
to be completely absent from the API, so that the WS implementation
can provide things that are better handled by the WS implementation
itself, so the application doesn't have to.

For example, reception of graceful close permits applications to
safely open a new connection and retry operations.

Without graceful close, applications cannot distinguish network errors
from remote end deciding to close an idle connection(*), which forces
applications to write *more* code to handle a normal situation.(**)

Without transport-level chunking, graceful close is not available in
some situations.

Greg's too-large-error scenario is also particularly interesting
because without handling it, some applications will continuously loop
retrying and failing.

(*) Not closing idle connections isn't an option on servers, because
they can't distinguish idle connections from dead ones, so will
consume all server resources over time if they don't do that.

(**) It could be application's own graceful close timeout messages,
keepalives, or sequence numbers with persistent history, or whatever.
Point is it forces the application to do something tricky, or be fragile.

>    My biggest fear is that a generic protocol will take a very high
>    burden of proof for implementation and deployment because it will
>    offer so many features and that this time will only delay solving the
>    one problem we do know of (fix hanging GET) while not really solving
>    any other specific problem either.  By contrast, if we stay focused on
>    the problem at hand, coming to agreement on the WebSockets protocol
>    seems quite achievable.

The subtext is that WebSockets doesn't solve the problem at hand.

See the recent post from Apache folks saying that WebSockets won't be
deployed by them in its current form because it creates more problems
than it solves.  Especially due to the large number of connections.

In its current form, WebSockets (after the handshake) seems to be a
fair match for the networks of the 80s and early 90s.  These days, you
cannot even rely on a TCP connection being reliable while you are
using it.  Remember that it's trying to go over port 80/443 because of
environments where all other ports are blocked by policy.  Those ports
often have proxies on them - and even on port 443, although they
cannot inspect the plaintext, proxies still expect a HTTP-like
communications pattern or they cut off the connection.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 12:33:49 +0100

> You make a good case... for SCTP :p

Yes, absolutely!

SCTP would be a much better match for the problem.

Meanwhile, we have to work over TCP.

Remember that Google's tests show you can only communicate from many
client sites over:

   - TCP port 80, using HTTP
   - TCP port 443, using CONNECT
   - No other TCP ports.
   - (Not shown in that test, but) no other IP protocols

It would be great to use SCTP when possible, but the basic protocol
has to use TCP if it's to support Web Applications over the Web Using Internet.

If you have a proposal for opportunistically upgrading to SCTP, that
would be great.

Perhaps if my ideas for proxies get taken seriously, we can put the
SCTP upgrade in there, transparent to WebSocket applications :-)

-- Jamie

---

From: Mike Belshe <[email-masked]>
Date: Mon, 19 Apr 2010 08:11:56 -0700

>   - TCP port 443, using CONNECT

Are you talking about the Google WebSocket test?  If so, that is not quite
right.
The Google Websockets test shows that using HTTP over non-port 80 has more
success than HTTP over port 80.  We believe this is due to proxies.

> SCTP upgrade in there, transparent to WebSocket applications :-)

The problem with SCTP is NAT.  NAT services generally only work for TCP and
UDP.  And what percentage of the net uses NAT today?  Everyone.

I suppose SCTP might be deployable those using IPv6.

Mike

---

From: Mike Belshe <[email-masked]>
Date: Mon, 19 Apr 2010 08:25:17 -0700

> error handling with large (or unlimited) frames.

OK - this approach seems like the simplest one; each side sends a
"max-frame-size" in its headers during the upgrade.  These values do not
need to be the same - each side declares the maximum size it is willing to
receive.

If this is the only use case which requires a complex frame header, then
this seems simpler than fragmentation/reassembly/chunking approaches, no?

Mike

---

From: Scott Ferguson <[email-masked]>
Date: Mon, 19 Apr 2010 10:59:03 -0700

> here, because the full string (and it's length) is already available. 
Yes, you forgot the server side :)

Buffering the message on the client is essentially unavoidable and not a 
major problem, because the client is only managing a small number of 
connections. The server is running 100s or possibly 1000s and really 
doesn't want to buffer entire messages before sending it out.
> support the WebSockets API.
The chunking is needed to support the client WebSockets API to receive 
messages.

Since the WebSockets API needs to receive the entire message, not the 
individual frames, the chunking is needed to support the API on the 
receiving end. It's true that the browser client buffers the entire 
response (not the only client, BTW).

In other words, it is the simplest way to support the WebSockets API.
> framework?  (if so, let's list them!) 
Technically, every application will invent its own protocol. 
"tic-tac-toe" uses a tic-tac-toe protocol. XMPP (or HMTP) over 
WebSockets. REST over websockets.  WebSockets isn't a full protocol 
stack, its just a layer (though an important one.)
> the problem at hand, coming to agreement on the WebSockets protocol 
We're talking about a very simple protocol. No one wants to make it 
complicated. Here's essentially the entire frame grammar (for something 
like Martin's proposal.)

  stream ::= message*
  message ::= non-final-frame* final-frame
  non-final-frame ::= (0x4000 | length) <octet-data>
  final-frame ::= (0x0000 | length) <octet-data>

Done.

If we do add the keepalive capabilities, close messages, error 
conditions (and the NAT discussion is convincing me that 
heartbeat/keepalive might be needed), those are a few, enumerated, 
specific packets.

0xff00 - keepalive packet
0xffff - graceful close
0xff01 - negotiation (?) e.g. requesting a server heartbeat/keepalive 
timeout if keepalives are implemented as unidirectional heartbeats.

etc. (I'd think these would be extremely limited in number.)

The concepts are more complicated than the bytes will be.

-- Scott
> I'm relatively new to this group and may be completely off base - so 
> my apologies if I am :-)  But I was at the IETF mtg on the topic, and 
> it didn't seem that we were  in agreement on what we're doing there 
> either...

---

From: Mike Belshe <[email-masked]>
Date: Mon, 19 Apr 2010 12:15:52 -0700

On Mon, Apr 19, 2010 at 10:59 AM, Scott Ferguson <[email-masked]> wrote:

> connections. The server is running 100s or possibly 1000s and really doesn't
> want to buffer entire messages before sending it out.
>
>
> receiving end. It's true that the browser client buffers the entire response
> (not the only client, BTW).
>
> Technically, every application will invent its own protocol. "tic-tac-toe"
> uses a tic-tac-toe protocol. XMPP (or HMTP) over WebSockets. REST over
> websockets.  WebSockets isn't a full protocol stack, its just a layer
> (though an important one.)
>
> complicated. Here's essentially the entire frame grammar (for something like
> Martin's proposal.)
>
>  stream ::= message*

You've convinced me.

Thanks,
Mike

>
> If we do add the keepalive capabilities, close messages, error conditions
> (and the NAT discussion is convincing me that heartbeat/keepalive might be
> needed), those are a few, enumerated, specific packets.
>

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 02:53:39 +0100

Mike Belshe wrote:
> 
>    On Mon, Apr 19, 2010 at 4:33 AM, Jamie Lokier <[1][email-masked]>
>    wrote:
> 
>    Thomson, Martin wrote:
>    Are you talking about the Google WebSocket test?  If so, that is not
>    quite right.
>    The Google Websockets test shows that using HTTP over non-port 80 has
>    more success than HTTP over port 80.  We believe this is due to
>    proxies.

Yes, that test.  Yes it shows more success over non-port 80, but there
is still a significant minority of sites where non-port 80/443 did not work.

That is what I mean by "you can only communicated from many client
sites [...] no other TCP ports".  Because there are many client sites
for which that is true according to the Google WebSocket test.

Sorry if my language was unclear.

>    The problem with SCTP is NAT.  NAT services generally only work for
>    TCP and UDP.  And what percentage of the net uses NAT today?
>    Everyone.  I suppose SCTP might be deployable those using IPv6.

Well, and that problem of sites where non-port 80/443 are unavailable.
I don't see them all letting SCTP through...

NAT is a problem, but there is also something called SPI, "stateful
packet inspection".  That is, firewalls which don't translate
addresses, but still track connections and block unrecognised
connections or those originated in the wrong direction.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 03:03:46 +0100

> implementations and not in the protocol.

One little flaw in the plan:

Won't all lazy implementors just cut and paste the handshaking code
and then put their minimal server code after it?

-- Jamie

---

From: Lars Eggert <[email-masked]>
Date: Tue, 20 Apr 2010 10:58:53 +0300

> (intercepting) TCP relays, so it's a total non-starter.

And the TCPM WG is saying that new applications should not use the urgent mechanism: http://tools.ietf.org/html/draft-ietf-tcpm-urgent-data

Lars

---

From: Jamie Lokier <[email-masked]>
Date: Wed, 21 Apr 2010 00:54:54 +0100

Mike Belshe wrote:
>    "max-frame-size" in its headers during the upgrade.  These values do
>    not need to be the same - each side declares the maximum size it is
>    willing to receive.
>    If this is the only use case which requires a complex frame header,
>    then this seems simpler than fragmentation/reassembly/chunking
>    approaches, no?

A message may be too large to receive for other reasons than exceeding
a fixed length known in advance.

-- Jamie