Subject: [hybi] hum #3: Message

From: Salvatore Loreto <[email-masked]>
Date: Thu, 05 Aug 2010 18:39:25 +0200

based on slide #8 in http://www.ietf.org/proceedings/78/slides/hybi-1.pdf

as Chairs we asked for a hum on the following statements:

    *   "Message" is a protocol unit with an end
    * A message may be composed of one or more "frame"s
    * Each frame has a length indication, encoded in a fixed number of
      bits (where that lengths is fixed in the specification to be written)

There was consensus of the room for the above statements.

The we have also asked  an hum on whether:

    * a message should be made of one or more frames?
    * each message should be composed on precisely one frame

The sense of the room was for the first question: a message should be 
made of one or more frames.

finally as Chairs asked for a hum on the following statements:

    * Should the message has a length?
    * Each frame has a length indication encoded in a fixed number of
      bits (where the length is fixed in the specification to be written).

and it was the room consensus on the fact that that Each frame has a 
length indication.

So unless people will raise good technical reasons against (during the 
next week):
we will declare consensus on them.

Joe Hildebrand and Salvatore Loreto
as chairs

---

From: Ian Hickson <[email-masked]>
Date: Thu, 5 Aug 2010 18:22:24 +0000 (UTC)

>      bits (where that lengths is fixed in the specification to be written)

When transfering large blocks of data, chunking is quite inefficient. This 
means that we'd need to make sure that the fixed width field is at least 
long enough to handle large chunks of data.

At terabit Ethernet speeds, 32 bits is not enough since you could transfer 
that much in mere seconds, so presumably we'll use 64 bits.

Do we really want a length that long? 

The alternative is variable width lengths, which are no more complicated 
to implement than fragmentation, and which do not preclude using 
fragmentation either, and which remain compact at any length.

>    * A message may be composed of one or more "frame"s

I think we should obtain real-world experience with how people use 
WebSockets before we add fragmentation support to the technology. Since it 
can easily be worked around at the application layer (by having 
fragmentation in the subprotocol), we can easily collect information on 
how people are using this after we have initial deployments.

Other relevant e-mails on the subject from other people:

http://www.ietf.org/mail-archive/web/hybi/current/msg02841.html
http://www.ietf.org/mail-archive/web/hybi/current/msg02830.html
http://www.ietf.org/mail-archive/web/hybi/current/msg02892.html

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: John Tamplin <[email-masked]>
Date: Thu, 5 Aug 2010 14:42:03 -0400

> Do we really want a length that long?

Let's assume the most recent proposal, which is a 4-byte header including a
2-byte length field.  So, if you need to send a message of 2^32 bytes, it
will be sent as a 65538 frames, incurring an overhead of 262152 bytes for
header overhead, or a total of 4295229448 bytes with .0006% overhead over
the payload data.  It is true that this is higher overhead than a single
large frame with a variable-length length field, but it hardly rises to
"quite inefficient".  Also, other reasons have been given for why you don't
want to send a single frame that long anyway, and for why you don't want to
have to keep an arbitrarily large buffer in the sender just so you can count
the bytes of the entire message before you send it.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: Douglas Otis <[email-masked]>
Date: Thu, 05 Aug 2010 11:46:59 -0700

On 8/5/10 11:42 AM, John Tamplin wrote:
> On Thu, Aug 5, 2010 at 2:22 PM, Ian Hickson <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> Let's assume the most recent proposal, which is a 4-byte header 
> including a 2-byte length field.  So, if you need to send a message of 
> 2^32 bytes, it will be sent as a 65538 frames, incurring an overhead 
> of 262152 bytes for header overhead, or a total of 4295229448 bytes 
> with .0006% overhead over the payload data.  It is true that this is 
> higher overhead than a single large frame with a variable-length 
> length field, but it hardly rises to "quite inefficient".  Also, other 
> reasons have been given for why you don't want to send a single frame 
> that long anyway, and for why you don't want to have to keep an 
> arbitrarily large buffer in the sender just so you can count the bytes 
> of the entire message before you send it.
+1

-Doug

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 05 Aug 2010 12:04:51 -0700

On Aug 5, 2010, at 11:42 AM, John Tamplin wrote:

> Let's assume the most recent proposal, which is a 4-byte header including a 2-byte length field.  So, if you need to send a message of 2^32 bytes, it will be sent as a 65538 frames, incurring an overhead of 262152 bytes for header overhead, or a total of 4295229448 bytes with .0006% overhead over the payload data.  It is true that this is higher overhead than a single large frame with a variable-length length field, but it hardly rises to "quite inefficient".  Also, other reasons have been given for why you don't want to send a single frame that long anyway, and for why you don't want to have to keep an arbitrarily large buffer in the sender just so you can count the bytes of the entire message before you send it.

In HTTP, it's common to send large entities with a length, even though a chunked encoding is available. In my experience, an explicit length is used much more often than chunked encoding, and I don't believe I've ever seen an HTTP message greater than 2^32 bytes in length sent with chunked transfer-encoding. Since both length up front and chunked encoding are available in HTTP, we can infer a revealed preference for explicit length for HTTP's use cases.

What reason is there to believe that WebSocket will be different? Why would we make the common and preferred way to send large messages over HTTP completely unavailable in WebSocket? That strikes me as hubris.

That being said, I think you're misunderstanding Ian's point. He expressed concern about the overhead for small messages to having a large fixed length, not the overhead to large messages of a chunked encoding.

Regards,
Maciej

---

From: Ian Hickson <[email-masked]>
Date: Thu, 5 Aug 2010 19:31:30 +0000 (UTC)

On Thu, 5 Aug 2010, Maciej Stachowiak wrote:
> 
> That being said, I think you're misunderstanding Ian's point. He 
> expressed concern about the overhead for small messages to having a 
> large fixed length, not the overhead to large messages of a chunked 
> encoding.

I was trying to express both.

Having to split a 2GB file into a bazilion pieces and write each one out 
individually with frame headers, compared to just writing the whole thing 
at once, seems inefficient far beyond simply the concern about the 
on-the-wire overhead.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Aug 2010 21:52:53 +0200

> at once, seems inefficient far beyond simply the concern about the

Hum, where exactly do you see the additional overhead? It's not like you 
need to split it into inidividual *files* - you just interleave the data 
with the frame headers while writing to the stream...

Best regards, Julian

---

From: John Tamplin <[email-masked]>
Date: Thu, 5 Aug 2010 15:53:56 -0400

> at once, seems inefficient far beyond simply the concern about the

Assuming you don't care about buffering requirements on the client and you
only care about the case where you already have the entire data to send in
memory and you don't care about compressing the output, then I agree just
doing write(socket, buf, len) is easier.  However, it isn't much easier
than:

while(len > MAX_FRAME_SIZE) {
  writeHeader(socket, true, MAX_FRAME_SIZE);
  write(socket, buf, MAX_FRAME_SIZE);
  buf+=MAX_FRAME_SIZE;
  len-= MAX_FRAME_SIZE;
}
writeHeader(socket, false, len);
write(socket, buf, len);

Also, you have previously stated that you expect compression to always be
used once we get around to supporting it.  I have given pseudocode
previously of what the compression loop looks like -- if I have to send the
length for everything up front, I have to complete compressing the entire
data before I can send one byte over the wire.  I have to have an
arbitrarily large output buffer (which means reallocating it with multiple
copies), which is inefficient.

Finally, there are cases where the WebSocket server isn't the ultimate
source of the data to send, so requiring a length up-front means buffering
every byte from the source before you can write a single byte on the
WebSocket connection.  If instead I have fragments, I can use a fixed-size
buffer that I never have to reallocate, read the data from the server, and
write individual WebSocket frames as that buffer fills.

Regarding the small-frame case -- a variable-length length field is only
more efficient than fixed length if the payload data is between 0 and 127
bytes.  In that case, the extra byte required seems insignificant to the
TCP/IP overhead for a small packet, where even mobile devices don't use IP
header compression.  Again, this doesn't seem to rise up to "quite
inefficient".

When considering inefficient beyond bytes on the wire, consider these two
receivers:

int len = 0;
int c;
do {
  c = readByte(socket);
  len = (len << 7) + (c & 127);
} while(c & 128);
(re)allocate buf[len]
read(socket, buf, len);

vs

struct WSHeader hdr;
read(socket, &hdr, sizeof(hdr));
fixup byteorder
// buf is preallocated to be the maximum size frame
read(socket, buf, hdr.len);

[ignoring error handling of course]

The former again requires either many allocations/frees of the buffer or
reallocating it as larger frames are seen, and more system calls.

---

From: Jack Moffitt <[email-masked]>
Date: Thu, 5 Aug 2010 13:55:51 -0600

> at once, seems inefficient far beyond simply the concern about the

Processing a 2GB file as a stream (or rather as a sequence of
variously sized chunks) is how it is normally done. You don't read in
a 2GB file in C, and you don't read in such a file in Python either.
You typically read either in fixed size amounts or in varying amounts
in a nonblocking fashion.

Sure, lots of people write code initially that reads everything at
once and writes it at once, but most of that code is thrown away or
changed as soon as the person realizes it as a bottleneck. It will
fail by running out of memory, being extremely slow, or locking up the
system in I/O when processing large files.

I think both concerns are relatively minor.

a) The overhead is pretty small on the wire (though some care should
be taken this applies to many small messages as well). Any framing for
getting to messages so far introduces at least one byte per message of
overhead. I think all the length-based proposals so far are 2 bytes.
If the message is small, it's not really much different except in the
extreme case of 1 byte payloads. If the message is large, 2 bytes
wasted out of every 64k or so hardly seems a high price.

b) The extra work to deal with chunks is already happening (in all but
the most trivial implementations), and is pretty minimal for writing
chunks, and only slightly harder for reading them.

jack.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 05 Aug 2010 22:08:31 +0200

* Maciej Stachowiak wrote:
>In HTTP, it's common to send large entities with a length, even though a
>chunked encoding is available. In my experience, an explicit length is
>used much more often than chunked encoding, and I don't believe I've
>ever seen an HTTP message greater than 2^32 bytes in length sent with
>chunked transfer-encoding. Since both length up front and chunked
>encoding are available in HTTP, we can infer a revealed preference for
>explicit length for HTTP's use cases.
>
>What reason is there to believe that WebSocket will be different?

One reason is that HTTP responses can indicate their end by closing the
connection. I've certainly seen many very large HTTP responses that did
not indicate a length at all (finite and infinite responses). This is
very noticable because the download's progress indicator appears broken.

(I'd also attribute very little value to such observations, for
instance, it is apparently possible to conduct in-depth studies of
billions of web pages without ever encountering a single site that
delivers different content based on the Accept-Language header you send,
even if your employers main web site does do that on its main page. Not
to mention that if you have static resources you wish to add to a view,
you are likely better off sending a HTTP reference than pushing it over
WebSockets each time, so you can benefit from caching.)
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Aug 2010 22:09:19 +0200

On Thu, Aug 05, 2010 at 12:04:51PM -0700, Maciej Stachowiak wrote:
> In HTTP, it's common to send large entities with a length, even though a chunked encoding is available.
> In my experience, an explicit length is used much more often than chunked encoding, 

It is only available in HTTP/1.1, not in 1.0, which is one reason many
software have to implement both methods, and when they want only one
method, they implement the common one (length). But when the length
cannot be determined in advance, they simply don't advertise the length
an use the close mode (check how some compression plugins work, you'll
notice this).

> and I don't believe I've ever seen an HTTP message greater than 2^32 bytes in length sent with chunked transfer-encoding.

Streamed audio or video is sometimes sent with chunked encoding
and can be of infinite length.

In fact, the analogy with HTTP is a proof that we'd better implement
chunked encoding from the beginning, otherwise even if we add it later,
it may get limited use due to compatibility concerns.

Willy

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 05 Aug 2010 22:18:51 +0200

>at once, seems inefficient far beyond simply the concern about the 

In order to determine how efficient the whole process would be, we would
have to factor in the overhead of the underlying protocols. For instance
how widely the IPv6 "jumbo payload" option is available in a couple of
years and how it carries through the networks.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 05 Aug 2010 13:21:25 -0700

On Aug 5, 2010, at 12:52 PM, Julian Reschke wrote:

> Hum, where exactly do you see the additional overhead? It's not like you need to split it into inidividual *files* - you just interleave the data with the frame headers while writing to the stream...

Sending a file from disk to the network without processing it in any way is more efficient than cutting it into slices and interleaving with headers. The IP layer is already going to cut it into slices, in a way that almost certainly won't line up with WebSocket-level fragment boundaries.

It's also more efficient for the receiver if they know the length up front. That way they can determine whether to allocate a large enough buffer, or just start spooling to persistent storage immediately.

Regards,
Maciej

---

From: Julian Reschke <[email-masked]>
Date: Thu, 05 Aug 2010 22:33:45 +0200

> Sending a file from disk to the network without processing it in any way is more efficient than cutting it into slices and interleaving with headers. The IP layer is already going to cut it into sl…

There's no way to "send a file over the network without processing". You 
need to read it into memory, and then write it to the network. Of course 
adding frame headers *is* overhead, but it's really identical or close 
to the overhead on the wire, nothing more.

> It's also more efficient for the receiver if they know the length up front. That way they can determine whether to allocate a large enough buffer, or just start spooling to persistent storage immed…

Absolutely. When the length is known (see video streaming and on-the-fly 
compression). Thus, it might be good to be able to announce the eppected 
size upfront.

Best regards, Julian

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 13:40:01 -0700

On Thu, Aug 5, 2010 at 1:33 PM, Julian Reschke <[email-masked]>wrote:

>> Sending a file from disk to the network without processing it in any way
>> is more efficient than cutting it into slices and interleaving with headers.
>> The IP layer is already going to cut it into slices, in a way that almost
>> certainly won't line up with WebSocket-level fragment boundaries.
>>
>
> adding frame headers *is* overhead, but it's really identical or close to
> the overhead on the wire, nothing more.

Well, that isn't entirely true. There are now kernel interfaces for doing
such things without userspace involvement. I doubt that we're at the level
of DMAing directly from the disk to the network device, so we're probably
hitting main-memory at some point, but there is certainly the potential for
higher thruput using these interfaces. My experience has been that this
theoretical increase isn't yet realized, but it *should* be soon.

-=R

>
>  It's also more efficient for the receiver if they know the length up
>> front. That way they can determine whether to allocate a large enough
>> buffer, or just start spooling to persistent storage immediately.
>>
>
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 05 Aug 2010 13:40:16 -0700

> an use the close mode (check how some compression plugins work, you'll

All browsers have supported HTTP/1.1 for over a decade. For content that's primarily aimed at delivery via a browser, I doubt compatibility is the main reason for avoiding chunking. I think it's simply because it is easier on the sender when they know the size up front, which tends to be the case for gigantic resources. Resources that are completely dynamically generated tend to be relatively small in size - multi-gigabyte dynamically generated resources are quite rare.

> it may get limited use due to compatibility concerns.

You are correct that chunking has useful use cases (although the "indefinite length stream" use case is one that can't be supported by the WebSocket API). But for the use case of "giant piece of static content that exists as a file on disk", I bet servers would still prefer to use an explicit length instead of chunked transfer-encoding.

My conclusion so far remains that we need large lengths even if we also have chunking.

Question for the server people here: for those of you serving large, fixed-length files over HTTP (for example software packages in a dmg or zip file, or large video or audio files), would you prefer to use HTTP chunked transfer-encoding if you could be sure of the compatibility? I haven't heard this particular form of feedback in my conversation with server teams that are serving large files, but I'm not that much of an expert on the server side of things.

Regards,
Maciej

---

From: Jack Moffitt <[email-masked]>
Date: Thu, 5 Aug 2010 14:43:03 -0600

> the overhead on the wire, nothing more.

 The kernel's sendfile() is responsible for a lot of the HTTP server
optimization for static resources, and for this case, I think it is
true that doing any framing that shows up in the middle of the file
would lead to a big inefficiency.

I'm not sure how useful this pattern is with dynamic data though. I
think people will continue to send most static resources via regular
HTTP because it is very, very good at it.

jack.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Aug 2010 22:55:38 +0200

On Thu, Aug 05, 2010 at 01:40:16PM -0700, Maciej Stachowiak wrote:
> All browsers have supported HTTP/1.1 for over a decade. For content that's
> primarily aimed at delivery via a browser, I doubt compatibility is the main
> reason for avoiding chunking. I think it's simply because it is easier on the
> sender when they know the size up front, which tends to be the case for
> gigantic resources.

obviously it's a lot easier (and cheaper) when you know the size in
advance, but sometimes (eg: compression) you simply don't know.

> You are correct that chunking has useful use cases (although the "indefinite length stream" use case is one that can't be supported by the WebSocket API). But for the use case of "giant piece of st…

yes in fact I remember when I saw chunk on large files, it precisely was
on DVD images. Those are generally more than 2^32 bytes and you never
know how that large a content length will be parsed by browsers and
intermediaries. At least chunked encoding solves that.

> Question for the server people here: for those of you serving large, fixed-length files over HTTP (for example software packages in a dmg or zip file, or large video or audio files), would you pref…

In my opinion as long as you can, you'd prefer to use content-length
because it's easier and causes less work on the server. But as I said
above it's not always possible.

Regards,
Willy

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 22:04:59 +0100

> HTTP because it is very, very good at it.

Right - the only case it'll be faster over a WebSocket is where the  
latency of a new TCP connection (to transport the HTTP) outweighs the  
transfer time. Full-on mega sneakiness could cause even that to go.

I think a key reason why chunking hasn't caught on in the HTTP space  
is actually because - since it didn't exist to begin with - it's  
never gained the momentum. When things are available and supported by  
a protocol as the "norm" from the get-go, you tend to see that  
feature used much more.

Otherwise, you'd be perfectly entitled not all that long ago, to say  
that virtually all binary data was base64 encoded anyway, so we  
didn't need 8-bit capable protocols. But FTP, to an extent, and HTTP  
more so, meant that binary data became expected there. Yet it's still  
a rarity to see it in email, where it was added later (yet still  
years and years ago).

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 05 Aug 2010 14:06:54 -0700

On Aug 5, 2010, at 1:33 PM, Julian Reschke wrote:

> There's no way to "send a file over the network without processing". You need to read it into memory, and then write it to the network. Of course adding frame headers *is* overhead, but it's really identical or close to the overhead on the wire, nothing more.

You don't necessarily have to read it into memory in user space. Two ways to avoid it:
(1) sendfile() or similar system calls will just copy from a file descriptor directly to network output buffers, in kernel space.
(2) You can mmap() the file and then write() the whole thing - again all the reading happens in kernel space.

> Absolutely. When the length is known (see video streaming and on-the-fly compression). Thus, it might be good to be able to announce the eppected size upfront.

After thinking about it more, I think the ability to send the size up front, and perhaps even a default expectation that the size is provided whenever possible, is more important than whether the transfer uses a single WebSocket frame or multiple.

Regards,
Maciej

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 05 Aug 2010 14:09:01 -0700

> HTTP because it is very, very good at it.

I think there's definitely use cases for transferring large static resources over WebSocket. Consider syncing a media library where items may have been added on either end.  Sync seems like a pretty likely target for WebSocket. You will want to do large static transfers in both directions, and it seems better to use the existing warmed-up WebSocket TCP connection than to make fresh HTTP connections.

Regards,
Maciej

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 22:09:26 +0100

On Thu Aug  5 22:06:54 2010, Maciej Stachowiak wrote:
> After thinking about it more, I think the ability to send the size  
> up front, and perhaps even a default expectation that the size is  
> provided whenever possible, is more important than whether the  
> transfer uses a single WebSocket frame or multiple.

FWIW, my [Really Great] framing proposal doesn't preclude this. You'd  
shove such data into the first frame as extension data, and  
intermediaries which didn't understand that would ignore it.

So yay, even that is taken care of.

Dave.

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 05 Aug 2010 14:18:01 -0700

> I think there's definitely use cases for transferring large static resources over WebSocket. Consider syncing a media library where items may have been added on either end.  Sync seems like a prett…

If it's an optional extension that intermediaries could ignore, that makes it unlikely that there will be a default expectation that it should be sent and have a correct value.

Regards,
Maciej

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 22:25:00 +0100

On Thu Aug  5 22:18:01 2010, Maciej Stachowiak wrote:
> > I think there's definitely use cases for transferring large  
> static resources over WebSocket. Consider syncing a media library  
> where items may have been added on either end.  Sync seems like a  
> pretty likely target for WebSocket. You will want to do large  
> static transfers in both directions, and it seems better to use the  
> existing warmed-up WebSocket TCP connection than to make fresh HTTP  
> connections.
> 
> If it's an optional extension that intermediaries could ignore,  
> that makes it unlikely that there will be a default expectation  
> that it should be sent and have a correct value.

I'm fine withdefining the extension in the core spec, and making this  
a SHOULD. I can see cases meaning it cannot be a MUST. Would this  
satisfy?

Dave.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 07:50:31 +1000

Eitherway, I'd like to express strong support for the main parts of the hum:

    *  "Message" is a protocol unit with an end
    * A message may be composed of one or more "frame"s
    * Each frame has a length indication

But I agree with everybody about the arguments for and against the
various length encodings. I think the reason is that everybody is
right!

Sometimes it is more efficient to send big content without chunking
it.  eg in Jetty we use the NIO gather write mechanism to send a
header from a memory buffer with a file mapped buffer of content.
This means the file is sent to the network interface without going via
user space memory.
But then I also very very rarely see any server side developers set
the content-length of dynamic content, so that is almost always best
sent as chunks.  You can have some very large dynamic content coming
out of DB selects etc.

I think the best solution is  fragmentation plus a variable length
encoding since this allows:

 + chunking of dynamic content
 + sending of big content as a single frame
 + minimal cost in bytes for the most frequent little messages.
 + having fragments in the base protocol means that all extensions
must handle them and thus it makes it easier to add extensions like
multiplexing without breaking other things.

The main down side I can think of is that it makes it harder to word
align header and data, but then I don't see a large demand for that.

There is some extra complexity on the receiver for fragments, but I do
not think it is significant.

regards

On 6 August 2010 07:25, Dave Cridland <[email-masked]> wrote:
>> > I think there's definitely use cases for transferring large static
>> > resources over WebSocket. Consider syncing a media library where items may
>> > have been added on either end.  Sync seems like a pretty likely target for
>> > WebSocket. You will want to do large static transfers in both directions,
>> > and it seems better to use the existing warmed-up WebSocket TCP connection
>> > than to make fresh HTTP connections.
>>
>> If it's an optional extension that intermediaries could ignore, that makes
>> it unlikely that there will be a default expectation that it should be sent
>> and have a correct value.
>
> I'm fine withdefining the extension in the core spec, and making this a
> SHOULD. I can see cases meaning it cannot be a MUST. Would this satisfy?
>
> Dave.
> --

---

From: Dave Cridland <[email-masked]>
Date: Thu, 05 Aug 2010 23:06:56 +0100

> align header and data, but then I don't see a large demand for that.

Fixed width fields obviously allow that. Additionally, they're  
simpler to encode and decode by a very large margin.

Finally, it means that you can build very dumb intemediaries which  
read in the frame header in a single network read(), check the stream  
identifier (for instance), and then hardware copy the payload between  
interfaces. Fast reliable demultiplexing and other front-ending seems  
like quite compelling to me.

Dave.

---

From: Jack Moffitt <[email-masked]>
Date: Thu, 5 Aug 2010 16:17:32 -0600

> I think there's definitely use cases for transferring large static resources over WebSocket. Consider syncing a media library where items may have been added on either end.  Sync seems like a prett…

After doing some research, it appears that sendfile() and its ilk use
ranges anyway, and that many implementations already chunk them (nginx
seems to do so first to get to a sector boundary, and then at a size
of 2G-1). Losing sendfile() completely would be bad, but even in the
chunked case, you can still do sendfile() for each chunk. Of course,
more calls to sendfile() are worse than fewer, but I imagine most of
the efficiency is retained as all memory copies in userland are still
eliminated.

I don't have the data on the difference, but if it comes down to this
particular issue being the deciding factor for consensus, such data
can be obtained.

With this in mind, I'm in favor of fixed length headers, which implies
chunking and multiple frames per message in some cases.

jack.

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 5 Aug 2010 16:59:43 -0700

The first hum I think may hopefully be slightly less contentious if people
read it carefully? The hum was not agnostic as to whether we use fixed or
variable length encoding. The hum was whether a message is composed of one
or more frames. In some proposals, e.g. Proposal V [1], a message could be
either one frame (of variable length) or multiple frames, depending on what
makes sense in terms of efficiency. If it's more efficient for the client to
just sendfile() in a single frame and it's not interested in things like
multiplexing, that's supported, via a variable length "length". If the
server is generating data dynamically and wants to send it as it becomes
available to reduce latency, that's also supported. It does require that
clients be aware of fragmentation and able to receive fragmented data, but
does not require the client to send fragmented data. Thus, I would hope that
if people are willing to accept "must be able to receive fragmented data but
are not required to send fragmented data" they could agree on that as a
middle ground, which is in line with the first hum.

[1] http://www.ietf.org/mail-archive/web/hybi/current/msg02924.html

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Thu, 5 Aug 2010 17:00:20 -0700

Sigh, perils of editing. Should read "The hum _was_ agnostic as to whether
we use fixed or variable length encoding."

2010/8/5 Ian Fette (イアンフェッティ) <[email-masked]>

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 01:49:07 +0100

Jack Moffitt wrote:
> > resources over WebSocket. Consider syncing a media library where
> > items may have been added on either end.  Sync seems like a pretty
> > likely target for WebSocket. You will want to do large static
> > transfers in both directions, and it seems better to use the
> > existing warmed-up WebSocket TCP connection than to make fresh
> > HTTP connections.

Also dynamically generated but cacheable in process messages, such as
HTML fragments and DB query results.  The memory for those dynamic
messages can be shared among sockets with sendfile(), or it's
relative, vmsplice().

> of 2G-1).

Note: 2G-1, not every few kB!

> Losing sendfile() completely would be bad, but even in the chunked
> case, you can still do sendfile() for each chunk. Of course, more
> calls to sendfile() are worse than fewer, but I imagine most of the
> efficiency is retained as all memory copies in userland are still
> eliminated.

It's time to explain sendfile() better.

0. Nearly all OSes used for web serving have something called
   sendfile() or under a different name.  It has become a standard
   method used in high performance static servers, and is beginning to
   be used in dynamic servers that do smart caching.  It is given an
   open file handle and range, and writes content of that file to a
   TCP socket as efficiently as possible.  Some OSes allow contents of
   process memory to be sent as well.

1. The overhead avoided using sendfile() is *not* just the copy.

   It's also the *memory shared* between thousands of sockets
   transmitting the same content from filesystem cache, db cache, even
   dynamic content cache.

2. It's not so much the act of copying, as avoiding moving data to and
   from the CPU.  Typically the data goes direct from disk -> RAM ->
   network, without the CPU reading and writing it. This reduces
   memory bus load and also frees up CPU cache.

3. Good network interface cards (NICs) allow the data for a packet to
   be given to the hardware as a TCP/IP header (generated by the
   kernel) and a single block of data - the data is checksummed by the
   NIC itself, and the NIC inserts it into the TCP/IP header, in hardware.

   If the data part is the result of using sendfile(), the data is
   copied (DMA'd) directly from RAM to the network without involving
   the CPU, freeing up CPU resources including cache, time, and
   checksum calculation.

4. With most NICs, it's only possible to perform the above hardware
   optimisation if there's _one_ data block following the TCP/IP header.

   This means lots of small sendfile() chunks, whose boundaries lie in
   the middle of TCP/IP packets, _prevent_ this hardware optimisation
   in those packets, forcing the CPU to copy the data itself, and
   preventing the memory sharing.

5. Better network cards do "TCP segmentation offload".  This is where
   the card is given a very large "virtual packet", and splits it up
   into the smaller ones actually needed for TCP/IP, adding headers
   itself to each packet.

   In order to fully utilise this hardware optimisation, the
   sendfile() chunks need to be somewhat larger than the virtual
   packet size, as opposed to the real packet size.

In a nutshell: If sendfile() is used with small chunks, interleaved
with other bytes, it's relatively inefficient compared with using
large chunks.  This is not because of memory copies.  It's because of
the way network hardware works, especially better hardware in servers,
combined with unpredictable TCP boundaries.

We do need to interleave and split sometimes, but we should also leave
room for sending large chunks efficiently, when a socket is not shared.

I suspect (but don't know) that a 64kB-1 maximum chunk size is too
small for best sendfile() performance on some current hardware, and
would push for at least 24 bits (or so) of size, if it's going to be a
fixed size field.

(Thinking of the future we'll probably be moving large TCP packets around
at some point.  100Gbit ethernet is just around the corner, and 1Gbit
home broadband is already available in some countries.)

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 01:55:34 +0100

> like quite compelling to me.

+1.  All sensible multiplexing puts the addressing and length at the
beginning to enable cut-through forwarding without having to examine
the data itself, and allows errors to be signalled by the forwarder
during sending, in case it receives an error in the middle.

(Cut-through means forwarding data as it arrives, without waiting for
all of it first.)

If large chunks are permitted, you *must* support cut-through
forwarding otherwise the memory required for buffering is large too,
as is the forwarding latency.

-- Jamie

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 5 Aug 2010 18:09:34 -0700

>   process memory to be sent as well.

Unfortunately, you should also be aware that nearly all OSes have
severe kernel bugs in their implementations of sendfile and seem
to be getting progressively worse over time, to the point where
Apache httpd needs to disable it by default.  I don't know why.

....Roy

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 02:20:45 +0100

> multiplexing without breaking other things.

I agree.  Being able to send big content as a single frame is
certainly simpler than having to split.

And I think that's likely to result in fewer buggy implementations
than a fixed header.  If there's a small maximum chunk, poor
implementations may implement send-splitting wrong, but go undetected
for years if most applications use messages that fit in a single
chunk.  Whereas it's hard to screw up a variable length encoding, when
you have to get it right for common usages.

I don't think multiplexing is helped or hindered by having fragments
in the base protocol.  A multiplexing implementation is the same
either way.  Proxying between a multiplexed and non-multiplexed stream
is much the same either way.

But I notice fragments are necessary in any situation where you might
want to:

   + abort a message in progress
   + inject a control message

For example when forwarding a large message, if an error is detected
by the forwarder, it might want to abort the message and follow it by
a control message describing the error it has received and start an
orderly close.

Without fragments, the only thing a forwarder could do would be to
close the socket, which gives very little information, not even saying
if it's the forwarder or the earlier sender that failed.

> align header and data, but then I don't see a large demand for that.

I don't think there is any practical overhead to frame header parsing
on modern hardware.  All variations we've talked about are extremely
simple.  However, there may be overhead in moving the data to some
place that benefits from alignment.  Even simple copying can benefit.

   + Answer: Allow a 1-byte padding message.  Let's call it 0x00 for
     simplicity.  Senders can pad if they think it'll improve efficiency.
     Receivers ignore it.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 02:28:45 +0100

> Apache httpd needs to disable it by default.  I don't know why.

Thanks.  Ugh.  (I would like to know more, if you can point me at the details.)

For WebSocket, details of specific current implementations aren't so
important, as acknowledging the principles that hardware may continue
to imply for some years, in those rare cases of OSes / kernels that
aren't too buggy to take advantage :-)

I.e. it just means "don't mandate that chunks must be small".
It's not asking much.

Who knows, we might end up with WebSocket multiplexing implemented in
a kernel, which is likely to avoid sendfile() bugs but still be
subject to NIC constraints.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 11:41:51 +1000

> Thanks.  Ugh.  (I would like to know more, if you can point me at the details.)

For jetty using java's file mapped buffers, we've encountered 2 key
problems that cause us to turn it off.
Firstly because the file is mapped to address space, even if it is not
actually loaded into user space, there was a problem with leaking
address space. Once mapped the address range was never recovered, even
if the buffer was discarded. It was possible (on 32 bit machines) to
consume all the available address space by repeatedly creating and
sending large files.  I'm not sure if this was an OS flaw or just the
way the JVMs expose the feature as a memory mapped buffer.
We also have issues with the windows implementation which causes the
file system to lock the file if it is mapped.  This can prevent hot
redeploys and efficient development cycles.

But even if the send file semantic is not always available, it is a
big gain when it is available.  So having support for large frames is
good.

> I.e. it just means "don't mandate that chunks must be small".

So is 16 exebytes enough for you (see Ian's email).
Not sure how I'm going to write a unit test for that one:)

cheers

---

From: Roberto Peon <[email-masked]>
Date: Thu, 5 Aug 2010 18:47:48 -0700

Thus my statement of "theoretical" benefits.
I've not seen the benefits of either splice or sendfile yet.
-=R

---

From: Scott Ferguson <[email-masked]>
Date: Thu, 05 Aug 2010 19:12:26 -0700

> long enough to handle large chunks of data.

Instead of guessing, I measured chunking vs non-chunking for large 
(16meg) HTTP requests (10000 requests measured by ab on localhost, 
single client) using Resin.

The biggest difference I saw was 3% after several runs, most were within 
1%, and in some cases the chunked beat length (yay GC.)

That was HTTP chunking (8 bytes with 4 encoded as ascii hex), a 8k 
chunk/buffer size, and zero application time (write a pre-filled 64k 
buffer 256 times). In other words, a bigger overhead than the proposed 
frame header.

In other words, the overhead of chunking is so small that it's barely 
measurable, even in the most extreme case.

-- Scott

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 03:43:16 +0100

> problems that cause us to turn it off.

Little note: Mapped buffers aren't sendfile (unless they aren't what
they sound like; I don't know Java too well).  Assuming they're mmap()
plus send(), they are a different technique that achieves _some_ of
the same benefits.  Mapped buffers reduce memory use and some copying,
but don't result in sockets sharing the same memory, nor eliminate
copying, nor are they affected in the same way by NIC hardware
acceleration constraints - unless mapped buffers can combine forces with AIO
(which certainly isn't possible on Linux, at least).

> way the JVMs expose the feature as a memory mapped buffer.

That sounds like a silly JVM or NIO bug.  Address space is recovered
when a file is unmapped at the OS level.  At least, in the OSes I know
about :-)

> redeploys and efficient development cycles.

Windows locks a file just for being open.  This isn't really a
sendfile/mapped buffer issue, so much as a "keeping a file open" issue.

> big gain when it is available.  So having support for large frames is

For a different perspective, see Roberto Peon's message which has the
same thread parent as yours - he says there's no gain from sendfile.

I don't think we should revolve around current implementation
issues/bugs, though, so much as what is realistic to encounter in the
years to come.

> Not sure how I'm going to write a unit test for that one:)

(24-bit is enough for me, from the lowlevel sendfile-ish perspective).

Ian's email pleases me.  63/64-bit sounds good to me.  Separate thread coming.

-- Jamie

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 6 Aug 2010 14:18:46 +1000

> (which certainly isn't possible on Linux, at least).

this is a little off topic, but...

The JVM API allows for memory map buffers to be sent to a channel and
it is meant to use sendfile like mechanisms so that the file content
is sent directly to the interface device.

I think the need to memory map to a buffer is a side effect of the
java API, which is all about channels and buffers and totally hides
file descriptors.  It may be that in this case java uses the memory
mapped buffer only as a holder of a file descriptor plus offset and
length.

> same thread parent as yours - he says there's no gain from sendfile.

Probably the biggest benefit from the java point of view is not being
able to drive the network interface at maximum speed, but rather
avoiding using user memory and the garbage collects that result.

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 6 Aug 2010 08:29:28 +0200

> I've not seen the benefits of either splice or sendfile yet.

At least I've seen it. Haproxy happily forwards traffic at 10 Gbps
with only 25% of one CPU core on a 2.66 GHz Core2Duo using splice().
And some file sharing sites make a big use of it ! Without splice,
it hardly achieves 10 Gbps at 100% CPU.

Anyway, it's unclear to me why we would need to send large amounts
of data over WebSockets. File serving could well be performed by
the web server itself.

Regards,
Willy

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 06 Aug 2010 02:08:28 -0700

> of data over WebSockets. File serving could well be performed by

I already cited the use case of sync. It's likely that in a case like filesystem sync or media sync, large data objects will be sent in both directions, and it will likely be advantageous to send them over WebSocket rather than using a separate HTTP exchange for each file transfer.

Regards,
Maciej

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 6 Aug 2010 11:11:10 +0200

> I already cited the use case of sync. It's likely that in a case like filesystem sync or media sync, large data objects will be sent in both directions, and it will likely be advantageous to send t…

Ah that's a good example. I didn't see your mail, but I agree that it
would be a suited case of massive data exchange.

Regards,
Willy

---

From: Pieter Hintjens <[email-masked]>
Date: Fri, 6 Aug 2010 11:15:24 +0200

> It's also more efficient for the receiver if they know the length up front. That way they can determine whether to allocate a large enough buffer, or just start spooling to persistent storage immed…

Agree with both these points.  If a sender does fragment a large
message, they ALSO need to specify the full size up front, for the
same reasons we specify the frame size up front.  You want to avoid
sentinel-based reads.

Content framing sounds simple but is actually one of those things that
gets complex to implement.

There is a very simple solution that addresses both efficiency for
small messages, and easy of framing for very large messages.

Rather than a fixed two-byte length, use a 1-byte length which escapes
into an 8-byte length for frames longer than 254 bytes.  Small
messages, which are the most important for high latency and high
frequency scenarios, are optimally encoded.  Long messages are most
trivially encoded.

If you really want a 4-byte header, use a two-byte length that escapes
into an 8-byte length.

I'd strongly advise, based on experience, to avoid frame fragmentation
as a way of encoding long frames.  Multiple frames per message IS
valuable but for other reasons (multipart).

-Pieter

---

From: Arman Djusupov <[email-masked]>
Date: Fri, 06 Aug 2010 14:26:22 +0300

> same reasons we specify the frame size up front.  You want to avoid

If the sender knows the size of the message upfront then there is no point to fragment it. The sender can just specify the total length of the message and keep pumping data into the connection, while the reader can count the received bytes and so know where the message ends. The whole reason of having fragmentation and framing is to remove the need to know the full size of the message prior to sending it. 

Arman

---

From: Pieter Hintjens <[email-masked]>
Date: Fri, 6 Aug 2010 13:35:22 +0200

I don't think this is accurate. Framing is to allow the reading of messages
by a layer that does not need to know their semantics. Also to allow
preallocation and protection against overflows. Fragmentation is to allow
long messages to be handled in smaller buffers. I think in all cases we can
assume the sender knows the length upfront.

- Pieter

On Aug 6, 2010 1:26 PM, "Arman Djusupov" <[email-masked]> wrote:

Pieter Hintjens wrote:
> messag...
If the sender knows the size of the message upfront then there is no point
to fragment it. The sender can just specify the total length of the message
and keep pumping data into the connection, while the reader can count the
received bytes and so know where the message ends. The whole reason of
having fragmentation and framing is to remove the need to know the full size
of the message prior to sending it.
Arman

---

From: Julian Reschke <[email-masked]>
Date: Fri, 06 Aug 2010 13:45:31 +0200

On 06.08.2010 13:26, Arman Djusupov wrote:
> If the sender knows the size of the message upfront then there is no
> point to fragment it. The sender can just specify the total length of
> the message and keep pumping data into the connection, while the reader
> can count the received bytes and so know where the message ends. The
> whole reason of having fragmentation and framing is to remove the need
> to know the full size of the message prior to sending it.
> Arman

There are more reasons. Just stating the length upfront and then sending 
the data as-is does not allow signaling an error when something bad 
happens (except by dropping the connection). This requires chunking, or 
some other way to signal the error condition.

Best regards, Julian

---

From: Arman Djusupov <[email-masked]>
Date: Fri, 06 Aug 2010 16:02:14 +0300

> assume the sender knows the length upfront.

Preallocating resources for the entire message is not always needed in order to consume it (e.g. forward only XML parsers can consume/transform GBs of XML data using a buffer of a few KBs). The same applies to chunking which does not require a single frame to fit into the input buffer since the implementation can read frames of arbitrary size using whatever input buffer is available. So the size of the frame/message is not always related to the size of the buffer.

We cannot assume that the sender necessarily knows the length of the data beforehand because there are cases when responses are getting generated on the fly by the server side script/code. In such cases the server side cannot predict the final size of the message unless it already has the response cached or buffered. So it either spends time waiting for the full response to be generated or starts sending it right way by using the chunked transfer encoding thus saving time and memory.

Arman

---

From: Jack Moffitt <[email-masked]>
Date: Fri, 6 Aug 2010 07:49:41 -0600

> it hardly achieves 10 Gbps at 100% CPU.

It's not clear how much of the benefit we'd lose with fixed size
lengths (and therefore mandatory chunking). At this point, it seems
that kind of data would be valuable.

The tradeoff *seems* to be simple intermediaries versus
sendfile()/splice() efficiency. If sendfile() is severely affected, it
may make sense to make the intermediaries life slightly more
complicated. If not, I think we keep their job simple.

jack.

---

From: John Tamplin <[email-masked]>
Date: Fri, 6 Aug 2010 10:17:19 -0400

Ok, does this summarize the various positions regarding chunking and
lengths?

   - We want to have a length of the complete message when it is available
   to make buffer management simpler for the receiver
   - We want to be able to fragment messages to avoid extra buffering when
   the length of the entire message is not known up front, such as when sending
   dynamic content, compressing frames, etc.
   - We want to keep small frames having small overhead (though there is a
   limit on how much this matters given TCP/IP overhead)
   - We want to be able to send very large files in a single fragment so we
   can just hand it off to sendfile or equivalent

What about the following:

   - 1 bit - Initial fragment
   - 1 bit - Final fragment
   - 1 bit - reserved
   - 5 bits - opcode
   - 8 bits - short length
   - [if length = 255: 8 bytes - length]
   - [if Initial = 1 and Final = 0: 8 bytes - overall message length, with 0
   meaning "unknown"]
   - length bytes of payload

This could be extended with any of the extension or padding proposals out
there, just trying to get the basic idea of having initial/fragment bits
with an optional overall message length up for discussion.  The length could
be a variable-length integer as in v76 if you prefer, likewise for the
message length.

---

From: Arman Djusupov <[email-masked]>
Date: Fri, 06 Aug 2010 17:59:13 +0300

>    can just hand it off to sendfile or equivalent

Yes.

> be a variable-length integer as in v76 if you prefer, likewise for the

I would agree with such a proposal. 

Arman

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 06 Aug 2010 08:39:27 -0700

John Tamplin wrote:
>     * 1 bit - Initial fragment
>     * 1 bit - Final fragment
>     * 1 bit - reserved
>     * 5 bits - opcode
>     * 8 bits - short length
>     * [if length = 255: 8 bytes - length]
>     * [if Initial = 1 and Final = 0: 8 bytes - overall message length,
>       with 0 meaning "unknown"]
>     * length bytes of payload
>
> This could be extended with any of the extension or padding proposals 
> out there, just trying to get the basic idea of having 
> initial/fragment bits with an optional overall message length up for 
> discussion.  The length could be a variable-length integer as in v76 
> if you prefer, likewise for the message length.
+1

(The initial fragment bit still looks redundant to me, but it's harmless.)

-- Scott
> ------------------------------------------------------------------------
>

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 06 Aug 2010 08:49:03 -0700

Arman Djusupov wrote:
> Preallocating resources for the entire message is not always needed in 
> order to consume it (e.g. forward only XML parsers can 
> consume/transform GBs of XML data using a buffer of a few KBs). The 
> same applies to chunking which does not require a single frame to fit 
> into the input buffer since the implementation can read frames of 
> arbitrary size using whatever input buffer is available. So the size 
> of the frame/message is not always related to the size of the buffer.
>
> We cannot assume that the sender necessarily knows the length of the 
> data beforehand because there are cases when responses are getting 
> generated on the fly by the server side script/code. In such cases the 
> server side cannot predict the final size of the message unless it 
> already has the response cached or buffered. So it either spends time 
> waiting for the full response to be generated or starts sending it 
> right way by using the chunked transfer encoding thus saving time and 
> memory.

Exactly.

Chunking also helps browser implementors with their current websocket 
API because their string needs to be serialized to UTF-8.

With chunking, you can serialize a large UTF-8 string with a fixed 
buffer in chunks using one pass, and then fill in the length when you're 
done.

Without chunking, you either need an extra pass over the string to 
calculate its length, or need to preallocate a buffer 3 times the 
character-length of the string.

-- Scott

---

From: Patrick McManus <[email-masked]>
Date: Fri, 06 Aug 2010 10:56:40 -0500

> it may get limited use due to compatibility concerns.

++1

the concern of not being able to express a large length when it is known 
(i.e. a known value > 16 bits) should be addressed through an option. It 
is useful optional information - on a large message the overhead of 
adding it in an option is inconsequential.

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 06 Aug 2010 09:01:35 -0700

>     * length bytes of payload

As a practical matter, a 16-bit short length would be better because 
buffer sizes are typically in the 1k to 64k range, and a 16-bit length 
lets the implementation pre-allocate the frame header for any message/chunk.

With an 8-bit length, the implementation either needs to shift bytes in 
the buffer when the message is 255 bytes long or it needs to preallocate 
the long frame header.

-- Scott

---

From: Patrick McManus <[email-masked]>
Date: Fri, 06 Aug 2010 11:07:39 -0500

> (2) You can mmap() the file and then write() the whole thing - again all the reading happens in kernel space.

I think zero copy is going to be very important in the semi-near future. 
But chunking doesn't mean you cannot effectively use it. A ZC write API 
can have a ptr and a length just like a copy based write does. As long 
as the chunk size is large enough (and 64KB is probably the minimum 
value of large enough) you can certainly still use zero copy techniques 
to send ranges.. there is no fundamental mismatch there even if ZC isn't 
ready for prime time everywhere today.

If there are a couple bits left over in the frame header, it might cover 
a lot of common message sizes to extend the max chunk size to 256KB.

---

From: Roberto Peon <[email-masked]>
Date: Fri, 6 Aug 2010 09:33:28 -0700

sendfile() doesn't require sending the entire file.
At some frame size, the overhead due to interleaving sendfile() with write()
should be negligible.
I don't think that sendfile() is a good reason to require total message
length up-front.
I still like having the total-message-length whenever possible-- it does
allow for a quick decision about where to store or process the data.
-=R

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 6 Aug 2010 19:01:26 +0100

> of data over WebSockets. File serving could well be performed by

Some examples:

  - You've got a web app which carries user-generated short messages,
    that should not be fragile when a user decides to send a long one.
    For example an IM client, when a user tries cutting-and-pasting 20
    pages of text (70kB) into an entry, there is no good reason for
    that to crash the app.  (Certainly not just because of a WebSocket
    limit being hit.)  Yes, users occasionally do this sort of thing.

  - Dynamic HTML updates sent from the server (expect lots of this in
    web apps).  Live wiki editing, shared documents etc.  They send
    HTML fragment update messages to update the browser.  Normally
    they'll be small, but under various circumstances (including
    temporary network outage so the user queues up a lots of deltas),
    an HTML update may occasionally be larger than usual.  It should
    just work, without invoking rarely-used code paths.

  - The old IM again, with inline smileys and avatars.  If you need to
    get the latency down you may encode these small images into an
    update message.  They're small, but vary in size and may
    occasionally exceed some arbitrary small limit, if there is one.

In a nutshell: The WS API should not hold surprises, cause
unnecessarily fragility in the apps using it, or require apps to have
exceptional code paths, where that's easy to avoid.

-- Jamie

---

From: Dave Cridland <[email-masked]>
Date: Fri, 06 Aug 2010 21:19:44 +0100

>    - length bytes of payload

I'm fine with this.

It feels like a solid starting point.

Dave.

---

From: Dave Cridland <[email-masked]>
Date: Fri, 06 Aug 2010 21:26:49 +0100

On Fri Aug  6 17:01:35 2010, Scott Ferguson wrote:
> As a practical matter, a 16-bit short length would be better  
> because buffer sizes are typically in the 1k to 64k range, and a  
> 16-bit length lets the implementation pre-allocate the frame header  
> for any message/chunk.
> 
> 
I'm happier with the "short" being 255 octets, because the only case  
we really need to minimize overhead is in mobile cases, where the  
power levels possible by transferring packets <128 octets are  
substantial and important.

I'm not wed to this, mind.

A good argument in favour is that it could be expected that more  
messages would be on both sides of the division with a 16-bit short  
length width. I'm not sure the predicate is true though.

> With an 8-bit length, the implementation either needs to shift  
> bytes in the buffer when the message is 255 bytes long or it needs  
> to preallocate the long frame header.

Indeed. I'm told, however, that this is not such a big deal. FWIW,  
unaliagned access to wrong-endian data is something I have experience  
of, and as far as I can tell, is not a substantial overhead.

Dave.

---

From: Douglas Otis <[email-masked]>
Date: Fri, 06 Aug 2010 13:44:40 -0700

+1

-Doug

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 06 Aug 2010 13:52:40 -0700

> complicated. If not, I think we keep their job simple.

It's not only the sender and intermediaries that are affected by mandatory chunking. The receiver is affected too. In the case of the browser as receiver, it is highly valuable to have the length up front, so the browser can either pre-allocate a memory buffer of the appropriate size, or decide to start spooling to disk right away if it knows the message will be too large to fit in memory. I imagine this will be useful in at least some cases of the server as receiver too.

Let me note also, that even without chunking at the WebSocket protocol level, receivers that would rather read a fixed size chunk at a time can still do so.

Regards,
Maciej

---

From: Jack Moffitt <[email-masked]>
Date: Fri, 6 Aug 2010 15:14:11 -0600

> It's not only the sender and intermediaries that are affected by mandatory chunking. The receiver is affected too. In the case of the browser as receiver, it is highly valuable to have the length u…

Having a length is orthogonal to chunking. There's little reason not
to include it with the first chunk when it is known. Several of the
proposals have added that now.

If the field is dynamic and we have the machinery for chunking, then
everything is possible. It becomes only a question of whether both
methods will be reliably implemented. It seems feasible to believe
that they will, and so I hope everyone will be satisfied by this
compromise.

jack.

---

From: Pieter Hintjens <[email-masked]>
Date: Fri, 6 Aug 2010 23:43:31 +0200

I also like the simplicity of 1 byte for short messages, 8 bytes for
anything over 254 bytes. It gives the most savings where it matters.

-Pieter

Sent from my Android mobile phone.

On Aug 6, 2010 10:26 PM, "Dave Cridland" <[email-masked]> wrote:
> Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]<[email-masked]>

---

From: Maciej Stachowiak <[email-masked]>
Date: Fri, 06 Aug 2010 14:54:02 -0700

> that they will, and so I hope everyone will be satisfied by this

I agree that supporting both chunking and large lengths is reasonable.

Regards,
Maciej

---

From: Scott Ferguson <[email-masked]>
Date: Fri, 06 Aug 2010 18:46:19 -0700

Dave Cridland wrote:
>> buffer sizes are typically in the 1k to 64k range, and a 16-bit 
>> length lets the implementation pre-allocate the frame header for any 
>> message/chunk.
>>
>>
>> With an 8-bit length, the implementation either needs to shift bytes 
>> in the buffer when the message is 255 bytes long or it needs to 
>> preallocate the long frame header.
>
> Indeed. I'm told, however, that this is not such a big deal.

16 bits is a very helpful length for implementing chunking efficiently 
for dynamically-produced content. This is not a theoretical concern, but 
based on experience implementing similar framing protocols and chunking 
for HTTP. So I'm telling you as a developer who will implement chunking 
for WebSockets, that this choice affects my implementation significantly.

Any of 16/64, or 8/16/64 or 8/32/64 would be fine, but 8/64 is a poor 
choice for chunking because 256 bytes is too small to be a reasonable 
chunk, and 64 bits is a waste. Suppose you're using a 8k buffer. With 
the 8-bit length, you need 32 chunks just to send the buffer. With the 
64-bit length, you're wasting 6 bytes. With those as the only options, 
you're stuck between two bad choices.

The advantage of 16/64 is that it's still efficient for small messages, 
and is also efficient for chunking, and for large static files. The 
extra byte over the minimum 2-byte header is insignificant (and people 
were happy with 3 bytes.)

Since you were suggesting a 6-byte header earlier, I can't see why a 
3-byte header would not be acceptable.

> FWIW, unaliagned access to wrong-endian data is something I have 
> experience of, and as far as I can tell, is not a substantial overhead.

Yes, endian is not a major overhead. Endian is a non-issue because it 
doesn't affect the length, and never forces a buffer copy (or tricks 
like multiple buffers to avoid the copy) like expanding over a dynamic 
length boundary does.

-- Scott

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 7 Aug 2010 06:33:59 +0200

> you're stuck between two bad choices.

You're wasting 6 bytes per 8kB, so less than 0.07%. It's even less than
the single byte you waste for 256 bytes (0.4%). The worst case is when
you send 2048 bytes. You waste exactly 8 bytes by either 8 8-bit chunks
or one 64-bit chunk, which in both cases represent 0.39%. But this is
still equal to the minimum 8-bit chunk overhead. So in fact if we wanted
to always have a lower overhead, we'd have to completely remove 8-bit
chunks, but I don't think that would be a good idea.

So all in all, I think that the 64-bit csize always offers lower overhead
than the 8-bit ones so it will always be more efficient above 2 kB than
anything sent in 8-bit blocks, and the overhead is still small.

Regards,
Willy

---

From: Scott Ferguson <[email-masked]>
Date: Sat, 07 Aug 2010 12:21:35 -0700

> chunks, but I don't think that would be a good idea.

Let's look at the full comparison between 8/64 and 16/64. The values in 
the table below are frame overhead for various message/chunk sizes 
(8-bit length is a 2 byte frame, 16-bit length is 3 bytes, and 64-bit 
length is 9 bytes):

          8/64      16/64     64
  --------------------------------
  32      6.3%      9.3%     28.1%
  128     1.6%      2.3%      7.0%  
  256     3.5%      1.2%      3.5%
  512     1.8%      0.6%      1.8%
  2048    0.4%      0.1%      0.4%
  8192    0.1%      0.04%     0.1%

(Note that 8/64 only beats 16/64 in the upper left.)

Three ways of looking at the table:

  a) optimal length given a priori knowledge of the message size
  b) practical length given dynamic content. (When it's not known if a 
32 byte echo/print/write is the full message or the first part of a 
bigger message.)
  c) chunking

a) For optimal, it all depends on the message-length distribution. When 
messages are tweets, 8/64 is better (1.6% vs 2.3%). When messages are 
between 256 and 64k, 16/64 is better (0.6% vs 1.8% for a 512-byte 
message, and 0.15% vs 0.44% for the 2048-byte chunk example).

b) For dynamic content, the upper-right table values are the problem. 
The 28.1% and 7.0% are more than they need to be. Even so, it's still 
probably not worth shifting the content. So the optimal 8/64 becomes a 
practical 64-bit fixed length because it's not worth the effort to 
repack the data.

c) Chunks would be between 512 and 64k, depending on the 
implementation's buffer size. In that range, 16/64 is always superior. 
It's only the final chunk that might benefit from an 8-bit length.

Even though 16/64 is a better choice overall (given dynamic-content 
pragmatic issues, chunking needs, and uncertainty about the real-world 
message-length distribution), 8/64 is workable. The 28% and 7% overheads 
in the upper right are inelegant, and the extra chunking overhead is 
less efficient, but it's workable.

-- Scott

---

From: Roberto Peon <[email-masked]>
Date: Sat, 7 Aug 2010 12:38:11 -0700

If we're going over TCP/IP, you may as well think about the overhead for the
TCP/IP headers.
It ends up coming down to transaction rate as much as anything else.
-=R

On Sat, Aug 7, 2010 at 12:21 PM, Scott Ferguson <[email-masked]> wrote:

>>> choice for chunking because 256 bytes is too small to be a reasonable chunk,
>>> and 64 bits is a waste. Suppose you're using a 8k buffer. With the 8-bit
>>> length, you need 32 chunks just to send the buffer. With the 64-bit length,
>>> you're wasting 6 bytes. With those as the only options, you're stuck between
>>> two bad choices.
>>>
>>>
>>
> Let's look at the full comparison between 8/64 and 16/64. The values in the
> table below are frame overhead for various message/chunk sizes (8-bit length
> is a 2 byte frame, 16-bit length is 3 bytes, and 64-bit length is 9 bytes):
>
>         8/64      16/64     64
>  128     1.6%      2.3%      7.0%   256     3.5%      1.2%      3.5%
>  512     1.8%      0.6%      1.8%
>  2048    0.4%      0.1%      0.4%
>  8192    0.1%      0.04%     0.1%
>
>  b) practical length given dynamic content. (When it's not known if a 32
> byte echo/print/write is the full message or the first part of a bigger
> message.)
>  c) chunking
>
> between 256 and 64k, 16/64 is better (0.6% vs 1.8% for a 512-byte message,
> and 0.15% vs 0.44% for the 2048-byte chunk example).
>
> b) For dynamic content, the upper-right table values are the problem. The
> 28.1% and 7.0% are more than they need to be. Even so, it's still probably
> not worth shifting the content. So the optimal 8/64 becomes a practical
> 64-bit fixed length because it's not worth the effort to repack the data.
>
> c) Chunks would be between 512 and 64k, depending on the implementation's
> buffer size. In that range, 16/64 is always superior. It's only the final
> chunk that might benefit from an 8-bit length.
>
> message-length distribution), 8/64 is workable. The 28% and 7% overheads in
> the upper right are inelegant, and the extra chunking overhead is less
> efficient, but it's workable.
>
> -- Scott
>
>

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 7 Aug 2010 22:23:31 +0200

> It ends up coming down to transaction rate as much as anything else.

Yes, but if/when we add multiplexing, it might be nice to be able to push
as many transactions as possible in a single TCP segment.

Scott, in my opinion, what is important is not to compare overheads together
but efficiency (payload/total), which is what really matters. Indeed, halving
the overhead when it's already not noticeable won't save much bandwidth.
However if we can keep it low for small packets, it helps :

            8/64      16/64     64          8/64 vs 16/64
     --------------------------------
     16     88.9%     84.2%     64.0%          +5.6%
     20     90.9%     87.0%     69.0%          +4.5%
     32     94.1%     91.4%     78.0%          +3.0%
     128    98.5%     97.7%     93.4%          +0.8%
     255    99.2%     98.8%     96.6%          +0.4%
     256    96.6%     98.8%     96.6%          -2.2%
     512    98.3%     99.4%     98.3%          -1.1%
     1024   99.1%     99.7%     99.1%          -0.6%
     2048   99.6%     99.9%     99.6%          -0.3%
     8192   99.9%     99.96%    99.9%          -0.06%

Now we see that above 512 bytes, we're fighting in the same yard with every
model. However, I think that it's interesting to note that below 128, 8/64
has a significant advantage (and 64 alone is a total waste).

I have intentionally added the 255-bytes message so see the maximal
theorical efficiency on 8-bit. In fact it will often be between 0.5% and
2% more efficient than the 16-bit model for tweets. Also, in my opinion,
we should theorically plan to send messages up to 510 bytes as 2 chunks
of 255, which finally improves efficiency to 98.5 for the 256-bytes message
(and up 99.2 for 512-bytes). This finally results in -0.3% for 8 vs 16
instead of -2.2%.

So in fact with 8 bit we could get savings of up to a few percent compared
to 16-bit on very short messages, and losses of at most 1.1% (511-bytes).
That's why I think it should still get an advantage, eventhough I admit
both tie on medium to large messages.

Regards,
Willy

---

From: Scott Ferguson <[email-masked]>
Date: Sat, 07 Aug 2010 15:46:13 -0700

>> It ends up coming down to transaction rate as much as anything else.

That's a good point. It's also an argument for a fixed 64-bit length.

> as many transactions as possible in a single TCP segment.

True, but you need a high message rate, even with multiplexing, to make 
it worth delaying. (I can't estimate how often that will happen. Maybe 
someone has a good feel for this?)

>             8/64      16/64     64          8/64 vs 16/64

Yes. That's a good way of looking at the data.

> has a significant advantage (and 64 alone is a total waste).

It's a space vs complexity trade-off.

Is it worth the extra implementation complexity of the 256-byte boundary 
to save 5% space? For me, no. I would happily pay that 5% to keep my 
code simple.

> both tie on medium to large messages.

This is all true, but I'd still prefer the savings in code complexity 
because the difference in space is close enough, imo.

-- Scott

---

From: Pieter Hintjens <[email-masked]>
Date: Sun, 8 Aug 2010 23:23:06 +0200

I'd point out that high volume tiny messages are a real use case in
asynchronous messaging. E.g. market data, which can be just a dozen bytes.
This is the range where overhead matters the most. Above 254 bytes it's much
less of a concern.

- Pieter

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 9 Aug 2010 10:50:28 +1000

All,

I do not disagree that message length (as opposed to frame length),
will often be a useful bit of meta-data for the receiver.

However, I think there is plenty of other meta-data that is related to
the total message that could be useful for the receiver: content-type,
content-encoding, expiry, etc are all things that a smart receiving to
potentially use to process/discard fragments on the fly without
waiting for the full message.

So I think it far better to support message length with a generic meta
data mechanism rather than give it a special spot in the first frame.
  Defining a mime-message op-code would be sufficient to allow us to
send a message with length and other meta data, that the framing layer
could then fragment into multiple chunks.

However, I do like having a very small minimal frame size for short
messages, so I would modify Johns proposal to:

 1 bit - fragment
 2 bits - reserved
 5 bits - opcode
 8 bits - short length
 [if length = 255: 6 bytes length]
 length bytes of payload

With opcodes defined for text, binary, mime and ping/pong frames

I've proposed a 48 bit length to make the total header 64bits, but I'm
ok with it being 64 bit length as well.

cheers

On 7 August 2010 00:17, John Tamplin <[email-masked]> wrote:
> We want to have a length of the complete message when it is available to
> make buffer management simpler for the receiver
> We want to be able to fragment messages to avoid extra buffering when the
> length of the entire message is not known up front, such as when sending
> We want to keep small frames having small overhead (though there is a limit
> on how much this matters given TCP/IP overhead)
> We want to be able to send very large files in a single fragment so we can
> just hand it off to sendfile or equivalent
>
> 1 bit - Initial fragment
> 1 bit - Final fragment
> 1 bit - reserved
> 5 bits - opcode
> 8 bits - short length
> [if length = 255: 8 bytes - length]
> [if Initial = 1 and Final = 0: 8 bytes - overall message length, with 0
> meaning "unknown"]
> length bytes of payload
>

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 9 Aug 2010 18:39:21 +0200

> With opcodes defined for text, binary, mime and ping/pong frames

What we've also found in ZeroMQ is that we need two fragment bits, not
one.  That is, a 'first' and 'last' bit.  That lets a receiver catch
frames in mid-stream and decide whether to collect them, or discard
them.  I'm not sure WebSockets has current use cases for that.
However if we *ever* want WebSockets' framing to work on UDP or
multicast I'd recommend two fragment bits.  It may also reduce
somewhat the state needed in clients.

> ok with it being 64 bit length as well.

I think history teaches us that "big enough" never is.  Data gets a
bit larger every 18 months.  (I.e. data sizes double every 18 months.)
There is great benefit in allowing 1 frame = 1 message.  So I'd use 64
bits now and expect that we upgrade to 128 bits in 30 years or so.

-Pieter

---

From: Salvatore Loreto <[email-masked]>
Date: Thu, 12 Aug 2010 21:14:54 +0200

Today ends the call for consensus/verification on the three hum(s) asked during
the F2F meeting in Maastricht as per mail:
http://www.ietf.org/mail-archive/web/hybi/current/msg02954.html

the list has been really active during the last week, with a lot of discussions and proposals.
Thank to all of you for the time and the energy you are putting to move the protocol forward.

1) for the hum#1 asked in the mail:
http://www.ietf.org/mail-archive/web/hybi/current/msg02960.html

nobody raised technical reason against.

so the chairs declare consensus on it!

2) for the hum#2 asked in the mail:
http://www.ietf.org/mail-archive/web/hybi/current/msg02961.html

nobody raised technical reason against.

so the chairs declare consensus on it!

3) the hum#3 asked in the mail:
http://www.ietf.org/mail-archive/web/hybi/current/msg02964.html

has received a lot of traffic in the ml (around 60 mails from more then 15 distinct contributors)
the discussion has reached a state where there is a rough consensus for including within
the framing :

- the ability to fragment a message because it helps in several scenarios
   (e.g. avoid extra buffering when the length of a message is not known upfront; allows signaling
    an error when something bad happes; etc.)

there is also a rough consensus for:

- having a large length field within the framing, so to be able to send very large files in a single frame
   (taking advantages of sendfile or equivalent methods)

The chairs ask Gabriel and Maciej (the requirements draft authors) to update their draft to reflect the consensus
and submit a new version soon.

S.Moonesamy as WG secretary will take care to update the issues tracker.

The huge discussion in the mailing list has also highlight the need to check consensus about
- Multiplexing
- Compression
- Metadata
- Mechanism for extensions

I will send out tomorrow separate emails to check if there is or not consensus for each of the above items
and also to check which of them receive consensus to be included already in the protocol and which can
eventually be added in the future.

cheers
Sal

---

From: Pieter Hintjens <[email-masked]>
Date: Thu, 12 Aug 2010 23:32:22 +0200

> Thank to all of you for the time and the energy you are putting to move the

Sal, I just wanted to say "thanks" for pushing this process forwards.

-Pieter

---

From: Maciej Stachowiak <[email-masked]>
Date: Thu, 12 Aug 2010 16:35:17 -0700

> and submit a new version soon.

Can you indicate which of these decisions should affect the requirements document, as opposed to the actual protocol draft, and if so what changes are appropriate? It seems to me that most of these are technical details which are relevant to the spec, but not the requirements document, at least directly.

For example, the following makes sense to me as a requirement:
(A) The protocol must support sending of large unknown-size messages without buffering.

But the following does not make sense to me as a requirement:
(B) The protocol must support the ability to fragment a message.

As I see it, (A) is a requirement based on use cases, (B) is an implementation strategy to meet requirement (A), among others. It is certainly within the WG's mandate to choose implementation strategies, but it does not make sense to me to record those choices in the requirements document.

Could the chairs please clarify what specifically should be changed in the requirements document?

Regards,
Maciej

---

From: Salvatore Loreto <[email-masked]>
Date: Fri, 13 Aug 2010 22:44:07 +0200

> Could the chairs please clarify what specifically should be changed in the requirements document?

hi Maciej,

here we are not talking of technical details, we are talking about 
general requirements that have been checked for consensus and have 
received it.
Now we have just include them in the requirement draft and yes this wg 
has decided that

"The protocol must support the ability to fragment a message"

is an acceptable and clear requirement.

How they will be implemented is still under discussion, indeed the 
actual framing has not been decided.

br
/Sal