Subject: [hybi] Extensibility mechanisms?

From: Justin Erenkrantz <[email-masked]>
Date: Thu, 15 Apr 2010 23:45:35 -0700

Hi all,

I think we're starting to come to a consensus that there is some type
of "base" protocol with some type of "sub-protocol" or "extensibility"
mechanisms for Web Sockets.  (Please correct me if I'm wrong, but
Hixie's latest post seems to indicate a shift towards accepting that
type of model.)

During the IETF-77 session, I pointed out that we really haven't
started any discussions around what those mechanisms would look like
in practice.  Given how much we're apparently placing into
"sub-protocols", I'd like to kick-off that discussion.

So...does anyone have any thoughts to get the ball rolling?  *grin*  -- justin

---

From: "Thomson, Martin" <[email-masked]>
Date: Fri, 16 Apr 2010 15:02:01 +0800

The base protocol need only support feature negotiation for most of these.  It's only when the sub-protocol wants to modify the framing or do something fancy that you need anything more.

Since there are too many degrees of freedom, it might help if someone is willing to propose a strawman, so...I'm with Justin: let's see it.

(I think that Greg promised a BWTP over Web Sockets proposal, that might be good candidate.)

--Martin

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Justin Erenkrantz
> Sent: Friday, 16 April 2010 4:46 PM
> To: Hybi
> Subject: [hybi] Extensibility mechanisms?
> 
> Hi all,
> 
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Ian Hickson <[email-masked]>
Date: Fri, 16 Apr 2010 07:16:22 +0000 (UTC)

On Thu, 15 Apr 2010, Justin Erenkrantz wrote:
> 
> I think we're starting to come to a consensus that there is some type of 
> "base" protocol with some type of "sub-protocol" or "extensibility" 

As far as I'm aware, this is the same model I've been proposing since 
TCPConnection in 2007, before Michael even proposed the Upgrade approach 
and before we renamed this work to Web Sockets. But if we have come to 
consensus on that then great! :-)

> During the IETF-77 session, I pointed out that we really haven't started 
> any discussions around what those mechanisms would look like in 
> practice.  Given how much we're apparently placing into "sub-protocols", 
> I'd like to kick-off that discussion.

There are two "extension" points of importance, as far as I can tell, 
covering three kinds of "extensions" (I use scare quotes here because 
I'm not sure I would call them all extensions):

1. Subprotocols that run over Web Sockets as implemented by authors.
   These would be protocols based on Web Sockets in the same way that, for 
   example, HTTP, XMPP, IMAP, or SSH are based on TCP; currently based 
   just on strings of text sent in both directions, the format of which is 
   the subprotocol.

2. Opt-in features from future version of the protocol.
   For example, a future version might introduce compression. For 
   compression to be used on a connection, both client and server would 
   have to include some defined field in their handshake; the peers would 
   then use different mechanisms to send text than the current mechanism 
   (e.g. using frmae 0x80 with gzipped data).

3. Opt-in experimental non-standard features implemented by Web browsers 
   and servers involved with the experiment.
   This would use the same mechanism as #2, but would be limited to 
   experimental work, not widely deployed. For example, Firefox could 
   define a field that, when used by both client and server, switches the 
   connection from using the Web Socket framing to using an entirely 
   different framing based on HTTP chunking or BWTP or BEEP or something.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 16 Apr 2010 14:56:00 +0200

Martin,

here is my strawman BWTP extension.

The key need is for it to be able to use the spare bits in
the frame type byte.

Note, that xml2rfc is giving me grief, so I've attached
the XML and a horrid PDF dump.   As soon as I get a ms
free I'm sure I'll find out what stupid thing I've done
to the xml.

Note that this is has not yet been submitted as a draft,
but I'm happy to do so when I've got the format right.
Remember that BWTP is little more than a thought
experiment - but I think a worthwhile one to test out
these concepts.

cheers

>> So...does anyone have any thoughts to get the ball rolling?  *grin*  --

<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes" ?>
<?rfc compact="yes" ?>
<?rfc sortrefs="yes"?>
<?rfc symrefs="yes" ?>
<?rfc linkmailto="yes"?>
<?rfc strict="yes"?>
<?rfc subcompact="no"?>
<rfc category="info" docName="draft-wilkins-hybi-bwtp-01" ipr="trust200902">
  <front>
    <title abbrev="BWTP/1.0">Bidirectional Web Transfer Protocol --
    BWTP/1.0</title>

    <author fullname="Greg Wilkins" initials="G." surname="Wilkins">
      <organization>Webtide</organization>

      <address>
        <email>[email-masked]</email>
      </address>
    </author>

    <author fullname="Pieter Hintjens" initials="P." surname="Hintjens">
      <organization></organization>

      <address>
        <email>[email-masked]</email>
      </address>
    </author>

    <date day="12" month="March" year="2010" />

    <area>Applications</area>

    <keyword>I-D</keyword>

    <keyword>Internet-Draft</keyword>

    <keyword></keyword>

    <keyword></keyword>

    <keyword></keyword>

    <abstract>
      <t>The Bidirectional Web Transfer Protocol (BWTP) is an proposed
      websocket subprotocol that will provide richer transport semantics.</t>
    </abstract>
  </front>

  <middle>
    <section anchor="Intro" title="Introduction">
      <section anchor="Purpose" title="Purpose">
        <t>This IETF draft is intended for further discussion in the
        [email-masked] mailing list for "Bidirectional communication for
        hypertext".</t>

        <t>The Hypertext Transfer Protocol (HTTP) is an application-level
        protocol for distributed, collaborative, hypermedia information
        systems. HTTP has been in use by the World-Wide Web global information
        initiative since 1990, but it is intrinsically a request/response
        protocol and cannot well service the growing use-case for data to be
        pushed from a server to a client.</t>

        <t>The websocket protocol is a message framing protocol that can be
        used to upgrade a HTTP/1.1 connection to be able to exhange
        bidirectional messages. The protocol is currently being standardized
        by the IETF HyBi working group.</t>

        <t>The Bidirectional Web Transfer Protocol (BWTP) is an proposed
        websocket subprotocol that will provide richer transport semantics.
        Some specific use-cases for BWTP are:</t>

        <t><list style="symbols">
            <t>A browser providing the websocket API that wishes to
            transparently provide features such as compression, multiplexing.
            The javascript using the the weebsocket API would be unaware that
            any additional transport features are being used.</t>

            <t>A browser could use BWTP to provide an enhanced websocket API
            that includes support features such as binary content and meta
            data.</t>

            <t>A websocket intermediary in cooperation with a server, could
            use BWTP to transparently enhance websocket connections received
            from clients with features such as connection aggregation, SSL
            offload, load balancing.</t>

            <t>A non browser client that wishes to use websocket framing
            protocol to transit the internet, could use BWTP to achieve a
            richer messaging semantic than that provided by the javascript
            API.</t>

            <t>A browser and server could use BWTP to transport HTTP requests
            and responses over a websocket connection. Not only would this
            allow a single connection to be used, but also features (many
            inspired by SPDY) such as header compression, multiple responses
            and responses sent in anticipation.</t>
          </list>This draft of BWTP differs significantly from the first draft
        of BWTP, in that it uses websocket and length encoded framing. However
        the semantics and capabilities expressed in the first draft are
        retained as well as new features inspired SPDY have been
        incorporated.</t>
      </section>
    </section>

    <section title="Terminology">
      <t>In this document, the key words "MUST", "MUST NOT", "REQUIRED",
      "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT
      RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described
      in BCP 14, <xref target="RFC2119">RFC 2119</xref> and indicate
      requirement levels for compliant implementations.</t>

      <t>An implementation is not compliant if it fails to satisfy one or more
      of the MUST or REQUIRED level requirements for the protocols it
      implements. An implementation that satisfies all the MUST or REQUIRED
      level and all the SHOULD level requirements for its protocols is said to
      be "unconditionally compliant"; one that satisfies all the MUST level
      requirements but not all the SHOULD level requirements for its protocols
      is said to be "conditionally compliant".</t>

      <t>This specification uses a number of terms to refer to the roles
      played by participants in, and objects of, the HTTP or BWTP
      communication.</t>

      <t>In this document, as short hand notation is used to represent
      websocket frames. Each websocket frame is delimited by the characters
      '{' and '}' and the character '|' is used to separate elements within
      the frame. The contents of each element is informally described and the
      context and surrounding text may need to be consulted for
      interpretations. For example the string "Hello World" sent as a
      WebSocket UTF-8 frame could be expressed as {0x00|Hello World|0xFF}. The
      same string sent as a length encoded binary frame can be represented as
      {0x80|length|Hello World}. The characters '[' and ']' are used to
      enclose frames that are sent as part of the same batch.</t>

      <t><list style="hanging">
          <t hangText="connection:"></t>
        </list></t>
    </section>

    <section anchor="overall" title="Overall Operation">
      <t>The BWTP protocol is designed to be a subprotocol of websocket and
      makes use of the unused bits of the frame type byte to send meta data
      and channel control frames. This section provides a quick overview of
      the operation of the protocol and it's main features.</t>

      <t>The following diagram shows how the User Agent (UA) initiates the
      BWTP connection by sending an upgrade request (an HTTP request) to the
      Origin Server (OS), over a connection (v):</t>

      <figure>
        <artwork><![CDATA[
  UA -------------------v------------------- OS
     WS upgrade request ------------------->
     <-------------- 101 WS upgrade response
     <------------ BWTP Frames ------------>
]]></artwork>
      </figure>

      <t>Following the upgrade request-response, both server and client can
      immediately start to send each other frames. The inbound and out- bound
      frames streams are independent and asynchronous (frames do not have
      responses).</t>

      <section title="Frame Types">
        <t>BWTP messages are batches of one or more websocket frames. The
        websocket frame type byte is used to indicate frames that represent
        semantic boundaries and/or batch boundaries:</t>

        <figure>
          <artwork><![CDATA[+------------+-------------+----------+-----+-----+-----+-----+-----+
| 7          | 6           | 5        | 4   | 3   | 2   | 1   | 0   |
+------------+-------------+----------+-----+-----+-----+-----+-----+
| Framing:   | Fragment:   | Batch:   |     | BWTP Frame Type       |
|  1=binary  |  1=Fragment |  1=Batch |     | xxxx=Meta Data        | 
|  0=UTF-8   |  0=Complete |  0=End   |     | 00000=Data            |
+------------+-------------+----------+-----+-----------------------+
]]></artwork>
        </figure>

        <t>The highorder bit of the WebSocket frame type is defined by the
        websocket protocol to determine if UTF-8 framing or length encoded
        binary framing is used. A BWTP implementation MUST accept both framing
        types, but it SHOULD only send frames that use the length encoded
        binary framing. </t>

        <t>The 2nd highest order bit of the WebSocket frame type is used by
        BWTP to indicate a frame that has been fragmented into multiple
        frames. The receiver should aggregate fragment frames until a non
        fragmented frame is received. It is hoped that the WebSocket
        specification will adopt the definition of this bit.</t>

        <t>The 3rd highest order bit of the WebSocket frame type is use by
        BWTP to indicate batches of frames. A frame with the batch bit set
        does not need to be transmitted towards the destination until a
        subsequent frame with the batch bit clear is available. A BWTP message
        is a batch of one or more websocket frames. It is hoped that the
        WebSocket specification will adopt the definition of this bit.</t>

        <t>BWTP defines several non data frame types that are used to
        coordinate the protocol or to send meta. The 4 low order bits of the
        WebSocket frametype are used the carry the BWTP frame type. The
        following BWTP frame types are defined:<list style="hanging">
            <t hangText="xxxx0000">Data</t>

            <t hangText="xxxx0001"><spanx>undefined</spanx></t>

            <t hangText="xxxx0010">Message Header Name</t>

            <t hangText="xxxx0011">Message Header Value</t>

            <t hangText="xxxx0100">Sequence number</t>

            <t hangText="xxxx0101">Acknowlege number</t>

            <t hangText="xxxx0110"><spanx>undefined</spanx></t>

            <t hangText="xxxx0111"><spanx>undefined</spanx></t>

            <t hangText="xxxx1000">Channel Identifier</t>

            <t hangText="xxxx1001">Channel State</t>

            <t hangText="xxxx1010">Channel Header Name</t>

            <t hangText="xxxx1011">Channel Header Value</t>

            <t hangText="xxxx1100"><spanx>undefined</spanx></t>

            <t hangText="xxxx1101"><spanx>undefined</spanx></t>

            <t hangText="xxxx1110"><spanx>undefined</spanx></t>

            <t hangText="xxxx1111"><spanx>undefined</spanx></t>
          </list></t>

        <section title="Data">
          <t>The simplest BWTP message is a single WebSocket data frame,
          either UTF-8 encoded frame:</t>

          <figure>
            <artwork><![CDATA[{00000000|Hello Beautiful World|0xFF}]]></artwork>
          </figure>

          <t>or length encoded binary frame:</t>

          <figure>
            <artwork><![CDATA[{10000000|0x15|Hello Beautiful World}]]></artwork>
          </figure>
        </section>

        <section title="Fragmentation">
          <t>A BWTP frame may be fragmented into multiple frames, with each
          incomplete frame having bit 6 of the frame type byte set to 1. For
          example, the following three frames represent a single data message
          "Hello Beautiful World":</t>

          <figure>
            <artwork><![CDATA[{11000000|0x06|Hello }
{11000000|0x0A|Beautiful }
{10000000|0x05|World}]]></artwork>
          </figure>
        </section>

        <section title="Message Meta Data">
          <t>A BWTP message may include meta data by including meta data
          frames in a batch with a data frame. The following example show a
          BWTP message with a content type set as meta data header and
          value:</t>

          <figure>
            <artwork><![CDATA[[{10100010|0x0C|Content-Type}
 {10100011|0x09|text/html}
 {10000000|0x14|<h1>Hello World</h1>}]]]></artwork>
          </figure>

          <t>A BWTP message batch may also be fragmented:</t>

          <figure>
            <artwork><![CDATA[[{10100010|0x0C|Content-Type}
 {10100011|0x09|text/html}
 {11000000|0x0A|<h1>Hello }
 {10000000|0x0A|World</h1>}]]]></artwork>
          </figure>

          <t>Note that in order to avoid the overheads associated with verbose
          redundant headers in HTTP, BWTP allows headers to be be sent
          associated with a channel, so that all messages in channel will
          inherit the meta data, which will not need to be sent with every
          message.</t>
        </section>
      </section>

      <section title="Channels">
        <t>BWTP connections may contain multiple Channels. Each Channel is an
        ordered sequence of frames and all BWTP endpoints and intermediaries
        must preserve the order of frames within a Channel. However, the
        frames of one channel may be interleaved with the frames of another
        and intermediaries are free to change the interleaving.</t>

        <t>A Channel is identified by a 32 bit signed integer and is created
        by the first usage of the channel. Channels that are created by a
        client MUST be assigned an even ID and channels created by the server
        MUST be assigned an odd ID, so as to avoid creation collissions [Note
        that this is opposite to the similar convention in SPDY, because
        channel 0 is implicitly opened by the client]. Once created, a channel
        is bidirectional.</t>

        <t>A Channel ID is sent in a Channel Identifier frame and once sent,
        all subsequent frames transmitted are associated with the channel,
        until a subsequent Channel Identifier frame is sent. This avoid the
        overhead of sending a channel identifier with every message.</t>

        <t>The following example shows messages for channel 5 interleaved with
        messages for channel 6:</t>

        <figure>
          <artwork><![CDATA[[{10001000|0x04|0x00 0x00 0x00 0x05}]
[{00000000|Hello Channel 5|0xff}]
[{00000000|Hello Channel 5 again|0xff}]
[{10001000|0x04|0x00 0x00 0x00 0x06}]
[{10100010|0x0C|Content-Type}
 {10100011|0x09|text/html}
 {10000000|0x14|<h1>Hello Chan 6</h1>}]
[{10001000|0x04|0x00 0x00 0x00 0x05}]
[{00000000|Hello Channel 5 yet again|0xff}]]]></artwork>
        </figure>

        <t>[ Option? A channel ID that is part of a batch could only effect
        the frames in that batch ??? but what if the channel ID was not the
        first in the batch ??? ]</t>

        <t>Channels have a simple state machine which applies to both the
        inbound and outbound parts of the Channel. The state machine allows
        for channels to be Open, Blocked by flow control, Closing, Closed and
        Reset. The state of a channel is sent in a Channel State frame. The
        following example shows a BWTP message to initiate an orderly close of
        a channel:</t>

        <figure>
          <artwork><![CDATA[[{11001000|0x04|0x00 0x00 0x00 0x05}
 {10001001|0x01|CLOSED}]]]></artwork>
        </figure>
      </section>

      <section title="HTTP over BWTP">
        <t>A BWTP Channel can be used to carry HTTP requests and responses as
        follows:<list style="symbols">
            <t>The request line is sent as 2 value frames at the start of a
            message batch (HTTP/1.1 is assumed).</t>

            <t>Headers are sent as header frame and value frame pairs</t>

            <t>Headers that are not expected to change (eg User-Agent) are
            sent once as channel header frames and value frame pairs</t>

            <t>The response status lines is sent as 3 value frames at the
            start of a message batch.</t>

            <t>The order of responses is the same as the request order, only
            within the same channel.</t>
          </list></t>

        <t>The following example shows a channel being opened and two
        pipelined GET requests:</t>

        <figure>
          <artwork><![CDATA[[{10001000|0x04|0x00 0x00 0x00 0x02}]
[{10101010|0x0A|User-Agent}
 {10101011|0x14|Chromezilla Exploder}
 {10101010|0x04|Host}
 {10101011|0x0B|example.org}
 {10101010|0x06|Cookie}
 {10001011|0x15|JSESSIONID=1234567890}]
[{10100011|0x03|GET}
 {10100011|0x01|/}
 {10100010|0x??|If-Modified-Since}
 {10000011|0x??|12:34 5 January 1964}]
[{10100011|0x03|GET}
 {10000011|0x??|/favicon.ico}]

]]></artwork>
        </figure>
      </section>

      <section title="WebSocket over BWTP">
        <t>A BWTP Channel can be used to carry a WebSocket connection, so that
        multiple WebSocket connections can be multiplexed over the a shared
        BWTP Websocket connections. The WebSocket handshake is sent as HTTP
        over BWTP, so that URL and sub protocol can be established.</t>
      </section>
    </section>

    <section title="new section">
      <t></t>
    </section>
  </middle>

  <back>
    <references title="Normative References">
      <?rfc include="reference.RFC.2119"?>

      <?rfc include="reference.RFC.2616"?>
    </references>
  </back>
</rfc>

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 16 Apr 2010 15:07:09 +0200

>    different framing based on HTTP chunking or BWTP or BEEP or something.

I do not think that extensions should be free to change framing.

The while point of having a standard like websocket, is so that
the network infrastructure can implement the RFC and then we
have known semantics over that connection that intermediaries
can at least handle correctly, but may also do clever stuff with.

If after an upgrade from HTTP to Websocket, the endpoints are
free to agree to send any old framing, then we've not upgraded
to websocket, we've downgraded to TCP/IP.

For me, a prime requirement of the base protocol is to
establish exact framing/transport semantics - so that
intermediaries can be well behaved and we can get websocket
penetrating more than 70% of the internet.

If we come up with something that was a HTTP connection,
but then it was a Websocket connection, but we are really
using it as a BEEP connection.... then the intermediaries
are just going to be lost and most corporate environments
will just block it as uncontrollable.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 16 Apr 2010 17:29:42 +0200

All,

Julian corrected my xml2rfc goofup.   Here is a more
readable version of the BWTP draft (still not submitted yet).

cheers

> experiment - but I think a worthwhile one to test out
Network Working Group

G. Wilkins

Internet-Draft

Webtide

Intended status: Informational

P. Hintjens

Expires: September 13, 2010

March 12, 2010

Bidirectional Web Transfer Protocol -- BWTP/1.0
draft-wilkins-hybi-bwtp-01

Abstract

The Bidirectional Web Transfer Protocol (BWTP) is an proposed websocket subprotocol that will provide richer transport semantics.

Status of this Memo

This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress”.

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire in September 13, 2010.

Copyright Notice

Copyright © 2010 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the BSD License.

Table of Contents

1.   Introduction

1.1   Purpose

2.   Terminology

3.   Overall Operation

3.1   Frame Types

3.1.1   Data

3.1.2   Fragmentation

3.1.3   Message Meta Data

3.2   Channels

3.3   HTTP over BWTP

3.4   WebSocket over BWTP

4.   new section

5.   Normative References

Authors' Addresses

1. Introduction

1.1 Purpose

This IETF draft is intended for further discussion in the [email-masked] mailing list for "Bidirectional communication for hypertext".

The Hypertext Transfer Protocol (HTTP) is an application-level protocol for distributed, collaborative, hypermedia information systems. HTTP has been in use by the World-Wide Web global information initiative since 1990, but it is intrinsically a request/response protocol and cannot well service the growing use-case for data to be pushed from a server to a client.

The websocket protocol is a message framing protocol that can be used to upgrade a HTTP/1.1 connection to be able to exhange bidirectional messages. The protocol is currently being standardized by the IETF HyBi working group.

The Bidirectional Web Transfer Protocol (BWTP) is an proposed websocket subprotocol that will provide richer transport semantics. Some specific use-cases for BWTP are:

A browser providing the websocket API that wishes to transparently provide features such as compression, multiplexing. The javascript using the the weebsocket API would be unaware that any additional transport features are being used.

A browser could use BWTP to provide an enhanced websocket API that includes support features such as binary content and meta data.

A websocket intermediary in cooperation with a server, could use BWTP to transparently enhance websocket connections received from clients with features such as connection aggregation, SSL offload, load balancing.

A non browser client that wishes to use websocket framing protocol to transit the internet, could use BWTP to achieve a richer messaging semantic than that provided by the javascript API.

A browser and server could use BWTP to transport HTTP requests and responses over a websocket connection. Not only would this allow a single connection to be used, but also features (many inspired by SPDY) such as header compression, multiple responses and responses sent in anticipation.

This draft of BWTP differs significantly from the first draft of BWTP, in that it uses websocket and length encoded framing. However the semantics and capabilities expressed in the first draft are retained as well as new features inspired SPDY have been incorporated.

2. Terminology

In this document, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described in BCP 14, RFC 2119 [RFC2119] and indicate requirement levels for compliant implementations.

An implementation is not compliant if it fails to satisfy one or more of the MUST or REQUIRED level requirements for the protocols it implements. An implementation that satisfies all the MUST or REQUIRED level and all the SHOULD level requirements for its protocols is said to be "unconditionally compliant"; one that satisfies all the MUST level requirements but not all the SHOULD level requirements for its protocols is said to be "conditionally compliant".

This specification uses a number of terms to refer to the roles played by participants in, and objects of, the HTTP or BWTP communication.

In this document, as short hand notation is used to represent websocket frames. Each websocket frame is delimited by the characters '{' and '}' and the character '|' is used to separate elements within the frame. The contents of each element is informally described and the context and surrounding text may need to be consulted for interpretations. For example the string "Hello World" sent as a WebSocket UTF-8 frame could be expressed as {0x00|Hello World|0xFF}. The same string sent as a length encoded binary frame can be represented as {0x80|length|Hello World}. The characters '[' and ']' are used to enclose frames that are sent as part of the same batch.

connection:

3. Overall Operation

The BWTP protocol is designed to be a subprotocol of websocket and makes use of the unused bits of the frame type byte to send meta data and channel control frames. This section provides a quick overview of the operation of the protocol and it's main features.

The following diagram shows how the User Agent (UA) initiates the BWTP connection by sending an upgrade request (an HTTP request) to the Origin Server (OS), over a connection (v):

Following the upgrade request-response, both server and client can immediately start to send each other frames. The inbound and out- bound frames streams are independent and asynchronous (frames do not have responses).

3.1 Frame Types

BWTP messages are batches of one or more websocket frames. The websocket frame type byte is used to indicate frames that represent semantic boundaries and/or batch boundaries:

The highorder bit of the WebSocket frame type is defined by the websocket protocol to determine if UTF-8 framing or length encoded binary framing is used. A BWTP implementation MUST accept both framing types, but it SHOULD only send frames that use the length encoded binary framing.

The 2nd highest order bit of the WebSocket frame type is used by BWTP to indicate a frame that has been fragmented into multiple frames. The receiver should aggregate fragment frames until a non fragmented frame is received. It is hoped that the WebSocket specification will adopt the definition of this bit.

The 3rd highest order bit of the WebSocket frame type is use by BWTP to indicate batches of frames. A frame with the batch bit set does not need to be transmitted towards the destination until a subsequent frame with the batch bit clear is available. A BWTP message is a batch of one or more websocket frames. It is hoped that the WebSocket specification will adopt the definition of this bit.

BWTP defines several non data frame types that are used to coordinate the protocol or to send meta. The 4 low order bits of the WebSocket frametype are used the carry the BWTP frame type. The following BWTP frame types are defined:

xxxx0000

Data

xxxx0001

undefined

xxxx0010

Message Header Name

xxxx0011

Message Header Value

xxxx0100

Sequence number

xxxx0101

Acknowlege number

xxxx0110

undefined

xxxx0111

undefined

xxxx1000

Channel Identifier

xxxx1001

Channel State

xxxx1010

Channel Header Name

xxxx1011

Channel Header Value

xxxx1100

undefined

xxxx1101

undefined

xxxx1110

undefined

xxxx1111

undefined

3.1.1 Data

The simplest BWTP message is a single WebSocket data frame, either UTF-8 encoded frame:

{00000000|Hello Beautiful World|0xFF}

or length encoded binary frame:

{10000000|0x15|Hello Beautiful World}

3.1.2 Fragmentation

A BWTP frame may be fragmented into multiple frames, with each incomplete frame having bit 6 of the frame type byte set to 1. For example, the following three frames represent a single data message "Hello Beautiful World":

{11000000|0x06|Hello }
{10000000|0x05|World}

3.1.3 Message Meta Data

A BWTP message may include meta data by including meta data frames in a batch with a data frame. The following example show a BWTP message with a content type set as meta data header and value:

[{10100010|0x0C|Content-Type}
{10100011|0x09|text/html}
{10000000|0x14|<h1>Hello World</h1>}]

A BWTP message batch may also be fragmented:

[{10100010|0x0C|Content-Type}
{10100011|0x09|text/html}
{11000000|0x0A|<h1>Hello }
{10000000|0x0A|World</h1>}]

Note that in order to avoid the overheads associated with verbose redundant headers in HTTP, BWTP allows headers to be be sent associated with a channel, so that all messages in channel will inherit the meta data, which will not need to be sent with every message.

3.2 Channels

BWTP connections may contain multiple Channels. Each Channel is an ordered sequence of frames and all BWTP endpoints and intermediaries must preserve the order of frames within a Channel. However, the frames of one channel may be interleaved with the frames of another and intermediaries are free to change the interleaving.

A Channel is identified by a 32 bit signed integer and is created by the first usage of the channel. Channels that are created by a client MUST be assigned an even ID and channels created by the server MUST be assigned an odd ID, so as to avoid creation collissions [Note that this is opposite to the similar convention in SPDY, because channel 0 is implicitly opened by the client]. Once created, a channel is bidirectional.

A Channel ID is sent in a Channel Identifier frame and once sent, all subsequent frames transmitted are associated with the channel, until a subsequent Channel Identifier frame is sent. This avoid the overhead of sending a channel identifier with every message.

The following example shows messages for channel 5 interleaved with messages for channel 6:

[{00000000|Hello Channel 5 yet again|0xff}]

[ Option? A channel ID that is part of a batch could only effect the frames in that batch ??? but what if the channel ID was not the first in the batch ??? ]

Channels have a simple state machine which applies to both the inbound and outbound parts of the Channel. The state machine allows for channels to be Open, Blocked by flow control, Closing, Closed and Reset. The state of a channel is sent in a Channel State frame. The following example shows a BWTP message to initiate an orderly close of a channel:

[{11001000|0x04|0x00 0x00 0x00 0x05}
{10001001|0x01|CLOSED}]

3.3 HTTP over BWTP

A BWTP Channel can be used to carry HTTP requests and responses as follows:

The request line is sent as 2 value frames at the start of a message batch (HTTP/1.1 is assumed).

Headers are sent as header frame and value frame pairs

Headers that are not expected to change (eg User-Agent) are sent once as channel header frames and value frame pairs

The response status lines is sent as 3 value frames at the start of a message batch.

The order of responses is the same as the request order, only within the same channel.

The following example shows a channel being opened and two pipelined GET requests:

[{10001000|0x04|0x00 0x00 0x00 0x02}]

3.4 WebSocket over BWTP

A BWTP Channel can be used to carry a WebSocket connection, so that multiple WebSocket connections can be multiplexed over the a shared BWTP Websocket connections. The WebSocket handshake is sent as HTTP over BWTP, so that URL and sub protocol can be established.

4. new section

5. Normative References

[RFC2119]

Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels”, BCP 14, RFC 2119, March 1997.

[RFC2616]

Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, “Hypertext Transfer Protocol -- HTTP/1.1”, RFC 2616, June 1999.

Authors' Addresses
Greg WilkinsWilkinsGregWebtideEMail: [email-masked] HintjensHintjensPieterEMail: [email-masked]

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 16 Apr 2010 17:39:33 +0200

All,

the issue that I have with the current subprotocol field allowed
in the handshake is that it is a) singled valued and b) exposed
in the application API.

So we have to assume that applications and frameworks are going
to invent all sorts of subprotocols and put all sorts
of values into this field... probably frequently for usages
that have nothing to do with sub protocols.

So if a browser, server and/or intermediary wish to negotiate
other extensions (eg for compression, multiplexing, etc. etc.),
then the protocol field is effectively unavailable for that
purpose.

Note that I don't think this is a bad thing.  I think it is
good that the application is able to effectively send arbitrary
meta data along with a handshake (I can imagine B64 encoded json
and/or XML being sent).   I think the protocol field is great
for that purpose (even if calling it "protocol" is probably
not exactly right).

We just need another field, so that entities below the
level of the API can negotiate and apply extensions.
This field needs to be multi valued, so that multiple
extensions can be applied.  However, I expect that some
extensions will be mutually incompatible.

cheers

---

From: Mike Belshe <[email-masked]>
Date: Fri, 16 Apr 2010 09:43:57 -0700

I have some possibly noobie questions.  I apologize if these have been
answered before:

a) In 3.1.2, what is the structure for the length?  I assume its a 4byte
integer?
b) It would be nice if every frame had a length in a fixed position within
the frame. Having the length in different places for different frame types
is just mechanically annoying.  With exception of the UTF-8 frame, it looks
like bytes 2-5 of any frame are the length?
c) Is the fragment really needed?
d) although not stated, I assume that if the first bit of a frame is 0 (e.g.
UTF-8), that the remaining 7 bits of the frame block header must be 0?  Do
the batch/fragment/type fields have meaning for UTF-8 frames?
e) 3.2 I like the channels. More details are needed on the channel lifetime
& state.  I assume you're looking at a half-closed type of thing.

My biggest complaint:
The logical unit of operation for a receiver of data over BWTP appears to be
a "frame batch", not a "frame".   Because frame types are as granular as
"header name", "channel id", etc, they are really more like field types
within a batch than a frame-id.  By having such fine-grained identifiers,
doesn't it allow for creation of very strange frames? It also creates the
opportunity for each implementation to encode equivalent frames in very
different ways.  In general, it is nice if equivalent frames end up being
the same sequence of bytes.

About layering of HTTP over BWTP:
* The example in 3.3 doesn't close the channel.
* Should HTTP over BWTP use a single channel?  Or should many channels be
used?
* The example in 3.3 - how does the receiver determine the end of a request
block?  Is it required to be a single batch?  If so, how does a large post
work?  Is this where a fragment would work.

One last question, and I hope this won't sound like pride, because it is not
- I just want to understand. Why not use SPDY for BWTP?  BWTP is definitely
aimed at the same goals as SPDY, but SPDY is much further along.  Is there
something where SPDY fails which makes BWTP necessary?  Aside from syntax,
there are no features in BWTP not already present in SPDY?  By contrast,
there are several features in SPDY not available in BWTP.  I propose we
understand such differences and unify these projects.

It seems odd to me that first, BWTP defines itself as a "sub-protocol" for
WebSockets, but then section 3.4 defines how to run WebSockets over BWTP :-)
 It sounds like indecision.

Mike

On Fri, Apr 16, 2010 at 8:29 AM, Greg Wilkins <[email-masked]> wrote:

> Network Working GroupG. WilkinsInternet-DraftWebtideIntended status:
> InformationalP. HintjensExpires: September 13, 2010March 12, 2010
>
> Abstract <#1280740bc6667cea_rfc.abstract>
>
> The Bidirectional Web Transfer Protocol (BWTP) is an proposed websocket
> subprotocol that will provide richer transport semantics.
> Status of this Memo <#1280740bc6667cea_rfc.status>
>
> Internet-Drafts are working documents of the Internet Engineering Task
> Force (IETF), its areas, and its working groups. Note that other groups may
> also distribute working documents as Internet-Drafts.
>
> Internet-Drafts are draft documents valid for a maximum of six months and
> may be updated, replaced, or obsoleted by other documents at any time. It is
> inappropriate to use Internet-Drafts as reference material or to cite them
> other than as “work in progress”.
>
> The list of current Internet-Drafts can be accessed at
> http://www.ietf.org/ietf/1id-abstracts.txt.
>
> Copyright Notice <#1280740bc6667cea_rfc.copyrightnotice>
>
> This document is subject to BCP 78 and the IETF Trust's Legal Provisions
> Relating to IETF Documents (http://trustee.ietf.org/license-info) in
> effect on the date of publication of this document. Please review these
> documents carefully, as they describe your rights and restrictions with
> respect to this document. Code Components extracted from this document must
> include Simplified BSD License text as described in Section 4.e of the Trust
> Legal Provisions and are provided without warranty as described in the BSD
> License.
> ------------------------------
> Table of Contents <#1280740bc6667cea_rfc.toc>
>
>    - 1.   Introduction <#1280740bc6667cea_Intro>
>       - 1.1   Purpose <#1280740bc6667cea_Purpose>
>    - 2.   Terminology <#1280740bc6667cea_rfc.section.2>
>    - 3.   Overall Operation <#1280740bc6667cea_overall>
>       - 3.1   Frame Types <#1280740bc6667cea_rfc.section.3.1>
>          - 3.1.1   Data <#1280740bc6667cea_rfc.section.3.1.1>
>          - 3.1.2   Fragmentation <#1280740bc6667cea_rfc.section.3.1.2>
>          - 3.1.3   Message Meta Data <#1280740bc6667cea_rfc.section.3.1.3>
>       - 3.2   Channels <#1280740bc6667cea_rfc.section.3.2>
>       - 3.3   HTTP over BWTP <#1280740bc6667cea_rfc.section.3.3>
>       - 3.4   WebSocket over BWTP <#1280740bc6667cea_rfc.section.3.4>
>    - 4.   new section <#1280740bc6667cea_rfc.section.4>
>    - 5.   Normative References <#1280740bc6667cea_rfc.references>
>    - Authors' Addresses <#1280740bc6667cea_rfc.authors>
>
> 1. <#1280740bc6667cea_rfc.section.1> Introduction<#1280740bc6667cea_Intro>
> 1.1 <#1280740bc6667cea_rfc.section.1.1> Purpose<#1280740bc6667cea_Purpose>
>
> This IETF draft is intended for further discussion in the [email-masked] list for "Bidirectional communication for hypertext".
>
> The Hypertext Transfer Protocol (HTTP) is an application-level protocol for
> distributed, collaborative, hypermedia information systems. HTTP has been in
> use by the World-Wide Web global information initiative since 1990, but it
> is intrinsically a request/response protocol and cannot well service the
> growing use-case for data to be pushed from a server to a client.
>
> The websocket protocol is a message framing protocol that can be used to
> upgrade a HTTP/1.1 connection to be able to exhange bidirectional messages.
> The protocol is currently being standardized by the IETF HyBi working group.
>
> The Bidirectional Web Transfer Protocol (BWTP) is an proposed websocket
> subprotocol that will provide richer transport semantics. Some specific
> use-cases for BWTP are:
>
>    - A browser providing the websocket API that wishes to transparently
>    provide features such as compression, multiplexing. The javascript using the
>    the weebsocket API would be unaware that any additional transport features
>    are being used.
>    - A browser could use BWTP to provide an enhanced websocket API that
>    includes support features such as binary content and meta data.
>    - A websocket intermediary in cooperation with a server, could use BWTP
>    to transparently enhance websocket connections received from clients with
>    features such as connection aggregation, SSL offload, load balancing.
>    - A non browser client that wishes to use websocket framing protocol to
>    transit the internet, could use BWTP to achieve a richer messaging semantic
>    than that provided by the javascript API.
>    - A browser and server could use BWTP to transport HTTP requests and
>    responses over a websocket connection. Not only would this allow a single
>    connection to be used, but also features (many inspired by SPDY) such as
>    header compression, multiple responses and responses sent in anticipation.
>
> This draft of BWTP differs significantly from the first draft of BWTP, in
> that it uses websocket and length encoded framing. However the semantics and
> capabilities expressed in the first draft are retained as well as new
> features inspired SPDY have been incorporated.
> 2. <#1280740bc6667cea_rfc.section.2> Terminology
>
> In this document, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL",
> "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
> "MAY", and "OPTIONAL" are to be interpreted as described in BCP 14, RFC
> 2119 <#1280740bc6667cea_RFC2119> [RFC2119] and indicate requirement levels
> for compliant implementations.
>
> An implementation is not compliant if it fails to satisfy one or more of
> the MUST or REQUIRED level requirements for the protocols it implements. An
> implementation that satisfies all the MUST or REQUIRED level and all the
> SHOULD level requirements for its protocols is said to be "unconditionally
> compliant"; one that satisfies all the MUST level requirements but not all
> the SHOULD level requirements for its protocols is said to be "conditionally
> compliant".
>
> This specification uses a number of terms to refer to the roles played by
> participants in, and objects of, the HTTP or BWTP communication.
>
> In this document, as short hand notation is used to represent websocket
> frames. Each websocket frame is delimited by the characters '{' and '}' and
> the character '|' is used to separate elements within the frame. The
> contents of each element is informally described and the context and
> surrounding text may need to be consulted for interpretations. For example
> the string "Hello World" sent as a WebSocket UTF-8 frame could be expressed
> as {0x00|Hello World|0xFF}. The same string sent as a length encoded binary
> frame can be represented as {0x80|length|Hello World}. The characters '['
> and ']' are used to enclose frames that are sent as part of the same batch.
> connection:3. <#1280740bc6667cea_rfc.section.3> Overall Operation<#1280740bc6667cea_overall>
>
> The BWTP protocol is designed to be a subprotocol of websocket and makes
> use of the unused bits of the frame type byte to send meta data and channel
> control frames. This section provides a quick overview of the operation of
> the protocol and it's main features.
>
> The following diagram shows how the User Agent (UA) initiates the BWTP
> connection by sending an upgrade request (an HTTP request) to the Origin
> Server (OS), over a connection (v):
>
> Following the upgrade request-response, both server and client can
> responses).
> 3.1 <#1280740bc6667cea_rfc.section.3.1> Frame Types
>
> BWTP messages are batches of one or more websocket frames. The websocket
> frame type byte is used to indicate frames that represent semantic
> boundaries and/or batch boundaries:
>
> The highorder bit of the WebSocket frame type is defined by the websocket
> protocol to determine if UTF-8 framing or length encoded binary framing is
> used. A BWTP implementation MUST accept both framing types, but it SHOULD
> only send frames that use the length encoded binary framing.
>
> The 2nd highest order bit of the WebSocket frame type is used by BWTP to
> indicate a frame that has been fragmented into multiple frames. The receiver
> should aggregate fragment frames until a non fragmented frame is received.
> It is hoped that the WebSocket specification will adopt the definition of
> this bit.
>
> The 3rd highest order bit of the WebSocket frame type is use by BWTP to
> indicate batches of frames. A frame with the batch bit set does not need to
> be transmitted towards the destination until a subsequent frame with the
> batch bit clear is available. A BWTP message is a batch of one or more
> websocket frames. It is hoped that the WebSocket specification will adopt
> the definition of this bit.
>
> BWTP defines several non data frame types that are used to coordinate the
> protocol or to send meta. The 4 low order bits of the WebSocket frametype
> are used the carry the BWTP frame type. The following BWTP frame types are
> defined:
> xxxx0000Dataxxxx0001*undefined*xxxx0010Message Header Namexxxx0011Message
> Header Valuexxxx0100Sequence numberxxxx0101Acknowlege numberxxxx0110*
> undefined*xxxx0111*undefined*xxxx1000Channel Identifierxxxx1001Channel
> Statexxxx1010Channel Header Namexxxx1011Channel Header Valuexxxx1100*
> undefined*xxxx1101*undefined*xxxx1110*undefined*xxxx1111*undefined*3.1.1<#1280740bc6667cea_rfc.section.3.1.1>
>  Data
>
> 3.1.2 <#1280740bc6667cea_rfc.section.3.1.2> Fragmentation
>
> A BWTP frame may be fragmented into multiple frames, with each incomplete
> frame having bit 6 of the frame type byte set to 1. For example, the
> following three frames represent a single data message "Hello Beautiful
> World":
>
> {11000000|0x06|Hello }
> 3.1.3 <#1280740bc6667cea_rfc.section.3.1.3> Message Meta Data
>
> A BWTP message may include meta data by including meta data frames in a
> batch with a data frame. The following example show a BWTP message with a
> content type set as meta data header and value:
>
> Note that in order to avoid the overheads associated with verbose redundant
> headers in HTTP, BWTP allows headers to be be sent associated with a
> channel, so that all messages in channel will inherit the meta data, which
> will not need to be sent with every message.
> 3.2 <#1280740bc6667cea_rfc.section.3.2> Channels
>
> BWTP connections may contain multiple Channels. Each Channel is an ordered
> sequence of frames and all BWTP endpoints and intermediaries must preserve
> the order of frames within a Channel. However, the frames of one channel may
> be interleaved with the frames of another and intermediaries are free to
> change the interleaving.
>
> A Channel is identified by a 32 bit signed integer and is created by the
> first usage of the channel. Channels that are created by a client MUST be
> assigned an even ID and channels created by the server MUST be assigned an
> odd ID, so as to avoid creation collissions [Note that this is opposite to
> the similar convention in SPDY, because channel 0 is implicitly opened by
> the client]. Once created, a channel is bidirectional.
>
> A Channel ID is sent in a Channel Identifier frame and once sent, all
> subsequent frames transmitted are associated with the channel, until a
> subsequent Channel Identifier frame is sent. This avoid the overhead of
> sending a channel identifier with every message.
>
> [ Option? A channel ID that is part of a batch could only effect the frames
> in that batch ??? but what if the channel ID was not the first in the batch
> ??? ]
>
> Channels have a simple state machine which applies to both the inbound and
> outbound parts of the Channel. The state machine allows for channels to be
> Open, Blocked by flow control, Closing, Closed and Reset. The state of a
> channel is sent in a Channel State frame. The following example shows a BWTP
> message to initiate an orderly close of a channel:
>
> 3.3 <#1280740bc6667cea_rfc.section.3.3> HTTP over BWTP
>
>    - The request line is sent as 2 value frames at the start of a message
>    batch (HTTP/1.1 is assumed).
>    - Headers are sent as header frame and value frame pairs
>    - Headers that are not expected to change (eg User-Agent) are sent once
>    as channel header frames and value frame pairs
>    - The response status lines is sent as 3 value frames at the start of a
>    message batch.
>    - The order of responses is the same as the request order, only within
>    the same channel.
>
> 3.4 <#1280740bc6667cea_rfc.section.3.4> WebSocket over BWTP
>
> A BWTP Channel can be used to carry a WebSocket connection, so that
> multiple WebSocket connections can be multiplexed over the a shared BWTP
> Websocket connections. The WebSocket handshake is sent as HTTP over BWTP, so
> that URL and sub protocol can be established.
> 4. <#1280740bc6667cea_rfc.section.4> new section5.<#1280740bc6667cea_rfc.section.5>Normative References
> *[RFC2119]*Bradner, S. <[email-masked]>, “Key words for use in RFCs to
> Indicate Requirement Levels <http://tools.ietf.org/html/rfc2119>”, BCP 14,
> RFC 2119, March 1997.*[RFC2616]*Fielding, R. <[email-masked]>, Gettys,
> J. <[email-masked]>, Mogul, J. <[email-masked]>, Frystyk, H.<[email-masked]>,
> Masinter, L. <[email-masked]>, Leach, P. <[email-masked]>,
> and T. Berners-Lee <[email-masked]>, “Hypertext Transfer Protocol --
> HTTP/1.1 <http://tools.ietf.org/html/rfc2616>”, RFC 2616, June 1999.
> Authors' Addresses <#1280740bc6667cea_rfc.authors>Greg WilkinsWilkinsGreg
> WebtideEMail: [email-masked] HintjensHintjensPieterEMail:
> [email-masked]
>

---

From: Ian Hickson <[email-masked]>
Date: Fri, 16 Apr 2010 20:13:09 +0000 (UTC)

On Fri, 16 Apr 2010, Greg Wilkins wrote:
> 
> The whole point of having a standard like websocket, is so that the 
> network infrastructure can implement the RFC and then we have known 
> semantics over that connection that intermediaries can at least handle 
> correctly, but may also do clever stuff with.

That wasn't even on my list of reasons for originally coming up with 
TCPConnection, let alone the main reason.

The main reason for the Web Socket protocol is to have one protocol that 
the browsers can all implement in an interoperable fashion so that servers 
can implement things once and have them work with all browsers. In the 
ideal deployment the connection is wrapped in end-to-end TLS, so the 
intermediaries can't do anything with it. There were only two reasons for 
even allowing unencrypted Web Socket originally: requiring amateurs to 
figure out TLS seemed like too high a bar, and there is still some concern 
that when the connection is being used for entirely public information, 
the TLS connection overhead is too expensive to justify.

---

From: Mike Belshe <[email-masked]>
Date: Fri, 16 Apr 2010 14:21:57 -0700

> the TLS connection overhead is too expensive to justify.

We've done extensive tests on TLS.  There are four latency issues:
  a) CPU on the handshake. TLS is now cheap on modern hardware, and it gets
cheaper every day.  I'm sure there are a few sites which might have some
trouble migrating to 100% TLS, but when designing for protocols of the
future, this is not an issue (Moore's law!).  Round trips are orders of
magnitude more expensive than the CPU overhead of the handshake.  We do need
to make sure TLS implementations pick up modern extensions like client-side
session tickets.
  b) CPU on bulk encryption.  Once you get beyond the handshake, you can
negotiate RC4; we've benchmarked it, and it is the same speed as using the
null-cipher.  Using other ciphers are sometimes expensive, each is
different.
  c) Handshake Round Trips.  The extra round trip for TLS is the worst part
of TLS from a latency perspective (100ms to resume a session).  But,
WebSockets has a handshake of its own which is also 100ms!  So, if you're
going to take a handshake, you may as well get crypto while you do it - *it's
free!*
  d) Bandwidth.  TLS does use 2-3% more bandwidth than non-TLS on typical
page loads.  This is largely weighted in the handshake overhead, so it gets
amortized if you use the websocket over time.

Another common argument is the "I don't want to buy a certificate" argument,
but there are now CAs offering free certs which are trusted in all browsers'
root-ca lists.

Mike

---

From: Justin Erenkrantz <[email-masked]>
Date: Sat, 17 Apr 2010 00:13:29 -0700

> intermediaries can't do anything with it. There were only two reasons for

It may be appropriate to say it is end-to-end TLS at an organizational
level (ie user to origin), but my experience is that most reverse
proxy deployments perform TLS termination on the edge of the network
so that load balancing techniques can be applied inside the network
without TLS overhead.  So, I believe it is very unlikely to expect
that there will always be end-to-end TLS at scale - the intermediaries
will be relied upon to provide critical load-balancing and failover
mechanisms even for Web Socket.  -- justin

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 17 Apr 2010 09:58:05 +0200

> TCPConnection, let alone the main reason.

It's always a problem when your children grow up and then decide not
to become doctors like you'd always planned for them.

I'm not doubting your agenda for starting this effort. Bravo!

I'm doubting that those reasons continue to be valid or are the
motivating factor in many major and minor companies investing
many millions of dollars of developer time and infrastructure
renewal so that we can have bidirectional conversations
between browser and server.

> the TLS connection overhead is too expensive to justify.

Between the bytes leaving a TLS endpoint and the application,
there will be infrastructure code written by experts.  On
the client side, the will be the browser.  On the server
side this will either be an enhanced HTTP server or
a dedicated websocket server.      That infrastructure may
include intermediaries such as load balancers etc.

You appear to believe that such intermediaries are proprietary
matters and not of concern of the WG.

But the standard process must consider these intermediaries
because in the real world, deployments are created from
the aggregation of many products from different vendors
that interoperate on the basis of open standards. - both on
the open web and within deployers data centres after
any TLS has been decoded.

I'm sorry but despite your original aims, enabling amateur
programmers to write protocols that avoid the need for vendor
supplied infrastructure is not the prime reason this WG has come
into existence.

The reason that most of us are here, and the reason that
our employers pay our salaries to be here, is so that
vendors can produce  software and/or appliances that
can be deployed by their clients with some remote hope of
interoperating with the products from other vendors.

regards

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 17 Apr 2010 10:50:27 +0200

> a) In 3.1.2, what is the structure for the length?  I assume its a 4byte

Nope - that's the websocket length encoding.  Go figure!

> My biggest complaint:
> The logical unit of operation for a receiver of data over BWTP appears
> to be a "frame batch", not a "frame".  

As I said, it's really just a thought experiment... to see how
websocket might be extended.  I'll ponder your feedback at length
and see if the experiment can be improved.

> One last question, and I hope this won't sound like pride, because it is
> not - I just want to understand. Why not use SPDY for BWTP?  BWTP is
> definitely aimed at the same goals as SPDY, but SPDY is much further
> along.  

I completely agree!   SPDY also has something very vital that BWTP
does not have - interest from a browser vendor!

Perhaps I should switch to thought experiments with SPDY rather
than BWTP - but I wanted to play with the concept of a frame
batch, as I think that is vital to fragmentation of websocket
messages - even though SPDY would not use it.

I think BWTP as a HTTP transport is not very likely.
SPDY over websockets would be a good outcome and we should
strive to make websockets good enough to carry SPDY well.

But perhaps there is some room for a BWTP-like layer between
base websockets and SPDY to provide the fragmentation and
multi-channel stuff as something common to all websocket
users of those facilities.  So maybe I should drop the
HTTP stuff and focus on fragmentation and multiplexing?

> It seems odd to me that first, BWTP defines itself as a "sub-protocol"
> for WebSockets, but then section 3.4 defines how to run WebSockets over
> BWTP :-)  It sounds like indecision.

No - that is running the websocket API over BWTP over websocket
protocol.    Ie aggregating multiple websocket "connections"
over a single real websocket connection, using BWTP.

cheers

---

From: Justin Erenkrantz <[email-masked]>
Date: Sat, 17 Apr 2010 06:04:51 -0700

> HTTP stuff and focus on fragmentation and multiplexing?

+1.  The more experimentation we have around fragmentation and
multiplexing, the more insights we'll gain there.  -- justin

---

From: Ian Hickson <[email-masked]>
Date: Sun, 18 Apr 2010 02:46:34 +0000 (UTC)

On Sat, 17 Apr 2010, Justin Erenkrantz wrote:
> level (ie user to origin)

Right, that's what I meant (though "origin" may be the wrong word to use 
here, given its meaning in the Web security sphere).

> but my experience is that most reverse proxy deployments perform TLS 
> termination on the edge of the network so that load balancing techniques 
> can be applied inside the network without TLS overhead.  So, I believe 
> it is very unlikely to expect that there will always be end-to-end TLS 
> at scale - the intermediaries will be relied upon to provide critical 
> load-balancing and failover mechanisms even for Web Socket.  -- justin

For all intents and purposes, the server is everything under control of 
the end point authority. It doesn't matter if the "server" conformance 
class is implemented by a 486 PC sitting under a desk using only userland 
Ruby code running on Linux or if it's implemented by a mixture of 
dedicated hardware in a datacenter providing load-balancing to an array of 
high-end servers running a custom clustering OS. The spec explicitly 
states that conformance requirements are essentially "black box" 
conformance requirements. It doesn't require the server to use a single 
processor; the TLS, Web Socket, and subprotocol handshakes could all be 
handled by completely different machines on different coasts, so long as 
the end result is indistinguishable from what the spec requires.

---

From: Ian Hickson <[email-masked]>
Date: Sun, 18 Apr 2010 02:55:22 +0000 (UTC)

On Sat, 17 Apr 2010, Greg Wilkins wrote:
> 
> I'm sorry but despite your original aims, enabling amateur programmers 
> to write protocols that avoid the need for vendor supplied 
> infrastructure is not the prime reason this WG has come into existence.

If that isn't a goal for this working group, then that's fine, but it 
means I'm in the wrong group.

> The reason that most of us are here, and the reason that our employers 
> pay our salaries to be here, is so that vendors can produce software 
> and/or appliances that can be deployed by their clients with some remote 
> hope of interoperating with the products from other vendors.

The reason I'm here is that the IETF asked me to work on Web Sockets in 
this working group. Web Sockets is a technology aimed as, as you put it, 
enabling amateurs programmers to write protocols that avoid the need for 
vendor supplied infrastructure (amongst other things). If the IETF is 
still interested in working on such a technology, what's the right working 
group for it? If the IETF is not interested in working on such a working 
group, then it would be in everyone's interests for that to be made clear, 
so that we don't act at cross purposes, and so that the work on that 
technology can resume elsewhere.

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 18 Apr 2010 09:20:33 +0200

> means I'm in the wrong group.

Please don't take your bat and ball and go home.

The charter is clear what this working group is for:

  The Hypertext-Bidirectional (HyBi) working group will seek
  standardization of one approach to maintain bidirectional
  communications between the HTTP client, server and intermediate
  entities, which will provide more efficiency compared to the current
  use of hanging requests.

However, your concerns about developer experience are included
in the charter:

  The Working Group should consider:
   * Implementer experience
   * Impact on existing implementations and deployments
   * Ability to achieve broad implementation
   * Ability to address broader use cases than those covered in
     the input document

> this working group. 

it would be really good if you took up that invitation,
but with a realization that it is a different process than
what you are used to.

The IETF has different goals to your own, due to the fact
that it is a collective organization and represents the
rough consensus of many peoples goals.

We keep ending up in threads like these because whenever
a dissenting view is posted is posted, one of your
most frequent responses is that you don't care about
that particular concern.

But if websockets is to be standardized at the IETF,
it is no longer your protocol and your concerns
are just some among many.    The ball park concerns
of the IETF community have been outlined in the
charter.   We are now narrowing them down in a proces
to produce a requirements document.

If you want you concern of enabling amateur programmers
to implement the protocol included, then please
propose it as a requirement and see if others
agree.

Similarly, I'm concerned about websockets transiting
intermediaries and I too will be proposing requirements
to that effect, and they too will need community
support.

regards

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 18 Apr 2010 09:28:01 +0200

Ian Hickson wrote:

> the end point authority. 

Even the smallest deployment will have to deal with
ISPs, Data centres or even home routers, which are not under
the authority of the application developer.

> The spec explicitly

End points are always going to be assembled from
multiple "boxes" black, grey and white, probably
from different vendors, that interoperate using
open standards.    These "boxes" may just be language
libraries, or ISP provided firewalls or full
corporate DMZ layers.

If interoperability is to be achieved, the standard
cannot end at the first box.

So back to the original objection I raised.  If
an intermediary sees a HTTP connection upgraded to
websocket, then it should be able to make the
reasonable assumption that protocol it will
subseqently see will be that described in the
websocket specification.   Allowing arbitrary
framing by sub protocols will break that
reasonable assumption.

regards

---

From: Maciej Stachowiak <[email-masked]>
Date: Sun, 18 Apr 2010 01:49:23 -0700

> to that effect, and they too will need community

I agree that everyone should advocate for what they feel to be  
requirements. That seems like a sensible way to drive the requirements  
document.

But in fairness, in the message Ian was responding to you said:

> infrastructure is not the prime reason this WG has come into  

...which seemed dismissive of Ian's goals.

We should try to be respectful of each other's proposed requirements,  
even when we disagree. The IETF operates by rough consensus, so it's  
totally acceptable for different people to be participating in this WG  
with different goals.

My own view: I think we need to support both lightweight "amateur"  
deployments and large-scale deployments involving complex  
infrastructure. Another way to put it is: the protocol should be  
compatible with advanced back-end infrastructure, but should not  
*require* complex infrastructure. Yet another way to put it is: the  
protocol should scale from very small experimental deployments, to  
very large deployments with worldwide reach.

Why do I take this view? A large part of the Web's success has come  
from the "long tail" of content that exists only because the  
underlying formats and protocols make it easy to do simple things.  
This has led to the explosive growth of the Web. At the same time,  
it's clearly important to consider the needs of larger organizations  
as well. A solution that works for Joe Hobbyist Programmer at home,  
but doesn't meet the needs of, say, Facebook or Yahoo, is not a  
complete solution. Large sites are clearly a critical part of the  
Web's success as well.

I think ultimately these two goals are not fundamentally at odds. It  
may be more challenging to satisfy both sets of needs, but we have a  
lot of smart people here so we should be able to figure it out.

Regards,
Maciej

---

From: Ian Hickson <[email-masked]>
Date: Sun, 18 Apr 2010 18:12:59 +0000 (UTC)

On Sun, 18 Apr 2010, Maciej Stachowiak wrote:
> protocol should scale from very small experimental deployments, to very 
> large deployments with worldwide reach.
> 
> Why do I take this view? A large part of the Web's success has come from 
> the "long tail" of content that exists only because the underlying 
> formats and protocols make it easy to do simple things. This has led to 
> the explosive growth of the Web. At the same time, it's clearly 
> important to consider the needs of larger organizations as well. A 
> solution that works for Joe Hobbyist Programmer at home, but doesn't 
> meet the needs of, say, Facebook or Yahoo, is not a complete solution. 
> Large sites are clearly a critical part of the Web's success as well.
> 
> I think ultimately these two goals are not fundamentally at odds. It may 
> be more challenging to satisfy both sets of needs, but we have a lot of 
> smart people here so we should be able to figure it out.

I completely agree with the above.

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 18 Apr 2010 22:06:25 +0200

Ian Hickson wrote:
>> infrastructure.
[...]

> I completely agree with the above.

Great!  so can we rewind this thread.

So as you don't preclude support for large-scale
deployments with complex infrastructure, my objection
to your proposed #3 extension can't be dismissed on
the grounds that your main reason for starting this
effort was to support amateur programmers.

You proposed 3 extension types.  I agree with 1 & 2,

> >    different framing based on HTTP chunking or BWTP or BEEP or something.

I believe that large scale deployments with complex
infrastructure will have significant problems if
a HTTP connection that has been upgraded to
websocket is suddenly found to be carrying BEEP
framing.

If you want to send BEEP over a HTTP connection,
then you should upgrade to BEEP, not websocket.

regards

---

From: Ian Hickson <[email-masked]>
Date: Sun, 18 Apr 2010 20:12:15 +0000 (UTC)

On Sun, 18 Apr 2010, Greg Wilkins wrote:
> I believe that large scale deployments with complex infrastructure will 
> have significant problems if a HTTP connection that has been upgraded to 
> websocket is suddenly found to be carrying BEEP framing.

Why would a large scale deployments with complex infrastructure that can't 
handle BEEP framing opt-in to an experimental non-standard feature that 
uses BEEP framing?

> If you want to send BEEP over a HTTP connection, then you should upgrade 
> to BEEP, not websocket.

If the handshake involves an optional opt-in to a non-standard 
experimental feature, and you accept it, then you're not upgrading to Web 
Socket, you're upgrading to a non-standard protocol influenced by Web 
Sockets, as defined by the convention you and the client are using.

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 18 Apr 2010 22:47:12 +0200

> handle BEEP framing opt-in to an experimental non-standard feature that 

because experiments are a precursor to real deployments.

> Sockets, as defined by the convention you and the client are using.

My point is that an intermediary can't possible know about
all subprotocols (experimental or otherwise), nor be expected
to upgrade everytime a new subprotocol is invented.

Intermediaries can know about websocket framing and so long as
all sub protocols and extensions are restricted to adhere to
websocket framing, then intermediaries can continue to work
regardless of sub protocols.

regards

---

From: Adam Barth <[email-masked]>
Date: Sun, 18 Apr 2010 14:03:35 -0700

> regardless of sub protocols.

I'm not sure I buy the premise that we want intermediaries to
understand and/or interfere with the websocket protocol.  It seems
more robust and more secure to establish a point-to-point encrypted
tunnel between the user agent and the server.

Adam

---

From: Mike Belshe <[email-masked]>
Date: Sun, 18 Apr 2010 14:06:05 -0700

> regardless of sub protocols.

Should intermediaries know about these protocols?  Why?

In my view, new protocols should be thinking not just about "right now" but
also about the future.  It takes a long time for protocols to be adopted, so
thinking only of 2010 is a mistake.  Think about 2020.

In the year 2020, should the guy sitting next to you in the coffee shop be
able to sniff your traffic to see what you're doing?  Sure, maybe you're
only watching a movie, but who's business is it but yours?  Should common
users be expected to understand when their data is private and when it is
not?  Why should they?  Is it realistic to expect them to?

If we run all websockets through SSL, the protocol is stronger.  And poor
users aren't left holding the bag wondering why they just got hacked.

But, even if we don't care about end-user security, surely we care about
deployability of the protocol.  Unlike any protocol layering on top of port
80, running with SSL is deployable today without updating the intermediaries
at all.

So, to review you get:
   a) A protocol designed for the future.
   b) A deployable protocol
   c) A more reliable and robust protocol, because it can't be accidentally
broken by intermediaries making incorrect assumptions about what it is
transporting.  (We've seen this so often in history, we ought to learn the
lesson!)

Mike

---

From: Adam Barth <[email-masked]>
Date: Sun, 18 Apr 2010 14:10:54 -0700

> transporting.  (We've seen this so often in history, we ought to learn the

Indeed.  SSL is on the cusp of being universally deployable today.  We
should be planning for a world where SSL is ubiquitous.

Now, we can have a discussion about certificates being a hassle and
the follies of PKI.  It's probably still worth having two modes: one
where we verify the servers certificate and one where we don't.

Adam

---

From: Scott Ferguson <[email-masked]>
Date: Sun, 18 Apr 2010 14:22:49 -0700

> complete solution. Large sites are clearly a critical part of the 
I think a little more precision is important here.

The client and server APIs must be suitable for amateur programmers, and 
the wire protocol must support those simple APIs, but it's _not_ 
important that the wire protocol be implementable by someone who can't 
understand buffering, chunking or encoding. After all, HTTP/1.1 requires 
those capabilities.

The "long tail" in HTTP is enabled by the server APIs: CGI, mod_xxx, 
PHP, Servlets, ASP, etc, and the client javascript API. It is indeed 
very important that the server-side and client-side APIs for WebSockets 
be usable by hobbyist programmers to do simple things.

However, the "long tail" are not writing hobbyist HTTP client or server 
protocol implementations.

In addition, WebSockets is intrinsically multithreaded, which makes the 
claim that it's aimed at amateur implementers bizarre. WebSockets 
requires threading/async handling on both ends, which is beyond the 
capabilities of these hypothetical amateur programmers who cannot encode 
a simple frame. Programmers who can implement a threaded/async server 
can handle the trivial framing we're discussing. If they can't handle 
the framing, the threading/async requirements are beyond them.

That's not a license to create CORBA 2, but does mean it's reasonable to 
require a basic level of competence from implementers.

-- Scott

---

From: "Thomson, Martin" <[email-masked]>
Date: Mon, 19 Apr 2010 07:21:53 +0800

> Here is a more

Thanks Greg,

From this I see that your assumption is that the sub-protocol is entirely in control over the remaining 7 bits of frame type octet.

The only concern that I have (and I see that you share this concern) is that this doesn't enable a modular approach to adding features.  It's a bit of all-or-nothing.

--Martin

---

From: Roberto Peon <[email-masked]>
Date: Sun, 18 Apr 2010 16:41:08 -0700

On Sun, Apr 18, 2010 at 2:22 PM, Scott Ferguson <[email-masked]> wrote:

>> the "long tail" of content that exists only because the underlying formats
>> and protocols make it easy to do simple things. This has led to the
>> explosive growth of the Web. At the same time, it's clearly important to
>> consider the needs of larger organizations as well. A solution that works
>> for Joe Hobbyist Programmer at home, but doesn't meet the needs of, say,
>> Facebook or Yahoo, is not a complete solution. Large sites are clearly a
>> critical part of the Web's success as well.
>>
> the wire protocol must support those simple APIs, but it's _not_ important
> that the wire protocol be implementable by someone who can't understand
> buffering, chunking or encoding. After all, HTTP/1.1 requires those
> capabilities.
>

> The "long tail" in HTTP is enabled by the server APIs: CGI, mod_xxx, PHP,
> Servlets, ASP, etc, and the client javascript API. It is indeed very
> important that the server-side and client-side APIs for WebSockets be usable
> by hobbyist programmers to do simple things.
>
> claim that it's aimed at amateur implementers bizarre. WebSockets requires
> threading/async handling on both ends, which is beyond the capabilities of
> these hypothetical amateur programmers who cannot encode a simple frame.
> Programmers who can implement a threaded/async server can handle the trivial
> framing we're discussing. If they can't handle the framing, the
> threading/async requirements are beyond them.
>

Thanks, Scott, this is exactly my view as well.
 -=R

---

From: Ian Hickson <[email-masked]>
Date: Sun, 18 Apr 2010 23:48:48 +0000 (UTC)

On Sun, 18 Apr 2010, Greg Wilkins wrote:
> Ian Hickson wrote:
> > 
> > Why would a large scale deployments with complex infrastructure that 
> > can't handle BEEP framing opt-in to an experimental non-standard 
> > feature that uses BEEP framing?
> 

I'm having great trouble understanding your scenario.

You are concerned that a browser might deploy an experimental non-standard 
feature that changes the framing of Web Sockets, because it is possible 
that large scale deployments with complex infrastructure that can't handle 
the new framing might want to opt in to such experiments because 
eventually they will want to do a real deployment of this new framing?

> >> If you want to send BEEP over a HTTP connection, then you should 
> >> upgrade to BEEP, not websocket.
> > 
> > experimental feature, and you accept it, then you're not upgrading to 
> > Web Socket, you're upgrading to a non-standard protocol influenced by 
> > Web Sockets, as defined by the convention you and the client are 
> > using.
> 
> My point is that an intermediary can't possible know about all 
> subprotocols (experimental or otherwise), nor be expected to upgrade 
> everytime a new subprotocol is invented.

We're not talking about subprotocols here, we're talking about Web Socket 
protocol extensions.

A server-side intermediary only needs to know about the protocol features 
and options that are in use at the site. This is similar to how the 
server-side intermediaries need to know about the version of HTTP in use 
at the site. If someone deploys a new experimental compression scheme in a 
Web browser, deployment sites that don't opt-in to using that compression 
scheme need not know anything about it.

> Intermediaries can know about websocket framing and so long as all sub 
> protocols and extensions are restricted to adhere to websocket framing, 
> then intermediaries can continue to work regardless of sub protocols.

What do you mean by "restricted"?

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 00:05:50 +0000 (UTC)

> understand buffering, chunking or encoding. After all, HTTP/1.1 requires 

IMHO, HTTP is a disaster in terms of getting multiple server-side 
implementations. Over three quarters of the market is dominated by two 
vendors, and the third largest vendor on a per-server basis is a company 
that runs a proprietary implementation and derives significant benefit 
from being able to do so. There are some custom servers, e.g. in some set 
top boxes or other network devices too small to run one of the few common 
servers, and many of those are highly incomplete or buggy implementations.

I would not consider HTTP a model solution here. On the contrary. Its 
complexity has led to a rather stagnant monoculture which is an attacker's 
dream. A single exploit can affect millions of servers.

> Servlets, ASP, etc, and the client javascript API. It is indeed very important
> that the server-side and client-side APIs for WebSockets be usable by hobbyist
> programmers to do simple things.
> 

That's because HTTP is too complicated. If we make Web Socket simple 
enough on the server that it's just as easy to roll your own Web Socket 
server as to use an HTTP framework today, then I see no reason why the 
long tail shouldn't write hobbyists Web Socket servers.

> claim that it's aimed at amateur implementers bizarre.

It's relatiely straight-forward to implement Web Sockets using a select(3) 
loop, no need for threads.

> WebSockets requires threading/async handling on both ends, which is 
> beyond the capabilities of these hypothetical amateur programmers who 
> cannot encode a simple frame.

The difference is that writing a select(3) loop is something you can learn 
from tutorials then implement with trial and error, and without which your 
server will not work at all; while getting text length encodings right 
requires asking questions that many ASCII-only authors will never know to 
even ask. (This is less of a problem in locales where non-ASCII characters 
are common, of course, at least for authors who use this feature to 
transmit natural language text.)

Actually measuring lengths is relatively straightforward in most languages 
once you know you have to do it.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 19 Apr 2010 09:45:11 +0200

> IMHO, HTTP is a disaster in terms of getting multiple server-side

It seems we have a fundamental disagreement about what is a "disaster".

> vendors, and the third largest vendor on a per-server basis is a company

Sounds like the browser market, btw.

> from being able to do so. There are some custom servers, e.g. in some set

Where's the problem with that?

> servers, and many of those are highly incomplete or buggy implementations.

There are also many servers running, for instances, one of the many 
Servlet/J2EE implementations. These may not be significant compared to 
the absolute number of servers out there, but how is that relevant???

 > ...

Yes. What does that have to do with the protocol, btw?

Best regards, Julian

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 19 Apr 2010 09:52:13 +0200

Mike Belshe wrote:

> If we run all websockets through SSL, the protocol is stronger.  And
> poor users aren't left holding the bag wondering why they just got hacked.

Mike,

I'm not arguing against SSL or having that the
default/preferred transport.

But in many deployments, SSL will be terminated on a box
remote from the application and the connection will go through
intermediaries provided by many vendors for load balancing,
aggregation, connection aggregation etc. The specification
needs to support such intermediaries in 2020.

But to make it to 2020, we also need something that is
deployable in 2010.  So that means respecting the HTTP
protocol.

What Ian is proposing is that a websocket client and server
can just agree on some new field, that can be used to
upgrade to any protocol they like eg BEEP!

HTTP has an upgrade mechanism and it's called Upgrade.
Websocket should not be allowed to subvert that with
something like:

   GET /something HTTP/1.1
   Host: somewhere:80
   Upgrade: WebSocket
   Connection: Upgrade
   My-Upgrade: BEEP

        HTTP/1.1 101 Web Socket Protocol Handshake
        Upgrade WebSocket
        Connection: Upgrade
        My-Upgrade: BEEP

   <-- Beep frames -->

Any intermediary involved will have seen that
as an upgrade to WebSocket, not to BEEP.
I can't see how that can be interpreted as
a compliant HTTP upgrade.

Finally,  if I  really try to imagine protocols
in 2020, I can't see secure connections being
allowed end to end in all environments.   For
better or worse, Corporations (and even countries)
like to monitor and apply policy to what their
users are doing.  I can imagine that every
hop will be encrypted, but I think that end
to end will not be universal (unless there
some some huge break out of progressive
libertarian sentiment in fortune 500 and
world governments).

regards

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 08:04:38 +0000 (UTC)

On Mon, 19 Apr 2010, Julian Reschke wrote:
> > > The client and server APIs must be suitable for amateur programmers, 
> > > and the wire protocol must support those simple APIs, but it's _not_ 
> > > important that the wire protocol be implementable by someone who 
> > > can't understand buffering, chunking or encoding. After all, 
> > > HTTP/1.1 requires those capabilities.
> > 

The browser market is even worse. It's incredibly difficult to enter the 
market. That's one of the reasons I've been working on making HTML5 as 
accurate a description of what a competitive browser needs to implement as 
possible -- to make it easier to enter that market. Unfortunately there's 
little more we can do -- we can make the specs more accurate, which makes 
it less expensive to create a Web browser, but we can't fundamentally make 
the browser market simpler, because of all the legacy content.

With Web Sockets, there's no legacy yet. We have a more or less fresh 
start. So we can ensure that what we do is simple enough that we don't 
create this problem.

> Where's the problem with that?

Your quoting is ambiguous here, but assuming you are referring to the 
third implementation being a proprietary implementation, it's not a 
problem. What I meant was that since that company benefitted from 
developing their own dedicated custom server, rather than using one of the 
three off-the-shelf solutions with more than 5% of the market (by domain), 
it might be the case that other organisations would benefit from the same 
ability, but find themselves unable to actually write their own server 
since it is such a complicated task. If this conclusion is correct, then 
it would lead to a further conclusion that we should probably ensure doing 
the same with Web Sockets is easier, so that one does not have to become a 
large company before one can reap the benefit.

> > top boxes or other network devices too small to run one of the few 
> > common servers, and many of those are highly incomplete or buggy 
> > implementations.
> 

I'm not sure what you're asking.

> > complexity has led to a rather stagnant monoculture which is an 
> > attacker's dream. A single exploit can affect millions of servers. ...
> 

A simple protocol is far more likely to end up with more implementations 
and thus a far more heterogeneous environment.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 19 Apr 2010 10:15:24 +0200

> and thus a far more heterogeneous environment.

There are many many implementations of HTTP. Some fast, some not so. 
Some complete, some not so.

The main reason we see two server implementations rule the market isn't 
necessarily because of protocol complexity, but because one of them 
ships with a popular server platform, and the other one is free and 
"good enough" for almost everything.

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 19 Apr 2010 17:24:25 +0900

> means I'm in the wrong group.

If what Greg wrote means that "you won't be able to deploy web sockets 
unless you pay one of these vendors a lot of money", then I clearly 
would disagree. But I sincerely hope that's not what Greg meant.

But I think there's a middle ground between "the protocol has to be 
really easy to implement for an amateur programmer in a weekend" and 
"the protocol needs heavy (and expensive) vendor support to be feasible".

In my view, at the low end, the amateurs will use the web sockets API on 
the client side, and should use a web sockets API (e.g. a Perl package, 
a Ruby gem, or some such) on the server side. There is a very important 
need to make it easy for 'amateurs' to use web sockets, but that doesn't 
mean that it has to be easy for them to implement a web sockets API. It 
is enough if an experienced programmer can create and implement a 
(server-side) API in a few days. Amateurs are much better at copy-pasting
   use 'webservice'; # Perl
or
   require 'webservice' # Ruby
than at counting characters or bytes.

Regards,   Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 08:36:53 +0000 (UTC)

On Mon, 19 Apr 2010, "Martin J. Dürst" wrote:
> 
> In my view, at the low end, the amateurs will use the web sockets API on the
> client side, and should use a web sockets API (e.g. a Perl package, a Ruby
> gem, or some such) on the server side. There is a very important need to make
> it easy for 'amateurs' to use web sockets, but that doesn't mean that it has
> to be easy for them to implement a web sockets API.

As is, the protocol is simple enough that you don't have to write an API 
for it on the server side -- for many simple deployments, you can just 
write the script including the Web Socket server all as one thing. I think 
that's valuable, for reasons I've explained at length today.

---

From: Mike Belshe <[email-masked]>
Date: Mon, 19 Apr 2010 01:40:01 -0700

> needs to support such intermediaries in 2020.
This is a fair point; but intermediaries within your (the site owner)
control is a solvable problem, whereas intermediaries outside your (the
user) control are not.

> libertarian sentiment in fortune 500 and
I think you're not thinking big enough.  There are solutions to all of these
problems.  I have yet to talk to anyone that doesn't think SSL everywhere is
the right idea; but many people think as you do - that it is just too hard.
 If we take that approach (the give up approach), then of course it will
prove true.  If you think about it - you're saying that policy should drive
protocol.  But isn't it the other way around?  The protocol should drive
policy?

There is only one question to answer:  what is the right approach for users?
 Then, once determined, figure out how to get there.  If security is in the
future, we should make it secure now.

Mike

---

From: Ian Hickson <[email-masked]>
Date: Mon, 19 Apr 2010 08:48:50 +0000 (UTC)

On Mon, 19 Apr 2010, Julian Reschke wrote:
> > A simple protocol is far more likely to end up with more 
> > implementations and thus a far more heterogeneous environment.
> 

I think we can get orders of magnitude more complete implementations of 
Web Sockets than of HTTP if we keep the protocol trivial.

> "good enough" for almost everything.

That is one possibility. I do not personally think it is the real reason. 
Unfortunately, I don't know of any way to test these hypotheses.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 19 Apr 2010 10:50:00 +0200

> servers, and many of those are highly incomplete or buggy implementations.

The HTTP server market is vibrant and dynamic and it is a
misrepresentation to characterize it as a disaster.

Unfortunately netcraft have stopped publishing the detail of
their server surveys.   But in 2008, when they were last published,
there were many many more than just "some custom servers".

Even back in the late nineties there were hundreds, if not thousands
of servers listed.  I remember being very excited when jetty broke
into the top 150, then top 100, then top 50 then top 10 for a while.

We'd fallen out of the top 10 on the last survey, not because we
had not increased market share, but because other servers had
come along and grown even more.

Nginx is now the 3rd top server that was originally written by
one guy for one website 5 years ago.

Even on the client site, HTTP is vibrant and diverse.

OK, there are limited browser choices, but that is more
a dynamic of desktop application choice rather than protocol
choice.     If you consider HTTP clients in general, there
are thousands of implementations, with many hundreds in
wide spread usage.

The protocol is simple enough that if you want to just
crack open a socket a GET some content, you can... and
many many developers do so.

Could HTTP be simpler or better designed?  Sure it could!
But I do not think you can characterize it as a disaster.

The complexities of HTTP are almost entirely optional,
but are there for very good reasons of efficiency,
scalability and portability.   They are there to solve
real world problems.  Having a simple protocol used
by simple developers will not make those problems
go away.

regards

PS. while I'm quoting Netcraft... I just noticed
this little gem:

  "More than 10 million websites were found running
   F5 BIG-IP devices, in our most recent Web Server
   Survey. F5's BIG-IP product family uses the TMOS
   platform to provide a modular approach to traffic
   management, and several distinct modules are
   available for tasks such as load balancing, SSL
   acceleration and fast caching."

So if websockets does not work well with intermediaries,
that's 10 million websites from just one vendor
that we disenfranchise.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 19 Apr 2010 10:59:57 +0200

> would disagree. But I sincerely hope that's not what Greg meant.

That is most definitely not what I meant.

As an open source developer, I've spent 15 years writing a
server that I give away for free.

What I meant is that I believe that the IETF is largely
compromised of individuals working for vendors who
have an interest in creating interoperable infrastructure.

The reason that we wish to create infrastructure, is not
part of some evil scheme to prevent application developers
from accessing the raw internet, and thus make millions
from selling our infrastructure products and services.
Rather it is because that we genuinely believe that
the best computer science solution to enable the
safe and efficient use of the network is to provide
some appropriate layer of infrastructure.

I don't oppose Ian's view of enabling application developers
to just crack open a socket and implement websockets because
I want to increase market share of Jetty.
I oppose it because it is a bad idea and will just result
in many insecure and bug ridden applications - not matter
how simple or trivial the protocol is.

regards

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 19 Apr 2010 11:08:54 +0200

Mike Belshe wrote:

> On Mon, Apr 19, 2010 at 12:52 AM, Greg Wilkins <[email-masked]

I believe that the vast majority of site owners to not control
the network infrastructure within the data centre in which
they are deployed.

But even if they do control them,  I agree that the problems of
non encrypted websocket interoperability between intermediaries
are definitely solvable.  I also think they should be solved by
this WG.    If not - who else?

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 19 Apr 2010 11:11:08 +0200

> have an interest in creating interoperable infrastructure.

s/compromised/comprised/

---

From: Mike Belshe <[email-masked]>
Date: Mon, 19 Apr 2010 02:14:46 -0700

> this WG.    If not - who else?

I agree a spec can be created.  But, existing proxies and intermediaries may
or may not work with it - they pre-exist the spec, so they aren't out of
compliance with anything.  Unfortunately, it takes many years to update all
the intermediaries, and it is just unknown what the ramifications of that
will be for websockets.  A similar historical point is HTTP/1.1 pipelining.
 In that case, the spec was ratified in 1999, and as of today, none of the
major browsers support it.  I hope that the same thing does not happen here.

But more importantly, SSL is better for everyone anyway (see previous
thread) and solves deployment too.  So we should just switch to it :-)  (Ok,
Ok...  I'll stop beating my drum now!)

Mike

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 10:17:37 +0100

> tunnel between the user agent and the server.

More secure, but higher latency and slower.

Going by the "each application on each open web page should open its
own connection" approach, and encrypted tunnels enforcing that, as
well as enforcing TLS setup latency for each one, *much* slower than
HTTP.  Not to mention the torrent of idle keepalives with so many
connections.

Less robust - I put together an alternative to Ian's "tic tac toe"
using long-GET HTTP.  It's a similar amount of code with a similar
structure, and as far as I can tell fails in all the exact same
scenarios, except for one: The HTTP version keeps working after a NAT
or SPI router times out in the middle.  The WebSocket version breaks
at that point, because it won't detect the broken TCP connection or
create a new one.

One of the not so obvious strengths of HTTP, which probably wasn't
deliberate design, is it recovers well from contemporary TCP-breaking
routers.

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Mon, 19 Apr 2010 11:20:13 +0200

> Web Sockets than of HTTP if we keep the protocol trivial.

Yes. That's a given. Make it less complex, and it will be easier to 
completely implement.

> Unfortunately, I don't know of any way to test these hypotheses.

OK. So let's get back to on-topic discussions.

Best regards, Julian

---

From: Mike Belshe <[email-masked]>
Date: Mon, 19 Apr 2010 02:22:48 -0700

> HTTP.  Not to mention the torrent of idle keepalives with so many

A few notes:
   - The bulk of the time goes to round trips.
   - WebSockets has a round trip for the upgrade header too.

> at that point, because it won't detect the broken TCP connection or

The NAT point is a great one.  I don't have any data on how often long-lived
connections are broken by NAT.  We need this.  Unfortunately, I bet may NATs
just drop the connection without notifying the endpoints (e.g. never sends a
RST).   This could become problematic for websockets, I'm not sure.
 Keepalives may help some, but I bet NATs are smart enough to break even
there too.  Data is needed....

> deliberate design, is it recovers well from contemporary TCP-breaking
True true true!  Lots of short connections, although less efficient, are
self-healing and robust.

Mike

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 19 Apr 2010 11:31:26 +0200

> misrepresentation to characterize it as a disaster.

This.  Having written multiple HTTP servers, I have to say that the
HTTP spec is just a bit complex but easy to implement securely, and
with perfect interoperability from the start.  It is hard to see how
this could _less_ of a disaster in terms of getting multiple
server-side implementations. There are literally thousands of them.

The market is dominated by a few because of natural monopolies on
back-end application infrastructure.  Platform monopolies, if you
like.  That has nothing whatsoever to do with HTTP.

-Pieter

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 19 Apr 2010 11:41:03 +0200

> and thus a far more heterogeneous environment.

You are IMO absolutely wrong in your assertion that HTTP's complexity
is the cause of market domination by a few products.

Have you actually implemented HTTP?  It is trivial to implement.  The
difficulty is not HTTP but writing a stable scalable server that can
handle all the diversity of use the real world demands.  You cannot
brush this under a select(3) carpet.

The web server market is based around a few dominant application
stacks.  Will these disappear with WebSocket?  You have an alternative
to mod_php?  To .NET?  What's your proposal to those who want to
connect that websocket to a database?  The web community has built
several mature answers to that question and those answers dominate the
market.

You are absolutely on the wrong track if you believe that a "simpler
protocol" will somehow be more successful.  Especially if you do not
understand the reasons for HTTP's success, and try to focus debate on
non-existent weaknesses.

-Pieter

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Mon, 19 Apr 2010 11:52:50 +0200

>"good enough" for almost everything.

The reason is more likely to be agglomeration; people settle where other
people settle; web servers require considerable work and skill to devel-
op, maintain, and integrate with other parts of the network, or the de-
velopment environment; the more you spread these resources across imple-
mentations, the higher the cost of individual deployments: educational
resources are harder to come by, extensions and adaptions have to be de-
veloped from scratch more often. The implementations consequently do not
particularily compete on matters of the underlying protocols, but e.g.
how they exploit system resources and support development resources.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: <[email-masked]>
Date: Mon, 19 Apr 2010 13:17:25 +0200

Hi,

Mike Belshe wrote:
On Mon, Apr 19, 2010 at 2:17 AM, Jamie Lokier <[email-masked]<mailto:[email-masked]>> wrote:

The NAT point is a great one.  I don't have any data on how often long-lived connections are broken by NAT.  We need this.  Unfortunately, I bet may NATs just drop the connection without notifying the endpoints (e.g. never sends a RST).   This could become problematic for websockets, I'm not sure.  Keepalives may help some, but I bet NATs are smart enough to break even there too.  Data is needed....

I agree this is an important point. If I'm writing an application on top of the WebSocket API, is it my responsibility to deal with keepalives and connection failures due to NAT timer resets or reboots, or does the protocol do somethig for me in this regard? I suppose the current draft (-75) is pretty silent on this, which means it's all up to the application programmer. That sounds a bit unreasonable if we want the app development to be really simple. Probably, even if the app developers figured out how to do it, they probably would not do it in an optimal manner. So, I suppose it would be good to have either an extension (or a subprotocol) to deal with this. At least it's a piece of code that all client side apps will need anyway. I am not sure to which level it needs to be standardized, but there has to be some way to get it done.

Is this still an open issue or has it been discussed and resolved in some way already?

Markus

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 13:10:00 +0100

[email-masked] wrote:
>    Unfortunately, I bet may NATs just drop the connection without
>    notifying the endpoints (e.g. never sends a RST).

Correct.  I've never seen a NAT send a RST.

In fact they *can't* send RST if the drop is due to routing update or
equipment reboots / failure.

Automatic routing updates are the most annoyingly subtle, because
typically nobody realises there's a problem, only that connections
drop from time to time.

>    This could become problematic for websockets, I'm not sure.

Yes, it will, unless applications use timeouts and automatic
reconnection.

>    Keepalives may help some,

Keepalives have two roles, and they actually have *distinct* timing
requirements:

    1. To tell NATs / SPIs / TCP relays / HTTP proxies (CONNECT)
       not to drop the connection.

    2. To tell the client application that the connection is still
       present so it knows when to abort and make a *new* connection.

The timeout needed for 1 is typically larger than the timeout for 2
but it depends on the app.

1 is network dependent; 2 is application dependent.
   
>    but I bet NATs are smart enough to break even there too.  Data is
>    needed....

I can give you personal experience.  Yes: NATs do break connections
with traffic on them.

   - I see about 1000 such breakages very day, admittedly on the same
     corporate national private network.

   - I see them occasionally elsewhere when connecting to servers.

   - Whenever a router is rebooted, this happens.
     This happens a lot in some houses with flaky ADSL routers.
     Even Apple's kit...

     (A very few routers are smart enough to persist NAT state across
     reboots, but it's rare, and I've never heard of it in a home router.)

>    I agree this is an important point. If I'm writing an application on
>    top of the WebSocket API, is it my responsibility to deal with
>    keepalives and connection failures due to NAT timer resets or reboots,
>    or does the protocol do somethig for me in this regard? I suppose the
>    current draft (-75) is pretty silent on this, which means it's all up
>    to the application programmer. That sounds a bit unreasonable if we
>    want the app development to be really simple. Probably, even if the
>    app developers figured out how to do it, they probably would not do it
>    in an optimal manner. So, I suppose it would be good to have either an
>    extension (or a subprotocol) to deal with this. At least it's a piece
>    of code that all client side apps will need anyway. I am not sure to
>    which level it needs to be standardized, but there has to be some way
>    to get it done.
> 

It is still being debated with no clear tracking and no resolution in sight.

There is currently no consensus on whether the application should
handle these network issues itself, or if the WebSocket implementation
should play a role.

Approximate divide: Browser implementers and WHATWG editors { ;-) }
appear to prefer the application handles these issues or ignores them
as it chooses choose, so that WebSocket implementation itself is simple.

Proxy and server implementers seem to prefer that WebSockets handles
it, so application programmers get a robust pipe without having to deal
with these issues (and it might use the network a bit better).

The current draft's direction is to push it to the application
programmer, either to keep WebSocket itself simple, or to make it a
thin wrapper around TCP (even though it's quite unlike TCP).

I suspect some supporters of latter approach are still thinking of the
80s and early 90s when a TCP connection was a reliable byte pipe.  It
isn't any more (it never was but the issues are more widespread now.)

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 13:21:36 +0100

Mike Belshe wrote:
> 
>    On Mon, Apr 19, 2010 at 2:17 AM, Jamie Lokier <[1][email-masked]>
>    wrote:
> 
>    Adam Barth wrote:
>    > On Sun, Apr 18, 2010 at 1:47 PM, Greg Wilkins <[2][email-masked]>
>    wrote:
>    > > Ian Hickson wrote:
> 
>    The NAT point is a great one.  I don't have any data on how often
>    long-lived connections are broken by NAT.

In my world, the long-lived ones are the ones broken most often :-)

I've seen a lot of NAT breakage in different situations: mobile
networks (30s timeout on some), home ADSL reboots, home NAT table
overloaded due to multiple users, corporate WANs daily at exactly 3pm
:-); it surprises me how much there is.

>    True true true!  Lots of short connections, although less efficient,
>    are self-healing and robust.

Even HTTP gets stuck, though, if the browser reuses a TCP connection
that has become black-holed.

The reason that isn't such a problem with Comet style is those
frameworks already have a timeout and reconnect on the client side to
deal with well-known brokenness :-) With some AJAXy apps, black-holed
persistent HTTP is still a problem, and it can take a while for the
apps to react and recover, if they recover at all.

I suspect a certain amount of "users knows the web is flaky and know
to reload the page if it's not working properly".  I certainly find
myself reloading Facebook from time to time because it's AJAXy
processing is reporting network errors, when on a mobile network.

-- Jamie

---

From: Vladimir Katardjiev <[email-masked]>
Date: Mon, 19 Apr 2010 15:34:25 +0200

> 1 is network dependent; 2 is application dependent.

#2 is the key, at least to me, and #1 a happy coincidence. The keepalive can tell the application when the connection is NOT there. To put this in the context, I have a very simple expectation of the WebSocket (browser) API.

onconnect() should tell me when the connection is ready for use
onclose() should tell me when the connection is dead. I can't use it anymore so I need to make a new one.

This is because after onconnect() I should be able to assume a persistent connection (that's the point of WebSockets, no?) until the browser tells me otherwise. All applications that depend on user-generated inputs (or unpredictable inputs in general) need it. Unpredictable inputs are things that the application can't know are going to happen, so the general application code is silent waiting for an event to occur.

Take, for no particular reason, the case of a web-based Tic Tac Toe application. Suppose it is the opponent's turn to move, but he went afk. Meanwhile, some NAT on the path decided it wants to close the connection. Since it's not my turn to move, my client should not have any reason to send traffic over the connection. It also doesn't have any set time it expects a response (because the other party is human the application has no way to know when he'll move). 

Suppose instead we added some way for one point (for the purposes of this discussion, let's say client but I don't want to exclude server /or intermediary/) to send a keepalive message as part of the protocol. This would be propagated to the endpoint in the given direction where the specification just says "echo it back". The burden of implementation on the server end is minimal, but it opens up for a number of improvements.

- Browsers can opt to include an adaptive keepalive algorithm that automatically keeps tabs on connection idling and sends a keepalive after a certain amount of time of no traffic. Trivial JS implementation is a function that has setTimeout(sendkeepalive(), 30000) and thus unequivocally sends a keepalive every 30 seconds even if there's activity on the line. 

- Nodes that know their connections won't be interrupted can opt to drop the keepalives between themselves and just replicate them at the edges. This may not be sexy, but now apply it between a mobile phone client and its first-hop proxy. If it's a WS-aware proxy, and there's some form of keepalive negotiation, they can determine that keepalives are unnecessary, or have a longer timer than is usual (within a controlled network this is possible) and not send them. For a mobile phone, this can incur significant battery savings (or, rather, not doing it may kill a smartphone's battery in less than a day for only keepalives). The keepalives can be regenerated at the edges so the target server wouldn't see anything different.

- The client can actually be told when a connection is dropped by onclose(wasClean=false) being triggered, and thus reestablish the connection. I mean, if you open a connection, surely you want to know if it's no longer available...

- Our hypothetical amateur programmer, testing his websocket connection against localhost, won't see the need for keepalives, but the protocol requiring them will stop his application from failing on the Wild Web. All he needs to do is echo back the bytes the browser sent, and the expert browser programmer handles the rest.

So I'd rather say I am having trouble seeing what applications would _not_ want keepalive functionality as part of the base offering of WebSockets.

> I can give you personal experience.  Yes: NATs do break connections

Of course, that's not the only thing that can fail silently. Lost connections due to other issues than NATs are also prevalent, anything from a cord being cut to someone using a wireless connection and going into a tunnel.

(This only appears to contradict what I said above on mobile keepalives if you assume all mobile networks are equal. They're not. What I want to say though is that even though we assume a general case where the connection WILL fail if it's left on its own, we should make it possible to make the transfer more efficient)
[...]
> with these issues (and it might use the network a bit better).

This really depends on how you define "robust". I'm okay with WebSockets failing because the recovery conditions aren't necessarily easy to define, and for some values of robustness you need to do stuff like deferring messages on the server-side, and then you need to identify the connection that requested them, and then authenticate it (if needed) and then you need to determine when you're NOT waiting for the robustness recovery, and it just goes on and on forever, much like this sentence.

So, yeah. Failure is good. My preference, though, is that the protocol itself takes care of the failing part if the failure is due to the network conditions, so anything written on top of the protocol doesn't have to keep doing the same old networking traps every. single. time. Though it doesn't hurt if the protocol also minimises the risk of failure, and, go figure, keepalives do both. Keepalives for President!

Vladimir

---

From: Jamie Lokier <[email-masked]>
Date: Mon, 19 Apr 2010 15:04:23 +0100

Vladimir Katardjiev wrote:
> Take, for no particular reason, the case of a web-based Tic Tac Toe
> application. Suppose it is the opponent's turn to move, but he went
> afk. Meanwhile, some NAT on the path decided it wants to close the
> connection. Since it's not my turn to move, my client should not have
> any reason to send traffic over the connection. It also doesn't have
> any set time it expects a response (because the other party is human
> the application has no way to know when he'll move).

Yes, that's the main way in which WebSocket Tic Tac Toe breaks.

> - The client can actually be told when a connection is dropped by
> onclose(wasClean=false) being triggered, and thus reestablish the
> connection. I mean, if you open a connection, surely you want to
> know if it's no longer available...

Sounds like a good plan :-)

> - Our hypothetical amateur programmer, testing his websocket
> connection against localhost, won't see the need for keepalives, but
> the protocol requiring them will stop his application from failing
> on the Wild Web. All he needs to do is echo back the bytes the
> browser sent, and the expert browser programmer handles the rest.

Keepalives are great, but this isn't a nice way to implement them.

If they are required by the protocol, then the protocol should
implement them itself.  (Besides, echo-reply isn't always bandwidth
efficient, depending on keepalive the needs of client and server).

> So I'd rather say I am having trouble seeing what applications would _not_ want keepalive functionality as part of the base offering of WebSockets.

Well, there may be disagreement over what type of keepalive, because
the optimal choice is both application dependent _and_ network link
dependent.

At the moment, it can be implemented by the WebSocket application
(albeit suboptimally, and any network link dependency has to be known
to the application as well).

> Of course, that's not the only thing that can fail silently. Lost
> connections due to other issues than NATs are also prevalent,
> anything from a cord being cut to someone using a wireless
> connection and going into a tunnel.

Yes, although those might be considered less transient errors :-)

The thing with NAT reset is that TCP won't recover, but reconnection
works immediately.

If you go into a tunnel and come out, TCP will recover.  If you cut a
cord, TCP won't recover but neither will reconnection work immediately.

So NAT reset is a bit different from other network faults.
Quick detection and recovery is useful with it.

> (This only appears to contradict what I said above on mobile keepalives if you assume all mobile networks are equal. They're not. What I want to say though is that even though we assume a general c…

Making the failure event more efficient by pushing keepalive or other
link detection down to the lowest reliable level is generally a good idea.

> This really depends on how you define "robust". I'm okay with
> WebSockets failing because the recovery conditions aren't
> necessarily easy to define, and for some values of robustness you
> need to do stuff like deferring messages on the server-side, and
> then you need to identify the connection that requested them, and
> then authenticate it (if needed) and then you need to determine when
> you're NOT waiting for the robustness recovery, and it just goes on
> and on forever, much like this sentence.

In general you *can't* make a protocol fully robust.  There is always
a failure mode.  But:

> So, yeah. Failure is good. My preference, though, is that the
> protocol itself takes care of the failing part if the failure is due
> to the network conditions, so anything written on top of the
> protocol doesn't have to keep doing the same old networking traps
> every. single. time.

There are very good reasons to distinguish certain classes of failure:

   - Non-failures!  (Peer closes connection due to timeout / resource limit)

   - Those you can recover from quickly and safely (NAT reset)

   - Those which are recoverable but need user conformation (POST data again?)

   - Those which are treated as unrecoverable (DNS failed, or 404 Not Found)

Some of the above list is best done in the WebSocket using
application, but graceful close and NAT reset detection are debatably
better helped by lower layers.

The key characteristic is which ones result in different *desirable*
behaviour from WebSocket application code.

When a condition is an expected behaviour of the network (server
closes connection, NAT reset) *and* it's possible to reopen a
connection and repeat the same messages without causing any problems -
those are particularly useful to distinguish.

Server timeout is also useful to distinguish for a user-interface
reason: User should not be told there is an error when something
happens in normal operation and is not an error.

-- Jamie

---

From: Vladimir Katardjiev <[email-masked]>
Date: Mon, 19 Apr 2010 16:29:23 +0200

> efficient, depending on keepalive the needs of client and server).

Yeah, it's not the best but it fulfils the design pattern that has been pre-eminent throughout the WebSocket spec -- that everything should be verifiable by the client, because that is the only "trusted" side. While I don't agree with the assumption that only the client should ever be trusted, it is a sufficient base case and only incurs trivial complexity increase by the server. Better an agreement on _a_ keepalive than no keepalive agreement?
> the optimal choice is both application dependent _and_ network link

I'd love to have that discussion. It means there'd be an acceptance of keepalives as part of the WebSocket spec.
> to the application as well).

Yeah... question is if this functionality isn't better handled by WebSockets for a majority of the applications, and the rest can opt out of it.

Vladimir

---

From: Vladimir Katardjiev <[email-masked]>
Date: Mon, 19 Apr 2010 16:47:24 +0200

> the TLS connection overhead is too expensive to justify.

Fair enough, it wasn't originally. But wouldn't it be better if it did? Let's take a couple of hypothetical scenarios where an amateur programmer can _transparently_ gain additional, desirable functionality by an intermediary "messing" with a well-defined protocol.

1. Compression
For all the talk about transferring 5gb using WebSockets, the FCC still maintains a fairly conservative definition of "broadband" at 768kbps. I believe the US average broadband speed was a handy contribution to the speed savings that SPDY's compression of data made possible. In other words, it's not unlikely that a future iteration of WebSockets adds compression (let's say gzip) to the equation.

Obviously, for this to work it needs to be optional, because otherwise we raise the barrier of entry. But in some deployments it is a desirable thing for the ISP to add. An ISP might want to compress the traffic to lower the load on their edge network (or their 768kbps broadband customers) In the case of a mobile network, the shorter time the radio is kept alive, the more battery savings, so it's highly desirable to compress WebSocket connections if appropriate.

2. Optimisation
Suppose we had a way of determining that a client cannot receive messages larger than 1mb, due to memory restrictions. (Feel free to substitute it with 5gb for the 2020 scenario). Wouldn't it be a waste of time to send all that data to the remote client only for it to discover that state on its own? For a mobile client, if the behaviour on too large frames is well-defined, we may even handle it without waking the client up (or close the connection if that's the defined behaviour) potentially saving a lot of bandwidth and battery.

3. Error checking
A properly written intermediary is also expertly written. So it would be able to prevent protocol exploits that may appear. In addition, these types of intermediaries would be maintained for the purpose of security and thus updated faster than you could hypothetically update client(s) and server(s). While it doesn't eliminate the spread of exploits and malware, it can reduce it. If we are supposed to be as concerned as we are about security, we should also cover the possibility that there may be a security hole in the future (one that we couldn't consider now) and make sure it can be protected. An intermediary can be a part of such a solution, but that also implies we must have intermediaries that can talk WebSockets.

4. Deployment
This falls into the category of "making the protocol actually work", but I do count it as a benefit to the amateur programmer because, yeah, I don't quite think TLS is so trivial to implement either. I don't remember where the original reference is, but I don't think creating a protocol whose design goal is to be wrapped end to end in TLS to simulate having open network access is going to work.

For one, you can't guarantee having no intermediaries with TLS (for the amateur programmer). All you need to do to MITM a TLS connection is, well, an intermediary box (I'd wager most ISPs can arrange one of those) and a signed certificate (shouldn't be an issue either). Since the WebSocket connection is user transparent they won't notice that the certificate no longer says Bank.com but ISP.com. Or Corporation.com for that matter. (I won't discuss ethics/morality/legality, just technology)

(course, this won't work if the TLS box shares https and wss traffic, but then we're not dealing with the amateur programmer are we?)

I don't think working on bypassing intermediaries for all traffic is a productive avenue. Rather, I think it'd just lead to a pointless arms race. It was my impression that what this wg was out to make was a protocol that played "nice" with both nonexistent, current and future intermediaries.

Don't get me wrong, the actual data passed through the WebSocket connection should be (possible to be?) opaque to any intermediaries, but the fact that it is a message, or a keepalive, or a graceful close, should be information available for intermediaries. 

Vladimir

---

From: Roberto Peon <[email-masked]>
Date: Mon, 19 Apr 2010 08:17:40 -0700

> HTTP.  Not to mention the torrent of idle keepalives with so many

I think that opening up many sockets to one endpoint is crazy-- having to
overcome Tcp-slowstart that many times is latency suicide.

-=R

---

From: Roberto Peon <[email-masked]>
Date: Mon, 19 Apr 2010 08:22:15 -0700

> Yes. That's a given. Make it less complex, and it will be easier to

This isn't true! Make it (the protocol) less complex and it will be easy to
implement something which *conforms to the spec*, but not necessarily
something which scales and is robust, reliable, and scalable in the face of
all the stuff that happens out there.

The latter part is what really worries me. We really need to be sure that
the protocol that we create allows for an implementation of a server to do
these things. If the (hopefully small) added complexity is too much for the
amateur programmer, then they should use the API level. If we don't focus on
these things, we'll vastly increase the cost of infrastructure while
decreasing reliability. I don't see that as a positive tradeoff.
-=R

---

From: Julian Reschke <[email-masked]>
Date: Mon, 19 Apr 2010 17:33:35 +0200

On 19.04.2010 17:22, Roberto Peon wrote:
>
>
> On Mon, Apr 19, 2010 at 2:20 AM, Julian Reschke <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> This isn't true! Make it (the protocol) less complex and it will be easy
> to implement something which *conforms to the spec*, but not necessarily
> something which scales and is robust, reliable, and scalable in the face
> of all the stuff that happens out there.

Yes, I agree with that. So the "complexity of implementing the protocol 
completely" isn't the only aspect we should consider.

> The latter part is what really worries me. We really need to be sure
> that the protocol that we create allows for an implementation of a
> server to do these things. If the (hopefully small) added complexity is
> too much for the amateur programmer, then they should use the API level.
> If we don't focus on these things, we'll vastly increase the cost of
> infrastructure while decreasing reliability. I don't see that as a
> positive tradeoff.
> -=R
> ...

Indeed.

Best regards, Julian

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 19 Apr 2010 17:57:03 +0200

> Web Sockets than of HTTP if we keep the protocol trivial.

This is a meaningless goal.  "Number of complete implementations" does
not measure success.

> Unfortunately, I don't know of any way to test these hypotheses.

As CEO of Wikidot.com, I can explain exactly how we chose our web
servers as our service grew to 0.5M users.  We used FastCGI to avoid
dependency on any specific server.  We started with Apache for obvious
reasons (natural monopoly in skills, it's the default server in
Linux).  We then switched to lighttpd because Apache did not scale.
We then switched to nginx because lighttpd would crash.

Nowhere does protocol complexity come into it.  We have more than
enough choice of excellent implementations.  However we do not choose
randomly but start with the default server.  We then change only when
there is an issue we cannot resolve otherwise.

Imagine a simpler protocol (that magically provides as much value as
HTTP) that has 1M complete implementations... it's so easy that every
student coder writes a complete implementation before breakfast, for
fun.

You will still find a market dominated by two implementations: the
default Windows one, and the default Linux one.

Protocol complexity has precisely zero to do with usage patterns, and
platform defaults have everything to do with them.

-Pieter

---

From: <[email-masked]>
Date: Mon, 19 Apr 2010 21:53:30 +0200

Hi,

How about this:

I think the websocket spec should define two special frames explicitly for keepalive purpose:
- Keepalive request frame; this is sent by the client
- Keepalive response frame; this is sent by the server as a response to the keepalive request
Depending on what the websocket protocol syntax allows, these frames could be actually just single bytes (frames starting between 0x80 and 0xFF seem to be reserved for something special, so we would use two of these code points).

The client would generate the keepalive requests as often as needed when there is no other traffic frequently enough. By the response from the server the client would know that the transport connection (TCP) is still up, the NAT and intermediary state is thus also up, and actually the server is also alive on the app layer.

How often the keepalives are sent is a tricky question, and may be beyond what is defined in the websocket protocol, as it depends so much in the environment (the middleboxes in the access networks, the web intermediaries and so on), and on the other hand sending too often consumes battery on wireless devices. The client can find out the optimal rate in its current environment by trial and error. What could be done is that the server might be able to hint the client on a suitable rate, although typically the server has no idea on e.g. NAT timeouts in the network the client is located (but in principle it could remember which values worked last time from that particular IP range). That would make the keepalive frames slightly more complicated but still within a few bytes. Yet another option is actually to standardize some kind of keepalive rate iteration algorithm.

Basically all other protocols that need to maintain long-lived TCP connections have done this: SIP and XMPP exchange 'CRLF's between client and server, for instance. HTTP based long-polling protocols send entire HTTP requests.

Markus

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of ext Vladimir Katardjiev
Sent: 19 April, 2010 17:29
To: Hybi
Subject: Re: [hybi] NAT reset recovery? Was: Extensibility mechanisms?

Vladimir

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 19 Apr 2010 22:09:13 +0200

On Mon, Apr 19, 2010 at 9:53 PM,  <[email-masked]> wrote:

> I think the websocket spec should define two special frames explicitly for
> keepalive purpose...

For what it's worth, the keep alive design from AMQP used only one
frame.  Senders are responsible for issuing KAs at regular intervals.
Receivers treat these as no-ops except to register "activity" on the
connection.  Any other frame also acts as activity, obviously.  If no
activity is received within X seconds, the connection can be killed.
There is no notion of request-response for KA in AMQP.

-Pieter

---

From: <[email-masked]>
Date: Mon, 19 Apr 2010 22:17:43 +0200

Pieter Hintjens wrote:
>> I think the websocket spec should define two special frames 
>explicitly 
>> for keepalive purpose...
>
>For what it's worth, the keep alive design from AMQP used only 
>one frame.  Senders are responsible for issuing KAs at regular 
>intervals.
>Receivers treat these as no-ops except to register "activity" 
>on the connection.  Any other frame also acts as activity, 
>obviously.  If no activity is received within X seconds, the 
>connection can be killed.

This is another possibility, and it's even simpler. But how fast would the sender be able to detect that the keepalive did not go through and there's something wrong with the connection? Only after TCP has given up? Also an explicit response would tell something more about the liveliness of the server than a mere TCP ack. 

I'd prefer request-response pattern. 

Markus

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 19 Apr 2010 22:26:17 +0200

> I'd prefer request-response pattern.

Which works only if the protocol is fully synchronous.  I thought
WebSockets was async...?

-Pieter

---

From: <[email-masked]>
Date: Mon, 19 Apr 2010 22:54:54 +0200

Hi Pieter,

Pieter Hintjens wrote:
>Which works only if the protocol is fully synchronous.  I 
>thought WebSockets was async...?
>

Hmm... Even if the protocol as a whole is asynchronous (both ends can generate frames at any time), isn't it still possible to exchange requests and responses as well over the same transport connection?

I guess a special situation arises when a client sends out a keepalive request and before responding, the server starts to send a data frame in the other direction (since these two events can happen asynchronously). But then the server just has to put the keepalive response in its sending queue. (In that case such a response would seem a bit redundant, since the client can see that the connection is up when it receives the data frame). I assume it would still possible to match the response to the request. 

Or am I missing something obvious here? 

Markus

---

From: Pieter Hintjens <[email-masked]>
Date: Mon, 19 Apr 2010 23:05:10 +0200

> Hmm... Even if the protocol as a whole is asynchronous (both ends can generate frames at any time), isn't it still possible to exchange requests and responses as well over the same transport connec…

Here is the logic we used when designing AMQP's KA mechanism.

Let's say we use a req-resp model over a symmetric async connection...

* if the other party is sending us stuff, KA responses are redundant
* If we're sending stuff, KA requests are redundant
* Since we're symmetric, both peers would send requests when idle
* Thus a response is always redundant

And it boils down to sending NOOPs when otherwise idle, to ensure the
other party knows we're still alive (as a process).

Incidentally this simplified async KA is also much easier to
implement, requiring no state at the recipient.

-Pieter

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 19 Apr 2010 23:17:03 +0200

> Hmm... Even if the protocol as a whole is asynchronous (both ends can generate frames at any time), isn't it still possible to exchange requests and responses as well over the same transport connec…

Another niceness of having request/response KA's on the async connection is
that it gives us a cheap and simple form of message acknowledgement.

If we receive a response to a KA, then we know that all messages
sent prior to the KA request have been delivered to the other endpoint.

cheers

---

From: Vladimir Katardjiev <[email-masked]>
Date: Mon, 19 Apr 2010 23:36:19 +0200

> implement, requiring no state at the recipient.

Don't both sender and receiver need state? Each node needs to know:

- When it last sent an outgoing message (so it can send a noop when it's time)
- When it last received an incoming message (so it knows when to treat the connection as "dead")

Having said that, from a mobile perspective this is disastrous. In the worst-case scenario, if your keepalive interval is 5 minutes, you could end up sending a keepalive message every 2.5 minutes. This would wake up the radio twice as often, and, consequently, halve battery life. In a scenario where the server's keepalive is triggered by the client's, the message from the server would likely arrive within the timeframe that the radio is "awake" anyway (3-5 seconds on smartphones) and thus avoid waking it up again. 

The usage of keepalive as an ack was an interesting proposal, although I'm not sure if the websocket level can properly define an ack... It would only mean the message had been received, not necessarily processed, so I'm not sure what the benefit is. From a data integrity perspective, the interesting ack is "processing complete", not "delivery performed". The latter we have on some level via TCP already.

Vladimir

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 02:16:36 +0100

> - When it last received an incoming message (so it knows when to treat the connection as "dead")

Yes.  But they don't need to remember if they've sent a "KA request"
or how many they have sent.

Request-response versus "async KA": Efficiency depends on timeouts
-----------------------------------------------------------------

Adding to the above: Request-response is three TCP packets (data,
data+ACK, ACK).  Mutual async keepalive is four packets in the same
time interval (data, data+ACK; data, data+ACK).

So it's more bandwidth efficient on any network when the keepalive
intervals are *symmetric*.

But you don't usually need symmetric keepalive times.  The client
often needs faster notification of lost connection than the server,
so that the client can reconnect to ensure it's up to date with
pending messages.  Whereas the server doesn't really care, except to
clean up unused connections (usually).

(That may even be true for symmetric peers once connection is
established: if it's always the same side which needs to know when to
reconnect and there is no other application state requirement.)

Let's use an example of client needing confirmation every 30 seconds
(to ensure client state is never >30 seconds out of date), and the
server needing 5 minutes (to clean up old connections).

In that situation, "async keepalive" is more efficient, because the
server sends KA messages every 30 seconds (when idle), and the client
sends a TCP ACK equally often, plus another exchange every 5 minutes -
that is more bandwidth efficient *and* more radio-power efficient than
request-response.

A smarter combination of both is for the slower KAs to be timed to
coincide with acknowledges from the faster KAs, to piggyback upon them.

Point is, there is an application-led aspect to keepalives (the 30
seconds in that example), a network-led aspect (whatever is needed for
NATs), and a WS-level aspect (whatever is needed by the server
cleaning up idle connections, 5 minutes in that example).

TCP ACK - no, sorry
-------------------

> The usage of keepalive as an ack was an interesting proposal,
> although I'm not sure if the websocket level can properly define an
> ack... It would only mean the message had been received, not
> necessarily processed, so I'm not sure what the benefit is. From a
> data integrity perspective, the interesting ack is "processing
> complete", not "delivery performed". The latter we have on some
> level via TCP already.

You can't rely on TCP ACK for this, because of unknown TCP relays and
HTTP proxies which might sit in between.

However, if you can confirm somehow that there is no TCP relay in the
way, then perhaps it can be used to be a little more bandwidth efficient.

Per-hop versus end-to-end
-------------------------

The most efficient place to put these keepalives is often per-hop (or
the shortest hops between active participants), not end-to-end - but
it does depend on what exactly is being confirmed.  For keeping NATs
open, per-hop is always good.  It lets you use fewer keepalive packets
for the same NAT timeout due to not needing to subtract end-to-end
latency, and it can be adapted with knowledge of each hop.  For
applications, per-hop isn't always suitable - it depends what semantic
the application associates with keepalive.

End-to-end adds more latency and jitter.  Depending on the various
requirements (including NAT), that may cause you to need a smaller
keepalive interval to achieve the same thing.

Connection aggregation
----------------------

For applications, sometimes keepalives can be shared over multiple
connections on the same hop (even between independent apps).  This is
because keepalive is a way of telling the app that not only is the
connection open, certain conditions have not changed since before.

For NATs, and for detecting specific connection failures, keepalives
cannot be shared.

If there are any multiplexing proxies which are aggregating
connections into fewer connections, then it is very good to defer
keepalives to the proxy (as in per-hop), so that it can use fewer
keepalives than one per original connection.  This can be quite a
significant saving.

Even if WebSocket never defines a multiplexing extension, this is a
particularly good reason to allow keepalive to be implemented per-hop,
so that a proxy using a different protocol to tunnel multiple WebSockets
can provide the keepalive function efficiently.

Radio power optimisation
------------------------

> Having said that, from a mobile perspective this is disastrous. In
> the worst-case scenario, if your keepalive interval is 5 minutes,
> you could end up sending a keepalive message every 2.5 minutes. This
> would wake up the radio twice as often, and, consequently, halve
> battery life. In a scenario where the server's keepalive is
> triggered by the client's, the message from the server would likely
> arrive within the timeframe that the radio is "awake" anyway (3-5
> seconds on smartphones) and thus avoid waking it up again.

Interesting point about the radio.  Now consider when you have 10
browser windows open on your phone (this is normal on my phone...),
each with 5 WebSocket-using applications...  The poor radio will be on
constantly if keepalives aren't synchronised.

Per-hop is a great place to put mobile radio optimisation, because you
can use a completely different mechanism.  Even a non-IP mechanism.
SMS has been mentioned; I don't know if there are other mobile network
signalling mechanisms available.

With multiple connections, if they need separate keepalives, then
there might be big power benefits to synchronising their timing, to
fit in the same power slot.

The request-response pattern has been noted as potentially good for
radio power consumption, provided it occurs fast enough to fit into a
single power slot.  In fact it's like TCP packet usage described
earlier: with symmetric timeouts, request-response is good for radio
power too.  With highly asymmetric timeouts, "async keepalive" is
better for radio power too.

Summary
-------

Keepalive optimisation is a fairly complicated area in general.  What
is good for one application can be expensive for another.  Their
aggregation and timing synchronisation are also potentially significant.

We should avoid over-designing it in WebSocket - we'd get it wrong
anyway - and we should avoid having a mandatory, over-simplistic
keepalive type because there are so many useful variations.

Unfortunately some aspects of efficient keepalive are impossible for
applications to implement on top of WebSocket.  For example
synchronisation among different applications to minimise radio power.

So, imho, we should make sure there is a place to hook in different
mechanisms as they are needed, by each application and link type, in a
future-extensible sort of way.

That probably means defining some API events (like heard-from or
not-heard-from), a generic way to request keepalive parameters
(strings & values go a long way!), and a place in the protocol for
transport-level negotiation of them, including per-hop.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 02:26:28 +0100

Greg Wilkins wrote:
> > Hmm... Even if the protocol as a whole is asynchronous (both ends
> > can generate frames at any time), isn't it still possible to exchange
> > requests and responses as well over the same transport connection?
> 

Nice, except for the cost when you don't need that acknowledgement...

This is a good reason to have a "ping" request, guaranteed to be fully
end-to-end, in addition to any other KA type.

"ping" is also useful for when you haven't received a KA for a little
while (less than the KA interval, but might still be minutes), and you
want to confirm that the connection is actually still alive right now.

Note that if KAs are handled by WebSocket itself, you only know the
message has been delivered to the endpoint's transport queue.  You
don't know that it will reach the endpoint's *application* unless it's
the application which will produce the response, and often that is
something you would like to know.

For these reasons I suggest that "ping" functionality be defined
separately from keepalive functionality, and its semantics made
explicit.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 02:32:20 +0100

[email-masked] wrote:
> This is another possibility, and it's even simpler. But how fast
> would the sender be able to detect that the keepalive did not go
> through and there's something wrong with the connection? Only after
> TCP has given up? Also an explicit response would tell something
> more about the liveliness of the server than a mere TCP ack.

An endpoint knows that the connection is down because it hasn't
*received* a keepalive after too long.

The speed of discovery depends on an agreed timing with the other end.

> I'd prefer request-response pattern. 

You obviously don't want both ends sending requests, so at least one
of them has to passively timeout waiting to receive a keepalive, and
then decide the connection is down.

AMQP just does that at both ends, instead of one.

Request-response from one end only does not remove the need for a
passive timeout agreed between the endpoints.  (It can be negotiated,
or pre-agreed).

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 02:40:41 +0100

> Or am I missing something obvious here? 

Perhaps the missing thing is that request-response is more complicated
in this kind of protocol, because you also have to define a way to
_associated_ each response with the request.  That means sequence
numbers, or identifiers, or being careful with counting response
messages distinct from non-response messages.

There are also subtleties: If the other end sends a graceful close
message, and then receives a KA request, does it send a KA response,
or does the graceful close mean that KA response is among the message
types that won't be sent?

Because it's more complicated, it's not necessarily a great pattern
compared with async keepalive, which is quite simple, and doesn't even
need to send a KA message from the server in your example.  (The data
frame is enough by itself).

However for packet efficiency, synchronising timing by using
request-response, or another synchronisation method, may still be
advantageous.  Even though it's more complicated than async keepalive.

-- Jamie

---

From: Pieter Hintjens <[email-masked]>
Date: Tue, 20 Apr 2010 08:01:10 +0200

> passive timeout agreed between the endpoints.  (It can be negotiated,

In AMQP each peer tells the other, during connection negotiation,
"please send me KA's at this interval".  Peers that do not respect
that (don't sent KAs) are disconnected.

-Pieter

---

From: Pieter Hintjens <[email-masked]>
Date: Tue, 20 Apr 2010 08:03:53 +0200

> messages distinct from non-response messages.

This is pretty much the problem we solved in AMQP by using async KAs.

> types that won't be sent?

Indeed... having to respond to a KA request makes a graceful close
much more complex.

> frame is enough by itself).

Yes.  A no-op frame is sufficient.

> advantageous.  Even though it's more complicated than async keepalive.

Complexity is always worth removing unless the overhead is significant
and since KAs can be entirely switched off when there is traffic at
all, and then reduced to once per 5 or 10 or 30 seconds, packet
efficiency seems irrelevant here.

-Pieter

---

From: "Thomson, Martin" <[email-masked]>
Date: Tue, 20 Apr 2010 14:17:21 +0800

> that (don't sent KAs) are disconnected.

That doesn't seem helpful.  Each peer knows what it wants and it can initiate keep-alives based on its own needs.  Forcing the other peer to acquiesce doesn't add value.

Furthermore, you don't need a response, TCP provides ACKs.  A one-way exchange of as little as a single byte should be enough to keep any bindings alive in (TCP) intermediaries.

If you want to step a layer up to WebSocket intermediaries the ping concept works there.  I'm not convinced that this is necessary.

--Martin

---

From: <[email-masked]>
Date: Tue, 20 Apr 2010 08:50:56 +0200

Hi,

Martin Thomson wrote:
>
>Furthermore, you don't need a response, TCP provides ACKs.  A 
>one-way exchange of as little as a single byte should be 
>enough to keep any bindings alive in (TCP) intermediaries.
>

This is true. 

I would say there are two options to do keepalives:
i) Define a keepalive frame that the client sends. There is no websocket level response but the TCP ACK is still enough to keep NATs, firewalls etc. open. 
Ii) Define a keepalive request frame that the client sends and keepalive response that the server uses to respond to it. The response is not really necessary for NATs etc., but might help the client to notice a broken connection faster. (I'm not sure if this is true but I remember this was an argument in another protocol, SIP, why keepalive over TCP was done in that way.) The response may make the protocol more complicated, though, as people have commented.

I would not want to make the keepalives peer-to-peer in a sense that also the server could send keepalives. I think the client should be in charge. The reason is that as a client developer for a mobile device I would like to be able to send keepalives as infrequently as possible. We have had some bad experiences with IMAP servers sending stuff frequently back to the client (presumably because not all clients are clever enough to do any keepalives themselves - this is a risk of course if we don't specify this at all in websocket spec), and the client has no way to affect their rate, which has been fixed.

If there really is a requirement for the server to see keepalives at a certain minimum rate (for instance so that it can abandon stale connections), I would then have that negotiated so that the server can tell that back to the client who would still do the sending. But I'm not sure if that's needed.

Markus

---

From: "Thomson, Martin" <[email-masked]>
Date: Tue, 20 Apr 2010 15:09:39 +0800

Responses are of some use at the application layer, so the second option might make some sense.

However, negotiation seems like overkill for this sort of mechanism.  Each peer has some expectations and constraints that it knows best.  Let them each take whatever action they need to ensure that the connection stays alive.

If a keep-alive frame is defined, either peer could unilaterally initiate it.  There's no technical reason why this shouldn't be allowed.  If the server cares, and it doesn't see any activity from the client, it could use the keep-alive too.

As Jamie points out, the server frequently doesn't care.  More to the point - the server can't do little about it if the connection does go away.  (I can imagine use of other channels to prod the client into wakefulness, but that's not something inherent to WS.)  Rather than outright prohibition, I'd instead prefer to discourage server use of keep-alive (SHOULD NOT).

Servers might know better - we have a server that has a 5 minute TCP "go dead" interval on the load balancer in front of it.  That server should not be outright prevented from sending a 4:55 keep-alive if it sees nothing from a client.

--Martin

> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, 20 April 2010 4:51 PM
> To: Thomson, Martin; [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: RE: [hybi] NAT reset recovery? Was: Extensibility mechanisms?
> 
> Hi,
> 
> Martin Thomson wrote:
> >
> i) Define a keepalive frame that the client sends. There is no
> websocket level response but the TCP ACK is still enough to keep NATs,
> firewalls etc. open.
> Ii) Define a keepalive request frame that the client sends and
> keepalive response that the server uses to respond to it. The response
> is not really necessary for NATs etc., but might help the client to
> notice a broken connection faster. (I'm not sure if this is true but I
> remember this was an argument in another protocol, SIP, why keepalive
> over TCP was done in that way.) The response may make the protocol more
> complicated, though, as people have commented.
> 
> I would not want to make the keepalives peer-to-peer in a sense that
> also the server could send keepalives. I think the client should be in
> charge. The reason is that as a client developer for a mobile device I
> would like to be able to send keepalives as infrequently as possible.
> We have had some bad experiences with IMAP servers sending stuff
> frequently back to the client (presumably because not all clients are
> clever enough to do any keepalives themselves - this is a risk of
> course if we don't specify this at all in websocket spec), and the
> client has no way to affect their rate, which has been fixed.
> 
> If there really is a requirement for the server to see keepalives at a
> certain minimum rate (for instance so that it can abandon stale
> connections), I would then have that negotiated so that the server can
> tell that back to the client who would still do the sending. But I'm
> not sure if that's needed.
> 
> Markus

---

From: Vladimir Katardjiev <[email-masked]>
Date: Tue, 20 Apr 2010 09:15:19 +0200

> This is pretty much the problem we solved in AMQP by using async KAs.

Is it really necessary to match responses to requests? The point is just to see if the connection is alive, no? All the keepalive would do is tell the server "hey, can you send me something". Logically, if the server already is sending data then it shouldn't need to send a keepalive back (because there's no need to establish liveness). So what interest would we have to be mapping these keepalive messages to each-other? What semantics would that derive? The best I can figure out is give the req/resp latency...

I mean, I can't really see why you'd need to match keepalive "responses" to which keepalive "request" that triggered them. Since we're mainly interested about ascertaining liveness, doesn't the fact that we receive a keepalive suffice? If there's traffic on the line, we can already deduce liveness, and if there's no traffic on the line, we can't send two keepalives in a row (because we'd timeout if we didn't receive some traffic in response to the first one). 

The reason I suggested "echo" though (at least for a basic case) is because it fulfils the "amateur programmer" requirement. The server is completely stateless, no timers, no nothing. This assumption even includes the server not keeping the state of if it's sending or not.

(To put it in RFC terms, if the server receives a KA, it MUST send some data to the client; if the server is already sending application data it MAY skip sending a keepalive back)
> Indeed... having to respond to a KA request makes a graceful close

I think this problem applies generally. Graceful close was supposed to let the remote side empty its buffer. But suppose it has 500gb in its sending buffer. That's going to take a while to gracefully close, so keepalives are still needed during this time period. And since the side that already sent graceful close won't send any more _data_ it needs to send keepalives.

Once both sides have sent a graceful close, what would be the point of replying to a keepalive? If the connection is alive, it would be closed first. If the connection is dead it's not going to make a world of difference.

(Of course, there is an argument for async KAs here, because if the server sent the graceful close, and the client is flushing a single 500gb frame, it can't send any KA frames, so the server would never echo back a KA in synchronous mode. Perhaps some sort of hybrid solution? It's a tricky question though, because my next point is also a concern...)
> efficiency seems irrelevant here.

Well, to be precise, my main concern isn't _packet_ efficiency so much as _power_ efficiency. Async keepalive keeps two separate timers on when to send, so in the worst case scenario (both intervals are equally long) some side will send a keepalive at a frequency that will be double that which synchronised keepalives would be. 

In a mobile scenario, what costs the most is radio uptime, not bandwidth. Whenever you need to set up a radio link to receive data, you need to spend a certain amount of time doing control signaling (during which time you're drawing power, but not getting any data). To minimise this, after a data link is set up it stays active for a couple of seconds, before going idle again.

For the purposes of keepalives, uncoordinated keepalives will almost always force the radio to wake up. This is bad enough for uncoordinated keepalives between different applications, but the problem would be doubled if each connection had two uncoordinated keepalives. All it takes is three uncoordinated keepalive timers to drain a modern smartphone's battery in the timespam of a single day -- and this is doing nothing but sending keepalives. If each connection has two uncoordinated timers, that means all it takes is two connections to empty the battery in a single day, and it only goes downhill from there.

This leaves us in the paradoxical scenario where sending keepalives from the client (slightly) more frequently than the keepalive timeout is actually better efficiency, because they can be sent when the radio is already up. But this is way overkill for a trivial WebSocket implementation. My interest is to have a base case of keepalives that is simple enough so that every server supports it, but flexible enough that I can have a smart client (smart phone? har har har) that can coerce the server into behaving well. Or that an even smarter client could cooperate with an intermediary on liveness and avoid many keepalives.

Vladimir

PS: Thanks Jamie for your in-depth replies! I'd like to look into more the per-hop keepalive option, especially if it can be done in an implicit manner.

---

From: Dave Cridland <[email-masked]>
Date: Tue, 20 Apr 2010 11:04:25 +0100

On Tue Apr 20 07:50:56 2010, [email-masked] wrote:
> I would not want to make the keepalives peer-to-peer in a sense  
> that also the server could send keepalives. I think the client  
> should be in charge. The reason is that as a client developer for a  
> mobile device I would like to be able to send keepalives as  
> infrequently as possible. We have had some bad experiences with  
> IMAP servers sending stuff frequently back to the client  
> (presumably because not all clients are clever enough to do any  
> keepalives themselves - this is a risk of course if we don't  
> specify this at all in websocket spec), and the client has no way  
> to affect their rate, which has been fixed.

I've seen this work in two different ways.

First, in IMAP, we've found that the client needs to be in control,  
as it often knows better - however, others (including Mark Crispin)  
found that having the server send keepalives when it'd otherwise be  
shutting down the session is useful. Note that in IMAP, there are  
technically no actual keepalives, but servers can send an ignorable  
"* OK", and in extreme cases, can fake a delivery to cause a client  
to do something. Clients have a NOOP command. I suspect a lot of the  
issues with IMAP relate to there being no simple way for the server  
to ping the client built-in to the protocol.

In XMPP, however, we've found that having the ability to ping the  
client from the server is very useful. In XMPP, there's two distinct  
protocol features which enable TCP-level keepalive. Firstly, either  
side may send a SP character between stanzas. Secondly, each side may  
actually ping the other. Having the server send a SP on an idle  
session is useful for defeating NAT timeouts, whereas the ping  
(XEP-0199) is extremely useful to definitively check reachability.

> If there really is a requirement for the server to see keepalives  
> at a certain minimum rate (for instance so that it can abandon  
> stale connections), I would then have that negotiated so that the  
> server can tell that back to the client who would still do the  
> sending. But I'm not sure if that's needed.

I think we need all cases possible, then some sensible advice to  
implementors. In the XSF, we've learned a lot about how to defeat NAT  
timeouts and get early failure detection, and the main thing we've  
learned is that things change - only a year or two back using  
long-lived TCP over mobile networks was really difficult, whereas now  
it's much better.

What's more interesting, to me, is what to do if the connection does  
break.

If each connection is granted an identifier, and - moreover - the  
client can request a specific identifer to be reattached during the  
HTTP Upgrade negotiation at a known sequence point, then in principle  
a WebSocket needn't be conceptually broken even if connectivity is  
entirely lost due to a NAT reboot.

Something very similar happens with XEP-0198, where XMPP clients can  
reacquire a running XMPP session after TCP is lost.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: <[email-masked]>
Date: Tue, 20 Apr 2010 14:26:57 +0200

Hi Martin,

What you say probably makes sense. So, it would be good to define the keepalive frame so that both the client and the server can send it. I'm just worrying about the following tradeoff:

Mobile clients would often want to send keepalives as infrequently as possible to save battery. They may use intelligent (unstandardized) mechanisms to learn what the mimimum rate acceptable for their access network NATs and firewalls are and use it. (As of today I believe the ballpark for TCP is something like a once per 10 minutes of idleness.) The servers on the other hand want to get rid of unused connections as fast as possible and may want to check the connection liveliness even faster. (I heard that some major XMPP servers send keepalives every 30 seconds.) For many clients (running in PCs etc.) this is probably fine but really bad for the clients in mobile devices. If the same device has multiple apps (websockets, XMPP, SIP, IMAP, ...) running it can even synchronize the keepalives it sends for all of them (so that radio is active only once during the cycle), but for the server initiated keepalives it can't do the same.

I've always wondered if all those warnings and nice words in RFCs help, but if/when we specify the server-initiated keepalive frame, the above case should be explained, so the servers would not use if carelessly. I think the gist should be so that the client is in charge of keeping NATs, firewalls, proxies and all that stuff (which is mostly related to the clients current point of attachment) happy, while the server should only worry about its own resources and those related to it, such as the load balancers. 

Markus

>From: ext Thomson, Martin [mailto:[email-masked]] 
>Sent: 20 April, 2010 10:10
>To: Isomaki Markus (Nokia-CIC/Espoo); [email-masked]; 
>[email-masked]
>Cc: [email-masked]
>However, negotiation seems like overkill for this sort of 
>mechanism.  Each peer has some expectations and constraints 
>that it knows best.  Let them each take whatever action they 
>need to ensure that the connection stays alive.
>
>If a keep-alive frame is defined, either peer could 
>unilaterally initiate it.  There's no technical reason why 
>this shouldn't be allowed.  If the server cares, and it 
>doesn't see any activity from the client, it could use the 
>keep-alive too.
>
>As Jamie points out, the server frequently doesn't care.  More 
>to the point - the server can't do little about it if the 
>connection does go away.  (I can imagine use of other channels 
>to prod the client into wakefulness, but that's not something 
>inherent to WS.)  Rather than outright prohibition, I'd 
>instead prefer to discourage server use of keep-alive (SHOULD NOT).
>
>Servers might know better - we have a server that has a 5 
>minute TCP "go dead" interval on the load balancer in front of 
>it.  That server should not be outright prevented from sending 
>a 4:55 keep-alive if it sees nothing from a client.
>
>--Martin
>
>> >Furthermore, you don't need a response, TCP provides ACKs.  
>A one-way 
>> >exchange of as little as a single byte should be enough to keep any 
>> >bindings alive in (TCP) intermediaries.
>> >
>> 
>> This is true.
>> 
>> over TCP was done in that way.) The response may make the protocol 
>> more complicated, though, as people have commented.
>>

---

From: <[email-masked]>
Date: Tue, 20 Apr 2010 14:42:47 +0200

Hi,

Dave Cridland wrote:
>
>Note that in IMAP, there are technically no actual keepalives, 
>but servers can send an ignorable
>"* OK", and in extreme cases, can fake a delivery to cause a 
>client to do something. Clients have a NOOP command. I suspect 
>a lot of the issues with IMAP relate to there being no simple 
>way for the server to ping the client built-in to the protocol.
>

Yes, this "* OK" is what I have seen. I agree that the lack of an explicit in-built mechanism is an issue and we should learn from that for WebSocket.

>Having the server send a SP on an idle session is 
>useful for defeating NAT timeouts, 

Here I'd really prefer the client to do it, as it should know better. And the host can potentially optimize between different connections etc.

>whereas the ping

Is that being used today by servers? 

>I think we need all cases possible, then some sensible advice 
>to implementors. In the XSF, we've learned a lot about how to 
>defeat NAT timeouts and get early failure detection, and the 
>main thing we've learned is that things change - only a year 
>or two back using long-lived TCP over mobile networks was 
>really difficult, whereas now it's much better.
>

Yes, things have improved. For instance, despite the looming IPv4 runout, many mobile operators have actually gotten rid of NATs and are giving out public addresses. There's however still a firewall with typically 10-60 minute timeout for TCP.

>What's more interesting, to me, is what to do if the 
>connection does break.
>
>If each connection is granted an identifier, and - moreover - 
>the client can request a specific identifer to be reattached 
>during the HTTP Upgrade negotiation at a known sequence point, 
>then in principle a WebSocket needn't be conceptually broken 
>even if connectivity is entirely lost due to a NAT reboot.
>
>Something very similar happens with XEP-0198, where XMPP 
>clients can reacquire a running XMPP session after TCP is lost.
>

This is definitely something to consider.

Markus

---

From: Dave Cridland <[email-masked]>
Date: Tue, 20 Apr 2010 14:32:27 +0100

On Tue Apr 20 13:42:47 2010, [email-masked] wrote:
> Yes, this "* OK" is what I have seen. I agree that the lack of an  
> explicit in-built mechanism is an issue and we should learn from  
> that for WebSocket.
> 
> 
What you'll also see on some IMAP servers is a "* EXISTS"/"* EXPUNGE"  
sequence sent at about the 29th minute of a client in IDLE mode, as  
that turns out to be the only way to "wake" some clients and check if  
they're there. This is a hideous hack, but the thing is that it's now  
too late (by 20 years) to mandate a server-initiated NOOP.

> Here I'd really prefer the client to do it, as it should know  
> better. And the host can potentially optimize between different  
> connections etc.
> 
> 
I'm open to suggestions. If the server sends a simple keepalive only  
on silent connections on a reasonably lengthy interval, this seems to  
be to cover most options - it in effect means that clients can do it  
sooner if they wish.

> Is that being used today by servers?
Absolutely, if you connect an XMPP client to jabber.org and leave it  
idle and silent, the server will eventually wonder if you're still  
there and ping you (it'll take about 5-7 minutes). Other servers may  
well do similar things.

Dave.

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 17:55:42 +0100

> >enough to keep any bindings alive in (TCP) intermediaries.

It doesn't work.

It's fine for intermediaries: they must be willing to accept data in
either direction as sufficient to prevent disconnection, because it's
quite common that data flows in one direction only for long periods.

But the client won't be able to detect that the connection is down and
and open a new one in a reasonable time with that mechanism.  Two reasons:

1. There is usually no operating system interface for the client to
detect TCP ACKs.

2. Any intermediaries will send TCP ACKs.  These are not confirmation
that the next hops and server are responding.

Remember that keepalive isn't just to keep the intermediaries alive,
it's to inform the endpoints that the link is still alive and doesn't
need recreating too.

> websocket level response but the TCP ACK is still enough to keep
> NATs, firewalls etc. open.

Works for intermediaries, but fails in multiple ways as described above.

> keepalive response that the server uses to respond to it. The
> response is not really necessary for NATs etc., but might help the
> client to notice a broken connection faster. (I'm not sure if this
> is true but I remember this was an argument in another protocol,
> SIP, why keepalive over TCP was done in that way.) The response may
> make the protocol more complicated, though, as people have
> commented.

Yes, massively faster.  TCP will take about 20 minutes if I recall
correctly, to detect send failures.  That's too long for most
interactive applications to detect and recreate broken connections.

But that reponse is redundant, if the server has sent data within the
timeout interval, in most situations.  So why insist on a
request-response pattern if it's adding redundant messages?

> also the server could send keepalives. I think the client should be
> in charge. The reason is that as a client developer for a mobile
> device I would like to be able to send keepalives as infrequently as
> possible. We have had some bad experiences with IMAP servers sending
> stuff frequently back to the client (presumably because not all
> clients are clever enough to do any keepalives themselves - this is
> a risk of course if we don't specify this at all in websocket spec),
> and the client has no way to affect their rate, which has been
> fixed.

No!  That's too 2010 and too application-specific.  What about when
the server is on your mobile phone and the client is in a data centre?
Think it won't happen?  What about that new Opera thingy where the
browser contains a web server for other browsers to use...

You can't assume the client side will always be the one with stricter
requirements, even in usual case of a user's web browser.  Servers
have resource limitations and need to manage idle connections.
1M+ connections to a single cluster is a fair implementation target;
then it matters.

> If there really is a requirement for the server to see keepalives at
> a certain minimum rate (for instance so that it can abandon stale
> connections), I would then have that negotiated so that the server
> can tell that back to the client who would still do the sending. But
> I'm not sure if that's needed.

This is not optional.

It is essential to abandon stale connections, otherwise servers run
out of resources over time - no matter how much memory they have.
Then they crash.

The only way to detect stale connections is send something or expect
to receive something.  TCP won't detect staleness if there's no activity.
(Useful word for blackholed connections, "stale", btw.  Thanks.)

Servers can send their own keepalive probes using the request-response
pattern, or by negotiated request to the client to send periodic
keepalives during idle periods, or by the packet-optimal synchronised
combination describe in an earlier mail from me.

All types have pros and cons, the same as the client side.

If both sides are sending request-response periodically, that's a high
overhead.  (Nb: In many applications, keepalives are the most expensive
bandwidth cost!)

Don't make the mistake of thinking a client-only regular ping is the
simplest keepalive to code...  It sounds simpler than it really is, when
you add what's needed for stale detection on both sides.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 19:52:35 +0100

> (XEP-0199) is extremely useful to definitively check reachability.

That is an excellent description, thank you.  I think you got it spot
on with XMPP, having both idle one-way keepalives and a ping option
that either side can use at any time.

The ability to definitively check reachability with a ping is useful
in some situations.  Especially with long keepalives giving a
"probably online (may be 2 minutes out of date)" state, sometimes
you need to upgrade that to "definitely online".

Due to keepalives being the primary bandwidth cost in mostly-idle
applications, I've studied optimising them.  I found one-way keepalive
on idle connections are most packet-efficient if there are asymmetric
timeout requirements; expediting a keepalive when you think the local
TCP is about to send a delayed ACK (i.e. when you just received
something after an idle period) increases packet-efficiency, and that
magically settles into a one-way request-response pattern when there
is a symmetric timeout requirement and no real data happening.  That
can be assembled into a single simple algorithm, but it's not the one
people tend to come up with when they first think about it.

> long-lived TCP over mobile networks was really difficult, whereas now  

+1, twice!

> reacquire a running XMPP session after TCP is lost.

I've looked at something very similar in my multiplexing/tunnelling
applications, and that is indeed a useful feature.

I'm not sure if it's stated as a requirement, but some of the
WebSocket discussion implies it is *intended* to provide a
quasi-reliable connection between web applications and server.

Unfortunately TCP does not provide that reliability on current networks.

I suspect automatic session resumption would be welcomed with open
arms by application developers, especially those who don't want
anything complicated, just something that's reliable.

HTTP and XHR effectively does provide it: The browser retries GET
requests automatically after a timeout if it doesn't get a response.
So web application authors are used to expecting some level of
automatic robustness.

It looks like XMPP people have valuable implementation experience that
is directly relevant.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 20:24:49 +0100

> >better. And the host can potentially optimize between different  

The client only knows about it's nearest hop, usually.  Upstream
knowledge is known to intermediaries and servers.  In a server farm,
reverse proxies usually have the best local knowledge.  With mobile
data, if a 3G router is used, the router knows better than the client
what to do - especially for optimising radio power.

What do you mean about optimizing between different connections?

> I'm open to suggestions.

I really want intermediaries to have an *optional* way to intercept
the keepalive function - so they can optimise with link/hop-level
knowledge.  For example mobile links using a different mechanism that
uses less radio power, proxies at specific locations extending the
keepalive interval for specific hops, timing synchronisation between
connections at a hop to minimise radio power, and multiplexers (not
necessarily using WebSocket, but carrying it) aggregating multiple
connection's keepalives into fewer.  Far fewer.

Aall that's needed is the keepalive message types are well known and
allowed to be adjusted by intermediaries.  And negotiation is also
well known.  Simple intermediaries will just pass them on.

> If the server sends a simple keepalive only on silent connections on
> a reasonably lengthy interval, this seems to be to cover most
> options - it in effect means that clients can do it sooner if they
> wish.

For the server to know when a connection is stale, why not tell the
client to send an SP after 29 minutes (like AMQP)?  Client
compatibility, or is there a technical reason to prefer ping if the
protocol were designed from scratch?

Isn't 30 minutes a rather long time to detect a user's presence is
lost?  I expect a lot of WebSocket applications to have a "presence"
pattern like XMPP's, but with different timing requirements.

Interactive web sites tracking online users may want an aggressive
timeout on the server side - as low as a few seconds in some cases,
more likely tens of seconds or a few minutes.

If the transport does KA at all, it's better if the application can
request specific KA timings from the transport, instead of sending a
second layer of "application KAs", which wouldn't be able to
participate in timing/power optimisations and hop/link-level substition.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 20:44:12 +0100

Thomson, Martin wrote:
> However, negotiation seems like overkill for this sort of mechanism.
> Each peer has some expectations and constraints that it knows best.
> Let them each take whatever action they need to ensure that the
> connection stays alive.

If they don't negotiate, the only option is both sides using ping requests.

That is the most expensive KA pattern.

Remember that KA is the *biggest* bandwidth cost in many applications.

It makes the biggest difference when client and server both need to
rapidly detect a stale connection.  (Depends on application - see below).

> If a keep-alive frame is defined, either peer could unilaterally
> initiate it.  There's no technical reason why this shouldn't be
> allowed.  If the server cares, and it doesn't see any activity from
> the client, it could use the keep-alive too.

I agree that a ping request should be defined, and allowed to be sent at
any time by either side.

It is important to treat it as a *request* with other attributes that
generic requests might use, such as matching requests with responses.

Otherwise you get a condition where you send a ping request, get a
ping response, but the response might be an *old* response from 30
seconds ago (queued somewhere) and that doesn't tell you the endpoint
is alive now.

> As Jamie points out, the server frequently doesn't care.  More to
> the point - the server can't do little about it if the connection does
> go away.

Quite untrue.  The server cares a lot.  It *usually* has a less
aggressive timing requirement than the client - but not always, if the
application running on top has "presence" needs or some kind of cloudy
write-back caching (let your imagination run wild ;-).

If it sees no activity, the server must close the connection to avoid
running out of memory in the long run.  It can't make a new connection
(as usual there are exceptions for some applications), but it may
still do something useful, for example deleting an entry from a
"user's online" list.

> (I can imagine use of other channels to prod the client into
> wakefulness, but that's not something inherent to WS.)

Mobile phones, SMS signalling...  I'm guessing a big power saver in
mostly-idle applications.  Not inherent to WS, but WS should be
designed to make those methods possible to use.

> Rather than outright prohibition, I'd instead prefer to discourage
> server use of keep-alive (SHOULD NOT).

I'd discourage the use of SHOULD NOT, because it will only encourage
widespread clients to fail to implement the KA response, and thus make
it impossible for any server to use the feature should the need arise.

Just let implementors decide what to use, and give them good advice.

> Servers might know better - we have a server that has a 5 minute TCP
> "go dead" interval on the load balancer in front of it.  That server
> should not be outright prevented from sending a 4:55 keep-alive if
> it sees nothing from a client.

There is no point in sending the 4:55 keepalive, if you *know* all
working clients send KAs at shorter intervals.  Or is there something
I've missed?

(4:55 is too late, btw, on a mobile network with 6s latency...)

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 20:57:31 +0100

[email-masked] wrote:
> Mobile clients would often want to send keepalives as infrequently
> as possible to save battery. They may use intelligent
> (unstandardized) mechanisms to learn what the mimimum rate
> acceptable for their access network NATs and firewalls are and use
> it. (As of today I believe the ballpark for TCP is something like a
> once per 10 minutes of idleness.)
>
> The servers on the other hand want to get rid of unused connections
> as fast as possible and may want to check the connection liveliness
> even faster. (I heard that some major XMPP servers send keepalives
> every 30 seconds.)

That depends a lot on the server, the application, and how loaded it is.

> For many clients (running in PCs etc.) this is probably fine but
> really bad for the clients in mobile devices. If the same device has
> multiple apps (websockets, XMPP, SIP, IMAP, ...)  running it can
> even synchronize the keepalives it sends for all of them (so that
> radio is active only once during the cycle),

Excellent idea :-)

> but for the server initiated keepalives it can't do the same.

That's an excellent reason to use per-hop keepalives with a proxy
on the other side of the mobile link...

Timing needs are highly application dependent.  If you are offering a
web-based chat service, the server might want to know aggressively so
it can update "users online", but if it's just a channel to send
interactive wiki updates out, the server might have a long timeout -
reducing when the server is loaded.

> I've always wondered if all those warnings and nice words in RFCs
> help, but if/when we specify the server-initiated keepalive frame,
> the above case should be explained, so the servers would not use if
> carelessly. I think the gist should be so that the client is in
> charge of keeping NATs, firewalls, proxies and all that stuff (which
> is mostly related to the clients current point of attachment) happy,
> while the server should only worry about its own resources and those
> related to it, such as the load balancers.

Exactly: Both sides should worry about their own resources, and their
own application-specific needs.

I don't think the client is more special than the server in that way.

Negotiation is not essential, and it could be made an optional
extension.  It's useful because it reduces the overheads compared with
pings.  Up front negotation is not as good as a "send me keepalives
every N seconds of idle time" which can be sent by either side at any
time and override the previous setting.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 21:21:21 +0100

> efficiency seems irrelevant here.

I'm talking about when there is no traffic.  Some applications are
idle nearly all the time; then KAs are by far the dominant cost.

Three packets exchanged every 30 seconds is a ~26GB per thousand
user-months.  You might transfer 1GB of real data in the same time.

If you have a million connections doing KAs all the time, you can
go bankrupt paying for KAs while the real data is affordable.
Same with lots of open tabs and WS apps on a mobile browser.

It's not worth obsessing over a few percent, but halving the cost is
sometimes worth it.  But anyway, leave that to each implementation.

Just make sure the protocol supports (a) ping requests, and (b)
request to set async KA idle time, and all will be fine.  [(b) is
as easy as negotiation and more versatile.]

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 20 Apr 2010 21:43:38 +0100

Vladimir Katardjiev wrote:
> Is it really necessary to match responses to requests? The point is
> just to see if the connection is alive, no? All the keepalive would
> do is tell the server "hey, can you send me something". Logically,
> if the server already is sending data then it shouldn't need to send
> a keepalive back (because there's no need to establish liveness). So
> what interest would we have to be mapping these keepalive messages
> to each-other? What semantics would that derive? The best I can
> figure out is give the req/resp latency...

If you receive a message, it could have been queued in the network for
up to 2 minutes.  So you don't know if the connection is really alive.

I see 20 second transient delays often on mobile networks; and 50
second delays occasionally elsewhere.

Whether you care depends on the application.

> The reason I suggested "echo" though (at least for a basic case) is
> because it fulfils the "amateur programmer" requirement. The server
> is completely stateless, no timers, no nothing. This assumption even
> includes the server not keeping the state of if it's sending or not.

It does involve a timer: You have to decide when to send the request.

Both sides need a timer to clean up stale connections.  The server
cannot be stateless; it must do this.

> (To put it in RFC terms, if the server receives a KA, it MUST send
> some data to the client; if the server is already sending
> application data it MAY skip sending a keepalive back)

If the server doesn't send a KA response, and the client receives the
data, the client doesn't know if that data was queued in the network
for up to 2 minutes, so it doesn't know if the server's been dead for
2 minutes.

Whether you care depends on the application.

Usually you don't in this case, but sometimes you want to know if the
server is still alive at the specific moment you send the ping
request.  Either to update an online status accurately, or to confirm
delivery of prior messages.

> I think this problem applies generally. Graceful close was supposed
> to let the remote side empty its buffer. But suppose it has 500gb in
> its sending buffer. That's going to take a while to gracefully
> close, so keepalives are still needed during this time period. And
> since the side that already sent graceful close won't send any more
> _data_ it needs to send keepalives.

I think you're right, keepalives are probably needed in this case, as
long as the closing side is prepared to drain incoming data.  (After
that, it becomes a connection error and graceful application recovery
may not be possible.)

However senders shouldn't queue 500GB if it will take a long time to
send (in 2020 maybe it will be quick), precisely because they can't
react to things like a graceful close message during the safe time :-)

> Once both sides have sent a graceful close, what would be the point
> of replying to a keepalive? If the connection is alive, it would be
> closed first. If the connection is dead it's not going to make a
> world of difference.

Depends on the semantics of graceful close.  For some it means "close"
:-) For some it means "I acknowledge receipt of messages up to
sequence number X or all those I've replied to, and will definitely
ignore further messages".  For some it means "requesting graceful
close to save resources, but we might re-open if you still want to use
this connection".

> [mobile radio considerations]

+1 :-)

Synchronised, coordinated keepalives are best for radio power.

Coordination between multiple connections is better.

Best is multiplexing with a single keepalive shared among them ;-)

> PS: Thanks Jamie for your in-depth replies! I'd like to look into more the per-hop keepalive option, especially if it can be done in an implicit manner.

I'm glad someone finds them useful.  I've just re-read them and worry
about spamming the list. (?)

-- Jamie