Subject: [hybi] I-D Action:draft-ietf-hybi-thewebsocketprotocol-05.txt

From: [email-masked]
Date: Tue, 08 Feb 2011 15:15:02 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the BiDirectional or Server-Initiated HTTP Working Group of the IETF.

	Title           : The WebSocket protocol
	Author(s)       : I. Fette
	Filename        : draft-ietf-hybi-thewebsocketprotocol-05.txt
	Pages           : 51
	Date            : 2011-02-08

The WebSocket protocol enables two-way communication between a user
agent running untrusted code running in a controlled environment to a
remote host that has opted-in to communications from that code.  The
security model used for this is the Origin-based security model
commonly used by Web browsers.  The protocol consists of an initial
handshake followed by basic message framing, layered over TCP.  The
goal of this technology is to provide a mechanism for browser-based
applications that need two-way communication with servers that does
not rely on opening multiple HTTP connections (e.g. using
XMLHttpRequest or <iframe>s and long polling).

Please send feedback to the [email-masked] mailing list.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-hybi-thewebsocketprotocol-05.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---

From: Andy Green <[email-masked]>
Date: Wed, 09 Feb 2011 09:03:06 +0000

> http://www.ietf.org/internet-drafts/draft-ietf-hybi-thewebsocketprotocol-05.txt

Nice, the per-frame SHA-1 is gone and no AES junk; syntax for extension 
specification in the client header is defined.  If you want to (in 
violation though) as a client you can compatibly set the frame nonce to 
00000000 and skip munging altogether.

I added in 05 framing support for client and server in libwebsockets, 
the support is:

  Server: 76/00, 04 and 05 selected dynamically per-client
  Client: 04 or 05 selectable

There's a -v switch on the test client and ping applets to set spec 
version to use, eg, -v4

Extension header should be recognized and rejected since it doesn't 
support any.

Ran out of time to do the new close stuff before "real work" must start 
but otherwise it's probably a useful sparring partner for other client 
or server implementations.

http://git.warmcat.com/cgi-bin/cgit/libwebsockets/

-Andy

---

From: Andy Green <[email-masked]>
Date: Thu, 10 Feb 2011 09:49:09 +0000

> http://git.warmcat.com/cgi-bin/cgit/libwebsockets/

The 05 CLOSE stuff on client and server is added earlier thismorning so 
it should be a reasonably complete 05 implementation now.

-Andy

---

From: [email-masked]
Date: Thu, 10 Feb 2011 02:59:12 -0800

Hi,

I have just started studying details of ws protocol on my new given task. 
I read -05  and 
I would like to give you some of my thoughts as feedback, with a few 
questions about the 
protocol. I would assume some of my questions have already been 
discussed/resolved. 
In such case, if you could kindly  refer me to relevant discussions made 
in the past, 
that would be great.

o Section 1.3 - discrepancy in SHA1 hash value:

My python says:
>>> h = hashlib.sha1()
>>> h.update('dGhlIHNhbXBsZSBub25jZQ==258EAFA5-E914-47DA-95CA- 
C5AB0DC85B11')
>>> h.hexdigest()
'1b334d8e1b8d57c1c7a214517c106abc434282b8'

which is different from what the example says in -05. Am I missing 
something?

o Section 1.9 - subprotocols

It would be better if the document describes expected behavior of server 
and client
when Sec-Websocket-Protocol header is not present.

o Section 2.1 - typo?

What is IRI? You mean, "URL"?

o Section 4.3 - Opcodes
In the description of 'Opcode', I thought it would be better to have a 
list of currently defined
opcodes with a brief description. (It's trivial though it helps for a 
reader like myself.)

e.g.)
    0x0: continuation frame
    0x1: connection close (A control frame)
    0x2: ping (A control frame)
    0x3: pong (A control frame)
    0x4: text frame
    0x5: binary frame
    0x6-F: reserved

o  Section 4.3 - Continuation frame (opcode=%x0)

It is not clear to me in the document how this frame (opcode=0) is used 
while there's
FIN bit for fragmentation/defragmentation.

Section 4.4 says, 
"..., and terminated by a single frame with the FIN bit set and an opcode 
of 0"

Considering a case where there are text and binary frames that are both 
fragmented
and alternating over the wire, how can application tell which frame (text 
or binary) the
last frame with FIN=1 set belongs to?

For example:
[op=txt, fin=0] [op=bin, fin=0] [op=0, fin=1] <== is this for txt or bin?

Is this kind of multiplexing support intended in the protocol design? If 
the doc explains
why and how fragmentation is used.

o Section 4.5.1 - the term "body"

For consistency, it should be replaced with Application Data, or "body" be 
defined explicitly in the document.

o Section 4.5.1 - Close handshake

It is no clear to me what 0x43 and 0x53 are trying to achieve. In TCP, 
there is FIN with no 
details of why/who is closing for graceful shutdown.

o Section 4.5.2 and 4.5.3

It would be better if there is some description for why we have ping and 
pong frames, 
especially for implementers. For example, ping/pong can be used to keep 
TCP 
connection alive at application level which can be transparent to 
application. But it can 
also be used for app to detect 'presence' of a client for the use in the 
app.
(# I myself can not tell what the original intention is by just reading 
-05.)

o Section 7.2 - _cleanly_

Very very trivial, but why not use "gracefully" which is widely used in 
TCP land.

o Overall

As I was reading though, it took some time to find out which feature is 
for which entities
role as in protocol layering. Ping/pong protocol for example, I am still 
not sure if
this is done by WebSocket protocol internally or exposed to application. 
Another 
examples would be the extension data filed - who is going to parse it?

Most of this ambiguity may be clear in the use of JS under WebSocket API. 
But I am
coming from non-browser perspectives and I would probably need to figure 
out
how to implement *protocol API* for non-application developers for 
example.

If there's a layer structure with interfaces defined in the document, that 
would be a
great help - something like:

===== WebSocket API ======
+------------------------+
|  WS-JS Bridge (if any) |
+------------------------+
+--------++--------------+
|        ||              |
|  HTTP  |+--+---+ Web   |                      -- peer -- 
|        |   ^   | Socket|
|        |  (HS) |       | <== WS protocol ==> [WS entity]
|        |   v   |       |
|        +---+--+|       |
|               |<========== HTTP ===========> [HTTP entity]
+---------------++-------+
+------------------------+
|         TCP/IP         |
+------------------------+

I wish I could help constructing more specific proposal rather than just 
giving you
comments, but I am still trying to catch up with the current status, to be 
honest. I hope
above would be of a little help at least for the WG.

Best regards,
--
Yutaka Takeda

---

From: Julian Reschke <[email-masked]>
Date: Thu, 10 Feb 2011 12:27:01 +0100

On 10.02.2011 11:59, [email-masked] wrote:
> ...
> ...

I see

>    The term "URL" is used in this section in a manner consistent with
>    the terminology used in HTML, namely, to denote a string that might
>    or might not be a valid URI or IRI and to which certain error
>    handling behaviors will be applied when the string is parsed.  [HTML]

This needs to be fixed to be either "URI" or "IRI" (depending on what we 
actually want, I *suspect* URI); and of course the bogus HTML reference 
needs to go.

Best regards, Julian

---

From: Andy Green <[email-masked]>
Date: Thu, 10 Feb 2011 11:35:28 +0000

On 02/10/11 10:59, Somebody in the thread at some point said:

こんばんわ、ゆたかさん

> when Sec-Websocket-Protocol header is not present.

I guess it's just the degenerate case that the server only supports one 
protocol.

> What is IRI? You mean, "URL"?

It's "internationalized"

http://www.w3.org/International/O-URL-and-ident.html

> 0x3: pong (A control frame)

Yes it's a nice idea.

> FIN bit for fragmentation/defragmentation.

Right it wasn't clear to me either.

> why and how fragmentation is used.

It's not defined yet that muxing is allowed, it'll be an extension it 
seems.  When muxing is allowed, there will be a stream index in each 
fragment to allow deterministic reassembly.

> details of why/who is closing for graceful shutdown.

I guess it at least is documentation if you are looking in tcpdump.

> especially for implementers. For example, ping/pong can be used to keep TCP

Yeah.

> this is done by WebSocket protocol internally or exposed to application.

In libwesockets

the library takes care of all control frames.  It passes a message to 
the user callback to notify PONG receipt but it just returns by default. 
  The actual CONTINUATION, CLOSE, PING, PONG service is done in the 
library without bothering user code.

BTW libwebsockets is C, LGPL2, maybe it can help you.

> examples would be the extension data filed - who is going to parse it?

For extensions that are transparent, like gzip or mux, then the 
websocket library can hide it from the user.  Anything more complicated 
will have to be a co-operation between library and user handling.

> Most of this ambiguity may be clear in the use of JS under WebSocket
> API. But I am
> how to implement *protocol API* for non-application developers for example.

Yeah that is my situation too.

> If there's a layer structure with interfaces defined in the document,
> that would be a

I think that is probably out of scope, the protocol stands alone and 
it's best to restrict what it tries to define to what goes on the wire.

-Andy

---

From: John Tamplin <[email-masked]>
Date: Thu, 10 Feb 2011 10:57:20 -0500

> FIN bit for fragmentation/defragmentation.

The reasonable options are:
 1) define a different frame format for continuation frames that doesn't
have opcodes
 2) use the same opcode as the original frame
 3) use a new opcode

#1 adds complexity and wastes bits for little benefit.  #3 has the benefit
of allowing you to detect the four states of a frame (single-frame message,
initial frame of fragmented message, middle frame of fragmented message,
final frame of fragmented message) if you care, and also provides
opportunity for more error checking.

> why and how fragmentation is used.

No, see 4.4.  Only control frames can be sent in the middle of a fragmented
message (I believe the "open question" line is out of date -- I think the
consensus was that control frames can be injected in the middle of a
fragmented message, otherwise the latency of responding to ping, for
example, would be very long if behind a large message), and control frames
cannot be fragmented.  Multiplexing will be defined as an extension and will
include logical channels.  Even in #2, you would have to maintain the frame
type from the original frame to avoid attacks like [BINARY "invalid UTF8"]
[TEXT/FIN "UTF8"] which could exploit bugs in UTF8 decoders.

o Overall

> As I was reading though, it took some time to find out which feature is for
> which entities

The WebSocket implementation will be the one to at least extract the
extension data from the frame, but the interpretation of that data will be
defined by the extension.  It may be processed by the WebSocket layer or the
application depending on the extension.

-- 
John A. Tamplin
Software Engineer (GWT), Google

---

From: [email-masked]
Date: Thu, 10 Feb 2011 10:49:35 -0800

> こんばんわ、ゆたかさん
コメントありがとう、アンディさん

> > What is IRI? You mean, "URL"?

I did not know it until now... - thanks!

> I guess it at least is documentation if you are looking in tcpdump.

I would prefer not to reveal any details over the wire for security reason 

if there's not much gain with the details, and also it is always clear 
who initiated Close with tcpdump anyway.

> BTW libwebsockets is C, LGPL2, maybe it can help you.

Wow, that would be helpful as a reference for validation of implementation
and protocol itself. I will look into it, when I get ready for it. Good 
job!

> > that would be a great help - something like:
> 

I see. I still think that a layered block diagram will help defining 
more clearly the scope of this document for readers to understand 
relationship between this protocol spec and other entities, IMHO.

Best,
- Yutaka

---

From: [email-masked]
Date: Thu, 10 Feb 2011 12:06:27 -0800

Hello John,

Thanks for your comments.
My comments inline..

> It is not clear to me in the document how this frame (opcode=0) is 
> used while there's 
>  1) define a different frame format for continuation frames that 
> doesn't have opcodes
> #1 adds complexity and wastes bits for little benefit.  #3 has the 
> benefit of allowing you to detect the four states of a frame 
> (single-frame message, initial frame of fragmented message, middle 
> frame of fragmented message, final frame of fragmented message) if 
> you care, and also provides opportunity for more error checking.

It's on TCP - the order is guaranteed, therefore only 1-bit of 
information is necessary, IMO. {Just continuation or final.
'first or middle' is redundant over the steamed protocol)

Single frame: [fin=1]
Fragmented  : [fin=0]...[fin=1]

It works even in the case where different frames are multiplexed.
(I understand, it occurs only with Control frames in the current spec.)

>From the net: [op=txt, fin=0] [op=bin, fin=0] [op=txt, fin=1]
Text buffer : [op=txt, fin=0]                 [op=txt, fin=1]
Bin. buffer :                 [op=bin, fin=0] (waiting for opt=bin, fin=0 
or 1)

# any thoughts?

> Is this kind of multiplexing support intended in the protocol 
> design? If the doc explains 
> No, see 4.4.  Only control frames can be sent in the middle of a 
> fragmented message (I believe the "open question" line is out of 

Ah, I see. When I read 4.4 - second paragraph seems to imply that
other adjacent fragments share the same opcode... But with third
bullet, I was lost on it. As you pointed out, this section needs to 
be updated/fixed.

> date -- I think the consensus was that control frames can be 
> injected in the middle of a fragmented message, otherwise the 
> latency of responding to ping, for example, would be very long if 
> behind a large message), and control frames cannot be fragmented.

I see. Good to know the purpose of the fragmentation. (please add
in the document)

I have not read [WSAPI] yet, though, I would assume the ping/pong
will be done at WS protocol level. How can you inject "ping" when
the first portion of long text frame has just been transmitted? 
The payload length is already sent. You would need to implement 
some scheduler for ping with a large buffer to avoid that. If the 
ping is initiated by the app, then it would be another story...

The heart of the problem I can think of is the fact that the 
current protocol does not define maximum size of the frame for
text and binary. I mean, one solution would be to set up such 
maximum frame size (or MFS - which corresponds in a way to TCP's 
MSS) to a reasonable amount to allow multiplexed frames to flow 
all the time with a reasonable delay.
 
>  Multiplexing will be defined as an extension and will include 
> logical channels.  Even in #2, you would have to maintain the frame 
> type from the original frame to avoid attacks like [BINARY "invalid 
> UTF8"] [TEXT/FIN "UTF8"] which could exploit bugs in UTF8 decoders.

I have to admit that I could not decipher you point. Sorry. I 
will look for information about the UTF8 bug exploit, or could you
please point me to a related discussion?

> extension data from the frame, but the interpretation of that data 
> will be defined by the extension.  It may be processed by the 
> WebSocket layer or the application depending on the extension.

It sounds good to me - thank you.

- Yutaka

---

From: John Tamplin <[email-masked]>
Date: Thu, 10 Feb 2011 15:26:14 -0500

Single frame: [fin=1]
> Bin. buffer :                 [op=bin, fin=0] (waiting for opt=bin, fin=0

That would let you interleave different types of frames, but not the same
type of frames.  Since the initial uses are all going to be text frames,
that doesn't buy you anything.  A complete multiplexing solution also needs
to address flow control issues, so I don't see that the limited ability to
interleave frames with different opcodes is useful.

> I see. Good to know the purpose of the fragmentation. (please add

The primary purpose of fragmentation is to allow sending a message that is
of unknown size when it is started without having to buffer it.  If messages
couldn't be fragmented, then you would have to buffer the entire message so
you could count its length before you could send the first byte.  With
fragmentation, you choose a reasonable size buffer and when the buffer is
full you write a fragment to the network.

A secondary use-case for fragmentation is for mulitplexing, where you don't
want want a large message on one logical channel to monopolize the output
channel, so the MUX needs to be free to split the message into smaller
fragments to better share the output channel.

> all the time with a reasonable delay.

Read the archives -- the current solution is a hard-won compromise between
those who wanted simplicity of being able to write a large message in a
single operation (and making use of things like sendfile to do it in the OS)
and those who wanted fragmentation.

> please point me to a related discussion?

Imagine we copied the opcode into every frame, and there was an
implementation that decided it would avoid having to remember what opcode it
was processing since every frame includes the opcode.  Now, it receives a
fragment with a BINARY opcode, including non-UTF8 data.  The next fragment
is TEXT/FINAL, and so the receiving software verifies that the bytes sent is
valid UTF8 data, appends it to the current message buffer, and then delivers
it to the application as UTF8 data.  However, the first part of the buffer
was never validated to be UTF8 since it was received as type BINARY.  Thus,
you might exploit a bug in UTF8 handling in the application that didn't
expect to deal with invalid UTF8 sequences in a text message.

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 11 Feb 2011 06:04:16 +0900

Hi Yutaka-san,

> '1b334d8e1b8d57c1c7a214517c106abc434282b8'
remove a space between 95CA- and C5AB, and you get what's on the spec.

>>> import hashlib
>>> h = hashlib.sha1()
>>> h.update('dGhlIHNhbXBsZSBub25jZQ==258EAFA5-E914-47DA-95CA-C5AB0DC85B11')
>>> h.hexdigest()
'b37a4f2cc0624f1690f64606cf385945b2bec4ea'

> defined explicitly in the document.
agreed

> details of why/who is closing for graceful shutdown.
Actually, it's not intended to indicate who's closing, but to make sure what
the close frame really means, i.e. it's close initiation or close
acknowledgement. Using an analogy of TCP, close S from server is FIN, and
close S from client is ACK for server's FIN.

Without this, each peer cannot know
- the other peer just initiated close
or
- the other peer received close frame and acknowledged it

Imagine both peers send out close initiation at approximately same time.
They looks like acknowledgement but are not.

Once we can perfectly distinguish initiation and acknowledgement as
described above, we can assume "If A received close acknowledgement from B,
B has received all the data sent from A before close successfully".

If you have any idea about the usefulness of this closing handshake design,
please come to the thread "Clarify the role of closing handshake" and speak
out. We're discussing it now.

Takeshi

---

From: [email-masked]
Date: Thu, 10 Feb 2011 13:22:46 -0800

Thanks John for your comments which corrected my misunderstanding!

Just a comple of things, just to summarize...

> That would let you interleave different types of frames, but not the
> same type of frames.

You are right. This multiplexing based on opcode is half-way through,
and we need to introduce some kind of channel ID for complete 
multiplexing,
which is out of scope in the current document - now I understand.

The original point I was making though was if 'Continuation frame' is 
necessary for the current spec. It doesn't harm to have it and I can live 
with it, but it still looks redundant or misleading to me.

> The primary purpose of fragmentation is to allow sending a message 
> that is of unknown size...

This got me realize my misunderstanding. Please for get about MFS thing
I mentioned. WS layer is already in control in terms of setting frame 
size to an optimal. I agree, flow control would be an issue in the future.

Thanks again,

** todo: read more archive ** 

- Yutaka

John Tamplin <[email-masked]> wrote on 02/10/2011 12:26:14 PM:

> Bin. buffer :                 [op=bin, fin=0] (waiting for opt=bin, 
> fin=0 or 1) 
> 
> # any thoughts? 
> 
> That would let you interleave different types of frames, but not the
> same type of frames.  Since the initial uses are all going to be 
> text frames, that doesn't buy you anything.  A complete multiplexing
> solution also needs to address flow control issues, so I don't see 
> that the limited ability to interleave frames with different opcodesis 
useful.
> The primary purpose of fragmentation is to allow sending a message 
> that is of unknown size when it is started without having to buffer 
> it.  If messages couldn't be fragmented, then you would have to 
> buffer the entire message so you could count its length before you 
> could send the first byte.  With fragmentation, you choose a 
> reasonable size buffer and when the buffer is full you write a 
> fragment to the network.
> 
> A secondary use-case for fragmentation is for mulitplexing, where 
> you don't want want a large message on one logical channel to 
> monopolize the output channel, so the MUX needs to be free to split 
> the message into smaller fragments to better share the output channel.
>  
> Read the archives -- the current solution is a hard-won compromise 
> between those who wanted simplicity of being able to write a large 
> message in a single operation (and making use of things like 
> sendfile to do it in the OS) and those who wanted fragmentation.
>   

> implementation that decided it would avoid having to remember what 
> opcode it was processing since every frame includes the opcode. 
>  Now, it receives a fragment with a BINARY opcode, including non-
> UTF8 data.  The next fragment is TEXT/FINAL, and so the receiving 
> software verifies that the bytes sent is valid UTF8 data, appends it
> to the current message buffer, and then delivers it to the 
> application as UTF8 data.  However, the first part of the buffer was
> never validated to be UTF8 since it was received as type BINARY. 
>  Thus, you might exploit a bug in UTF8 handling in the application 
> that didn't expect to deal with invalid UTF8 sequences in a text 
message.

---

From: John Tamplin <[email-masked]>
Date: Thu, 10 Feb 2011 16:27:59 -0500

> with it, but it still looks redundant or misleading to me.

Right, and making all the frames have to be the same is redundant and
doesn't let you distinguish the four states.  The only way that wouldn't be
redundant is to actually have a message layer and a fragment layer, and
aside from the having two framing formats instead of one and rules for
interaction between them, there is no way to do it without wasting bits (or
switching to a bit-oriented protocol) and overcomplicating the simple cases.

It is all a compromise between competing goals, and we spent months coming
up with a framing that everyone could agree to.

---

From: Gabriel Montenegro <[email-masked]>
Date: Fri, 11 Feb 2011 03:12:50 +0000

Just responding to one point:

“No, see 4.4.  Only control frames can be sent in the middle of a fragmented message (I believe the "open question" line is out of date -- I think the consensus was that control frames can be injected in the middle of a fragmented message, otherwise the latency of responding to ping, for example, would be very long if behind a large message), and control frames cannot be fragmented. “

Agree, I think this can be deleted from 4.4:

   o  _Note: There is an open question as to whether control frames be
      interjected in the middle of a fragmented message.  If so, it must
      be decided whether they be fragmented (which would require keeping
      a stack of "in-progress" messages)._

I also believe we agreed that the answer to the first questions is “yes”. Regardless, it is logical. For example, you might want to send a Ping (the delay between fragments may be large as the data is produced) or more obviously, a Close.

And the answer to the second has already been settled in the draft for at least a couple of releases as “control messages cannot be fragmented”. Deleting the above means adding a sentence in 4.5 (control frames) the the effect that “Control frames can be interjected in the middle of a fragmented message.”

---

From: Julian Reschke <[email-masked]>
Date: Fri, 11 Feb 2011 10:48:30 +0100

> actually want, I *suspect* URI); and of course the bogus HTML reference

Clarifying - we currently have a normative reference to "HTML" as a 
document on the WHATWG site, which won't work here because it's a moving 
target (see <http://blog.whatwg.org/html-is-the-new-html5>).

We should look at the individual pieces we need, and cite them properly.

After a quick check, these may fall into these categories:

1) Actual references to HTML constructs, like forms -- simply cite W3C's 
HTML spec (either 4.01 or the latest HTML 5 WD; but I believe 4.01 would 
be sufficient).

2) Definition of URL -- simply don't. Stick with IETF terminology.

3) Credits for previous websockets work -- this can be done in the prose 
of the Acknowledgments section (potentially including a precise link to 
another doc's acknowledgments), but it doesn't need an entry under 
"References".

Best regards, Julian