Subject: [hybi] WebSockets: UDP

From: Erik Möller <[email-masked]>
Date: Tue, 01 Jun 2010 12:24:58 +0200

[originally (mis-)posted to WHATWG, reposting here]

The use case I'd like to address in this post is Real-time client/server
games.

The majority of the on-line games of today use a client/server model over
UDP and we should try to give game developers the tools they require to
create browser based games. For many simpler games a TCP based protocol is
exactly what's needed but for most real-time games a UDP based protocol is
a requirement. Games typically send small updates to its server at 20-30Hz
over UDP and can with the help of entity interpolation and if required
entity extrapolation cope well with intermittent packet loss. When a
packet loss occur in a TCP based protocol the entire stream of data is
held up until the packet is resent meaning a game would have to revert to
entity extrapolation possibly over several seconds, leading to an
unacceptable gameplay experience.

It seems to me the WebSocket interface can be easily modified to cope with
UDP sockets (a wsd: scheme perhaps?) and it sounds like a good idea to
leverage the work already done for WebSockets in terms of interface and
framing.

The most important distinction between ws: and wsd: is that messages sent
by send() in wsd: need not be acknowledged by the peer nor be resent. To
keep the interface the same to the largest possible extent I'd suggest
implementing a simple reliable 3-way handshake over UDP, keep-alive
messages (and timeouts) and reliable close frames. If these are
implemented right the interface in it's entirety could be kept. Only one
new readonly attribute long maxMessageSize could be introduced to describe
the min path MTU (perhaps only valid once in connected mode, or perhaps
set to 0 or 576 initially and updated once in connected mode). This
attribute could also be useful to expose in ws: and wss: but in that case
be set to the internal limit of the browser / server.

The actual content of the handshake for wsd: can be vastly simplified
compared to that of ws: as there's no need to be http compliant. It could
contain just a magic identifier and length encoded strings for origin,
location and protocol.

To minimize the work needed on the spec the data framing of wsd: can be
kept identical to that of ws:, though I'd expect game developers would
choose whatever the binary framing will be once the spec is done.

I'd be very interested to hear peoples opinion on this.

-- 
Erik Möller
Core Developer
Opera Software

---

From: Erik Möller <[email-masked]>
Date: Tue, 01 Jun 2010 15:00:50 +0200

On Tue, 01 Jun 2010 13:34:51 +0200, Philip Taylor  
<[email-masked]> wrote:

> On Tue, Jun 1, 2010 at 11:12 AM, Erik Möller <[email-masked]> wrote:
>> exactly what's needed but for most real-time games a UDP based protocol  
>> is a
>> requirement. [...]
>>
>> UDP sockets [...]
> As far as I'm aware, games use UDP because they can't use TCP (since
> packet loss shouldn't stall the entire stream) and there's no
> alternative but UDP. (And also because peer-to-peer usually requires
> NAT punchthrough, which is much more reliable with UDP than with TCP).
> They don't use UDP because it's a good match for their requirements,
> it's just the only choice that doesn't make their requirements
> impossible.
>
> There are lots of features that seem very commonly desired in games: a
> mixture of reliable and unreliable and reliable-but-unordered channels
> (movement updates can be safely dropped but chat messages must never
> be), automatic fragmentation of large messages, automatic aggregation
> of small messages, flow control to avoid overloading the network,
> compression, etc. And there's lots of libraries that build on top of
> UDP to implement protocols halfway towards TCP in order to provide
> those features:
> http://msdn.microsoft.com/en-us/library/bb153248(VS.85).aspx,
> http://opentnl.sourceforge.net/doxydocs/fundamentals.html,
> http://www.jenkinssoftware.com/raknet/manual/introduction.html,
> http://enet.bespin.org/Features.html, etc.
>
> UDP sockets seem like a pretty inadequate solution for the use case of
> realtime games - everyone would have to write their own higher-level
> networking libraries (probably poorly and incompatibly) in JS to
> provide the features that they really want. Browsers would lose the
> ability to provide much security, e.g. flow control to prevent
> intentional/accidental DOS attacks on the user's network, since they
> would be too far removed from the application level to understand what
> they should buffer or drop or notify the application about.
>
> I think it'd be much more useful to provide a level of abstraction
> similar to those game networking libraries - at least the ability to
> send reliable and unreliable sequenced and unreliable unsequenced
> messages over the same connection, with automatic
> aggregation/fragmentation so you don't have to care about packet
> sizes, and dynamic flow control for reliable messages and maybe some
> static rate limit for unreliable messages. The API shouldn't expose
> details of UDP (you could implement exactly the same API over TCP,
> with better reliability but worse latency, or over any other protocols
> that become well supported in the network).
>

I've never heard any gamedevs complain how poorly UDP matches their needs  
so I'm not so sure about that, but you may be right it would be better to  
have a higher level abstraction. If we are indeed targeting the game  
developing community we should ask for their feedback rather than guessing  
what they prefer. I will grep my linked-in account for game-devs tonight  
and see if I can gather some feedback.

I suspect they prefer to be "empowered with UDP" rather than "boxed into a  
high level protocol that doesn't fit their needs" but I may be wrong.  
Those who have the knowledge, time and desire to implement their own  
reliable channels/flow control/security over UDP would be free to do so,  
those who couldn't care less can always use ws: or wss: for their reliable  
traffic and just use UDP where necessary.

So the question to the gamedevs will be, and please make suggestions for  
changes and I'll do an email round tonight:

If browser and server vendors agree on and standardize a socket based  
network interface to be used for real-time games running in the browsers,  
at what level would you prefer the interface to be?
(Note that an interface for communicating reliably via TCP and TLS are  
already implemented.)
- A low-level interface similar to a plain UDP socket
- A medium-level interface allowing for reliable and unreliable channels,  
automatically compressed data, flow control, data priority etc
- A high-level interface with "ghosted entities"

Oh, and I guess we should continue this discussion on the HyBi list... my  
fault for not posting there in the first place.

-- 
Erik Möller
Core Developer
Opera Software

---

From: Erik Möller <[email-masked]>
Date: Tue, 01 Jun 2010 22:02:05 +0200

On Tue, 01 Jun 2010 21:14:33 +0200, Philip Taylor  
<[email-masked]> wrote:

> More feedback is certainly good, though I think the libraries I
> mentioned (DirectPlay/OpenTNL/RakNet/ENet (there's probably more)) are
> useful as an indicator of common real needs (as opposed to edge-case
> or merely perceived needs) - they've been used by quite a few games
> and they seem to have largely converged on a core set of features, so
> that's better than just guessing.
>
> I guess many commercial games write their own instead of reusing
> third-party libraries, and I guess they often reimplement very similar
> concepts to these, but it would be good to have more reliable
> information about that.
>

I was hoping to be able to avoid looking at what the interfaces of a high  
vs low level option would look like this early on in the discussions, but  
perhaps we need to do just that; look at Torque, RakNet etc and find a  
least common denominator and see what the reactions would be to such an  
interface. Game companies are pretty restrictive about what they discuss,  
but I think I know enough game devs to at least get some good feedback on  
what would be required to make it work well with their engine/game.

> If you put it like that, I don't see why anybody would not want to be
> empowered :-)

Yeah I wouldn't put it like that when asking :) I'm really not trying to  
sell my view, I just like to see real browser gaming in a not too distant  
future.

>
> But that's not the choice, since they could never really have UDP -
> the protocol will perhaps have to be Origin-based, connection-oriented
> (to exchange Origin information etc), with complex packet headers so
> you can't trick it into talking to a DNS server, with rate limiting in
> the browser to prevent DOS attacks, restricted to client-server (no
> peer-to-peer since you probably can't run a socket server in the
> browser), etc.
[...]
>
> That first option sounds like you're offering something very much like
> a plain UDP socket (and I guess anyone who's willing to write their
> own high-level wrapper (which is only hundreds or thousands of lines
> of code and not a big deal for a complex game) would prefer that since
> they want as much power as possible), but (as above) I think that's
> misleading - it's really a UDP interface on top of a protocol that has
> some quite different characteristics to UDP. So I think the question
> should be clearer that the protocol will necessarily include various
> features and restrictions on top of UDP, and the choice is whether it
> includes the minimal set of features needed for security and hides
> them behind a UDP-like interface or whether it includes higher-level
> features and exposes them in a higher-level interface.

So, what would the minimal set of limitations be to make a "UDP WebSocket"  
browser-safe?

-No listen sockets
-No multicast
-Reliable handshake with origin info
-Automatic keep-alives
-Reliable close handshake
-Socket is bound to one address for the duration of its lifetime
-Sockets open sequentially (like current DOS protection in WebSockets)
-Cap on number of open sockets per server and total per user agent

-- 
Erik Möller
Core Developer
Opera Software

---

From: Mark Frohnmayer <[email-masked]>
Date: Tue, 1 Jun 2010 16:07:48 -0700

On Tue, Jun 1, 2010 at 1:02 PM, Erik Möller <[email-masked]> wrote:
>
> I was hoping to be able to avoid looking at what the interfaces of a high vs
> low level option would look like this early on in the discussions, but
> perhaps we need to do just that; look at Torque, RakNet etc and find a least
> common denominator and see what the reactions would be to such an interface.
> Game companies are pretty restrictive about what they discuss, but I think I
> know enough game devs to at least get some good feedback on what would be
> required to make it work well with their engine/game.

Glad to see this discussion rolling!  For what it's worth, the Torque
Sockets design effort was to take a stab at answering this question --
what is the least-common-denominator "webby" API/protocol that's
sufficiently useful to be a common foundation for real time games.  I
did the first stab at porting OpenTNL (now tnl2) atop it; from my
reading of the RTP protocol that should easily layer as well, but it
would be worth getting the perspective of some other high-level
network stack folks (RakNet, etc).

> sell my view, I just like to see real browser gaming in a not too distant

Hmm... given the number of different approaches to higher-level game
networking I'd hate to see a high-level straightjacket where a
well-conceived low level API could easily support all of the existing
solutions out there.  The more complex the API the larger the attack
surface at the trusted level and the more difficulty in getting
existing stakeholders (game and browser makers) on board.

> So, what would the minimal set of limitations be to make a "UDP WebSocket"

Only feedback here would be I think p2p should be looked at in this
pass -- many client/server game instances are peers from the
perspective of the hosting service (XBox Live, Quake, Half-Life,
Battle.net) -- forcing all game traffic to pass through the hosting
domain is a severe constraint.  My question -- what does a "webby" p2p
solution look like regarding Origin restrictions, etc?

> -Reliable close handshake

While we're at it, I'd add key exchange, encryption and client puzzles
to reduce connection depletion/CPU depletion attacks to the handshake.
 The protocol you seem to be aiming for isn't UDP -- rather it's more
like an connected, unreliable packet stream between hosts.

> -Cap on number of open sockets per server and total per user agent

A single UDP socket can host multiple connections (indexed by packet
source address), so even a modest limit on actual number of sockets
wouldn't be a big impediment.

I'd also advocate for packet delivery notification to be a part of the
API -- with a known success or failure status on packet delivery many
of the higher level data transmission policies become trivial, and
should be essentially zero overhead at the protocol level.  Without
notification the higher level code has to do manual packet
acknowledgement as Phil mentioned.

Regards,
Mark

---

From: <[email-masked]>
Date: Wed, 2 Jun 2010 00:35:26 +0100

> wouldn't be a big impediment.

Um, NAT?

Lloyd Wood
[email-masked]
http://sat-net.com/L.Wood

---

From: Mark Frohnmayer <[email-masked]>
Date: Tue, 1 Jun 2010 17:12:15 -0700

>> wouldn't be a big impediment.

You would want to index by the NAT'd address.  In the case of peer
introduction and connection a third party is needed to provide the
external send-to address.

Is that what you were asking?

Regards,
Mark

---

From: Erik Möller <[email-masked]>
Date: Wed, 02 Jun 2010 10:47:57 +0200

> network stack folks (RakNet, etc).

For those who missed Mark's initial post on the subject his TorqueSocket  
API can be found here:
http://github.com/nardo/torque_sockets/raw/master/TorqueSocket_API.txt

Perhaps we could communally have a look at how this compares to other  
common network libraries to find a least common denominator of  
functionality?

> solution look like regarding Origin restrictions, etc?

Although it sure complicates things I don't see why WebSockets couldn't be  
extended to allow peer-to-peer connections if we really wanted to.
User agent A and B connects to WebSocket server C who keeps a list of  
clients connected to it. A and B are informed of the id's of connected  
clients and both decide to set up a peer-to-peer connection. A and B  
simultaneously call socket.connect_to_peer() passing a peer id and the  
call returns a new WebSocket in connecting mode. The user agents  
communicate with the server over the server-socket, retrieves the  
necessary information to connect to the peer including what origin to  
expect and off we go. (Server implementations would of course be free to  
not support this if they choose)
As long as the usual DOS/tamper protection is there it should be  
possible... then there's the success rate of UDP NAT hole punching...

-- 
Erik Möller
Core Developer
Opera Software

---

From: Erik Möller <[email-masked]>
Date: Thu, 03 Jun 2010 08:28:41 +0200

On Wed, 02 Jun 2010 19:48:05 +0200, Philip Taylor  
<[email-masked]> wrote:

I'm glad the discussion on this has taken off a bit. I've spoken to a few  
more game devs and even though it's still relatively few there's a slight  
majority that prefer the interface to be at the "Torque/RakNet-level"  
rather than at the "UDP-socket-wrapper-level". I'm hoping I can talk a few  
of them into joining the list and taking a more active part in the  
discussions.

> So they seem to suggest things like:
> - many games need a combination of reliable and unreliable-ordered and
> unreliable-unordered messages.

One thing to remember here is that browsers have other means for  
communication as well. I'm not saying we shouldn't support reliable  
messages over UDP, but just pointing out the option. I believe for example  
World of Warcraft uses this strategy and sends reliable traffic over TCP  
while movement and other real-time data goes over UDP.

> - many games need to send large messages (so the libraries do
> automatic fragmentation).

Again, this is probably because games have no other means of communication  
than the NW-library. I'd think these large reliable messages would mostly  
be files that need to be transferred asynchronously for which browsers  
already have the tried and tested XMLHttpRequest.

> - many games need to efficiently send tiny messages (so the libraries
> do automatic aggregation).

This is probably true for many other use-cases than games, but at least in  
my experience games typically use a bit-packer or range-coder to build the  
complete packet that needs to be sent. But again, it's a matter of what  
level you want to place the interface.

> Perhaps also:
> - Cap or dynamic limit on bandwidth (you don't want a single web page
> flooding the user's network connection and starving all the TCP
> connections)
> - Protection against session hijacking

Great

> - Protection against an attacker initiating a legitimate socket with a
> user and then redirecting it (with some kind of IP (un)hijacking) to a
> service behind the user's firewall (which isn't a problem when using
> TCP since the service will ignore packets when it hasn't done the TCP
> handshake; but UDP services might respond to a single packet from the
> middle of a websocket stream, so every single packet will have to be
> careful not to be misinterpreted dangerously by unsuspecting
> services).

I don't quite follow what you mean here. Can you expand on this with an  
example?

-- 
Erik Möller
Core Developer
Opera Software

---

From: Philip Taylor <[email-masked]>
Date: Tue, 1 Jun 2010 20:14:33 +0100

On Tue, Jun 1, 2010 at 2:00 PM, Erik Möller <[email-masked]> wrote:
> [...]
> I've never heard any gamedevs complain how poorly UDP matches their needs so
> I'm not so sure about that, but you may be right it would be better to have
> a higher level abstraction. If we are indeed targeting the game developing
> community we should ask for their feedback rather than guessing what they
> prefer. I will grep my linked-in account for game-devs tonight and see if I
> can gather some feedback.

Once you've got all that, a simple UDP-socket-like API might not be
the most natural or efficient way to implement a higher-level
partially-reliable protocol - the application couldn't cooperate with
the low-level network buffering to prioritise certain messages, it
couldn't use the packet headers that have already been added on top of
UDP, it would have to send acks from a script callback which may add
some latency after a packet is received from the network, etc. So I
think there's some tradeoffs and it's not a question of one low-level
protocol vs one strictly more restrictive higher-level protocol.

> network interface to be used for real-time games running in the browsers, at
> what level would you prefer the interface to be?

-- 
Philip Taylor
[email-masked]

---

From: Scott Hess <[email-masked]>
Date: Tue, 1 Jun 2010 16:34:05 -0700

> solution look like regarding Origin restrictions, etc?

Unix domain sockets allow you to pass file descriptors between
processes.  It might be interesting to pass a WebSocket endpoint
across a WebSocket.  If the clients can punch through NATs, it becomes
a direct peer-to-peer connection, otherwise it gets proxied through
the server.  Probably makes implementations excessively complicated,
though.  UDP-style would be easier (no need to worry about data
received by the server after it initiates pushing the endpoint to the
other client - just drop it on the floor).

-scott

---

From: Philip Taylor <[email-masked]>
Date: Wed, 2 Jun 2010 18:48:05 +0100

On Tue, Jun 1, 2010 at 9:02 PM, Erik Möller <[email-masked]> wrote:
> On Tue, 01 Jun 2010 21:14:33 +0200, Philip Taylor <[email-masked]>
> wrote:
>

I'm trying to think of them mainly as indirect examples of use cases,
rather than as direct examples of interfaces. Under the assumption
that most games either use a library like these or implement a
comparable one themselves, and that the library designs are driven by
the game requirements, if a feature is supported by most of the
libraries then it's probably needed by many games; and if a feature is
unsupported in many of the libraries then it's probably unnecessary
for most games. (Also an assumption: games running in web browsers
will have similar needs to native games (though lagging many years
behind state-of-the-art); and we only ought to aim to support the
needs of most games, not all games.)

So they seem to suggest things like:
- many games need a combination of reliable and unreliable-ordered and
unreliable-unordered messages.
- many games need to send large messages (so the libraries do
automatic fragmentation).
- many games need to efficiently send tiny messages (so the libraries
do automatic aggregation).
- many games need some kind of security (I have no idea exactly what,
or how much is still relevant when the client is JavaScript and
trivial to tamper with).
- many games need to prioritise certain messages when bandwidth is limited.
- most games don't need low-level control over individual datagrams
and precise packet loss feedback, they're okay with the socket details
being abstracted away.
- ... probably lots more (and/or less); I'm not very familiar with the
details of the libraries so this is unlikely to be an accurate list,
but I think it may be a useful way to analyse the requirements.

(The solution suggested in your initial post
(socket.send(data_smaller_than_mtu) going over UDP) seems to be one
extreme, which combines with higher-level JS libraries to satisfy
these needs. I think I initially suggested the other extreme of
encoding all the features into the browser API. I guess the best
tradeoff depends largely on what non-game use cases exist that should
be satisfied by the same solution.)

Perhaps also:

-- 
Philip Taylor
[email-masked]

---

From: Ben Garney <[email-masked]>
Date: Wed, 2 Jun 2010 11:35:21 -0700

On Wed, Jun 2, 2010 at 10:48 AM, Philip Taylor
<[email-masked]<[email-masked]>

> rather than as direct examples of interfaces. Under the assumption

This is a very valid approach. (Note that most serious games do not use
DirectPlay and it has not been updated by Microsoft in a while, so don't
burn a lot of braincells on it.)

Games always do weird stuff, focus on what will enable the 80% case and not
break the web! :)

Some thoughts point by point:

> unreliable-unordered messages.

Yes. But if you unreliable-unordered you can build the rest pretty easily.

> automatic fragmentation).

Yes. But many engines do fragmentation more explicitly, since they don't
spend a lot of time dealing with TCP streams. (The usual case is transfer of
asset files over the network connection, for instance, if a client doesn't
have a texture. I suspect for a JS app using UDP this would not be an issue,
they could just ask the web server.)

> do automatic aggregation).

Sort of. The common cases are most-recent state and message queues. Mostly
they define a packet format and have smart logic for incrementally sending
the most relevant state in each packet.

> or how much is still relevant when the client is JavaScript and

What is relevant for this discussion is to prevent man in the middle and DoS
activities. (Smart) game devs always assume the client is compromised - same
as smart web devs. :)

> - many games need to prioritise certain messages when bandwidth is limited.

See the aggregation comment. Also, games typically assume bandwidth is
limited - they will target and operate within a fixed bandwidth budget and
often a fixed packet rate. Say, 10 500 byte packets per second. This is
beneficial for many reasons, but primarily it is good because it optimizes
for realtime interaction. For some situations you will want fixed rate, for
others you want to have adaptive rate. (For instance, talking to a game
server in Counterstrike vs. playing an MMO.)

> and precise packet loss feedback, they're okay with the socket details

Disagree, for good networking you need to know about packet delivery state
at quite high levels of the game simulation.

> but I think it may be a useful way to analyse the requirements.

I think you got the major points. (Even though I don't necessarily agree
with all of your conclusions. :))

> be satisfied by the same solution.)

I think the best layer to provide is a notify protocol. That is, provide a
way to send lossy, smaller-than-MTU messages over a connection in realtime,
and notify the dev when a packet has made it (or hasn't). This way the
browser can provide a connection abstraction, which deals with the
DoS/security/origin issues. (For instance, two obvious pieces are connection
puzzles to prevent DoS, and PKI signing to prevent tampering with data. The
browser can also support arranged connections for the p2p case.)

It also keeps the interoperability manageable. Servers have a specific,
fairly small protocol to implement, and can provide some or all of a small
set of features. (For instance, caps might be: basic protocol support,
secure connections, arranged connections.) Beyond that it is application
specific blobs of small size (under a few kb).

All that said, I think looking at the notify protocol in TNL is a good
baseline. On that you can pretty easily implement every game scenario I can
think of. You can reuse a lot of the TCP connection security semantics, too,
since it is a connection - just one with lossy delivery.

Ben

---

From: Philip Taylor <[email-masked]>
Date: Thu, 3 Jun 2010 15:18:23 +0100

On Thu, Jun 3, 2010 at 7:28 AM, Erik Möller <[email-masked]> wrote:
> communication as well. I'm not saying we shouldn't support reliable messages
> over UDP, but just pointing out the option.

Yep - the relevant use cases ought to be supported decently by the
platform, but not necessarily by this extension to the platform (it
might be a different extension or it might be (probably is) supported
already).

> I don't quite follow what you mean here. Can you expand on this with an

I was thinking something like: A host at IP 11.11.11.11 on the public
internet runs some UDP service, like DNS or TFTP or something a bit
more secure. That service is restricted so it only responds to packets
received from IP 22.22.22.22 (a trusted user). The UDP Web Socket
handshake is carefully constructed so that it won't trigger dangerous
behaviour in any of those services (like how the TCP Web Socket uses a
safe HTTP-ish handshake).

An attacker hijacks the IP 11.11.11.11 from the perspective of the
user (by advertising new routes near the user), so the user's packets
to that address go to the attacker. The attacker gets the user to
visit a web page which sets up a UDP Web Socket with the attacker's
server at 11.11.11.11, doing all the handshake authentication
correctly.

The attacker then releases its hijacked address, so any subsequent Web
Socket packets will go to the original restricted service. Since
they're being received from the trusted user, the service will trust
them. Since the web browser has already done the Socket handshake, it
will believe it's talking to a legitimate Web Socket server and will
continue sending whatever data packets the attacker's script tells it
to.

The service will then be receiving and responding to
attacker-controlled packets, and will never have seen the carefully
constructed handshake that's designed to protect it. That's not a
danger for TCP services since they'll reject unexpected packets from
the middle of a TCP stream, but UDP services may accept packets from
the middle of a UDP Web Socket stream.

So it's not sufficient to carefully construct the Web Socket handshake
packets to not trigger unwanted behaviour in non-Socket services.
Every data packet sent on the Socket has to be carefully constructed
too.

(This might be a largely impractical or pointless attack, and there's
probably much easier ways to attack the exposed service, but I don't
know enough about security to judge that. Also I don't know what
packet construction would be sufficiently careful. But it seems like a
possible new concern that's introduced when using UDP in this
context.)

-- 
Philip Taylor
[email-masked]

---

From: Erik Möller <[email-masked]>
Date: Thu, 10 Jun 2010 21:35:39 +0200

During the Opera Network Seminar held in Oslo this week I discussed the  
possible addition of a new wsd: URL scheme to WebSockets that would allow  
relaxing the packet resends and enable demanding real-time applications to  
be written. I'd like to summarize some of the conclusions a few of us came  
to when discussing this (informally).

Regarding the discussions on at what level the API of a UDP-WebSocket  
should be: One of the most important aspects to remember are that for this  
to be interesting to application developers we need all the browser  
vendors to support this feature in a compatible way. Therefore it doesn't  
seem reasonable to standardize and spec a higher level network API akin to  
RakNet / Torque Socket and hope all vendors will be willing to spend the  
(quite large amount of) resources required for their own implementation of  
TCP over UDP, bandwidth throttling etc. In our opinion we're much better  
off standardizing a minimal UDP-like socket. For most application  
developers it seems likely they will be able to work with a mix of  
XMLHttpRequest, WebSockets and this new UDP-WebSocket to achieve the same  
functionality provided by those higher level APIs. If deemed necessary for  
an application the higher level network API can be written in JavaScript  
and work on-top of the much smaller hopefully cross-browser compatible  
UDP-WebSocket API.

As discussed the following features/limitations are suggested:
-Same API as WebSockets with the possible addition of an attribute that  
allows the application developer to find the path MTU of a connected  
socket.
-Max allowed send size is 65,507 bytes.
-Socket is bound to one remote address at creation and stays connected to  
that host for the duration of its lifetime.
-IP Broadcast/Multicast addresses are not valid remote addresses and only  
a set range of ports are valid.
-Reliable handshake with origin info (Connection timeout will trigger  
close event.)
-Automatic keep-alives (to detect force close at remote host and keep NAT  
traversal active)
-Reliable close handshake
-Sockets open sequentially (like current DOS protection in WebSockets) or  
perhaps have a limit of one socket per remote host.
-Cap on number of open sockets per host and global user-agent limit.

Some additional points that were suggested on this list were:
-Key exchange and encryption
  If you do want to have key exchange and encryption you really shouldn't  
reinvent the wheel but rather use a secure WebSocket connection in  
addition to the UDP-WebSocket. Adding key exchange and encryption to the  
UDP-WebSocket is discouraged.

-Client puzzles to reduce connection depletion/CPU depletion attacks to  
the handshake.
  If the goal is to prevent DOS attacks on the accepting server this seems  
futile. Client puzzles only raises the bar ever so slightly for an  
attacker so this is also discouraged.

-Packet delivery notification to be a part of the API.
  Again this is believed to be better left outside the UDP-WebSockets spec  
and implemented in javascript if the application developer requires it.

Best Regards,

-- 
Erik Möller
Core Developer
Opera Software

---

From: Mark Frohnmayer <[email-masked]>
Date: Thu, 10 Jun 2010 15:21:38 -0700

On Thu, Jun 10, 2010 at 12:35 PM, Erik Möller <[email-masked]> wrote:
> Regarding the discussions on at what level the API of a UDP-WebSocket should
> be: One of the most important aspects to remember are that for this to be
> interesting to application developers we need all the browser vendors to
> support this feature in a compatible way. Therefore it doesn't seem
> reasonable to standardize and spec a higher level network API akin to RakNet
> / Torque Socket and hope all vendors will be willing to spend the (quite
> large amount of) resources required for their own implementation of TCP over
> UDP, bandwidth throttling etc. In our opinion we're much better off
> standardizing a minimal UDP-like socket. For most application developers it

TorqueSocket is not in the same category as RakNet or OpenTNL -- the
point of the TorqueSocket effort was to find the web equivalent of a
minimal UDP'ish socket (understanding of course that it would be in
the context of a connected packet stream).  RakNet and OpenTNL provide
higher level data guarantees, RPC, object state replication, etc.
TorqueSocket is a simple send/recv packet protocol - it does no TCP
over UDP or bandwidth throttling.

> allows the application developer to find the path MTU of a connected socket.

I'd recommend doing some real-world testing for max packet size.  Back
when the original QuakeWorld came out it started by sending a large
connect packet (could be ~8K) and a good number of routers would just
drop those packets unconditionally.  The solution (iirc) was to keep
all packet sends below the Ethernet max of 1500 bytes.  I haven't
verified this lately to see if that's still the case, but it seems
real-world functionality should be considered.

> reinvent the wheel but rather use a secure WebSocket connection in addition
> to the UDP-WebSocket. Adding key exchange and encryption to the

If WebSocket supports an encrypted and unencrypted mode, why would the
real-time version not support data security and integrity?

>
> -Client puzzles to reduce connection depletion/CPU depletion attacks to the
> handshake.
> futile. Client puzzles only raises the bar ever so slightly for an attacker
> so this is also discouraged.

Client puzzles allow the host to allocate zero resources for a pending
connection until it knows that the source address of the client
request is valid and that the client has done some work; you could
still take a similar (though not client computationally expensive)
approach by having the host hash the client identity (IP/port) with a
server-generated secret.  Any approach that allocates memory or does
work on the host without verifying the client source address first is
vulnerable to a super-trivial DOS attack (connection depletion before
even any bandwidth overwhelm).

> and implemented in javascript if the application developer requires it.

I'd propose that doing this in the javascript level would result in
unnecessary extra overhead (sequence numbering, acknowledgements) that
could easily be a part of the underlying protocol.  Having implemented
multiple iterations of a high-level networking API, the notification
function is a critical, low-overhead tool for making effective
higher-level data guarantees possible.

Regards,
Mark

---

From: Lars Eggert <[email-masked]>
Date: Fri, 11 Jun 2010 07:25:41 +0300

Hi,

on a purely managerial level, let me point out that this work is far beyond the current charter of the HYBI WG. This defines an entirely new protocol, and will definitely require a charter discussion.

(If there is community/developer interest, we should by all means have that discussion.)

Lars

On 2010-6-10, at 22:35, Erik Möller wrote:

> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Erik Möller <[email-masked]>
Date: Fri, 11 Jun 2010 12:18:24 +0200

On Fri, 11 Jun 2010 00:21:38 +0200, Mark Frohnmayer  
<[email-masked]> wrote:

> TorqueSocket is not in the same category as RakNet or OpenTNL

Ah, sorry I got the names mixed up, I meant to say RakNet/OpenTNL and not  
RakNet/TorqueSocket.

> real-world functionality should be considered.

Absolutely, that's why the path-MTU attribute was suggested. The ~64k  
limit is an absolute limit though at which sends can be rejected  
immediately without even trying.

> real-time version not support data security and integrity?

The reasoning was that if you do need data security and integrity the  
secure websocket over TCP uses the same state-of-the-art implementation as  
the browsers already have implemented. Secure connections over UDP would  
either require a full TCP over UDP implementation (to use TLS) or a second  
implementation that would need to be maintained. That implementation would  
be either a very complex piece or software or clearly inferior to that  
users are accustomed to.
So what's a good use-case where you want a secure connection over UDP and  
cannot use a second TLS connection?

> even any bandwidth overwhelm).

Right, this is probably an area that needs to be looked more carefully at  
if/when real work is started on a spec.

> higher-level data guarantees possible.

Yes, no doubt it's useful for those implementing higher level APIs. As  
usual it's a matter of at what level to place the API.

-- 
Erik Möller
Core Developer
Opera Software

---

From: Erik Möller <[email-masked]>
Date: Fri, 11 Jun 2010 12:25:12 +0200

On Fri, 11 Jun 2010 06:25:41 +0200, Lars Eggert <[email-masked]>  
wrote:

> Hi,
>
> on a purely managerial level, let me point out that this work is far  
> beyond the current charter of the HYBI WG. This defines an entirely new  
> protocol, and will definitely require a charter discussion.
>

Sure, this just felt like the appropriate venue for these initial  
discussions. If not please say so and we'll move it somewhere else.

 From my personal experience when discussing this with developers and  
people from adjacent industries there's significant interest in the _final  
product_ but not so much interest in getting directly involved in the  
process.

-- 
Erik Möller
Core Developer
Opera Software

---

From: Mark Frohnmayer <[email-masked]>
Date: Fri, 11 Jun 2010 08:52:45 -0700

On Fri, Jun 11, 2010 at 3:18 AM, Erik Möller <[email-masked]> wrote:
> Absolutely, that's why the path-MTU attribute was suggested. The ~64k limit
> is an absolute limit though at which sends can be rejected immediately
> without even trying.

Ah, gotcha.  I was trying to separate the cases of MTU before
fragmentation and maximum packet size (fragmented) before some host
along the way silently discards the packet.

> The reasoning was that if you do need data security and integrity the secure
> websocket over TCP uses the same state-of-the-art implementation as the
> browsers already have implemented. Secure connections over UDP would either
> require a full TCP over UDP implementation (to use TLS) or a second
> be either a very complex piece or software or clearly inferior to that users
> are accustomed to.

I guess I was thinking more in the direction of a well conceived, well
reviewed protocol (TLS-level of scrutiny) for encryption of the
unreliable packet stream - a secure data stream is a useful primitive
and would otherwise have to be implemented at the application level.
I'd rather have security done well at the platform level (what
developers expect from a modern browser).  I'm not sure I'd agree that
it has to be super complex in order to be effective, and could likely
use much of the same functionality (ciphers, etc).

I'm also not assuming a second message channel -- if a developer is
using an API like TNL or RakNet that handles higher level data and
message guarantees then the addition of multiple message streams is
unneeded complexity.

>
> Yes, no doubt it's useful for those implementing higher level APIs. As usual
> it's a matter of at what level to place the API.

Agreed.  This goes back to Phil's point that UDP by itself isn't a
great primitive to build on -- the key element for real-time
applications is that dropped packets shouldn't stall the communication
stream nor be automatically retransmitted.  A slightly higher level
approach (connection handshake, security, etc), as long as its
overhead is low, could be a more effective foundation to build on.

Cheers,
Mark