Subject: [BEHAVE] TURN - some proposed changes for improvement and simplification

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 12 Sep 2007 12:59:42 -0400

As Dan mentioned at the last IETF meeting, we've formed a small team to
actively work on TURN over the next month, in order to accelerate its
completion and produce a version ready for WGLC by the end of September.

As part of this, we took a step back and looked a bit at some of the
more troubling aspects of TURN. Three in particular seemed important:

* There are several different methods of framing and encapsulation of 
messages with a number of exceptions based on transport protocol, 
contents of the data, and the active destination.  The overall effect is 
confusing and rather complex.

* The active destination concept is inelegant, confusing, complex, and 
during forking results in undesirable overhead and at least one 
timing-related race condition.

* the use of multiple TCP connections from the TURN server towards
peers, when there is one TCP connection from the TURN server to the
client, has been a concern on the list recently.

We have a proposal on some non-trivial changes to TURN on these topics.

Firstly, we can eliminate the entire need for active destinations (and
the Send and Data primitives as well) by introducing the concept of a
'channel'. Each channel is a stream of application data sent to and from
a remote IP and port. In the case of UDP, we'd add a SINGLE BYTE to the
front of every packet to and from the TURN server, which is used to
signal the channel number. Channel 0 is reserved for TURN signaling to
and from the client and TURN server. The remaining 255 channels are for
remote destinations and carry pure data after the 1-byte channel number.
So when a client does a Send or a Connect, the server allocates a
remaining channel number and passes this channel number back to the
client. From that point on, when the TURN server receives data from the
peer, it sends it to the client and adds only the byte up front
containing the channel number. Similarly, when the client wants to send
data, it just adds this byte up front indicating the channel number, and
the TURN server sends the data to the corresponding remote peer. There
is no encapsulation of data as a TURN message ever. Thus, the Data
indication is removed. TURN currently requires the server to check for
the magic cookie in packets received from remote peers, and when it sees
it, encapsulate the resulting data in a Data indication. That
requirement would also be eliminated.

Consider a forking use case where A calls a URI and this forks to B and
C. Both send a 200 OK. A will end up running ICE simultaneously with
both B and C. All of the STUN checks and any resulting RTP would just be
sent over different channels, all with identical overhead (1 byte).
There would be no need to re-invite, as there would typically be in the
current TURN mechanism.

This also fixes the complexity that was raised on the list for
packetcable; the current mechanism complicates their QOS mechanisms
because the bandwidth for the stream is not deterministic. Here, it is
deterministic. Every packet always has an extra 1 byte per packet overhead.

The main reason we've avoided any kind of framing for UDP up until now
has been overhead. So what kind of overhead does this introduce? Well if
we assume something really nasty like g729 with 20ms framing, thats 20
bytes of payload plus 40 for RTP/UDP/IP gives us 60, which now goes to
61. Thats a 1.6 percent penalty which is pretty good, considering
especially the huge overhead you are already paying for RTP/UDP. Anyway
a more realistic number is g711 with 30ms framing, and there is .35%
overhead. This cost seems reasonable given the simplifications it brings.

Another possible concern with the 1-byte overhead is path MTU. If a UDP
packet was sent from a remote peer and it was exactly equal to the path
MTU from the remote peer to the TURN server, this will now fragment on
the TURN server to client link. Its not clear that this is a real
problem. But if you are aware of UDP applications that send data exactly
at the MTU for some reason, please speak up.

For TCP, the channel mechanism would work the same. The current framing
already has a 'type' field which we'd just turn into a channel number.
Thus both UDP and TCP would have framing, the difference is only that
TCP includes a length whereas UDP does not. Protocol operation is
otherwise more or less the same, and this would forge closer alignment
between UDP and TCP. As with UDP, this allows multiple TCP connections
from the TURN server to remote peers, each of which would have its own
channel number, allocated to the client when permissions are created for
a remote IP address, or when a connect request is sent. Connection
status indications would all come over the control channel, and each
such indication would contain, as an attribute, the channel number that
the status indication is associated with.

Now, that said, TCP has some additional complexities that UDP does not.
First and foremost is that you need to be very careful with how you
handle buffering. We want to avoid the case where data arrives quickly
on the connection from the client to the TURN server, but the TCP
connection from the TURN server to the remote peer is slow, and so data
is lost at the TURN server when buffers overflow. To handle this, we
will specify a buffering requirement and a possiblemechanism. There 
would be a maximum buffer size  per connection set in the kernel (for 
example). The TURN server application code reads  off of this buffer and 
writes to the other TCP connection. If it cannot write because the 
window is full, it doesn't read data from the incoming
connection. This will eventually cause the buffers in the kernel to
fill, and then packets are dropped (and not ACKed), causing backpressure
on the TCP connections. This effectively bridges the congestion control
between the connections, and ensures that, TURN server failures aside,
no packets get dropped in the TURN server. We believe that this kind of
mechanism is used in other similar boxes like HTTP proxies.

Now, lets consider the case of multiple TCP connections to a
multiplicity of remote peers. There have been suggestions on the list to
just disallow this. We cannot. It happens as an artifact of forking.
Consider client A that sends an offer. Amongst its candidates is a
passive TCP candidate learned from the TURN server. THis INVITE forks to
B and C, each of which return a single active candidate. Now, A will
need to open permissions on the TURN server for B and C's IP addresses
listed in the SDP answer, and both B and C will open a TCP connection
towards the TURN server in order for ICE checks to proceed. Note well,
that since ICE processing happens ideally prior to call answering, this
case happens NOT JUST when both B and C answer at the same time, but
whenever there is forking at all with ICE-tcp. So this is actually a
common case and not a corner case.

In this case of multiple TCP connections to remote peers, the congestion
control bridging described above gets complicated, and head-of-line
blocking gets introduced. Congestion on the connection from the TURN
server to one peer can slow down the connection to the client, and thus
limit the throughput to the other peer. This is only a problem when we
are transferring real data, not so much for things like STUN messages or
TLS handshakes. So the recommendation is going to be that, if a client
ends up in a case with multiple TCP connections that survive past the
connectivity checks, in that case the spec will recommend the client
allocate a new TCP candidate from the TURN server and restart ICE with
one of the peers, in order to eliminate the HOL blocking.

We're looking for feedback on this proposal (and questions and
clarifications of course) ASAP - within the next week. If this seems
like a good direction, we can work these changes in and address the
other comments and still deliver a revision on schedule.

Thanks,
Jonathan R.
-- 
Jonathan D. Rosenberg, Ph.D.                   600 Lanidex Plaza
Cisco Fellow                                   Parsippany, NJ 07054-2711
Cisco Systems
[email-masked]                              FAX:   (973) 952-5050
http://www.jdrosen.net                         PHONE: (973) 952-5000
http://www.cisco.com

_______________________________________________
Behave mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/behave

---

From: "Kevin Johns" <[email-masked]>
Date: Wed, 12 Sep 2007 11:27:28 -0600

Jonathan,

Thanks for the lengthy write-up on the proposal. It does naturally raise
a few questions for clarification...

How does the channel concept work if the TURN server receives data from
a peer prior to the client setting up a channel for that peer? In the
current model, that data would be forwarded to the client with a receive
indication which does not require any action by the client. This was
both nice and not-so-nice in that a client did not have to communicate
with the TURN server if a call forked and could wait for the 200 OK to
set the active destination. It did create some QoS complexity, but
minimized client communication with the TURN server.

The other question is around the details of how a channel is setup
between the client and TURN server and how this interacts with ICE. When
in the call setup/ICE process would the client setup a channel with the
TURN server? Would the client setup a channel for each candidate
received or only those which are validated as part of the ICE
connectivity checks?

Thanks
Kevin

-----Original Message-----
From: Jonathan Rosenberg [mailto:[email-masked]] 
Sent: Wednesday, September 12, 2007 11:00 AM
To: Behave WG
Subject: [BEHAVE] TURN - some proposed changes for improvement
andsimplification

This also fixes the complexity that was raised on the list for
packetcable; the current mechanism complicates their QOS mechanisms
because the bandwidth for the stream is not deterministic. Here, it is
deterministic. Every packet always has an extra 1 byte per packet
overhead.

overhead. This cost seems reasonable given the simplifications it
brings.

Now, that said, TCP has some additional complexities that UDP does not.
First and foremost is that you need to be very careful with how you
handle buffering. We want to avoid the case where data arrives quickly
on the connection from the client to the TURN server, but the TCP
connection from the TURN server to the remote peer is slow, and so data
is lost at the TURN server when buffers overflow. To handle this, we
will specify a buffering requirement and a possiblemechanism. There
would be a maximum buffer size  per connection set in the kernel (for
example). The TURN server application code reads  off of this buffer and
writes to the other TCP connection. If it cannot write because the
window is full, it doesn't read data from the incoming connection. This
will eventually cause the buffers in the kernel to fill, and then
packets are dropped (and not ACKed), causing backpressure on the TCP
connections. This effectively bridges the congestion control between the
connections, and ensures that, TURN server failures aside, no packets
get dropped in the TURN server. We believe that this kind of mechanism
is used in other similar boxes like HTTP proxies.

Thanks,
Jonathan R.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 12 Sep 2007 14:40:31 -0400

Kevin Johns wrote:
> a peer prior to the client setting up a channel for that peer?

It has been the case for a long time that a TURN server won't just 
accept data from a random peer. You need to set up a permission by 
telling the TURN server the identity of the remote peer. So, when you do 
that, the TURN server will allocate the channel to you at that time.

> In the
> indication which does not require any action by the client. 

It had required the client to do a Send. It is the Send that will 
establish the channel.

> This was
> TURN server?

In ICE processing, once the O/A exchanges happens an agent starts to 
send checks. It does so by issuing a Send request to the TURN server. It 
would continue to do that, but instead would issue a message to create a 
channel to the remote peer and then send data over the channel that has 
been allocated. Its possible to combine these into one transaction also.

  Would the client setup a channel for each candidate
> received or only those which are validated as part of the ICE

For each remote candidate it will have a separate channel.

-Jonathan R.

---

From: "Kevin Johns" <[email-masked]>
Date: Wed, 12 Sep 2007 13:36:48 -0600

Some more comments in line, if we can keep the setting up of a channel a
byproduct of sending a connectivity check, this would appear to be a
better approach. I certainly like having a more deterministic overhead.

Kevin
-----Original Message-----
From: Jonathan Rosenberg [mailto:[email-masked]] 
Sent: Wednesday, September 12, 2007 12:41 PM
To: Kevin Johns
Cc: Behave WG
Subject: Re: [BEHAVE] TURN - some proposed changes for improvement
andsimplification

Kevin Johns wrote:
> Jonathan,
> 
> Thanks for the lengthy write-up on the proposal. It does naturally 
> raise a few questions for clarification...
> 
> How does the channel concept work if the TURN server receives data 
> from a peer prior to the client setting up a channel for that peer?

It has been the case for a long time that a TURN server won't just
accept data from a random peer. You need to set up a permission by
telling the TURN server the identity of the remote peer. So, when you do
that, the TURN server will allocate the channel to you at that time.
<kcj> Right, but I think a permission is automatically created when the
client sends a send request which also contains the binding request for
the connectivity check.

> In the
> current model, that data would be forwarded to the client with a 
> receive indication which does not require any action by the client.

> between the client and TURN server and how this interacts with ICE. 
> When in the call setup/ICE process would the client setup a channel 
> with the TURN server?

In ICE processing, once the O/A exchanges happens an agent starts to
send checks. It does so by issuing a Send request to the TURN server. It
would continue to do that, but instead would issue a message to create a
channel to the remote peer and then send data over the channel that has
been allocated. Its possible to combine these into one transaction also.
<kcj> Right, so the send request would add the remote endpoint to the
permissions list and forward the binding request at the same time. It
would be good if we could keep that as a single transaction.

For each remote candidate it will have a separate channel.
<kcj> Right and the channel would be setup as a byproduct of the binding
request (hopefully).

-Jonathan R.

---

From: Rémi Denis-Courmont <[email-masked]>
Date: Thu, 13 Sep 2007 11:41:52 +0300

Le Wednesday 12 September 2007 19:59:42 ext Jonathan Rosenberg, vous avez 
écrit :
(...)
> signal the channel number. (...)

That looks ok.

This would break natural alignment of the inner payload. But the overhead is 
fixed size, so that could be easily compensated.

(...)
> at the MTU for some reason, please speak up.

Given how path MTU discovery is unreliable through NATs/firewalls, that will 
hardly worsen the situation.

(...)
> TLS handshakes.

OK for connectivity checks, provided the output buffer of any TCP session is 
big enough to handle one connectivity check (I assume you don't send more 
than one check in TCP case). And it most certainly is big enough.

It might actually be a problem for TLS handshakes? Though it was my 
understanding ICE-TCP would do connectivity checks first. Would this not make 
TLS handshakes a non-issue?

> So the recommendation is going to be that, if a client 

Precisely.

-- 
Rémi Denis-Courmont

---

From: Marc Petit-Huguenin <[email-masked]>
Date: Tue, 18 Sep 2007 10:26:10 -0700

Hi Jonathan,

I like very much the simplification but I have some comments.

As I understand it, permissions contains only the IP address of the
peer.  So if a channel is created for a specific IP address and port,
when data will be coming from the peer from a different port they will
be accepted by the permission but there is no mechanism in the channel
to send the real port to the TURN client.

Also I think that perhaps the Send and Connect methods can be merged
into one request, like Create Channel (and add a Destroy Channel for
symmetry).

-- 
Marc Petit-Huguenin           [                                 ]
Home: [email-masked] [RFC1855-compliant space for rent ]
Work: [email-masked]            [                                 ]
[                                                               ]

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 18 Sep 2007 16:53:06 -0400

Yeah - we noticed this also. We've got a few ideas on how to deal with 
this. Stay tuned.

-Jonathan R.