Subject: [BEHAVE] RE: Review of draft-ietf-behave-turn-04.txt

From: "Dan Wing" <[email-masked]>
Date: Thu, 13 Sep 2007 12:14:48 -0700

Matt -- I don't recall seeing a response to your earlier email.  Please accept
my apologies for that.

There is a team of authors working on -05 (Rohan Mahy, Philip Matthews,
Jonathan Rosenberg, and me helping a bit).  As you can see from Jonathan's
post at http://www1.ietf.org/mail-archive/web/behave/current/msg02742.html,
the protocol may change if there is consensus in the working group for the
proposed changes.

See below for some specific responses.

> -----Original Message-----
> From: Matt Mathis [mailto:[email-masked]] 
> Sent: Thursday, September 13, 2007 11:46 AM
> To: Lars Eggert; Magnus Westerlund; Dan Wing
> Cc: [email-masked]; [email-masked]; 
> [email-masked]; Dan Wing; Transport Directorate; 
> [email-masked]
> Subject: Review of draft-ietf-behave-turn-04.txt
> 
> (Resend to include more participants and stakeholders.)
> 
> I've reviewed this document as part of the transport area 
> directorate's
> ongoing effort to review key IETF documents. These comments 
> were written
> primarily for the transport area directors, but are copied to 
> the document's
> authors for their information and to allow them to address 
> any issues raised.
> 
> Unfortunately I have not been a participating in the ongoing 
> discussion about
> STUN and TURN, so some of my comments may reflect my lack of 
> context, or
> otherwise repeat deeply reviewed features of these protocols.
> 
> Please forward this to the WG list as appropriate (I am not 
> subscribed).
> 
> My comments are primarily focused on 
> draft-ietf-behave-turn-04.txt, although
> some reflect architectural concerns about STUN itself.  These comments
> probably apply to additional documents.
> 
> Summary: TCP over TURN needs some additional work.  It would 
> improve the document not to intermix the TCP and UDP discussions.

We separated ICE into ICE-for-UDP (draft-ietf-mmusic-ice) and 
ICE-for-TCP, draft-ietf-mmusic-ice-tcp.  This was done to accelerate
ICE for UDP, as most implementors were interested in ICE for UDP,
as well as to simplify things.

At this time, we hope to keep TCP in the TURN specification, 
especially because TURN allows TCP-to-UDP interworking (see
section 4.1 of turn-04).  

> First I would like to use "/" notation to indicate various TURN
> configurations: UDP/TURN/TCP would be UDP over TURN over TCP
> transport.  I will use */TCP when */TLS/TCP also applies.
>
> I have no problems with UDP/TURN/UDP - the semantics of all three
> layer seem to be well matched.
>
> UDP/TURN/TCP is likely to cause problems for real time application
> because losses on the TCP connection will cause the UDP to be
> delivered very late rather than lost.  It should be noted that this
> is a non-preferred configuration, and why.

Good point, thanks.

> I have multiple concerns about the framing and multiplexing layer
> between TURN and it's transport.  Although I have only recently
> become aware of this mechanism, and are far from being a expert, it
> really strikes me as being non-robust.  I do understand why it
> exists and why it was done the way it is done.
>
> Particularly problematic for me is the idea that TCP/TURN/TCP can be
> used without framing, but without prohibiting further STUN messages.
> First it is a violation of TCP semantics to manage or expose segment
> boundaries.  TCP is not obligated to make segment boundaries match
> write() buffer edges and is even permitted to change segment
> boundaries on retransmission or for example as part of RFC 4821 MTU
> discovery.  Also read() buffers are not required to match segment
> boundaries.  This means that embedded TURN messages can not be
> assumed to be aligned with buffer edges and have to be discovered by
> scanning all client data.  This by itself is not a total show
> stopper, however, if I wanted to attack STUN, I would create data
> objects that resembled STUN messages. They could be placed anywhere:
> comments in web pages, etc., and although the authentication
> machinery would probably cause them to be discarded, the messages
> could first cause arbitrary data to be discarded, including possibly
> legitimate following control messages.  At this point is I do not
> believe that there is a robust or secure way to process any
> additional STUN messages, once a connection has carried any
> non-framed TCP/TURN/TCP data, and doing so should be forbidden.
> (The situation is much better for UDP/TURN/UDP, since the UDP syntax
> in not ambiguous with the STUN syntax).

TURN's TCP framing is described in sectoin 5 of turn-04, and
I do not believe suffers from the problems you describe above.

> Although I understand why STUN permits lots of flexibility in the
> framing/multiplexing layer, doing so creates the opportunity for
> unanticipated robustness problems that are the consequence of
> interactions between different STUN services specified in different
> documents.  There should be a unified analysis of threats against
> the weakly specified framing and multiplexing layer (please excuse
> me if I have overlooked one).  The simplest fix is to always require
> explicit framing for STUN/TCP, except perhaps a few specific events
> such as one transition to TCP/TURN/TCP, which would forbid all
> further STUN messages.

It would be helpful if you looked at the new 'channel' proposal
from Jonathan,
http://www1.ietf.org/mail-archive/web/behave/current/msg02742.html, 
and see if it suffers a similar problem or if the use of 'channels' would be
superior.

> The document does not describe how TURN would be used or deployed.

During IESG review, Jonathan added an "Operational Considerations"
section to ICE (it's in draft-ietf-mmusic-ice-18).  Perhaps something
akin to that would be useful for TURN.

> I will make the assumption that TURN is not designed to only be used
> by TURN specific applications.  Ordinary Internet application would
> be adapted to use TURN by partially replacing run-time libraries
> and/or system functions with TURN enabled versions of the same
> functions.  This places additional constraints on TURN.  Not only
> must there be a clear mapping between TURN messages and actions on
> the connection to the external client, there also has to be a clear
> mapping between events at the application API in the STUN client and
> corresponding TURN messages.  This implicitly requires that there is
> a reasonable mapping between TCP API semantics and TURN messages at
> the STUN client.
>
> There are a number of problems in this area.  For starters no TCP
> application ever knows what bandwidth it is going to use, so "should
> include a BANDWIDTH attribute" is pointless. 

Some TCP applications do; for example, if you're running RTP over
TCP and you know the codec you're using, you do know the bandwidth
you're going to consume.

> More serious, there
> are a number of places where it is specified that TCP payload data
> is silently discarded.  Under no circumstances does TCP silently
> discard data, it always reports errors through the API, usually
> synchronously at that.  Many application depend on this to shutdown
> failed connections.  It is guaranteed that there are applications
> that will hang or loop forever if data is silently discarded.

I concur that if a TURN server is doing TCP-to-TCP, it shouldn't 
be allowed to discard data.  However, if it's doing TCP-to-UDP,
I don't see the harm in it discarding data; such discarding can
even be helpful for realtime applications (packets that arrive
late are worthless).

> I have a feeling that I overlooked other API glitches.
> 
> It makes the document extra obtuse to have UDP and TCP discussions
> interleaved, when the underlying assumptions to the protocols 
> are totally different.  It would help to separate the descriptions 
> of TCP/TURN/* from UDP/TURN/*. Although the TURN messages look 
> similar for TCP and UDP, they have substantially different 
> semantics.
> 
> Finally, the question that triggered this review: a point 
> raise by Rémi Denis-Courmont and Rohan Mahy on about Aug 23rd.  
> The connection multiplexing mechanism does not implement per 
> stream flow control and is subject to head-of-queue blocking 
> if one of the streams is making insufficient progress
> for some reason. 

Yep.

> Put anther way, any application or context that causes "TCP
> persist" or "zero window probing" will cause deadlock like 
> TURN protocol freezes.  It is also possible to construct 
> examples that can exhibit true deadlocks.

This problem only occurs when we have SIP forking, and when
SIP forking occurs we're only likely to need ICE connectivity
checks and TLS (or DTLS) handshakes.  

In -05, we will provide reasons why the duration of this
activity needs to be limited, and describe how (with SIP)
re-Invites can be performed to regain 1-to-1 TCP connections
so that HOL blocking is avoided.

> Early versions of SSH port forwarding also had this this problem.
> To fix it each stream has it's own buffer at the demultiplexer, and
> there is an explicit flow control signaling mechanism to inform the
> multiplexer how much buffer space is available for each channel.
> The multiplexer is not permitted to send data for any channel that
> would overflow that channels buffer at the demultiplexer.  Search
> RFC4254 for "window".

Thank you very much for that pointer -- Rohan, Jonathan, Philip
and I were discussing that very problem Monday evening.

> BTW, user experience with the older non-flow-control SSH 
> was not good.  Head-of-queue blocking is a bad thing and 
> must be avoided.
> 
> To further complicate the matter, an optimal implementation 
> requires that each of the buffers at the demultiplexer be 
> the same size as the delay bandwidth product of the 
> underlying TCP connection itself.  This has been implemented in
> HPN-SSH http://www.psc.edu/networking/projects/hpn-ssh/ , 
> although not described in a standards document.
> 
> The security section for TURN (and STUN) are focused on 
> attacks against the protocols themselves, and do not adequately 
> address ways in which these protocols might be used to attack 
> other protocols or resources.  In particular a TURN/STUN server 
> that does weak Authentication, Authorization or Accounting
> (logging) can be used to implement untraceable anonymous 
> attacks on other resources.  These configurations must be 
> forbidden.

Good point.  We will add some text to the Operational 
Considerations section or, failing that, to the Security 
Considerations section.
 
> Although UDP/TURN might be ready to progress, TCP/TURN needs 
> to be reworked.
> It might make sense to divide (or restructure) the document.

We will consider it, especially that we split ICE-TCP long
ago.

> Please explicitly include me on any follow up discussion.

Thanks for your thorough comments!

-d

_______________________________________________
Behave mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/behave

---

From: Matt Mathis <[email-masked]>
Date: Thu, 13 Sep 2007 14:45:46 -0400 (EDT)

I've reviewed this document as part of the transport area directorate's
ongoing effort to review key IETF documents. These comments were written
primarily for the transport area directors, but are copied to the document's
authors for their information and to allow them to address any issues raised.

Unfortunately I have not been a participating in the ongoing discussion about
STUN and TURN, so some of my comments may reflect my lack of context, or
otherwise repeat deeply reviewed features of these protocols.

Please forward this to the WG list as appropriate (I am not subscribed).

My comments are primarily focused on draft-ietf-behave-turn-04.txt, although
some reflect architectural concerns about STUN itself.  These comments
probably apply to additional documents.

Summary: TCP over TURN needs some additional work.  It would improve the
document not to intermix the TCP and UDP discussions.

First I would like to use "/" notation to indicate various TURN
configurations: UDP/TURN/TCP would be UDP over TURN over TCP transport.  I
will use */TCP when */TLS/TCP also applies.

I have no problems with UDP/TURN/UDP - the semantics of all three layer seem
to be well matched.

UDP/TURN/TCP is likely to cause problems for real time application because
losses on the TCP connection will cause the UDP to be delivered very late
rather than lost.  It should be noted that this is a non-preferred
configuration, and why.

I have multiple concerns about the framing and multiplexing layer between TURN
and it's transport.  Although I have only recently become aware of this
mechanism, and are far from being a expert, it really strikes me as being
non-robust.  I do understand why it exists and why it was done the way it is
done.

Particularly problematic for me is the idea that TCP/TURN/TCP can be used
without framing, but without prohibiting further STUN messages.  First it is a
violation of TCP semantics to manage or expose segment boundaries.  TCP is not
obligated to make segment boundaries match write() buffer edges and is even
permitted to change segment boundaries on retransmission or for example as
part of RFC 4821 MTU discovery.  Also read() buffers are not required to match
segment boundaries.  This means that embedded TURN messages can not be assumed
to be aligned with buffer edges and have to be discovered by scanning all
client data.  This by itself is not a total show stopper, however, if I wanted
to attack STUN, I would create data objects that resembled STUN messages.
They could be placed anywhere: comments in web pages, etc., and although the
authentication machinery would probably cause them to be discarded, the
messages could first cause arbitrary data to be discarded, including possibly
legitimate following control messages.  At this point is I do not believe that
there is a robust or secure way to process any additional STUN messages, once
a connection has carried any non-framed TCP/TURN/TCP data, and doing so should
be forbidden.  (The situation is much better for UDP/TURN/UDP, since the UDP
syntax in not ambiguous with the STUN syntax).

Although I understand why STUN permits lots of flexibility in the
framing/multiplexing layer, doing so creates the opportunity for unanticipated
robustness problems that are the consequence of interactions between different
STUN services specified in different documents.  There should be a unified
analysis of threats against the weakly specified framing and multiplexing
layer (please excuse me if I have overlooked one).  The simplest fix is to
always require explicit framing for STUN/TCP, except perhaps a few specific
events such as one transition to TCP/TURN/TCP, which would forbid all further
STUN messages.

The document does not describe how TURN would be used or deployed.  I will
make the assumption that TURN is not designed to only be used by TURN specific
applications.  Ordinary Internet application would be adapted to use TURN by
partially replacing run-time libraries and/or system functions with TURN
enabled versions of the same functions.  This places additional constraints on
TURN.  Not only must there be a clear mapping between TURN messages and
actions on the connection to the external client, there also has to be a clear
mapping between events at the application API in the STUN client and
corresponding TURN messages.  This implicitly requires that there is a
reasonable mapping between TCP API semantics and TURN messages at the STUN
client.

There are a number of problems in this area.  For starters no TCP application
ever knows what bandwidth it is going to use, so "should include a BANDWIDTH
attribute" is pointless.  More serious, there are a number of places where it
is specified that TCP payload data is silently discarded.  Under no
circumstances does TCP silently discard data, it always reports errors through
the API, usually synchronously at that.  Many application depend on this to
shutdown failed connections.  It is guaranteed that there are applications
that will hang or loop forever if data is silently discarded.

It makes the document extra obtuse to have UDP and TCP discussions
interleaved, when the underlying assumptions to the protocols are totally
different.  It would help to separate the descriptions of TCP/TURN/* from
UDP/TURN/*. Although the TURN messages look similar for TCP and UDP, they have
substantially different semantics.

Finally, the question that triggered this review: a point raise by Rémi
Denis-Courmont and Rohan Mahy on about Aug 23rd.  The connection multiplexing
mechanism does not implement per stream flow control and is subject to
head-of-queue blocking if one of the streams is making insufficient progress
for some reason.  Put anther way, any application or context that causes "TCP
persist" or "zero window probing" will cause deadlock like TURN protocol
freezes.  It is also possible to construct examples that can exhibit true
deadlocks.

Early versions of SSH port forwarding also had this this problem.  To fix it
each stream has it's own buffer at the demultiplexer, and there is an explicit
flow control signaling mechanism to inform the multiplexer how much buffer
space is available for each channel.  The multiplexer is not permitted to send
data for any channel that would overflow that channels buffer at the
demultiplexer.  Search RFC4254 for "window".  BTW, user experience with the
older non-flow-control SSH was not good.  Head-of-queue blocking is a bad
thing and must be avoided.

To further complicate the matter, an optimal implementation requires that each
of the buffers at the demultiplexer be the same size as the delay bandwidth
product of the underlying TCP connection itself.  This has been implemented in
HPN-SSH http://www.psc.edu/networking/projects/hpn-ssh/ , although not
described in a standards document.

The security section for TURN (and STUN) are focused on attacks against the
protocols themselves, and do not adequately address ways in which these
protocols might be used to attack other protocols or resources.  In particular
a TURN/STUN server that does weak Authentication, Authorization or Accounting
(logging) can be used to implement untraceable anonymous attacks on other
resources.  These configurations must be forbidden.

Although UDP/TURN might be ready to progress, TCP/TURN needs to be reworked.
It might make sense to divide (or restructure) the document.

Thanks,
--MM--

---

From: Matt Mathis <[email-masked]>
Date: Thu, 13 Sep 2007 17:07:05 -0400 (EDT)

> checks and TLS (or DTLS) handshakes.

This comment implies that TURN might only be used with SIP - This is not a
good assumption at all.

If STUN and/or TURN run as stand alone services, they will come to carry
arbitrary other protocols.  SIP might tolerate head-of-queue blocking under
normal conditions, but that is not an acceptable assumption to make for a
protocol that might be appropriate for solving a wide range of problems.

TURN MUST be robust and free from deadlocks, no matter what client protocols
are using it.  It is ok to explicitly implement only a subset of the functions
(e.g. no general listen() facility), but this has to be stated up front. "It
will do weird things with unexpected client protocols" will be a show stopper
further down the standards track.

As for testing future versions of TURN for head-of-queue blocking, think about
pathologies when one channel has a sender with lots of data to send and a
receiver that dawdles.  Does this interfere with other channels at all?  Does
it interfere with the return data on the same or other channels?  If any of
these are true, you can be sure that there will be pathological behavior
(deadlocks?) under some unexpected conditions.

Thanks,
--MM--
-------------------------------------------
Matt Mathis      http://www.psc.edu/~mathis
Work:412.268.3319    Home/Cell:412.654.7529
-------------------------------------------
Evil is defined by mortals who think they know
"The Truth" and use force to apply it to others.