Subject: [MMUSIC] Updating the pacing of ICE connectivity checks

From: Ari Keränen <[email-masked]>
Date: Thu, 14 Feb 2013 15:49:24 +0200

Hi all,

Updating the pacing parameter, Ta, of ICE for non-RTP traffic (for 
background, see [1] and [2]), as discussed at the Atlanta meeting, will 
be one of the important changes in the new version of ICE.

Currently the RFC5245 says (for non-RTP traffic) "Ta SHOULD be 
configurable, SHOULD have a default of 500 ms, and MUST NOT be 
configurable to be less than 500 ms."

This results in very slow pace of starting connectivity checks if ICE is 
used with non-RTP traffic. With RTP, the Ta is calculated based on the 
expected bandwidth used by the RTP stream and can be down to 20 ms. 
Therefore, many current implementations apparently ignore the "MUST NOT" 
above and set Ta to something lower.

The quite conservative value of 500 ms was originally chosen because of 
the fear of congesting the links with the connectivity check messages. 
This is obviously still a valid concern, but would suggesting something 
lower still make sense? How low would it be reasonably safe to go? What 
is currently used in practice? Can we come up with a formula similar to 
RTP (perhaps based on expected bandwidth usage)?

Cheers,
Ari

[1] http://tools.ietf.org/html/rfc5245#appendix-B.1
[2] http://tools.ietf.org/html/rfc5245#section-16.2

---

From: "Dan Wing" <[email-masked]>
Date: Thu, 14 Feb 2013 07:20:13 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Ari Keränen
> Sent: Thursday, February 14, 2013 5:49 AM
> To: mmusic
> Subject: [MMUSIC] Updating the pacing of ICE connectivity checks
> 
> Hi all,
> 

That would explain failures claimed to justify the desire for running
over one port.  

The 20ms threshold for the initial connectivity checks is because some 
NATs could not create new mappings quickly.  This problems appears to
persist today, considering that both Chrome and Windows 8 limit their 
number of outstanding DNS queries in an apparent attempt to work around
that very same NAT UDP mapping behavior.

If trying to create NAT mappings with 0ms pacing, it will certainly 
cause NAT mapping failures and force retries.

> RTP (perhaps based on expected bandwidth usage)?

A good question for the transport area, which is the origination of the
suggestion to send ICE connectivity checks paced to not exceed the 
bandwidth of the subsequent media.  Perhaps the arguments that TCP 
IW=10 is not harmful could be brought to the party.

-d

> _______________________________________________
> mmusic mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mmusic

---

From: Ari Keränen <[email-masked]>
Date: Thu, 14 Feb 2013 17:32:55 +0200

> That would explain failures claimed to justify the desire for running

Interesting point. Does anyone know what kind of pacing was actually 
used when these failures were observed?

> cause NAT mapping failures and force retries.

Yes, definitely. So we shouldn't go below 20ms with non-RTP either. 
Perhaps need to add some more justification/warning text about this too.

> IW=10 is not harmful could be brought to the party.

FYI, I forwarded the original mail to the Transport ADs too.

Cheers,
Ari

---

From: Emil Ivov <[email-masked]>
Date: Thu, 14 Feb 2013 16:36:56 +0100

Any idea why there had to be different pacing for RTP and non-RTP flows
in the first place?

Emil

-- 
https://jitsi.org

---

From: Ari Keränen <[email-masked]>
Date: Thu, 14 Feb 2013 17:44:32 +0200

> Any idea why there had to be different pacing for RTP and non-RTP flows

Because with RTP you know the to-be-used bandwidth and can match checks 
with that (http://tools.ietf.org/html/rfc5245#section-16.1):
>    These formulas are aimed at causing STUN transactions to be paced at
>    the same rate as media.  This ensures that ICE will work properly
>    under the same network conditions needed to support the media as
>    well.

Cheers,
Ari

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 14 Feb 2013 16:51:13 +0100

> Any idea why there had to be different pacing for RTP and non-RTP flows

I think our arguments where that in the context of a non-congestion
controlled voice application it would send with 20 ms interval anyway.
For other transports and protocols it was not obvious for what intended
environment it would be used. For example a low power network
application that sends beyond 30 kbps could be problematic. Thus we took
one of our conservative values from the transport books, i.e. 500 ms.

I would point out that this is a difficult transport issue. Compared to
TCP IW10, that burst occurs first after the initial handshake when one
has actually established that there is a responder.

With ICE there is a clear issue with an attacker bloating a offer/answer
with candidates that recreates the voice hammer attack using ICE's STUN
checks towards a target address or domain.

Also the STUN checks will have different source address pairs which in
it self could be an issue for attack prevention firewalls etc.

Thus I would be careful with speeding up Ta for RTP to much, and I think
someone needs to fine empiric data that it would work.

For the the 500 ms that can likely be significantly lowered without any
issues. But some considerations around the wider usage needs to be taken
into account.

Cheers

Magnus
One of the ADs responsible for the pacing in ICE.

-- 

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: Ari Keränen <[email-masked]>
Date: Thu, 14 Feb 2013 18:12:36 +0200

Hi Magnus,

Thanks for clarifying the background on this! Some comments and 
questions inline.

> someone needs to fine empiric data that it would work.

I'd guess the 20 ms interval is quite OK for the RTP since, as Dan 
noted, you can't go much faster without overwhelming the NATs.

> issues. But some considerations around the wider usage needs to be taken

Would you have any recommendation for a good, less conservative value? 
If the bandwidth of the non-RTP stream is known, I guess that could be 
used in a similar way as with RTP to calculate Ta (with lower bound of 
20ms). For cases where the bw is not known, we could perhaps 
RFC2119-recommend some fairly-conservative value (say, around 100ms?) 
and "MUST NOT" for below 20ms?

Cheers,
Ari
(as individual)

---

From: Emil Ivov <[email-masked]>
Date: Thu, 14 Feb 2013 17:12:48 +0100

Thanks for explaining Magnus,

some more inline

> checks towards a target address or domain.

OK, but we already have 5.7.3 and the 100 check limitation, right?

I suppose the thing that is really problematic here is the strength of
the recommendation. Other sections in the spec seem to often refer to
often leave the choice to the application advising use of a rate that
matches the traffic that's about to be exchanged.

It seems that a similar recommendation would make sense here. WDYT?

Emil

-- 
https://jitsi.org

---

From: "Dan Wing" <[email-masked]>
Date: Thu, 14 Feb 2013 15:43:21 -0800

> From: Magnus Westerlund [mailto:[email-masked]]
> Sent: Thursday, February 14, 2013 7:51 AM
> To: Emil Ivov
> Cc: Ari Keränen; 'mmusic'; Dan Wing
> Subject: Re: [MMUSIC] Updating the pacing of ICE connectivity checks
> 
> > Any idea why there had to be different pacing for RTP and non-RTP
> > flows in the first place?
> 

Most of those concerns are for the ICE connectivity checks (to the remote
peer).  There are different concerns for the initial STUN (and TURN) 
communications to the STUN/TURN servers, which are the communications
that create the NAT (or firewall) mappings.  I noticed the two different
concerns are nicely written in
http://tools.ietf.org/html/rfc5245#appendix-B.1

Question, playing devil's advocate:  is the voice hammer attack (or a
STUN connectivity check hammer attack) worse than a web page full of
HTML IMG tags pointing at a victim's web server, which I believe has
no protection or pacing?  Has such an HTML IMG attack been a real
problem, or do we consider it will be a problem?

-d

> >>>> From: [email-masked] [mailto:[email-masked]] On
> >>>> Behalf Of Ari Keränen
> >>>> background, see [1] and [2]), as discussed at the Atlanta meeting,
> >>>> will be one of the important changes in the new version of ICE.
> >>>>
> >>>> This results in very slow pace of starting connectivity checks if
> >>>> ICE is used with non-RTP traffic. With RTP, the Ta is calculated
> >>>> based on the expected bandwidth used by the RTP stream and can be
> down to 20 ms.
> >>> That would explain failures claimed to justify the desire for
> >>> running over one port.
> >>
> >>> The 20ms threshold for the initial connectivity checks is because
> >>> some NATs could not create new mappings quickly.  This problems
> >>> appears to persist today, considering that both Chrome and Windows 8
> >>> limit their number of outstanding DNS queries in an apparent attempt
> >>> to work around that very same NAT UDP mapping behavior.
> >>>
> >>>> The quite conservative value of 500 ms was originally chosen
> >>>> because of the fear of congesting the links with the connectivity
> check messages.
> >>>> This is obviously still a valid concern, but would suggesting
> >>>> something lower still make sense? How low would it be reasonably
> >>>> safe to go? What is currently used in practice? Can we come up with
> >>>> a formula similar to RTP (perhaps based on expected bandwidth
> usage)?
> >>>
> >>> A good question for the transport area, which is the origination of
> >>> the suggestion to send ICE connectivity checks paced to not exceed
> >>> the bandwidth of the subsequent media.  Perhaps the arguments that
> >>> TCP

---

From: Matthew Kaufman <[email-masked]>
Date: Thu, 14 Feb 2013 15:46:40 -0800

> problem, or do we consider it will be a problem?

Most browsers do have a limit for the number of simultaneous IMG loads, some even more so if they all appear to be pointed at the same domain.

Matthew Kaufman

---

From: "Dan Wing" <[email-masked]>
Date: Thu, 14 Feb 2013 15:55:00 -0800

> From: Matthew Kaufman [mailto:[email-masked]]
> Sent: Thursday, February 14, 2013 3:47 PM
> To: Dan Wing
> Cc: Magnus Westerlund; Emil Ivov; Ari Keränen; mmusic
> Most browsers do have a limit for the number of simultaneous IMG loads,
> some even more so if they all appear to be pointed at the same domain.
> 
> Matthew Kaufman=

I guess you are referring to the per-server HTTP connection limit, but 
I am guessing on your reference.  If so, perhaps modeling the ICEbis 
limits (for ICE  connectivity checks) on that design is a straw man 
we should consider.  If not, perhaps you could clarify what Most
Browsers do regarding such a crafted web page (which of course need
not be IMG, but any HTML tag that pulls content from another host).

-d

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 15 Feb 2013 11:34:14 -0500 (EST)

> From: Matthew Kaufman <[email-masked]>
> 
> Most browsers do have a limit for the number of simultaneous IMG
> loads, some even more so if they all appear to be pointed at the
> same domain.

IIRC, browsers aren't supposed to have more than two TCP connections
to any one server name.  Though I suppose you can get around that by
creating a zillion A records in your private domain that all point to
the same IP address.

   RFC 2616 (HTTP/1.1) section 8.1.4

   Clients that use persistent connections SHOULD limit the number of
   simultaneous connections that they maintain to a given server. A
   single-user client SHOULD NOT maintain more than 2 connections with
   any server or proxy.

Dale

---

From: Gonzalo Camarillo <[email-masked]>
Date: Fri, 15 Feb 2013 18:47:18 +0200

Hi,

to add to what Magnus said, it is important that we understand what
implementations have been doing so far (regardless of what the RFC says)
and that we come up with recommendations implementers will actually
follow. Keep in mind we are here to write specifications, not fiction ;-)

Thanks,

Gonzalo

---

From: Emil Ivov <[email-masked]>
Date: Fri, 15 Feb 2013 18:16:34 +0100

Well for one, we hadn't even noticed that there was different pacing for
non-RTP traffic, so when we added support for Pseudo TCP to ice4j.org,
we just let it use the default timers. It really didn't cross our
collective minds that the same stack in the same application would have
the right of going down to 20ms if it used the magic word (and RTP
header) but then go back up to 500 ms if it didn't. It's almost as if it
were punished for having bad non-RTP thoughts ;)

Anyways, I think that, had we known about the "MUST NOT drop below 500
ms" limitation, we would have simply ignored it: these are user
perceptible delays after all, and no one would like to wayt 10+ seconds
for a file to start transferring.

Cheers,
Emil

-- 
https://jitsi.org

---

From: "Pal Martinsen (palmarti)" <[email-masked]>
Date: Fri, 15 Feb 2013 18:43:09 +0000

> were punished for having bad non-RTP thoughts ;)
Ha,ha. For us it was opposite. Never noticed the non-RTP stream limitation...

> for a file to start transferring.

On a side note I think simpler is better. Choosing a 20ms timer (or any value) is better than calculating something from what bandwidth you are using. It is hard to say what bandwidth a H.264 encoder is going to use, depends on how much movement in the picture and so on.

.-.
Pål-Erik

---

From: "Dan Wing" <[email-masked]>
Date: Fri, 15 Feb 2013 10:57:36 -0800

> From: Emil Ivov [mailto:[email-masked]]
> Sent: Friday, February 15, 2013 9:17 AM
> To: Gonzalo Camarillo
> Cc: Magnus Westerlund; Ari Keränen; 'mmusic'; Dan Wing

A quick a analysis of what browsers have been doing in this area:  The 
TCP retransmit timer on most OSs is 3 seconds.  But Firefox and Chrome 
have backup connection timers that are more aggressive (250ms and 300ms,
if I recall correctly).  IE does not have a backup connection timer.  So
they are not as aggressive as 20ms (which would be too aggressive on a lot
of networks) but they also do not strictly follow the IETF TCP transport
guidelines to retry a TCP connection after a long 3 seconds.

-d

> > implementations have been doing so far (regardless of what the RFC
> > says) and that we come up with recommendations implementers will
> > actually follow. Keep in mind we are here to write specifications, not
> > fiction ;-)
> >
> > Thanks,
> >
> > Gonzalo
> >
> >
> >> Thus I would be careful with speeding up Ta for RTP to much, and I
> >> think someone needs to fine empiric data that it would work.
> >>
> >> For the the 500 ms that can likely be significantly lowered without
> >> any issues. But some considerations around the wider usage needs to
> >> be taken into account.
> >
> >
> 
> --
> https://jitsi.org

---

From: Martin Thomson <[email-masked]>
Date: Fri, 15 Feb 2013 11:41:34 -0800

> guidelines to retry a TCP connection after a long 3 seconds.

That's good info.  I could only find that for Firefox easily.  Given
that they also ignore the limit Dale pointed out and create 6
connections to any given server, this leads to an aggregate rate of
SYN packets of one every 50ms, though it's probably more bursty than
the even pacing required by RFC 5245.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Fri, 22 Feb 2013 03:25:40 +0000

This is a hard problem to figure out but glad it is being tackled as the current draft was fiction created to get it approved more.

I'd be tempted to think of their being a few different things to conserve or constraints we need to not exceed 

1) nat timing 

2) total ports in usage on the NAT

3) bandwidth usage that is not congestion controlled 

For 1) I'm not seeing any reasons to move off the 20 ms lower bound - but if someone has empirical data that would be nice as the empirical data we are basing this on is old. 

For 2) some Carrier Grade Nats (CGNs) are restricting a given client to something like 1000 ports. There may be other limits too. It might be worth setting up ICE so that a given ICE client never used more than 100 ports at a time. I think this is a new type of constraint that is not a big deal today but may be a bigger deal in the future and we should consider it in an ICE bis. 

For 3) Its very hard to come up with something that is fast enough that the media guys like it and slow enough the transport folks are sure it is safe. My strawman proposal would be use more more bandwidth than G.711 with 20 ms packets as we have been doing that all over the place - without congestion control  - and the internet has not melted down. 

On Feb 14, 2013, at 6:49 AM, Ari Keränen <[email-masked]> wrote:

> Hi all,
> 
> Updating the pacing parameter, Ta, of ICE for non-RTP traffic (for background, see [1] and [2]), as discussed at the Atlanta meeting, will be one of the important changes in the new version of ICE.
> 
> Currently the RFC5245 says (for non-RTP traffic) "Ta SHOULD be configurable, SHOULD have a default of 500 ms, and MUST NOT be configurable to be less than 500 ms."
> 
> This results in very slow pace of starting connectivity checks if ICE is used with non-RTP traffic. With RTP, the Ta is calculated based on the expected bandwidth used by the RTP stream and can be down to 20 ms. Therefore, many current implementations apparently ignore the "MUST NOT" above and set Ta to something lower.
> 
> The quite conservative value of 500 ms was originally chosen because of the fear of congesting the links with the connectivity check messages. This is obviously still a valid concern, but would suggesting something lower still make sense? How low would it be reasonably safe to go? What is currently used in practice? Can we come up with a formula similar to RTP (perhaps based on expected bandwidth usage)?
> 
> 
> Cheers,
> Ari
>