Subject: [AVTCORE] Flash crowd related bug in AVPF

From: Magnus Westerlund <[email-masked]>
Date: Tue, 27 May 2014 09:43:37 +0200

WG,

In my work to simulate the behaviour of the aggregation scheduling
algorithm in draft-ietf-avtcore-rtp-multi-stream I found another bug in
AVPF.

This bug is quite serious if actually triggered, but is limited to
special deployments and situations. It's effects are also time limited.

I found this issue when performing a simulation with a flash crowd case.
I had 1000 endpoints joining at exactly the same time. I also had a
rather high propagation delay of 0.3 seconds that also was uniform.

So the bug in AVPF is the specification is all depending on what this
overview bullet says:

   d) Let Tmin be the minimal interval between RTCP packets as per [1].
      Unlike in [1], the initial Tmin is set to 1 second to allow for
      some group size sampling before sending the first RTCP packet.
      After the first RTCP packet is sent, Tmin is set to 0.

So what is happening in this scenario is that all the endpoints that had
one SSRC each for reporting on received packet from sender, i.e. a few
to Many distribution case. The SSRC in each of the endpoint is schedule
and due to the input values these 1000 endpoints are randomized over a
time interval that is within the interval [0.205, 0.615] with a
probability distribution that is exponentially growing. If you wonder
why Tmin of 1 second results in the above interval please remember that
this is the initial transmission, thus Tmin/2 is used.

So during the first 0.2 seconds no RTCP from these receivers are sent.
Then as some has drawn random numbers that allows them to go early,
these sends their first RTCP packet. When each of these endpoints has
sent, they set Tmin = 0 and reschedule. As the T-rr-int is 0 in this
case the result is that that the endpoint schedules the next RTCP packet
to utilize the full RTCP Receiver bandwidth (RR). As it doesn't know of
any other receiver SSRCs it will take its fair share, and if no media
source are sending RTP/RTCP either then it will individually consume the
RR by itself.

For the next 0.3 seconds while the RTCP packets propagate over to the
other participants in the session, all that are schedule to send their
first RTCP packet within the interval 0.205 and 0.505 will transmit
their initial packet and then start to consume the configured bandwidth
by themselves. This is approximately, half the sources that manages to
do that. After the 0.3 seconds, the RTCP packets starts arrive and the
group size grows rapidly with everyone, and the reconsiderations work
and everyone backs of to the fair share of RTCP bandwidth.

However, during the time of the propagation delay the RTCP bandwidth
consumptions goes through the roof. How much over use depends on the
propagation delay in relation to this scheduling interval and the
arrival pattern of the flash crowd. In this particular case the consumed
bandwidth just before it slows down is 500 times the configured one.

The circumstances in my simulations are almost worst case, I the
propagation delay would have been 410 ms or longer, all end-points would
have been sending, and the burst is no longer than the propagation
delay. Simultaneous arrival of endpoints and uniform propagation delay
maximizes the effect of this bug.

So a simple mitigation for anyone using AVPF in cases where flash crowd
may occur and having substantial propagation delay for the RTCP feedback
is to set the t-rr-int value to at least half that propagation delay, if
that otherwise works with the RTCP usage. This prevents each endpoint
from sending any more than ~3 RTCP packets during the period when they
are blind to the other participants. This still results in a burst but
significantly limits it and the RTCP bandwidth usage.

The problem with the above mitigation is that it assumes knowledge about
the propagation delay among the RTCP endpoints.

I think it might be time to consider revising AVPF to fix this as well
as the regular report suppression mechanism. And this time I think we
need to require simulation before writing it into the specification.

Until then take note of this issue and consider if you might be affected.

Magnus Westerlund

----------------------------------------------------------------------
Services, Media and Network features, Ericsson Research EAB/TXM
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Färögatan 6                 | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------