Subject: [Covidimpacts-workshop] Forward Error Correction Qs

From: "Livingood, Jason" <[email-masked]>
Date: Thu, 12 Nov 2020 16:37:57 +0000

In Wednesday’s session I was intrigued by Cullen’s comment about forward error correction (FEC) potentially leading to audio bitrates for conferencing exceeding that of video. I was curious if that potentially pertained to add A/V codecs or just one specific codec? And can this be quantified?

Lastly, if this is indeed as critical an issue as it seemed, how does the feedback loop to the codec developers work so that there could be a software fix leading to a real world improvement?

Thanks!
Jason

---

From: [email-masked]
Date: Thu, 12 Nov 2020 12:14:32 -0500

I  was also interested in this. Here is another angle. Remote broadcasters (who used to have a truck with a microwave dish to get their signal back to the studio) are now all using production quality codecs over the Internet, including cellular links. They expect flawless audio. As an example of what they use, look at Comrex products. Comrex uses AAC coding, and stresses that it can work over low bandwidth links.

At several thousand dollars a pop, I am not going to play with one of them. Here is one product description with discussion of latency and bandwidth. I think this is state of the art in quality audio. It does have noticeable latency (you can see that when a studio cuts over to a remote.)

Yes, we may be off topic. It is so easy to do...

https://www.bswusa.com/Codecs-Comrex-BRIC-Link-II-P11106.aspx

David Clark
On Nov 12, 2020, 11:38 AM -0500, Livingood, Jason <Jason_Livingood=[email-masked]>, wrote:
> --
> Covidimpacts-workshop mailing list
> [email-masked]
> https://www.iab.org/mailman/listinfo/covidimpacts-workshop

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 12 Nov 2020 20:55:52 -0700

To get a bit more specific about the webex FEC bandwidth …

I’ll talk about webex here but the the situation is similar for the other major web conferencing vendors. Note there is a lots of fine print to this… The amount of FEC bits that are  allocated is regularly changed and tuned (all the major vendors do this) and varies by things like geo, customer type, how much coffee the developers got, which version of of the product, A/B testing, and so on. What I am describing here would be the case for a significant portion of webex traffic in later summer / early fall time ranges. 

The audio codec is opus and the video codec is H.264 or slight variants of that. There are some special case use of AV1 but I'm talking just about the normal cases that are on H.264. This is for the audio / video flowing to and from a webex users desktop or mobile client. WebRTC in browsers is a different story. I’m including retransmission technique in this “FEC” bandwidth even though some people might argue it is not strictly FEC. 

The combined bandwidth for the audio + the FEC for audio is greater that the bandwidth for the video + the FEC for the video. 

Yep, I know that is not what most people would expect but it is where things ended up for some good reason, and probably some bad reasons too. The media is typically sent over SRTP which makes it pretty easy to verify theses numbers in wireshark. I don’t plan to do any tutorials on how to reverse engineer webex :-) 

The amount of audio FEC was not this high before COVID-19. This was added to deal with deteriorating network conditions relative that webex was operating in. I expect the amount of FEC to go down over time thought it is hard to guess what might happen in the short term with currently rising COVID-19 case numbers in many countries. 

I got asked about if a conference had 8 people and 5 of them had video turned off, was this a total statistic that added all the people with and without video together or something. It is not, it is about the bandwidth going from a conference to a single user that has both audio and video turned on. 

> On Nov 12, 2020, at 9:37 AM, Livingood, Jason <Jason_Livingood=[email-masked]> wrote:
> 
> [email-masked] <mailto:[email-masked]>
> https://www.iab.org/mailman/listinfo/covidimpacts-workshop <https://www.iab.org/mailman/listinfo/covidimpacts-workshop>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 13 Nov 2020 12:25:17 +0100

So you say the reaction to too few bandwidth was sending more data...?

Is the FEC fixed or is that somehow adopted dynamically?

> On 13. Nov 2020, at 04:55, Cullen Jennings <[email-masked]> wrote:
> 
>

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 13 Nov 2020 05:59:10 -0700

> On Nov 13, 2020, at 4:25 AM, Mirja Kuehlewind <[email-masked]> wrote:
> 

No, it’s not the case where as the packet loss goes up, the endpoints start sending more data. (that gets proposed on a regular basis but someone eventually points out why that is not such a hot idea)

At a high level the conceptual works along theses lines: 

1. The congestion control algorithm, or something else, provides an estimate of the bandwidth 

2. A rate controller decides what percentage of the bandwidth to allocate to each category of data that need to be sent: (audio, video, audio FEC, video FEC, feedback metrics, control). 

3. Given the bandwidth available for video, the rate controller will pick the resolution / frame rate to use for video 

4. Both opus and H.264 can dramatically adjust the bitrate they are sending at to match what the rate controller gives them as a target. 

That is a very gross over simplification of what any of the major systems do. What they actually do is a much more complicated but fits into that overall model. This area has a ton of IPR as well as trade secrets so it is not something that going to get discussed much on an open list. If you want to dig into details, I’m happy to do that outside an IETF context. The open source wbeRTC media engine is also a great example of what this looks like for people that want to dig deeper. 

One of the things that complicates the above is you have control loops running at very different times lines. The control loop changing the bitrate from the video decoder can not react to changing bandwidth as fast as the congestion control typically reacts. There are some great publications on how to deal with this at the USPTO. 

I also want to be a bit more nuanced about the quality of bandwidth. People tend to focus on how many bits per second users can get. Sure that matters but it is only one factor. If I had a 1.5 mbps connection that once a minute dropped all my packets for 1 second, it would be nearly impossible to have a good video call across that short of AI that predicts what I will say next. If I had the same connection and it lost twice as many packets as the first, but the losses were randomly distributed, that would deliver an awesome video conference without doing anything clever. Obviously jitter and latency are also important. The fact that we buy internet connections largely differentiated only by bandwidth in the advertising might be part of the focus on this.  I look forward 

The general quality of the internet that webex calls were running over was worse in March 2020 than it was in January 2020. VoIP apps have always had to deal with come users were on connections that were fast and some where slow - that was not new with covid-19. However the changes in other quality aspects of the internet were part of what caused significant shifts in how that bandwidth was allocated in step 2.

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 13 Nov 2020 14:41:19 +0100

So you are basically say that when the problem was detect and FEC adjusted, the intention was to use potentially the same amount of bandwidth but spend more of that for FEC and thus reduce quality/bitrate of the video instead. Do you have any measurement data on that?

> On 13. Nov 2020, at 13:59, Cullen Jennings <[email-masked]> wrote:
> 
> 
>

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 13 Nov 2020 11:06:56 -0700

> On Nov 13, 2020, at 6:41 AM, Mirja Kuehlewind <[email-masked]> wrote:
> 
> So you are basically say that when the problem was detect and FEC adjusted, the intention was to use potentially the same amount of bandwidth but spend more of that for FEC and thus reduce quality/bitrate of the video instead.

Yes - exactly. 

There is also a max bandwidth the app will use if there there was no network bandwidth limits. That limit has moved up over time too but that is a separate things. 

> Do you have any measurement data on that?
> 
>