Subject: [AVTCORE] Some (SFrame RTP encapsulation) questions to ponder

From: Bernard Aboba <[email-masked]>
Date: Wed, 10 Mar 2021 22:21:43 -0800

During the SFrame meeting earlier in the week, Colin Perkins noted the
potential for the E2E encryption problem to be considerably more complex
than might be appreciated today.  Also, Stefan Wenger noted the difficulty
in devising solutions that would work for all past, present and future
codecs.

This brought to mind another problem that turned out to be more
complex than we initially thought.... NAT traversal. We eventually realized
that NAT behavior was more subtle than we originally thought, and that we
needed revisions to STUN/TURN as well as the ICE framework.

So before setting out to climb the mountain in sneakers and a thin
windbreaker, it might be useful to get an idea of whether we are setting
out for a nearby hill with an elevation of 1000 feet, or whether we intend
to climb Mt. Everest.

In looking at the "generic packetization" slides, this brought a question
to mind:

Does a solution to "generic packetization" require standardization of
metadata and forwarding solutions that work for every codec without parsing
the payload?

Or can we settle for something less, such as specifying the E2E
packetization format on a per-codec basis, with some of the codecs
requiring payload parsing?

I ask this because there appear to be codecs that are widely deployed (e.g.
VP8) that require "hacks" for SFUs to forward, as noted in the presentation
at IETF 109.  In the case of VP8, the "hack" was required because of a
deficiency in the VP8 decoder specification (e.g. the requirement that
PictureIDs be consecutive, not just monotonically increasing).  This means
that the "hacks" described can't be eliminated just by coming up with a
better RTP forwarding extension.  You'd actually need to change the VP8
specification and upgrade VP8 decoders.

That seems like a pretty big hill to climb.  Do we really need to climb it,
or can we just walk around it?

Similarly, do we need to come up with a single set of metadata and a single
RTP header extension that will work with all codecs?

Or can we just continue to experiment and document solutions (and
non-solutions) on a per-codec basis, and see what we learn from that?

As an example, at the interim it was noted that framemarking was
implemented along with H.264/AVC with temporal scalability, and that it
worked satisfactorily.  So if you wanted to do SFrame RTP encapsulation of
H.264/AVC with temporal scalability, could you negotiate framemarking and
just use that?

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Thu, 11 Mar 2021 09:14:56 +0100

El jue, 11 mar 2021 a las 7:22, Bernard Aboba (<[email-masked]>)
escribió:

>
> I ask this because there appear to be codecs that are widely deployed
> (e.g. VP8) that require "hacks" for SFUs to forward, as noted in the
> presentation at IETF 109.  In the case of VP8, the "hack" was required
> because of a deficiency in the VP8 decoder specification (e.g. the
> requirement that PictureIDs be consecutive, not just monotonically
> increasing).  This means that the "hacks" described can't be eliminated
> just by coming up with a better RTP forwarding extension.  You'd actually
> need to change the VP8 specification and upgrade VP8 decoders.
>
>
The PictureID and the TL0PICIDX  are defined in rfc7741 RTP Payload Format
for VP8 Video and not on the VP8 spec. So, if using an agnostic packetizer
instead of the rfc7741 one, the issue just doesn't exist.

Best regards
Sergio

---

From: Alexandre GOUAILLARD <[email-masked]>
Date: Thu, 11 Mar 2021 15:46:28 +0700

Sergio,

I think you had done this analysis for h264, vp8 and vp9 two years ago. There are some existing  slides about it.

If Apple/Intel add a slide for h265 since they support it in safari, we add av1, Stefan bring in h266, we have more or less most of the codecs of interest and enough data to judge.

We could have three sections:
1 - what do you need to use to go into a generic packetizer and how much is codec specific. 
2 - what do You need to duplicate in an rtp header extension for an SFU to work.
I see three different things:
- base for encrypted non-SVC payload (frame marking)
- scalability structure (payload encrypted or not).
a/ I have not had the time to look at Stefan a proposal for the new MPEG codecs but his messages on the list were going in the same direction (higher complexity with full SVC than with temporal scalability only). 
b/ I do not know how far appart the AV1 structure and the vvc or ecc would be, if there is a way to make it generic, and if it a even a goal we would want to have.
c/ I understand there was originally a framemarking part to the vvc rtp payload proposal, I m wondering if there was a gap when trying to represent all SVC structures the vvc bistream otherwise provides.
- sfu optimisation structures, eg for av1 the Instant decidability decision structure (payload encrypted or not), whic is not critical for sfu operation but allow much faster adaptation and smarter CC.

If we go down that path that could mean a codec specific version of each of:
- Payload format
- Sframe rtp packetizer
- rtp extension header

With the following triplet already more or less working well together already:
[h264, generic, framemarking]
[VP8, generic, framemarking]
[VP9, generic, DD]
[AV1, generic, DD]

Now in terms of security for the encrypted case, that means the presence or absence of a given rtp header extension leaks info about e.g which codec is being used. I m not sure this is something w can completely avoid anyway. 

Sent from my iPhone

> On 11 Mar 2021, at 15:15, Sergio Garcia Murillo <[email-masked]> wrote:
> 
> 
> 
> 
>> El jue, 11 mar 2021 a las 7:22, Bernard Aboba (<[email-masked]>) escribió:
>> 
>> I ask this because there appear to be codecs that are widely deployed (e.g. VP8) that require "hacks" for SFUs to forward, as noted in the presentation at IETF 109.  In the case of VP8, the "hack" was required because of a deficiency in the VP8 decoder specification (e.g. the requirement that PictureIDs be consecutive, not just monotonically increasing).  This means that the "hacks" described can't be eliminated just by coming up with a better RTP forwarding extension.  You'd actually need to change the VP8 specification and upgrade VP8 decoders. 
>> 
> 
> The PictureID and the TL0PICIDX  are defined in rfc7741 RTP Payload Format for VP8 Video and not on the VP8 spec. So, if using an agnostic packetizer instead of the rfc7741 one, the issue just doesn't exist.
> 
> Best regards
> Sergio
> _______________________________________________
> Audio/Video Transport Core Maintenance
> [email-masked]
> https://www.ietf.org/mailman/listinfo/avt

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 11 Mar 2021 09:22:43 +0000

While updating frame marking to add back VP9, I checked the actual specs and discovered that VP9 and VP8 have identical language on TL0PICIDX, and both merely say it “MUST be incremented” but not necessarily by 1. Any implementation that requires +1 seems incorrect unless I’m missing something. There is an additional informative note in VP8: “This will likely require rewriting values of TL0PICIDX and KEYIDX after the splice.” This also does not require +1, and also seems to apply equally to VP9 for TL0PICIDX, and was likely removed from VP9 because it lacks KEYIDX, not because it has any different semantics for TL0PICIDX.

VP9: https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2
      TL0PICIDX:  8 bits temporal layer zero index.  TL0PICIDX is only
         present in the non-flexible mode (F = 0).  This is a running
         index for the temporal base layer pictures, i.e., the pictures
         with TID set to 0.  If TID is larger than 0, TL0PICIDX
         indicates which temporal base layer picture the current picture
         depends on.  TL0PICIDX MUST be incremented when TID is equal to
         0.  The index SHOULD start on a random number, and MUST restart
         at 0 after reaching the maximum number 255.

VP8: https://tools.ietf.org/html/rfc7741#section-4.2
      TL0PICIDX:  8 bits temporal level zero index.  TL0PICIDX is a
         running index for the temporal base layer frames, i.e., the
         frames with TID set to 0.  If TID is larger than 0, TL0PICIDX
         indicates on which base-layer frame the current image depends.
         TL0PICIDX MUST be incremented when TID is 0.  The index MAY
         start at a random value, and it MUST wrap to 0 after reaching
         the maximum number 255.  Use of TL0PICIDX depends on the
         presence of TID.  Therefore, it is RECOMMENDED that the TID be
         used whenever TL0PICIDX is.
…snip…
   Informative note:  Implementations doing splicing of VP8 streams will
      have to make sure the rules for incrementing TL0PICIDX and KEYIDX
      are obeyed across the splice.  This will likely require rewriting
      values of TL0PICIDX and KEYIDX after the splice.

Regards,
Mo

From: avt <[email-masked]> on behalf of Alexandre GOUAILLARD <[email-masked]>
Date: Thursday, March 11, 2021 at 3:47 AM
To: Sergio Murillo <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [AVTCORE] Some (SFrame RTP encapsulation) questions to ponder

Sergio,

Sent from my iPhone

El jue, 11 mar 2021 a las 7:22, Bernard Aboba (<[email-masked]<mailto:[email-masked]>>) escribió:

Best regards
Sergio