Subject: [AVTCORE] Finishing up framemarking....

From: Bernard Aboba <[email-masked]>
Date: Sun, 11 Jul 2021 08:04:36 -0700

On Thursday, March 11, 2021, Mo posted a question relating to VP9:
https://mailarchive.ietf.org/arch/msg/avt/JA4b0XAllPtrSiTRxr8JIeGWR7E/

I believe this is the only remaining issue on the framemarking document.
Since framemarking is holding up publication of the VP9 RTP payload as an
RFC, it would be good to move this toward resolution at IETF 111 (or
ideally, before).

--------------

VP9 experts,

While updating frame marking to add back VP9, I had difficulty finding
an indicator for discardable non-reference frames in the VP9 RTP
payload draft to map to the frame marking “D” bit. VP8 has the “N” bit
for this in the first payload byte (payload descriptor). But VP9 lacks
anything similar, although section 8 thinks this should be in section
4.2 (but there is nothing there or elsewhere).
https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-8
   ...Media aware network elements MAY use the information in the VP9
payload descriptor in Section
4.2<https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2>
<http://4.2%3Chttps//tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2%3E>
to identify non-reference frames and discard them in order to reduce
network congestion.  Note that discarding of non-reference frames
cannot be done if the stream is encrypted (because the non-reference
marker is encrypted).

Is this an oversight? Or was the non-ref indicator intentionally
removed? If so, why?

>From my work on AV1, I know how the 8 reference frames work in
AV1/VP9, so I devised a solution for frame marking. But it seems like
a hack to require this knowledge and go deep into the bitstream
uncompressed header (not RTP payload descriptor) to extract this
information. Is this really what we must do for VP9? Here is what I
put in frame marking version -12, which seems like a hack:
https://tools.ietf.org/html/draft-ietf-avtext-framemarking-12#section-3.3.1
   The D bit MUST be 1 if the refresh_frame_flags in the VP9 payload
uncompressed header are all 0, otherwise it MUST be 0.

Reading the refresh_frame_flags from the payload requires some payload
parsing and checks since it’s not a fixed offset, and requires knowing
subtle nuances like key frames implicitly force all 8 flags to 1. If
implementers have been confused by something as simple as TL0PICIDX++,
they will likely stumble on this as well.

Is it possible to revive the VP8 N bit? This would simplify frame
marking, AV1 DD, GCC, L2/L3 QoS mappings, or anything else that wants
to know about discardable data.

Regards,
Mo

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 11 Jul 2021 08:16:27 -0700

An update. The VP9 RTP payload no longer references framemarking, so
framemarking is not holding up publication (LRR is the MISREF).

The text that Mo references is now in Section 9 of the VP9 spec (
https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section9)

9 <https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-9>.
Congestion Control

   Congestion control for RTP SHALL be used in accordance with RFC
3550 <https://datatracker.ietf.org/doc/html/rfc3550>
   [RFC3550 <https://datatracker.ietf.org/doc/html/rfc3550>], and with
any applicable RTP profile; e.g., RFC 3551
<https://datatracker.ietf.org/doc/html/rfc3551>
   [RFC3551 <https://datatracker.ietf.org/doc/html/rfc3551>].  The
congestion control mechanism can, in a real-time
   encoding scenario, adapt the transmission rate by instructing the
   encoder to encode at a certain target rate.  Media aware network
   elements MAY use the information in the VP9 payload descriptor in
   Section 4.2 <https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-4.2>
to identify non-reference frames and discard them in
   order to reduce network congestion.  Note that discarding of non-
   reference frames cannot be done if the stream is encrypted (because
   the non-reference marker is encrypted).

Section 4.2 now includes definition of an N bit:

   Reference indices:  When P and F are both set to one, indicating a
      non-key frame in flexible mode, then at least one reference index
      MUST be specified as below.  Additional reference indices (total
      of up to 3 reference indices are allowed) may be specified using
      the N bit below.  When either P or F is set to zero, then no
      reference index is specified.

      P_DIFF:  The reference index (in 7 bits) specified as the relative
         PID from the current picture.  For example, when P_DIFF=3 on a
         packet containing the picture with PID 112 means that the
         picture refers back to the picture with PID 109.  This
         calculation is done modulo the size of the PID field, i.e.,
         either 7 or 15 bits.  A P_DIFF value of 0 is invalid.

      N:  1 if there is additional P_DIFF following the current P_DIFF.

On Sun, Jul 11, 2021 at 8:04 AM Bernard Aboba <[email-masked]>
wrote:

> While updating frame marking to add back VP9, I had difficulty finding an indicator for discardable non-reference frames in the VP9 RTP payload draft to map to the frame marking “D” bit. VP8 has the “N” bit for this in the first payload byte (payload descriptor). But VP9 lacks anything similar, although section 8 thinks this should be in section 4.2 (but there is nothing there or elsewhere).
>    ...Media aware network elements MAY use the information in the VP9 payload descriptor in Section 4.2<https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2> <http://4.2%3Chttps//tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2%3E> to identify non-reference frames and discard them in order to reduce network congestion.  Note that discarding of non-reference frames cannot be done if the stream is encrypted (because the non-reference marker is encrypted).
>
> Is this an oversight? Or was the non-ref indicator intentionally removed? If so, why?
>
> From my work on AV1, I know how the 8 reference frames work in AV1/VP9, so I devised a solution for frame marking. But it seems like a hack to require this knowledge and go deep into the bitstream uncompressed header (not RTP payload descriptor) to extract this information. Is this really what we must do for VP9? Here is what I put in frame marking version -12, which seems like a hack:
>    The D bit MUST be 1 if the refresh_frame_flags in the VP9 payload uncompressed header are all 0, otherwise it MUST be 0.
>
> Reading the refresh_frame_flags from the payload requires some payload parsing and checks since it’s not a fixed offset, and requires knowing subtle nuances like key frames implicitly force all 8 flags to 1. If implementers have been confused by something as simple as TL0PICIDX++, they will likely stumble on this as well.
>
> Is it possible to revive the VP8 N bit? This would simplify frame marking, AV1 DD, GCC, L2/L3 QoS mappings, or anything else that wants to know about discardable data.
>
> Regards,
> Mo
>
>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 12 Jul 2021 04:52:33 +0000

Thanks, Bernard, for reiterating this VP9 issue so we can resolve it.

Note the latest VP9 draft N bit is unrelated to the VP8 N bit or the Frame Marking D (Discardable) bit. The VP9 N bit signals which other frames this frame refers to, not whether or not any future frames will refer to this frame. It does NOT signal if the current frame is a discardable (non-reference) frame or a non-discardable (reference) frame, as in VP8 and Frame Marking.

VP9 currently requires using the refresh_frame_flags in the VP9 payload uncompressed header to determine if a frame is a discardable non-reference frame (when all bits are zero). The Frame Marking draft describes this. If that is acceptable to all, we can resolve both VP9 and Frame Marking as-is. If VP9 users want an easier way to determine if a frame is discardable without parsing everything before the refresh_frame_flags, that would need a substantive change in the VP9 draft, which would then be referenced in the Frame Marking draft.

Regards,
Mo

From: avt <[email-masked]> on behalf of Bernard Aboba <[email-masked]>
Date: Sunday, July 11, 2021 at 11:17 AM
To: "[email-masked]" <[email-masked]>
Cc: Justin Uberti <[email-masked]>, Jonathan Lennox <[email-masked]>
Subject: Re: [AVTCORE] Finishing up framemarking....

An update. The VP9 RTP payload no longer references framemarking, so framemarking is not holding up publication (LRR is the MISREF).

The text that Mo references is now in Section 9 of the VP9 spec (https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section9)

9<https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-9>.  Congestion Control

   Congestion control for RTP SHALL be used in accordance with RFC 3550<https://datatracker.ietf.org/doc/html/rfc3550>

   [RFC3550<https://datatracker.ietf.org/doc/html/rfc3550>], and with any applicable RTP profile; e.g., RFC 3551<https://datatracker.ietf.org/doc/html/rfc3551>

   [RFC3551<https://datatracker.ietf.org/doc/html/rfc3551>].  The congestion control mechanism can, in a real-time

   Section 4.2<https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-4.2> to identify non-reference frames and discard them in

On Sun, Jul 11, 2021 at 8:04 AM Bernard Aboba <[email-masked]<mailto:[email-masked]>> wrote:
On Thursday, March 11, 2021, Mo posted a question relating to VP9:
https://mailarchive.ietf.org/arch/msg/avt/JA4b0XAllPtrSiTRxr8JIeGWR7E/

I believe this is the only remaining issue on the framemarking document. Since framemarking is holding up publication of the VP9 RTP payload as an RFC, it would be good to move this toward resolution at IETF 111 (or ideally, before).

--------------

VP9 experts,

   ...Media aware network elements MAY use the information in the VP9 payload descriptor in Section 4.2<https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2><http://4.2%3Chttps/tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2%3E> to identify non-reference frames and discard them in order to reduce network congestion.  Note that discarding of non-reference frames cannot be done if the stream is encrypted (because the non-reference marker is encrypted).

Regards,

Mo

---

From: Jonathan Lennox <[email-masked]>
Date: Mon, 12 Jul 2021 11:17:52 -0400

I agree it’s unfortunate that it’s not directly possible to determine discardability from the RTP payload header, but as you say, a marker can parse the uncompressed VP9 header.  (It’s a fairly straightforward structure to parse, at least as video codecs go.)

Also, I’d imagine that in many cases the frame marking header is being generated by the original generator of the VP9 bitstream; in this case, if it knows the encoding structure in use, it can know what’s discardable that way.

In the worst case, it’s always safe to conservatively mark nothing at all as discardable, of course.

I don’t think doing a substantive change to VP9 would be a good idea.

> On Jul 12, 2021, at 12:52 AM, Mo Zanaty (mzanaty) <[email-masked]> wrote:
> 
> From: avt <[email-masked] <mailto:[email-masked]>> on behalf of Bernard Aboba <[email-masked] <mailto:[email-masked]>>
> To: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
> Cc: Justin Uberti <[email-masked] <mailto:[email-masked]>>, Jonathan Lennox <[email-masked] <mailto:[email-masked]>>
> The text that Mo references is now in Section 9 of the VP9 spec (https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section9 <https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section9>)
>  
> 9 <https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-9>.  Congestion Control
>  
>    Congestion control for RTP SHALL be used in accordance with RFC 3550 <https://datatracker.ietf.org/doc/html/rfc3550>
>    [RFC3550 <https://datatracker.ietf.org/doc/html/rfc3550>], and with any applicable RTP profile; e.g., RFC 3551 <https://datatracker.ietf.org/doc/html/rfc3551>
>    [RFC3551 <https://datatracker.ietf.org/doc/html/rfc3551>].  The congestion control mechanism can, in a real-time
>    Section 4.2 <https://datatracker.ietf.org/doc/html/draft-ietf-payload-vp9-16#section-4.2> to identify non-reference frames and discard them in
> On Sun, Jul 11, 2021 at 8:04 AM Bernard Aboba <[email-masked] <mailto:[email-masked]>> wrote:
>> https://mailarchive.ietf.org/arch/msg/avt/JA4b0XAllPtrSiTRxr8JIeGWR7E/ <https://mailarchive.ietf.org/arch/msg/avt/JA4b0XAllPtrSiTRxr8JIeGWR7E/>
>>  
>> https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-8 <https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-8>
>>    ...Media aware network elements MAY use the information in the VP9 payload descriptor in Section 4.2<https://tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2> <http://4.2%3Chttps/tools.ietf.org/html/draft-ietf-payload-vp9-11#section-4.2%3E> to identify non-reference frames and discard them in order to reduce network congestion.  Note that discarding of non-reference frames cannot be done if the stream is encrypted (because the non-reference marker is encrypted).
>>  
>> https://tools.ietf.org/html/draft-ietf-avtext-framemarking-12#section-3.3.1 <https://tools.ietf.org/html/draft-ietf-avtext-framemarking-12#section-3.3.1>

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Mon, 12 Jul 2021 15:40:26 +0000

Unless anyone objects to the current text in the VP9 draft or Frame Marking draft, then we can consider this issue resolved for both drafts.

Regards,
Mo

From: Jonathan Lennox <[email-masked]>
Date: Monday, July 12, 2021 at 11:18 AM
To: "Mo Zanaty (mzanaty)" <[email-masked]>
Cc: Bernard Aboba <[email-masked]>, "[email-masked]" <[email-masked]>, Justin Uberti <[email-masked]>
Subject: Re: [AVTCORE] Finishing up framemarking....

On Jul 12, 2021, at 12:52 AM, Mo Zanaty (mzanaty) <[email-masked]<mailto:[email-masked]>> wrote:

Regards,
Mo

From: avt <[email-masked]<mailto:[email-masked]>> on behalf of Bernard Aboba <[email-masked]<mailto:[email-masked]>>
Date: Sunday, July 11, 2021 at 11:17 AM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Cc: Justin Uberti <[email-masked]<mailto:[email-masked]>>, Jonathan Lennox <[email-masked]<mailto:[email-masked]>>
Subject: Re: [AVTCORE] Finishing up framemarking....

--------------

VP9 experts,

Regards,

Mo