Subject: [AVTCORE] Call for Framemarking implementation experience

From: Bernard Aboba <[email-masked]>
Date: Sun, 22 Nov 2020 07:24:23 -0800

During WGLC, questions were raised about implementation experience with draft-IETF-AVTEXT-framemarking. If you have implemented the specification, please respond to this thread with your observations. 

Some notes relating to WebRTC.org:

Frame marking support (-07) appears to have been first added to the WebRTC.org implementation in 2017:
https://codereview.webrtc.org/2954503002/
https://bugs.chromium.org/p/webrtc/issues/detail?id=7765

The commits refer appears to support temporal scalability with H.264/AVC. 

In 2019 support was added to the H.264 receiver in Chrome M76:
https://groups.google.com/g/discuss-webrtc/c/Y7TIuNbgP8M/m/UoXP-RuxAwAJ

In March 2020, Issue 7765 on support for framemarking was closed as ‘won’t fix’.

A security issue was found in April 8, 2020:
https://bugs.chromium.org/p/chromium/issues/detail?id=1042933

Framemarking support was removed from WebRTC.org in June, 2020 and from Chrome starting in M85:
http://bugs.webrtc.org/11637

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sun, 22 Nov 2020 22:24:22 +0100

I implemented an slightly modified version of an earlier draft of the 
Framemarking on both my SFU and a custom libwebrtc for supporting VP9 
SVC with perc-lite e2ee:

https://codereview.webrtc.org/2954503002/
https://github.com/murillo128/draft-berger-avtext-framemarking/commit/35cdb229108251f12bc1cdf77909df83a51d3994

I don't think we ever got to solve the question about the P/U bits on 
the vp9 payload draft:

https://mailarchive.ietf.org/arch/msg/avtext/ntfDsjMJjuruDPenp6-w0k0__Rk/

I plan to remove the framemarking code soon as it is no longer needed 
for SFrame with standard VP9 packetization and not supported by 
libwebrtc for other codecs neither (h264 with temporal scalability was a 
nice feature though).

Best regards
Sergio

On 22/11/2020 16:24, Bernard Aboba wrote:
> During WGLC, questions were raised about implementation experience 
> with draft-IETF-AVTEXT-framemarking. If you have implemented the 
> specification, please respond to this thread with your observations.
>
> <https://codereview.webrtc.org/2954503002/>
> <https://bugs.chromium.org/p/webrtc/issues/detail?id=7765>
>
> <https://groups.google.com/g/discuss-webrtc/c/Y7TIuNbgP8M/m/UoXP-RuxAwAJ>
>
> <https://bugs.chromium.org/p/chromium/issues/detail?id=1042933>
>
> _______________________________________________
> Audio/Video Transport Core Maintenance
> [email-masked]
> https://www.ietf.org/mailman/listinfo/avt

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 22 Nov 2020 13:54:45 -0800

Sergio --

Did the P/U bit issue in VP9 render framemarking completely unusable with
VP9?  Or were there some set of use cases (e.g. simulcast-only, temporal
scalability only, etc.) in which it still could be useful?  If the P/U
issue had been solved, would the list of VP9 use cases be expanded?

Did you implement support for H.264/AVC with temporal scalability based on
framemarking?  If so, did you consider it usable for that purpose?

Any implementation experience with VP8?

On Sun, Nov 22, 2020 at 1:24 PM Sergio Garcia Murillo <
[email-masked]> wrote:

> Framemarking on both my SFU and a custom libwebrtc for supporting VP9 SVC
> with perc-lite e2ee:
>
> I don't think we ever got to solve the question about the P/U bits on the
> vp9 payload draft:
>
> I plan to remove the framemarking code soon as it is no longer needed for
> SFrame with standard VP9 packetization and not supported by libwebrtc for
> other codecs neither (h264 with temporal scalability was a nice feature
> though).
>
> Best regards
> Sergio
>
>
> During WGLC, questions were raised about implementation experience with
> draft-IETF-AVTEXT-framemarking. If you have implemented the specification,
> please respond to this thread with your observations.
>
> Audio/Video Transport Core [email-masked]://www.ietf.org/mailman/listinfo/avt
>
>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sun, 22 Nov 2020 23:09:55 +0100

Adding Iñaki  to the loop (see below)

On 22/11/2020 22:54, Bernard Aboba wrote:
> Sergio --
>
> Did the P/U bit issue in VP9 render framemarking completely unusable 
> with VP9?  Or were there some set of use cases (e.g. simulcast-only, 
> temporal scalability only, etc.) in which it still could be useful?  
> If the P/U issue had been solved, would the list of VP9 use cases be 
> expanded?

The P/U bits allows the SFU to upswitch to the next temporal/spatial 
layer, so they are required for temporal and spatial scalability, but no 
for simulcast only. Anyway, I implemented that before the K-SVC modes 
where available and didn't implement the flexible modes neither, so I am 
afraid that even including those bits would not be enough to support all 
SVC modes but just the simplest ones.

> Did you implement support for H.264/AVC with temporal scalability 
> based on framemarking?  If so, did you consider it usable for that 
> purpose?

Framemarking could be interesting for adding temporal scalability with 
h264 nicely without having to fully support h264-svc. I implemented in 
on my SFU but never fully tested it because of the libwebrtc issues at 
the time:

https://github.com/medooze/media-server/blob/master/src/h264/H264LayerSelector.cpp#L40

I think Iñaki  also implemented it for mediasoup, so he might have some 
extra feedback:

https://github.com/versatica/mediasoup/issues/306

> Any implementation experience with VP8?

I have the code in my SFU as well, but it is not usable as libwebrtc 
requires to rewrite the picture ids on the vp8 payload in order for the 
temporal scalability to work.

>
> On Sun, Nov 22, 2020 at 1:24 PM Sergio Garcia Murillo 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     I implemented an slightly modified version of an earlier draft of
>     the Framemarking on both my SFU and a custom libwebrtc for
>     supporting VP9 SVC with perc-lite e2ee:
>
>     <https://github.com/murillo128/draft-berger-avtext-framemarking/commit/35cdb229108251f12bc1cdf77909df83a51d3994>
>
>     I don't think we ever got to solve the question about the P/U bits
>     on the vp9 payload draft:
>
>     <https://mailarchive.ietf.org/arch/msg/avtext/ntfDsjMJjuruDPenp6-w0k0__Rk/>
>
>     I plan to remove the framemarking code soon as it is no longer
>     needed for SFrame with standard VP9 packetization and not
>     supported by libwebrtc for other codecs neither (h264 with
>     temporal scalability was a nice feature though).
>
>     Best regards
>     Sergio
>
>
>>     During WGLC, questions were raised about implementation
>>     experience with draft-IETF-AVTEXT-framemarking. If you have
>>     implemented the specification, please respond to this thread with
>>     your observations.
>>
>>     http://bugs.webrtc.org/11637 <http://bugs.webrtc.org/11637>
>>
>>
>>
>>
>>
>>
>>     [email-masked]  <mailto:[email-masked]>
>>     https://www.ietf.org/mailman/listinfo/avt  <https://www.ietf.org/mailman/listinfo/avt>
>
>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/avt>
>

---

From: Jonathan Lennox <[email-masked]>
Date: Mon, 23 Nov 2020 16:49:55 -0500

(Responding as an individual.)

I implemented frame marking at my previous company (Vidyo), specifically for H.264/AVC temporal scalability.

Frame marking nicely retrofits H.264/AVC to support temporal scalability, in fact better than the H.264/SVC format does (because the H.264/SVC format doesn’t allow you to specify temporal layers on packets containing fragmentation units), and in much more backward-compatible way.  This is why we implemented it at Vidyo, and contributed it to the WebRTC.org <http://webrtc.org/> codebase.

For the reasons that Stephan discussed, however, I never felt that frame marking was particularly suitable for other codecs, where in-payload-header and even in-payload bits need to be inspected and in some cases rewritten. Additionally, frame marking isn’t semantically rich enough (on its own) to express all the information that a fully-featured SFU needs to know about spatially-scalable video, even before newer encoding structures such as K-SVC come into play.

It may be that the AV1 Descriptor Format will be rich enough, though even there I worry about features features of future codecs that aren’t envisioned by AV1.  A notable example would be (as discussed at last week’s meeting) H.266’s gradual decoder refresh (GDR), with its ambiguity of whether it should be considered as satisfying a keyframe request.

> On Nov 22, 2020, at 10:24 AM, Bernard Aboba <[email-masked]> wrote:
> 
> https://codereview.webrtc.org/2954503002/ <https://codereview.webrtc.org/2954503002/>
> https://bugs.chromium.org/p/webrtc/issues/detail?id=7765 <https://bugs.chromium.org/p/webrtc/issues/detail?id=7765>
> 
> https://groups.google.com/g/discuss-webrtc/c/Y7TIuNbgP8M/m/UoXP-RuxAwAJ <https://groups.google.com/g/discuss-webrtc/c/Y7TIuNbgP8M/m/UoXP-RuxAwAJ>
> 
> https://bugs.chromium.org/p/chromium/issues/detail?id=1042933 <https://bugs.chromium.org/p/chromium/issues/detail?id=1042933>
>

---

From: Bernard Aboba <[email-masked]>
Date: Mon, 23 Nov 2020 18:57:02 -0800

On Nov 23, 2020, at 13:49, Jonathan Lennox <[email-masked]> wrote:
> Frame marking nicely retrofits H.264/AVC to support temporal scalability, in fact better than the H.264/SVC format does (because the H.264/SVC format doesn’t allow you to specify temporal layers on packets containing fragmentation units), and in much more backward-compatible way.  This is why we implemented it at Vidyo, and contributed it to the WebRTC.org codebase.

[BA] How many temporal layers did you support? I take it that you had no problem determining upswitch points for all the supported temporal modes?

> It may be that the AV1 Descriptor Format will be rich enough, though even there I worry about features features of future codecs that aren’t envisioned by AV1.  A notable example would be (as discu…

[BA] Jonathan said:
“ where in-payload-header and even in-payload bits need to be inspected and in some cases rewritten”

This sounds like an issue that would be encountered by *any* RTP header extension (e.g.  framemarking, DD, DDv2, etc.). The info could be placed in an RTP header extension so as to allow the SFU to inspect and rewrite it, but then the RTP de-packetizer would need to modify the bitstream before sending it to the decoder to reflect the SFU modified values.  It’s a bit disconcerting to give an untrusted SFU the rights to modify a “protected” bitstream after reception.

---

From: Jonathan Lennox <[email-masked]>
Date: Tue, 24 Nov 2020 11:23:16 -0500

> On Nov 23, 2020, at 9:57 PM, Bernard Aboba <[email-masked]> wrote:
> 

We did three temporal layers.

We assumed temporal nesting, which I believe framemarking requires, so every frame was a valid upswitch point.

> This sounds like an issue that would be encountered by *any* RTP header extension (e.g.  framemarking, DD, DDv2, etc.). The info could be placed in an RTP header extension so as to allow the SFU to…

Yes, agreed.

Part of this would be ameliorated by receivers being more tolerant of SFU operations (for example, today, VP8 receivers need TL0PICIDX to be rewritten when if simulcast layers are spliced); some of it was additional optimizations which could probably safely be skipped.