Subject: [AVTCORE] VVC payload format, open issue cluster #1 (SDP for scalability)

From: Stephan Wenger <[email-masked]>
Date: Thu, 17 Dec 2020 16:30:54 +0000

Hi,
This is the first in a series of emails in which we authors will try to propose ways to solve open issues indicated in the draft (https://tools.ietf.org/html/draft-ietf-avtcore-rtp-vvc-06).  As we have many of those, we will try to organize those open issues into meaningful clusters get piecemeal agreement on each cluster.  Once received, we will update the draft and go for the next cluster.  Sometimes, we will overlap unrelated clusters.

Many of these emails are going to be long, sorry for that.  Many also require quite detailed knowledge of H.266.  Occasionally, we will include some explanatory language on the H.266 feature in question.  If you want to dive deeper, H.266 is now available for free download from https://www.itu.int/itu-t/recommendations/rec.aspx?rec=14336 –have fun reading the 500 pages of super-dense (and small font) text.

The first cluster concerns section 7.1, Media Type Registration, and therein specifically the codepoints and language related to non-temporal scalability support.  This is a hard problem, quite possibly the hardest we need to address in this exercise.  I apologize in advance for the word count.

As a very brief and oversimplified recap, H.266/VVC supports temporal scalability in all of its profiles except the still image profiles.  In that, H.266 is similar to H.265/HEVC.  However, H.266 also has two scalable profiles (one for 4:2:0 and the another for 4:4:4) that allow
for a powerful spatial/SNR scalability mode.  Many of us believe that these profiles have higher chances of success compared to previous video coding standards, because they are a) not requiring much in terms of decoder implementation complexity beyond non-scalable implementations, and b) are conceptually similar to what VP9 and AV1 use, and hence familiar to those using modern scalable codecs, like the webrtc folks.  Insofar, we want to support scalability in the payload format from the outset.

If only temporal scalability is supported, the selection of an appropriate sub-layer in the onion-shaped multi-sublayered stream by an SDP-based offer-answer is a one-dimensional problem, and has been solved in places like the HEVC payload format by an offer-answer exchange involving a single numeric parameter indicative of the highest supported sub-layer.  However, when having all temporal, spatial, and SNR scalability, the optimal selection of an appropriate output layer set (OLS) becomes a complex multi-dimensional problem, that may have more than one solution for different values of “optimal”, and also requires certain context.  We try to solve this multi-dimensional problem by a combination of a requirement for conditional presence of certain otherwise optional SDP codepoints when scalability is in use, and the introduction of an additional few code points.  Alternatives we rejected include not supporting scalability, or relying exclusively on the exchange of possibly many variants of parameter sets, as that could lead to excessively large SDP blobs (easily larger than the 1500 bytes still common as MTU).  Note that the latter option can still be implemented for cases where our restrictions are considered inappropriate.

The design philosophy is as follows:

  1.  When using scalability and OLS-based negotiation, the sender must include a Video Parameter Set (VPS) as a sprop-vps or have otherwise pre-arranged knowledge that the VPS is known by the receiver.  For those unfamiliar with H.266, the VPS includes a “table of content” of the layering structure, and without it, the values of new codepoints (sprop-ols-id and recv-ols-id) could not be interpreted.
  2.  sprop-ols-id will typically point to the “best” output layer set—the combination of layers providing the most pleasing reconstructed result—that the sender can support taking into account all other parameters known at this point of the O/A process.
  3.  The value of sprop-ols-id selects one of possibly several OLSs branches in the VPS.  In the reply, using recv-ols-id, the answerer can select that OLS or other OLSs from the same branch of scalable layers in the VPS to which the sprop-ols-id belongs, but not any other OLSs that may exist in the VPS.  The worst-case fallback will be base-layer only, which is always supported, assuming H.266 and the relevant profiles tiers, and levels are supported.
  4.  Sublayers may be negotiated as in the HEVC payload format and in parallel.
  5.  The interaction between what’s going on for the (temporal) sub-layers and for the (spatial/SNR) layers is an application logic problem and does not need to be specified, nor documented.  It’s very hard to grok for anyone not intimately familiar with H.266.
  6.  With these design principles, sprop-opi is not needed anymore and will be removed.

Based on this design, the following changes can be made in the draft:
-Remove edt note #5, and remove the mentioning of sprop-opi in the sentence above note 5.
-Remove edt note #6, and remove the mentioning of sprop-opi in the sentence above note 6.
-sprop-ols-id: add a sentence “if this optional parameter is present, sprop-vps must also be present or its content must be known a priori at the receiver”
-sprop-recv-id: add “sprop-recv-id, when present, must be included only when sprop-ols-id was received and must refer to an output layer set in the VPS that is in the same dependency tree as the OLS referred to by sprop-ols-id.  If this optional parameter is present, sprop-vps must have been received or its content must be known a priori at the receiver”
-Remove the sprop-opi language and edt note #17

Please provide comments to this design choice by Jan 6th.  Absent comments, we will implement the changes as described above, also any changes required for consistency, and we will add corresponding language in the currently empty offer-answer section of the draft.

Thanks,
Stephan

---

From: "Ye-Kui Wang" <[email-masked]>
Date: Thu, 17 Dec 2020 09:49:11 -0800

Thanks Stephan for the long email (which is sort of usual for you anyway 😊).

The suggestion looks good to me.

BR, YK

From: avt <[email-masked]> On Behalf Of Stephan Wenger
Sent: Thursday, December 17, 2020 8:31
To: [email-masked]
Subject: [External] [AVTCORE] VVC payload format, open issue cluster #1 (SDP for scalability)

Hi,

Thanks,

Stephan

---

From: "Sanchez de la Fuente, Yago" <[email-masked]>
Date: Fri, 18 Dec 2020 08:48:33 +0000

Thanks Stephan,

Your suggestion looks good to me as well.

Best regards,
Yago Sánchez

---
Department Video Coding & Analytics
Group Multimedia Communications

Fraunhofer HHI - Heinrich Hertz Institute
Einsteinufer 37, 10587 Berlin, Germany
http://www.hhi.fraunhofer.de/ip/mc

Tel.: +49 30 310 02663
[email-masked]<mailto:[email-masked]>

On 17. Dec 2020, at 18:49, Ye-Kui Wang <[email-masked]<mailto:[email-masked]>> wrote:

BR, YK

From: avt <[email-masked]<mailto:[email-masked]>> On Behalf Of Stephan Wenger
Sent: Thursday, December 17, 2020 8:31
To: [email-masked]<mailto:[email-masked]>
Subject: [External] [AVTCORE] VVC payload format, open issue cluster #1 (SDP for scalability)

Hi,

Thanks,
Stephan

_______________________________________________
Audio/Video Transport Core Maintenance
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/avt

---

From: Alexandre GOUAILLARD <[email-masked]>
Date: Fri, 18 Dec 2020 19:36:05 +0800

Hi stefan,

Some of the AVTCORE members who have worked on the design of RTP
payload for AV1, VP9 and other SVC / Simulcast features in webrtc, included
but not limited to SDP O/A, RID and others, are on vacation until january
7th.

I think your proposal would benefit a lot from their spending time and
providing feedback from an informed position, and I would humbly request
that you push back your deadline for them to participate in the discussion.
I do believe that the discussion can go for as long as the chairs want, and
there is usually no deadline per say anyway, but it's alway nicer to ask.

Also, I believe that anything related to SDP O/A and its modifications
needs to be proposed to the MMUSIC group.

Regards.

On Fri, Dec 18, 2020 at 4:48 PM Sanchez de la Fuente, Yago <
[email-masked]> wrote:

> [email-masked]
>
>
>
> On 17. Dec 2020, at 18:49, Ye-Kui Wang <[email-masked]> wrote:
>
> *From:* avt <[email-masked]> *On Behalf Of *Stephan Wenger
> *Sent:* Thursday, December 17, 2020 8:31
> *To:* [email-masked]
> *Subject:* [External] [AVTCORE] VVC payload format, open issue cluster #1
> (SDP for scalability)
>
> Hi,
> This is the first in a series of emails in which we authors will try to
> propose ways to solve open issues indicated in the draft (
> https://tools.ietf.org/html/draft-ietf-avtcore-rtp-vvc-06).  As we have
> many of those, we will try to organize those open issues into meaningful
> clusters get piecemeal agreement on each cluster.  Once received, we will
> update the draft and go for the next cluster.  Sometimes, we will overlap
> unrelated clusters.
>
> Many of these emails are going to be long, sorry for that.  Many also
> require quite detailed knowledge of H.266.  Occasionally, we will include
> some explanatory language on the H.266 feature in question.  If you want to
> dive deeper, H.266 is now available for free download from
> https://www.itu.int/itu-t/recommendations/rec.aspx?rec=14336 –have fun
> reading the 500 pages of super-dense (and small font) text.
>
> The first cluster concerns section 7.1, Media Type Registration, and
> therein specifically the codepoints and language related to non-temporal
> scalability support.  This is a hard problem, quite possibly the hardest we
> need to address in this exercise.  I apologize in advance for the word
> count.
>
> As a very brief and oversimplified recap, H.266/VVC supports temporal
> scalability in all of its profiles except the still image profiles.  In
> that, H.266 is similar to H.265/HEVC.  However, H.266 also has two scalable
> profiles (one for 4:2:0 and the another for 4:4:4) that allow
> for a powerful spatial/SNR scalability mode.  Many of us believe that
> these profiles have higher chances of success compared to previous video
> coding standards, because they are a) not requiring much in terms of
> decoder implementation complexity beyond non-scalable implementations, and
> b) are conceptually similar to what VP9 and AV1 use, and hence familiar to
> those using modern scalable codecs, like the webrtc folks.  Insofar, we
> want to support scalability in the payload format from the outset.
>
> If only temporal scalability is supported, the selection of an appropriate
> sub-layer in the onion-shaped multi-sublayered stream by an SDP-based
> offer-answer is a one-dimensional problem, and has been solved in places
> like the HEVC payload format by an offer-answer exchange involving a single
> numeric parameter indicative of the highest supported sub-layer.  However,
> when having all temporal, spatial, and SNR scalability, the optimal
> selection of an appropriate output layer set (OLS) becomes a complex
> multi-dimensional problem, that may have more than one solution for
> different values of “optimal”, and also requires certain context.  We try
> to solve this multi-dimensional problem by a combination of a requirement
> for conditional presence of certain otherwise optional SDP codepoints when
> scalability is in use, and the introduction of an additional few code
> points.  Alternatives we rejected include not supporting scalability, or
> relying exclusively on the exchange of possibly many variants of parameter
> sets, as that could lead to excessively large SDP blobs (easily larger than
> the 1500 bytes still common as MTU).  Note that the latter option can still
> be implemented for cases where our restrictions are considered
> inappropriate.
>
>    1. When using scalability and OLS-based negotiation, the sender must
>    include a Video Parameter Set (VPS) as a sprop-vps or have otherwise
>    pre-arranged knowledge that the VPS is known by the receiver.  For those
>    unfamiliar with H.266, the VPS includes a “table of content” of the
>    layering structure, and without it, the values of new codepoints
>    (sprop-ols-id and recv-ols-id) could not be interpreted.
>    2. sprop-ols-id will typically point to the “best” output layer
>    set—the combination of layers providing the most pleasing reconstructed
>    result—that the sender can support taking into account all other parameters
>    known at this point of the O/A process.
>    3. The value of sprop-ols-id selects one of possibly several OLSs
>    branches in the VPS.  In the reply, using recv-ols-id, the answerer can
>    select that OLS or other OLSs from the same branch of scalable layers in
>    the VPS to which the sprop-ols-id belongs, but not any other OLSs that may
>    exist in the VPS.  The worst-case fallback will be base-layer only, which
>    is always supported, assuming H.266 and the relevant profiles tiers, and
>    levels are supported.
>    5. The interaction between what’s going on for the (temporal)
>    sub-layers and for the (spatial/SNR) layers is an application logic problem
>    and does not need to be specified, nor documented.  It’s very hard to grok
>    for anyone not intimately familiar with H.266.
> -sprop-ols-id: add a sentence “if this optional parameter is present,
> sprop-vps must also be present or its content must be known a priori at the
> receiver”
> -sprop-recv-id: add “sprop-recv-id, when present, must be included only
> when sprop-ols-id was received and must refer to an output layer set in the
> VPS that is in the same dependency tree as the OLS referred to by
> sprop-ols-id.  If this optional parameter is present, sprop-vps must have
> been received or its content must be known a priori at the receiver”
> Please provide comments to this design choice by Jan 6th.  Absent
> comments, we will implement the changes as described above, also any
> changes required for consistency, and we will add corresponding language in
> the currently empty offer-answer section of the draft.
>
> Thanks,
> Stephan
>
>
>
>
>
> [email-masked]
> [email-masked]

-- 
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard

   -

---

From: Stephan Wenger <[email-masked]>
Date: Fri, 18 Dec 2020 21:55:32 +0000

Sure.  How does a week later (13th) sound for the next version?  Some of us will be busy with MPEG, but unless there is a deluge of useful comments during that week, I think we will cope.
No, regarding MMUSIC.  The design of the offer-answer part of an RTP payload spec is AVT territory.  We do not plan to change o/a per se.
S.

From: Alexandre GOUAILLARD <[email-masked]>
Date: Friday, December 18, 2020 at 03:36
To: "Sanchez de la Fuente, Yago" <[email-masked]>
Cc: Stephan Wenger <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [AVTCORE] [External] VVC payload format, open issue cluster #1 (SDP for scalability)

Hi stefan,

Some of the AVTCORE members who have worked on the design of RTP payload for AV1, VP9 and other SVC / Simulcast features in webrtc, included but not limited to SDP O/A, RID and others, are on vacation until january 7th.

I think your proposal would benefit a lot from their spending time and providing feedback from an informed position, and I would humbly request that you push back your deadline for them to participate in the discussion. I do believe that the discussion can go for as long as the chairs want, and there is usually no deadline per say anyway, but it's alway nicer to ask.

Also, I believe that anything related to SDP O/A and its modifications needs to be proposed to the MMUSIC group.

Regards.

On Fri, Dec 18, 2020 at 4:48 PM Sanchez de la Fuente, Yago <[email-masked]<mailto:[email-masked]>> wrote:
Thanks Stephan,

Best regards,
Yago Sánchez

---

Tel.: +49 30 310 02663

BR, YK

Hi,

Thanks,
Stephan

--
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard<http://sg.linkedin.com/agouaillard>

  *

---

From: Alexandre GOUAILLARD <[email-masked]>
Date: Mon, 21 Dec 2020 18:03:35 +0800

Hi stefan,

Sure.  How does a week later (13th) sound for the next version?

I think it should be ok, worse case scenario we can discuss it at the jan
28th interim meeting.

> Some of us will be busy with MPEG, but unless there is a deluge of useful
> comments during that week, I think we will cope.
>

Yes I do remember our discussion during the AV1 codec group weekly meeting
about being courteous to other standardization bodies by not having AV1
meetings when other bodies had some. Ironically that discussion happened
during an IETF meeting week, and you explained that MPEG was important, but
IETF was not, or not enough to extend the courtesy to IETF.

> No, regarding MMUSIC.  The design of the offer-answer part of an RTP
> payload spec is AVT territory.
>

If you are just defining the Mapping of Media Subtype Parameters to SDP and
not extending any SDP feature, or SDP O/A feature, you are good. This is
not always the case, and specifically in the case of simulcast and SVC
modes, work had to be done when writing the AV1 RTP payload specification.
I just wanted to make sure you were aware this could be the case for you
too since you are touching scalability modes.

Regards,

> S.
>
>
>
>
>
> *From: *Alexandre GOUAILLARD <[email-masked]>
> *Date: *Friday, December 18, 2020 at 03:36
> *To: *"Sanchez de la Fuente, Yago" <[email-masked]>
> *Cc: *Stephan Wenger <[email-masked]>, "[email-masked]" <[email-masked]>
> *Subject: *Re: [AVTCORE] [External] VVC payload format, open issue
> cluster #1 (SDP for scalability)
>
>
>
> Hi stefan,
>
>
>

   -

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 21 Dec 2020 17:37:38 +0000

Dr. Alex,
Pls see inline in blue.
S.

From: Alexandre GOUAILLARD <[email-masked]>
Date: Monday, December 21, 2020 at 02:03
To: Stephan Wenger <[email-masked]>
Cc: "Sanchez de la Fuente, Yago" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [AVTCORE] [External] VVC payload format, open issue cluster #1 (SDP for scalability)

Hi stefan,

I think it should be ok, worse case scenario we can discuss it at the jan 28th interim meeting.

Some of us will be busy with MPEG, but unless there is a deluge of useful comments during that week, I think we will cope.

Yes I do remember our discussion during the AV1 codec group weekly meeting about being courteous to other standardization bodies by not having AV1 meetings when other bodies had some. Ironically that discussion happened during an IETF meeting week, and you explained that MPEG was important, but IETF was not, or not enough to extend the courtesy to IETF.

Being cited out of context is all the more fun when the citation is incorrect :-)

No, regarding MMUSIC.  The design of the offer-answer part of an RTP payload spec is AVT territory.

If you are just defining the Mapping of Media Subtype Parameters to SDP and not extending any SDP feature, or SDP O/A feature, you are good. This is not always the case, and specifically in the case of simulcast and SVC modes, work had to be done when writing the AV1 RTP payload specification. I just wanted to make sure you were aware this could be the case for you too since you are touching scalability modes.

For those not active in the Alliance for Open Media (AOM) codec WG and RTC subgroup (which, I fear, is the majority here): AOM decided a long time ago to design the payload format for AV1 outside of the IETF.  While the deliberations of the subgroup are AFAIK open only to AOM members, the github with the draft spec is in the open, secured only by obscurity :-).  The current (I hope) and I believe close to final AV1 payload spec can be found here: https://aomediacodec.github.io/av1-rtp-spec/#723-usage-with-the-sdp-offeranswer-model

@Dr. Alex: I do not see anything in 7.2.3 (or, in fact, anywhere else in the AV1 payload spec), that would not normally be done here in AVTcore.  Can you please elaborate on your previous para?

Thanks,
S.

Regards,

S.

From: Alexandre GOUAILLARD <[email-masked]<mailto:[email-masked]>>
Date: Friday, December 18, 2020 at 03:36
To: "Sanchez de la Fuente, Yago" <[email-masked]<mailto:[email-masked]>>
Cc: Stephan Wenger <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [AVTCORE] [External] VVC payload format, open issue cluster #1 (SDP for scalability)

Hi stefan,

Regards.

Best regards,
Yago Sánchez

---

Tel.: +49 30 310 02663

BR, YK

Hi,

Thanks,
Stephan

  *

  *

---

From: Alexandre GOUAILLARD <[email-masked]>
Date: Tue, 22 Dec 2020 02:46:08 +0800

you seem to know what yo uare doing, we cand discuss further point at the
meeting on january 28th. or later.

On Tue, Dec 22, 2020 at 1:37 AM Stephan Wenger <[email-masked]> wrote:

> *Date: *Monday, December 21, 2020 at 02:03
> *To: *Stephan Wenger <[email-masked]>
> *Cc: *"Sanchez de la Fuente, Yago" <[email-masked]>, "
> [email-masked]" <[email-masked]>
> If you are just defining the Mapping of Media Subtype Parameters to SDP
> and not extending any SDP feature, or SDP O/A feature, you are good. This
> is not always the case, and specifically in the case of simulcast and SVC
> For those not active in the Alliance for Open Media (AOM) codec WG and RTC
> subgroup (which, I fear, is the majority here): AOM decided a long time ago
> to design the payload format for AV1 outside of the IETF.  While the
> deliberations of the subgroup are AFAIK open only to AOM members, the
> github with the draft spec is in the open, secured only by obscurity :-).
> The current (I hope) and I believe close to final AV1 payload spec can be
> found here:
> https://aomediacodec.github.io/av1-rtp-spec/#723-usage-with-the-sdp-offeranswer-model
>
>
>
> @Dr. Alex: I do not see anything in 7.2.3 (or, in fact, anywhere else in
> the AV1 payload spec), that would not normally be done here in AVTcore.
> Can you please elaborate on your previous para?
>
>
>
> Thanks,
> S.
>
>
>
>
>
> Regards,
>
>
>
> S.
>
>
>
>
>

   -