Subject: [AVTCORE] Comments on “RTP Payload Format for Versatile Video Coding (VVC)”

From: "Hannuksela, Miska (Nokia - FI/Tampere)" <[email-masked]>
Date: Mon, 23 Aug 2021 18:55:25 +0000

Dear VVC RTP payload I-D authors and AVT experts,

I am responding to the WGLC as a contributor to the VVC standard and with the knowledge gained by authorship in AVC and HEVC RTP payload formats. Please find my technical comments below.

A few comments below are asking questions on particular design choices. Apologies if these have been already discussed in the past – I have not followed the development of the VVC RTP payload draft closely. In many cases, this might just be a sign to consider adding an informative note on why a design choice was made.

I have also sent a number of editorial fixes and suggestions directly to the authors.

1) Section 4.2

   A lower value of TID indicates a higher importance.

   More-important NAL units MAY be better protected against transmission

   losses than less-important NAL units.

      For Discussion: quite possibly something similar can be said for

      the Layer_id in layered coding, but perhaps not in multiview

      coding.  (The relevant part of the spec is relatively new,

      therefore the soft language).  However, for serious layer pruning,

      interpretation of the VPS is required.  We can add language about

      the need for stateful interpretation of LayerID vis-a-vis

      stateless interpretation of TID later.

  *   Note that independent layers are also possible, e.g. for depth maps or alpha planes.

  *   In VVC, VPS does not indicate the type of scalability. Consequently, layer pruning in a MANE needs to be based on VPS and some additional signaling, such as the SEI messages currently under development in JVET. Since these SEI messages are work in progress, it is proposed simply to remove the paragraph for discussion and not discuss layer pruning or layer importance in the RTP payload text.

2) Section 7.1, profile-id semantics

         A profile_tier_level( ) syntax structure may be contained in an

         SPS, VPS, or DCI NAL units as specified in [VVC].  One of the

         following three cases applies to the container NAL unit of the

         profile_tier_level( ) syntax structure containing those PTL

         syntax elements used to derive the values of profile-id, tier-

         flag, level-id, sub-profile-id, or interop-constraints:

         ...

         3) The container NAL unit is a DCI NAL unit and the

         profile_tier_level( ) syntax structures in all DCI NAL units in

         the bitstream has the same values respectively for those PTL

         syntax elements.

  *   The first sentence above might give a wrong impression, since VPS and DCI may contain multiple PTL syntax structures. The sentence could be rephrased as follows: “As specified in [VVC], a profile_tier_level( ) syntax structure may be contained in an SPS NAL unit, and one or more profile_tier_level( ) syntax structures may be contained in a VPS NAL unit and in a DCI NAL unit.”

  *   I am a bit uncertain how to interpret the phrasing “the PTL syntax structures in all DCI NAL units in the bitstream has the same values respectively for those PTL syntax structures”.
     *   Let’s take an example: there are two DCI NAL units, dciA and dciB, in the bitstream, and both of them contain two PTL structures, i.e. dciA contains ptlA1 and ptlA2 structures (in this order) and dciB contains ptlB1 and ptlB2 structures (in this order).
     *   Does the phrasing mandate ptlA1 to be the same as ptlB1, and ptlA2 to be the same as ptlB2? If so, then why is the phrasing needed at all, since the VVC standard requires all DCI NAL units in a bitstream to have the same content.
     *   Or does the phrasing mandate ptlA1 to be the same as ptlA2, and ptlB1 to be the same as ptlB2? This interpretation would not be meaningful technically, since why would an encoder include multiple PTL structures in a DCI NAL unit, if they the same values.

  *   It seems that case 3 (DCI) would not work if a DCI NAL unit contains more than one PTL syntax structure. If there were multiple PTL structures in a DCI NAL unit, how would it be determined which one of them is used to derive profile-id, tier- flag, level-id, sub-profile-id, and interop-constraints? Would case 3 need to be limited to apply only if DCI NAL unit(s) contain only one PTL structure?

3) Section 7.1, tier-flag, level-id

         If the tier-flag and level-id are derived from the

         profile_tier_level( ) syntax structure in a DCI NAL unit, the

         following applies:

  *   According to VVC, a DCI NAL unit may contain multiple PTL syntax structures. This sentence seems to assume that there would be only one PTL structure in a DCI NAL unit.

4) Section 7.1, sprop-ols-id

         The value of sprop-ols-id MUST be in the range of 0 to 257,

         inclusive.

            Informative note: VVC allows having up to 258 output layer

            sets indicated in the VPS as the number of output layer sets

            minus 2 is indicated with a field of 8 bits.

  *   The maximum value of vps_num_output_layer_sets_minus2 is 255. Thus, VVC allows to have 257 output layer sets and the value range for OLS ID is from 0 to 256, inclusive.

5) Section 7.1, recv-ols-id

         When present, the value of recv-

         ols-id must be included only when sprop-ols-id was received and

         must refer to an output layer set in the VPS that is in the

         same dependency tree as the OLS referred to by sprop-ols-id.

  *   I don’t understand why sprop-ols-id is required in the offer in order to have recv-ols-id in the answer. Could you explain?
     *   Note that if this requirement is relaxed, Section 7.2.2.2 would also need changes.

  *   In my understanding:
     *   The absence of sprop-ols-id in an offer indicates that the offerer can transmit any OLS that is included in sprop-vps. Consequently, the answerer could respond with recv-ols-id equal to any OLS ID included in sprop-vps.
     *   The presence of sprop-ols-id in an offer indicates that the offerer only has a bitstream corresponding to the OLS indicated by sprop-ols-id and can transmit only that OLS or any OLS contains either the same layers as or a subset of the layers of the OLS indicated by sprop-ols-id.

  *   The term dependency tree is not defined and hence should either be avoided in the phrasing or defined.

6) Section 7.1, sprop-vps

         The sprop-vps parameter MAY contain one or more than one video

         parameter set NAL unit.  However, all other video parameter

         sets contained in the sprop-vps parameter MUST be consistent

         with the first video parameter set in the sprop-vps parameter.

         A video parameter set vpsB is said to be consistent with

         another video parameter set vpsA if any decoder that conforms

         to the profile, tier, level, and constraints indicated by the

         data starting from the syntax element general_profile_space to

         the syntax element general_level_idc, inclusive, in the first

         profile_tier_level( ) syntax structure in vpsA can decode any

         bitstream that conforms to the profile, tier, level, and

         constraints indicated by the data starting from the syntax

         element general_profile_space to the syntax element

         general_level_idc, inclusive, in the first profile_tier_level(

         ) syntax structure in vpsB.

  *   I don’t understand what makes the first PTL structure in a VPS special. Could you explain? Perhaps adding an informative NOTE in the text would help readers to understand this design choice.

  *   Shouldn't the consistency requirement to be stricter so that the number of OLSs in different VPSs must match and a decoder capable of decoding the PTL in vpsA for any OLS with index olsIdx can decode any CVSs referring to vpsB when TargetOlsIdx is equal to olsIdx?

  *   Shouldn’t “can decode any bitstream” rather be “can decode any CVS(s) referencing vpsB”? Referencing a new VPS does not necessarily start a new bitstream.

7) Section 7.1, sprop-sei

            ... the display orientation SEI message

            when the device is a handheld device (as the display

            orientation may change when the handheld device is turned

            around), ...

  *   Neither the VVC standard nor the VSEI standard currently includes the display orientation SEI message. It is proposed to remove this part of the informative note.

8) Section 7.1, related to sprop-sei

  *   It would be helpful to enable offer-answer negotiation on properties that are signaled with SEI messages. These properties could include e.g.:
     *   Whether film grain synthesis is supported (the film grain characteristics SEI message)
     *   Whether frame packing is supported and which frame packing arrangement type to use (the frame packing arrangement SEI message)
     *   How many subpictures to use and which levels they should correspond (subpicture level information SEI message)

  *   It is suggested to allow empty or truncated SEI message payloads in sprop-sei to indicate the capability of the offerer to encode these SEI messages with any SEI message payload that starts with the bits included in sprop-sei (if any). For example, when contained in sprop-sei:
     *   The film grain characteristics SEI message with a truncated payload that only contains data up to fg_model_id indicates that the offerer is capable of producing a bitstream containing film grain of that fg_model_id value.
     *   The frame packing arrangement SEI message with an empty payload indicates that the offerer is capable of producing a bitstream with any frame packing arrangement.
     *   The subpicture level information SEI message with an empty payload indicates that the offerer is capable of encoding content with subpictures and include subpicture level information SEI message(s) into the bitstream.

  *   It is suggested to introduce an optional recv-sei parameter along the following principles:
     *   The SEI messages included in recv-sei in an answer indicate which SEI messages must be present in the bitstream from the offerer to the answerer
     *   When recv-sei is present in an answer, the SEI message types in recv-sei must be the same as or a subset of those included in the sprop-sei of the offer.
     *   When recv-sei is present in an answer, the SEI message payload for a particular SEI message type in recv-sei must start with the SEI message payload bits present (if any) in recv-sei for the same SEI message type.
     *   It is suggested to allow empty or truncated SEI message payloads in recv-sei to indicate that the answerer has the liberty to encode any values for remaining of the SEI message payload (as long as they conform to the specification where the SEI message is specified). Note that the conditions in the bullets above still apply.
     *   When an SEI message with a particular SEI message type is present in sprop-sei of an offer and is absent in recv-sei in an answer, the answerer indicates that it does not support the processing the SEI message included in sprop-sei.

9) Section 7.2.2.1

      1) When sprop-sps parameter is present, all sequence parameter

      sets contained in the sprop-sps parameter in the SDP answer and

      all sequence parameter sets sent in-band for either the offerer-

      to-answerer direction or the answerer-to-offerer direction MUST be

      consistent with the first sequence parameter set in the sprop-sps

      parameter of the offer (see the semantics of sprop-sps in

      Section 7.1 of this document on one sequence parameter set being

      consistent with another sequence parameter set).

  *   The semantics of sprop-sps do not specify when an SPS is consistent with another SPS. It presumably means roughly the following: “An SPS spsB is consistent with an SPS spsA, when a decoder capable of decoding the PTL in spsA can decode any CLVSs referring to spsB.” Please check and add definition of consistent SPS into the semantics of sprop-sps.

10) Section 7.2.2.2, on sprop-sub-layer-id indicating the property of the highest layer only

   o  The offerer MAY include sprop-sub-layer-id which, in case of

      scalable VVC, is interpreted as the highest sub-layer of the

      highest enhancement layer in the OLS indicated by sprop-ols-id.

      The answerer MAY include recv-sub-layer-id which can be used to

      downgrade the sublayer of the highest enhancement layer.  This

      specification does not support downgrading the sub-layer of any

      layers in the OLS that are not the highest layer.

         Informative note: in other words, using this mechanism, an

         answerer can downgrade only the frame rate for the highest

         spatial/quality layer (typically corresponding to the highest

         resolution or bitrate, hence the most complex to decode), but

         not for lower spatial/quality layers.  The answerer must

         support all sublayers for lower layers in the OLS, or reject

         the offer.  That's not a big burden, as the receiver/decoder

         has the option to discard any sublayers it cannot decode,

         irrespective of what is being signalled through offer/answer.

  *   This design choice is not obvious. Note that pruning temporal sublayers from the highest layer but not from the reference layer(s) could have undesirable impacts, such as:
     *   The decoder outputs reference-layer pictures with TemporalId greater than sprop-sub-layer-id. In SNR/spatial scalability these pictures have lower quality/resolution than the pictures of the highest layer and hence cause temporally fluctuating quality/resolution.
     *   If sprop-sub-layer-id is used to input the Htid variable to the decoder as specified in Section 8.1.1 of VVC, the receiver would need to apply the sub-bitstream extraction process of VVC, C.6 to obtain a conforming bitstream.

  *   I would think that the maximum of sprop-sub-layer-id (when present) and recv-sub-layer-id (when present) MUST indicate the highest temporal sublayer to be decoded for the bitstream from the offerer to the answerer and MAY be provided as the value of Htid to the VVC decoding process as specified in Section 8.1.1 of VVC. Changes would be needed in Section 7.2.2.2 as well as in sprop-sub-layer-id and recv-sub-layer-id in Section 7.1.

11) Section 7.2.3

      *  Declaring actual configuration or bitstream properties:

         +  profile-id

         +  tier-flag

         +  level-id

         +  interop-constraints

         +  sub-profile-id

         +  sprop-dci

         +  sprop-vps

         +  sprop-sps

         +  sprop-pps

         +  sprop-max-don-diff

         +  sprop-depack-buf-bytes

         +  sprop-sub-layer-id

         +  sprop-ols-id

  *   Should the list also contain sprop-sei?

12) Section 10

   A media sender can remove NAL units

   belonging to higher temporal sublayers (i.e., those NAL units with a

   high value of TID) or higher spatio-SNR layers (as indicated by

   interpreting the VPS) until the sending bitrate drops to an

   acceptable range.

  *   VPS does not indicate the type of scalability. It is proposed to remove “(as indicated by interpreting the VPS)”.

Best Regards,
Miska Hannuksela

From: avt <[email-masked]> On Behalf Of Bernard Aboba
Sent: Monday, August 16, 2021 11:30 PM
To: IETF AVTCore WG <[email-masked]>
Subject: [AVTCORE] WGLC on “RTP Payload Format for Versatile Video Coding (VVC)”

This is an announcement of WG last call on "RTP Payload Format for Versatile Video Coding (VVC)”.

The document is available for inspection here:

https://datatracker.ietf.org/doc/draft-ietf-avtcore-rtp-vvc/

WG Last Call will end on August 30, 2021.

In response, please state one of the following:

* I support advancing the document to Proposed Standard

* I object to advancement to Proposed Standard, due to Issues

described below <Issue description or link>.

Bernard Aboba

For the Chairs