Subject: Re: [AVTCORE] I-D Action: draft-ietf-avtext-framemarking-08.txt

From: [email-masked] (Dale R. Worley)
Date: Sun, 08 Dec 2019 14:48:25 -0500

My apologies for my delay in responding to this.

draft-ietf-avtext-framemarking-10 is definitely better than -08 on a lot
of points, but there are some points that (in my opinion) need to be
more fully addressed before the document is fully ready for approval.

"Mo Zanaty (mzanaty)" <[email-masked]> writes:

[Moving this item first, as it is a design question which informs the
entire proposal:]

> Mo: It is impossible to capture and signal all the dependencies in a
> modern video stream efficiently.

That makes sense, but OTOH the *purpose* of this extension is to capture
enough of the dependency structure that e.g. a router can act on the
extension values when deciding how to drop packets from the video stream
without unduly messing up the video or wasting bandwidth -- all without
being able to understand the video packets themselves.  Which pretty
much means that the extension can contain false-positive dependencies
but should avoid false-negative dependencies.  Which implies that the
extension values model a particular dependency system, which can be
tailored to be a reasonable approximation of the dependency systems of
the video formats now in use.

(As an exception, the case "Complex structures also use TIDs and LIDs
but not necessarily in a clean nested hierarchy. The most complex
structures are total anarchy (dynamic, unpredictable) but could still
use TIDs and LIDs (within the codec payloads, but not in this header
extension)." -- In that case, the video stream's structure is encoded in
the trivial way (with many false-positive and no false-negative
dependencies), where all packets have TID=0 and LID=0.)

-----

>>There is some oddity in how the sections are structured.  [...]
>
> Mo: This structure was intentional to show that Layer ID mappings only
> apply to Scalable Streams.  [...]

What I didn't make clear was that when I read -10, I didn't realize that
the short form is simply what the long form reduces to when the stream
is not scalable (and thus the layer IDs are all 0).  The logical
structure of the actual proposal is that the extension is defined in
section 3.2, but if the stream is not scalable, it simplifies to the
short form described in section 3.1.  However, the text is organized as
if you are defining two logically independent extensions (which look a
lot alike), and which one was actually in use for a particular RTP
stream is dependent on whether the particular video format is scalable
or not.  (And as you've noted, we want this extension to work as an
annotation for encrypted video streams, where the e.g. router acting on
the extension may have no information about the underlying video
encoding.)

In this regard, I think the document needs to explicitly state that the
description in section 3.2 is the *definition* of the extension, and the
description in section 3.1 is just what the definition *reduces to* when
TID and LID are both 0 (i.e., the stream is not scalable).  It would
help a great deal if section 3.2 was moved to be first (as it is
logically antecedent), and section 3.1 was annotated as the simplified
case.

-----

>>Also, 3.2.1.3 (H264 (AVC) LID Mapping) and 3.2.1.4 (VP8 LID Mapping)
>>don't specify how the S, E, I, D, and B bits are determined from the
>>codec's output packets.
>
> Mo: This is specified in section 3.2.

What I am hoping for is that sections 3.2.1.1 to 3.2.1.4 give explicit
rules for computing the various flag bits from specific bits of the
video protocols.  Of course, the computation is *implied* by the rules
of section 3.2, and in some cases, the current text states explicit
rules, but I think it would help if all of the computations were stated
in ready-to-code form.

Also, text like this appears in several places:

   The S, E, I and D bits MUST match the corresponding bits in PACSI
   payload structures.

Does "the corresponding bits" mean "the bits named S, E, I, and D in
PACSI ..."?  If so, I think the latter phrasing is clearer, and if the
bits with those meanings have different names in PACSI, the names should
be stated.

-----

   o  TID: Temporal ID (3 bits) - The base temporal layer starts with 0,
      and increases with 1 for each higher temporal layer/sub-layer.  If
      no scalability is used, this MUST be 0.  It is implicitly 0 in the
      short extension format.
   o  LID: Layer ID (8 bits) - Identifies the spatial and quality layer
      encoded, starting with 0 and increasing with higher fidelity.  If
      no scalability is used, this MUST be 0 or omitted to reduce
      length.  When omitted, TL0PICIDX MUST also be omitted.  It is
      implicitly 0 in the short extension format or when omitted in the
      long extension format.

I notice that while TID has the restriction "increases with 1 for each
higher temporal layer", LID does not.  Is there a reason that LID
numbers aren't required to be sequential?

-----

> Mo: I fixed this unfortunate circumstance by forcing B=0 when TID=0. B is
> useless when TID=0, so we can force it to 0 or 1 arbitrarily. The
> "natural" value for something to ignore is 0, so I just enforced this in
> the following text.
> o B: Base Layer Sync (1 bit) - When TID is not 0, this MUST be 1 if
> the sender knows this frame only depends on the base temporal
> layer; otherwise MUST be 0. When TID is 0 or if no scalability is
> used, this MUST be 0.

Hmmm, it's not important enough to write out my full argument, but I
would have voted for B=1 when TID=0.

-----

The following item has a number of ramifications.  I include the prior
discussion for reference:

>>The switching of video streams is recommended to be done this way:
>>
>>   When an RTP switch wants to forward a new video stream to a receiver,
>>   it is RECOMMENDED to select the new video stream from the first
>>   switching point with the I (Independent) bit set in all spatial
>>   layers and forward the same.  An RTP switch can request a media
>>   source to generate a switching point by sending Full Intra Request
>>   (RTCP FIR) as defined in [RFC5104], for example.
>>
>>This is difficult to implement in general, as it requires the switch to
>>keep track of all the layer IDs that have been seen, then look ahead in
>>the stream to see if, over a narrow range of time, all of the layers
>>that have been seen have packets with I set.  If the fundamental purpose
>>of I is to signal the best points to switch streams, it would be better
>>to define its semantics to be that.  E.g., "If a switch intends to start
>>forwarding a video stream, and within that stream, transmitting all
>>frames with TID and LID less than or equal to certain values, it should
>>start forwarding the stream beginning with a packet within that layer
>>that has I set."  That is, I signals that at this point, the coming
>>frames of this layer and all layers with lesser TID/LID can be decoded
>>without dependency on any previous frames.
>
> Mo: Media switches already do this, but by deep inspection of the payload
> rather than simple inspection of a header extension. The I bit does not
> signal anything about lower layers, only this layer. That is what media
> switches want and expect.

The first point is that with this extension, a media switch should be
able to switch streams without inspecting the payload *at all*, so what
they now do by deep inspection of the video stream is not particularly
relevant.

Another point is to look at the bit definitions in section 3.2:

   o  S: Start of Frame (1 bit) - MUST be 1 in the first packet in a
      frame within a layer; otherwise MUST be 0.
   o  E: End of Frame (1 bit) - MUST be 1 in the last packet in a frame
      within a layer; otherwise MUST be 0.  Note that the RTP header
      marker bit MAY be used to infer the last packet of the highest
      enhancement layer, in payload formats with such semantics.
   o  I: Independent Frame (1 bit) - MUST be 1 for frames that can be
      decoded independent of temporally prior frames, e.g. intra-frame,
      VPX keyframe, H.264 IDR [RFC6184], H.265 IDR/CRA/BLA/RAP
      [RFC7798]; otherwise MUST be 0.  Note that this bit only signals
      temporal independence, so it can be 1 in spatial or quality
      enhancement layers that depend on temporally co-located layers but
      not temporally prior frames.
   o  D: Discardable Frame (1 bit) - MUST be 1 for frames the sender
      knows can be discarded, and still provide a decodable media
      stream; otherwise MUST be 0.

Note that bits S and E are defined in regard to "a frame within a
layer".  Section 3 defines "frame":

   A frame, in the context of this specification, is the set of RTP
   packets with the same RTP timestamp from a specific RTP
   synchronization source (SSRC).

I don't think it's stated explicitly, but clearly "layer" means "the
packets with a particular TID and LID value".

However, bits I and D are defined in terms of a frame alone, not "frame
within a layer", so they are *necessarily* the same for all packets
within a frame (i.e. with a particular timestamp) regardless of TID and
LID values.

Is this intended, or is the intention that I and D are defined for
"frame within a layer"?

This question interacts in a complicated way with this part of section
3.4:

   When an RTP switch wants to forward a new video stream to a receiver,
   it is RECOMMENDED to select the new video stream from the first
   switching point with the I (Independent) bit set in all spatial
   layers and forward the same.

Given that the I bit is the same for all layer IDs, this text is
extra-strict, and could be phrased "from the first switching point with
the I bit set".

I suspect you mean for the I bit to be defined in regard to "a frame
within a layer", in which case this text is phrased the way one would
expect it to be.

However, if you want to make life easier for RTP switches, I think you
want to define the I bit this way (and RTP switching seems to be the
only use of I bits) (and I think this would not burden RTP senders):

   o  I: Independent Frame (1 bit) - MUST be 1 for frames within layers
      that can be decoded independently of temporally prior frames,
      e.g. intra-frame, VPX keyframe, H.264 IDR [RFC6184], H.265
      IDR/CRA/BLA/RAP [RFC7798] and for which all layers within the
      frame with no larger TID or LID are similarly independent;
      otherwise MUST be 0.  Note that this bit only signals temporal
      independence, so it can be 1 in spatial or quality enhancement
      layers that depend on temporally co-located layers but not
      temporally prior frames.

That is, the I bit summarizes not only the independence of a packet of a
particular frame/layer, but of all the lower LID/TID layers in the frame
as well.  Then video switching becomes far simpler:

   When an RTP switch wants to forward a new video stream to a receiver,
   it is RECOMMENDED to select the new video stream from the first
   switching point with the I (Independent) bit set in the layer with
   the highest TID/LID that is being passed.

In a somewhat similar way, I think you want to modify the definition of
the D bit to:

   o  D: Discardable Frame (1 bit) - MUST be 1 for frames in layers the
      sender knows can be discarded, and still provide a decodable media
      stream for the layer; otherwise MUST be 0.

The advantage of this definition is that it allows, e.g., the packets
with LID=1 of a frame to be discardable even if the packets with LID=0
of the same frame are not.  (I do not know if any existing video format
generates frames with this property.)

(The reverse case cannot happen in practice, since the LID=1 packets of
a frame are implicitly dependent on the LID=0 packets of the same frame,
and if the LID=0 packets are discarded, the LID=1 packets are expected
to not be decodable.)

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 17 Dec 2019 22:57:09 -0500

In attempt to clarify a particular point:

The whole point of draft-ietf-avtext-framemarking is to provide
information to packet-handling devices on how to manipulate streams of
RTP packets containing encoded video, even when the device cannot
understand the payload of the RTP packets, either because they are
encrypted or they are in a video format that the device does not
understand.

Three typical operations are:
1) Routers dropping packets due to congestion, trying to determine the
least "costly" packets to drop.
2) Routers trying to "shape" the bandwidth demand of a video stream by
removing one or more highest-resolution layers from the video encoding.
3) RTP switches wanting to splice from one video stream to another,
looking for an "efficient" place to switch to the new stream.

In order for draft-ietf-avtext-framemarking to work well, the
significance of the extension data must be well-defined, so devices know
what the extension data tells about the RTP packets.  Inevitably, this
means that the extension data is interpreted within a model about the
packets and how they are related.

The fundamental structure seems to be the "frame-in-layer", the set of
packets that have the same SSRC, RTP timestamp, LID, and TID values.  A
frame-in-layer is assumed to encode a particular image at a particular
temporal and spatial resolution.

The remaining extension data seems to always encode dependencies between
frames-in-layer, that is, if the receiver is to successfully decode one
particular frame-in-layer, it needs all or most of the packets in
certain other frames-in-layer.

E.g., the D bit says that this frame-in-layer is not depended upon by
any other frame-in-layer.  The implication is that dropping these
packets is "low cost" compared to dropping packets within a
frame-in-layere that is depended on by another frame-in-layer.

There is an implication that if one frame-in-layer has the same SSRC,
same timestamp, no higher LID, and no higher TID than another
frame-in-layer, then the latter frame-in-layer depends on the former
frame-in-layer.

Within this framework, the algorithm for applying the extension to any
particular video encoding attempts to capture the actual dependency
structure of the video packets within the model that the extension data
can express.  There can be two sorts of mismatch:  "false positive",
where the extensions express a dependency not present in the video
encoding, and "false negative", where the extension do not express a
dependency which is present in the video encoding.  The extreme case is
"complex, irregular scalability structures that do not conform to
common, fixed patterns of inter-layer dependencies and referencing
structures."  In that case, using TID and LID is likely to not be
beneficial, and the extension data will tend to express a lot of "false
positive" dependencies.

What I'm pushing for is that all of this machinery be clearly stated,
especially exactly what dependencies are signaled by the extension data.
If those are left to the common intuitive understanding, we're likely to
have a lot of edge cases implemented differently by different devices,
leading to poor user experience (although probably not outright
non-interoperability).

Dale