Subject: [AVT] Comments on the major open issue of draft-ietf-avt-rtp-svc-07.txt, cross layer decoding order dependency

From: <[email-masked]>
Date: Fri, 8 Feb 2008 16:56:42 -0000

The problem to be solved can be summarised as follows. The video
encoder, or other source of coded video data, produces a sequence of
chunks of data known as NAL units. These are to be transmitted over two
or more RTP sessions. At the receiver, the data is to be put back into a
single sequence with the same order as in the original sequence from the
encoder or data source. This data is then input to the video decoder and
is decoded and output. There are other variants, where for example, the
receiver is not a decoder, but some other device such as a MANE, but the
core problem of re-establishing the original order of NAL units is the
same.

One of the solutions to this problem, the CL-DON solution, allocates a
monotonic increasing sequence number to each of the NAL units from the
encoder, transports these numbers through the network, and uses these
numbers to re-establish the original order of NAL units. The NAL units
received on the multiple RTP sessions are simply ordered according to
this monotonic increasing sequence of numbers.

The other solution to this problem, the classical solution, when
operated as in the rules in the current version of the draft operates as
follows. The NAL units from the encoder are grouped into NAL units and
associated with a non- monotonic number (the timestamp representing
output (display) order rather than decoding order). Effectively the NAL
units are being labelled with (almost) arbitrary labels. These labelled
NAL units are then separated into multiple RTP streams, and a monotonic
increasing sequence is applied independently in each RTP stream. Note
both of these steps are performed in the CL-DON solution, but do not
have to be used to restore decoding order. At the receiver, the
independent monotonic increasing sequence numbers are used to re-order
packets in each RTP stream. These are then grouped according to label
(timestamp) in each RTP stream. Then the NAL units from the lower layers
are "merged" with the NAL units in the highest enhancement layer,
grouping together NAL units with the same label (timestamp). Finally,
SEI NAL units must be moved to the start of each group (access unit), if
they were transmitted anywhere other than the base RTP session.

This suffers from the need for the highest layer to have NAL units at
every time instant for which there is a NAL unit in any lower layer. And
due to the need for this process to work regardless of how many of the
RTP sessions are received, the same has to apply to any layer with
regards to the layers below it. While this can be overcome by inserting
filler data NAL units, it does seem to have a problem with packet loss,
as this situation can not be guaranteed after loss. Given that the
highest layer may often be transmitted with the least error protection,
this is a major limitation of this approach.

But the classical solution can be operated in a different way at the
receiver to overcome this limitation, but with additional complexity. As
before, at the receiver, the independent monotonic increasing sequence
numbers are used to re-order packets in each RTP stream, and then these
are grouped according to label (timestamp) in each RTP stream. Then the
sequences of labels (timestamps) in each stream can be analysed, and in
many (but not all) cases, the decoding order of the labels (timestamps)
can be deduced, and then used to restore the decoding order.

In the example below, the top two RTP sessions operate at a given frame
rate and the base layer is operating at half the frame rate. Packet loss
has affected one access unit in the top layer and one access unit in the
middle layer.

 4     1  3  8  6  5  7 12 10

 4  2  1  3  8     5  7 12 10

 4  2        8  6       12 10

However, decoding order can be restored by noticing from the middle
layer that NAL units with label =2 are to be decoded before those with
label=1. Similarly, the top layer tells us that NAL units with label =6
are to be decoded before those with label=5.

But if both middle and top layers lost their NAL units with label=2, as
shown below, it would be more difficult to re-establish decoding order
as from the RTP and payload layer it is not possible to determine if
label=2 comes before or after label=1. It may be possible to determine
order by looking into pic_timing SEI messages, if present (not
guaranteed), or a best guess could be made by making assumptions based
on previous GOP structures (and the order of timestamps). Alternatively
it may be better to discard all NAL units with labels 1 and 3 rather
than to risk feeding data to the decoder in the wrong order.

 4     1  3  8  6  5  7 12 10

 4     1  3  8     5  7 12 10

 4  2        8  6       12 10

My conclusion is that while using a non-monotonic set of numbers
(timestamps) to re-establish decoding order is possible in many but not
all cases, it is a fairly complex process, particularly if it is to make
best use of all packets received when some are lost, as in the second
method above. And in practice I feel that the second method would be
implemented because the performance of the first in the case of packet
loss could be unacceptably poor.

The major weakness of the CL-DON method is that it is not backwards
compatible with the single NAL unit mode of RFC 3984.

One way to overcome this would be to use some backward compatible
mechanism to transport the CL-DON information in the base RTP session
operating in single NAL unit mode. The RTP header extension mechanism is
one way that this could be done, but I know that there are objections to
doing this.

However, the single NAL unit mode was introduced into RFC 3984 primarily
"for low-delay applications that are compatible with systems using ITU-T
Recommendation H.241". 

Hence, if there is a need for backwards compatibility with the single
NAL unit mode, and this is itself very debatable, then this need would
seem to be restricted to low delay applications, where it is unlikely
that access units would be encoded in a different order to output
(display) order.

Consequently, a solution to the whole problem of restoring the decoding
order of NAL units is define a class of receiver that supports the full
CL-DON method, and the classical method restricted to cases where the
timestamps are monotonic increasing. This restricted case of the
classical method is much simpler to implement than the general case, and
provides backwards compatibility with the intended applications of the
single NAL unit mode.

Best regards

Mike

Mike Nilsson 
Multimedia Analysis and Coding
BT Group Chief Technology Office
___________________________ 

Sirius House (B54-MH), Room 92 
Adastral Park, Martlesham Heath, Ipswich, IP5 3RE, UK 
Tel:    +44 1473 645413 
Mobile: +44 7917 025433 
Fax:    +44 1908 862365 
Email:  [email-masked] <BLOCKED::mailto:[email-masked]> 

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
http://www.ietf.org/mailman/listinfo/avt

---

From: "Jonathan Lennox" <[email-masked]>
Date: Mon, 11 Feb 2008 17:58:36 -0500

Another weakness of the CL-DON method is that it does not support NAL
unit fragmentation in mode 1.  RFC 3984 FU-A packets cannot include a
CL-DON, since mixing aggregation and fragmentation isn't allowed.

Though it's better to avoid it if you can, there are scenarios where
fragmentation is needed, i.e. video being transmitted over a network
with an MTU lower than the video encoder expected.  If retransmission or
FEC is in use, RTP-layer fragmentation is hugely preferable to IP-layer
fragmentation of the RTP packet.

More architecturally, I wonder if cross-layer decoding is a question
that should be addressed in a generic manner rather than per-media.
Several drafts have been presented recently for audio codecs which do
session multiplexing in a very similar way to H.264/SVC, and they also
need to be able to indicate a global decoding order.  Should this be
addressed as a generic problem?

-- 
Jonathan Lennox
Vidyo, Inc (formerly Layered Media)
[email-masked]

---

From: <[email-masked]>
Date: Tue, 12 Feb 2008 11:45:03 +0200

>Another weakness of the CL-DON method is that it does not 
>support NAL unit fragmentation in mode 1.  
 
You are free to use FU-A in the CL-DON mode when you use the interleaved
packetization mode. 

BR, YK

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of ext Jonathan Lennox
>Sent: Tuesday, February 12, 2008 12:59 AM
>To: [email-masked]
>Cc: [email-masked]
>Subject: Re: [AVT] Comments on the major open issue 
>ofdraft-ietf-avt-rtp-svc-07.txt, cross layer decoding order dependency
>
>Another weakness of the CL-DON method is that it does not 
>support NAL unit fragmentation in mode 1.  RFC 3984 FU-A 
>packets cannot include a CL-DON, since mixing aggregation and 
>fragmentation isn't allowed.
>
>Though it's better to avoid it if you can, there are scenarios 
>where fragmentation is needed, i.e. video being transmitted 
>over a network with an MTU lower than the video encoder 
>expected.  If retransmission or FEC is in use, RTP-layer 
>fragmentation is hugely preferable to IP-layer fragmentation 
>of the RTP packet.
>
>
>More architecturally, I wonder if cross-layer decoding is a 
>question that should be addressed in a generic manner rather 
>than per-media.
>Several drafts have been presented recently for audio codecs 
>which do session multiplexing in a very similar way to 
>H.264/SVC, and they also need to be able to indicate a global 
>decoding order.  Should this be addressed as a generic problem?
>
>--
>Jonathan Lennox

---

From: "Jonathan Lennox" <[email-masked]>
Date: Tue, 12 Feb 2008 10:21:34 -0500

[email-masked] writes:
> You are free to use FU-A in the CL-DON mode when you use the
> interleaved packetization mode.

Yes, but not in the non-interleaved packetization mode (mode 1), which
was my point.  There's been consensus that we need to support operations
in which the base layer, at least, isn't using interleaved mode.

---

From: <[email-masked]>
Date: Tue, 12 Feb 2008 17:34:06 +0200

Alright. I thought "NAL unit fragmentation in mode 1" meant FU-A. 

BR, YK 

>From: ext Jonathan Lennox [mailto:[email-masked]] 
>Sent: Tuesday, February 12, 2008 5:22 PM
>To: Wang Ye-Kui (Nokia-NRC/Tampere); [email-masked]
>Cc: [email-masked]
>Subject: RE: [AVT] Comments on the major open issue 
>Yes, but not in the non-interleaved packetization mode (mode 
>1), which was my point.  There's been consensus that we need 
>to support operations in which the base layer, at least, isn't 
>using interleaved mode.
>
>--
>Jonathan Lennox

---

From: Colin Perkins <[email-masked]>
Date: Sat, 16 Feb 2008 14:26:52 +0000

On 11 Feb 2008, at 22:58, Jonathan Lennox wrote:
...
> addressed as a generic problem?

That's an interesting idea. If there are sufficient commonalities  
between the different payload formats, it makes sense to use a common  
mechanism if such a thing is possible (perhaps either an extension to  
the RTP header, or a common building block used in different payload  
headers).

-- 
Colin Perkins
http://csperkins.org/

---

From: <[email-masked]>
Date: Sun, 17 Feb 2008 00:18:32 +0200

>which do 
>That's an interesting idea. If there are sufficient 
>commonalities between the different payload formats, it makes 
>sense to use a common mechanism if such a thing is possible 
>(perhaps either an extension to the RTP header, or a common 
>building block used in different payload headers).
>

There are at least three existing drafts (the SVC payload draft, the MVC
payload draft, and the EVBR payload draft) that are common in this
regard. 

Having an extension to the RTP header is what Mike has proposed earlier.
If this is viable, it would be the cleanest solution in the long run,
and it will reduce a lot the document complexity for scalable codec's
RTP payload drafts. The only problem is when an RFC for this can be
available to enable SVC deployment. 

Having a common block used in different payload headers is not easy or
impossible to achieve, because the SVC and MVC draft are both based on
RFC 3984 and are required to be backward compatible to RFC 3984. If all
the scalable codecs are, like the EVBR codec, developped from scratch
and not based an existing codec, then a common buidling block in payload
headers is then easy to achieve. 

BR, YK

---

From: Colin Perkins <[email-masked]>
Date: Tue, 19 Feb 2008 23:01:34 +0000

> available to enable SVC deployment.

We've had a requirement since RFC 1889 that the header extension be  
ignorable without affecting correctness, so I don't think we can use  
it for this role (although I understand the desire to do so).

> headers is then easy to achieve.

I'm less concerned with the format of the bits in the header, than in  
the semantics. Having common semantics across formats would be  
incredibly useful - the exact syntax of the bits is less important.

-- 
Colin Perkins
http://csperkins.org/