Subject: Re: [AVT] Sigalling for layered media -- do it generic or do itpayloadspecific

From: Andrea Lorenzo VITALI <[email-masked]>
Date: Fri, 02 Dec 2005 11:35:06 +0100

> I am in the opinion that a middlebox as well as a receiver has to have the
> full knowledge about the media and therefore it must be able to parse all
> binary SVC description. But it could make life easier if the
> dependency/relation would be signalled via e.g. extended group-semantics.
> But for SVC there is no strong demand on doing signalling in a generic way.

This is a good summary of available alternatives. Let me recap:
1) middlebox able to parse all binary description -> impractical
2) extended group-semantic (RFC 3388)
3) nothing -> not acceptable

Let me add that there is not only layered or scalable coding (SVC) but 
also other coding techniques as multiple description coding (MDC).

The 1st option is impractical. Middleboxes should be as simple as possible.

The 3rd option is not acceptable. It is easy indeed. But it does not 
exploit available features. E.g. dropping the least important layer 
(SVC) or one description (MDC).

The 2nd option is viable. Not easy, but also not impossible to do.

Wenger suggested a simple generic technique to signal dependency among 
layers in draft-wenger-avt-rtp-svc-00.txt

The technique was based on tagging layers and on signalling for each 
layer the "parent" layer. This works for SVC where each layer has only 
one parent layer. The base layer havo no parent. For your convenient I 
enclose here the example of Wenger (slide #9):

   base L0
   |     \
   |      \
   L1:0    L2:0
   |  \
   |   \
   L3:1 L4:1
        |
        |
        L5:4

It would be useful to add another field (beyond the parent field) for 
the layer importance. In this way the middlebox can select the best 
child layer, for a given parent layer that has many childs.

Also, I think that it is possible to accomodate also MDC in a natural 
way because childs of a given layer can be seen as multiple descriptions 
of the ehnancement layer.

The only modification required (for sake of generality) is to let 
several base layer coexist.

The rest (what to do with a given layer) can be codec-dependent or not.

We tried to define a codec-independent way to support multiple layers 
with draft-vitali-ietf-avt-mdc-lc-00.txt.

However, as emerged during the 64th IETF meeting in Vancouver: "The 
proposed attributes are an example of general SDP signalling, much
as described earlier in the session by Stephan Wenger, and demonstrate
the complexity of defining codec independent signalling for multiple
description coding."

So we are thinking about simplifying the proposed scheme.

   Andrea

-- 
  ______________________________________________________________

   Andrea VITALI                            [email-masked]

   tel (+39) 039 603 7244             mobile (+39) 347 93 08 433
   fax (+39) 039 603 6129

   STMicroelectronics
   Centro Direzionale COLLEONI - Palazzo "DIALETTICA"
   20041 Agrate Brianza (MI) ITALY
  ______________________________________________________________

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Thomas Schierl" <[email-masked]>
Date: Fri, 2 Dec 2005 17:22:41 +0100

Andrea,

What I tried to say is, we could signal the dependency within SDP in a
generic way, but in the case of SVC that would be redundant information
since for SVC more detailed knowledge of the stream will be required for
doing real useful things like shown in the slides of Stephan.
In my opinion MDC is no reason for having generic signalling. Since there is
neither an international standard nor a widely accepted solution for MDC
available, instead of that basically different approaches are proposed
within research papers. Sorry, but I don't expect anything being
standardized in the next years. But that's my opinion only! Let us ask the
other people.

Nevertheless, there could be a reason doing the generic signalling also for
JPEG2000.
I want to ask this question now, better than having discussions too late in
the standardization process of the SVC payload format.
I'm not an expert with JPEG2000, but I want to ask the people working on
that draft:
Does it make sense to signal a layer dependency for JPEG2000 in a generic
way? 
If the answer should be YES, then we would have two codecs, which have
similar properties and could be used in a similar way. Thus generic
signalling could be useful.
Anyway extending the "group-semantics" would be enough for SVC and hopefully
for JPEG2000 as well.
But if the answer is NO, I can see no other use-cases, in which today's
scalable video codecs would strongly require the generic SDP signalling.

Thomas. 

> -----Original Message-----
> From: Andrea Lorenzo VITALI [mailto:[email-masked]]
> Sent: Friday, December 02, 2005 11:35 AM
> To: Thomas Schierl
> Cc: 'Magnus Westerlund'; 'Stephen Casner'; 'AVT WG'; 'Colin Perkins'
> Subject: Re: [AVT] Sigalling for layered media -- do it generic or do
> itpayloadspecific
> 
>

---

From: Andrea Lorenzo VITALI <[email-masked]>
Date: Fri, 02 Dec 2005 18:36:16 +0100

> generic way, but in the case of SVC that would be redundant information

Ok. I got your point. The same information (dependency) would be sent 
twice. Out-of-band via SDP and in-band in the bitstream.

However, let's say that we want to do packet dropping because of 
congestion. It would indeed be useful to have middlebox capable of 
dropping the least important packets (packets in the enhancement layer). 
And this middlebox should not be SVC-aware. Right? Or am I missing some 
point?

> neither an international standard nor a widely accepted solution for MDC

You are completely right. And that's why we want to propose a 
standard-compatible MDC approach done in the pixel domain. It could be 
easily standardized as it is not codec-dependent.

SVC gives you efficient scalability. MDC gives you error-resilience. It 
would be good to have the best of both.

With standard-compatible MDC you create subsequences that can then be 
coded in a scalable way, eg. using SVC. Remember that SVC is also 
backward compatible because the base layer is a standard H.264 stream.

While pushing for MDC we believe to be in the same track as SVC.

> way? If the answer should be YES, then we would have two codecs, which have

Sorry. I am not following you. Why two codecs? One based one SVC and one 
based on JPEG2000? You mean: to code the enhancement layer?

   STMicroelectronics

---

From: David R Oran <[email-masked]>
Date: Sun, 4 Dec 2005 11:13:31 -0500

On Dec 2, 2005, at 12:36 PM, Andrea Lorenzo VITALI wrote:

> Ok. I got your point. The same information (dependency) would be  
> sent twice. Out-of-band via SDP and in-band in the bitstream.
>
> dropping the least important packets (packets in the enhancement  
> layer). And this middlebox should not be SVC-aware. Right? Or am I  
> missing some point?
>
That's what drop precedence is for in various diffserv PHBs.

> standard-compatible MDC approach done in the pixel domain. It could  
> be easily standardized as it is not codec-dependent.
>
> SVC gives you efficient scalability. MDC gives you error- 
> resilience. It would be good to have the best of both.
>
> With standard-compatible MDC you create subsequences that can then  
> be coded in a scalable way, eg. using SVC. Remember that SVC is  
> also backward compatible because the base layer is a standard H.264  
> stream.
>
> Sorry. I am not following you. Why two codecs? One based one SVC  
> and one based on JPEG2000? You mean: to code the enhancement layer?
>

---

From: "Stephan Wenger" <[email-masked]>
Date: Sun, 4 Dec 2005 22:23:57 +0200

Hi David, folks,

Long email below.  Sorry.

-----Original Message-----
>That's what drop precedence is for in various diffserv PHBs.

I fully agree.  But David's answer needs a bit more explanation, I think.
We also need to get back to application layer work.

It's a common misconception in media committees and by media coding
researchers that a middlebox can meaningfully manipulate an RTP stream
without knowledge of the signaling.  Clearly, it can not.  One of the more
obvious reasons is that a middlebox needs to be able to associate the PT
field in the RTP header with an RTP payload format.  There are many more
reasons.
(Note that this paragraph uses US-English spelling rules of "signaling" :-)

Since a middlebox needs to intercept and interpret signalling, per-packet
signalling of the importance of packets is only necessary or helpful if a)
packets of different importance (belonging to different layers? Not
necessarily.  See H.264/RFC 3984) are conveyed in the same stream, and b)
diffserv is (for whatever reason) not available.
(Note that this paragraph follows imperial English spelling rules :-)

To conclude: let's not talk about media-unaware middleboxes.  They make no
sense.  Andrea: that's the prime reason why blasted your I-D in Vancouver --
lack of applications  There are many more reasons, though :-)

When it comes to media-aware middleboxes (that intercept signaling and can
interpret RTP headers, payload headers, and perhaps parts of the payload),
then some form of "priority", "importance", ... in an easily accessible form
makes perfect sense to me.  

Example: it's stupid to have a middlebox whose task is stream-thinning to
parse a bit stream to figure out how much intra information is in a slice,
so to conclude the importance of the slice for error recovery.  The encoder
knows.  If he has a couple of bits in an easily accessible location, let him
code the "importance", "Priority" in these bits.  That's what the NRI field
in H.264 and RFC 3984 is for, and something like that will also be available
in the SVC payload.  No doubt.

Where this information is defined (media spec, RTP payload, both) is not
really a design problem.  I suggest using the definition procedure that's
least painful for the standardization process.  For H.264, that has been
that JVT defined the bits and RFC3984 referenced the definition (and
informatively explained them).

So how much of this information is necessary?  That's payload dependent.  In
H.264, academic work (by myself, the VCIP 2000 conference paper) showed that
three levels were sufficient for the then valid working draft.  That's where
the two NRI bits came from.  We now have four levels, and no defined mapping
(and my old work is no more valid), but nevertheless, there was at some
point of time a reason for having 2 bits and not 7 or none whatsoever.  

I could imagine technically valid arguments for more than two bits in SVC.

I don't think the traditional MPEG-2 and H.263 Annex O style enhancement
layers (spatial and SNR scalability in the latter case) need any
"importance" or "priority" information.  In these cases, when you loose a
packet you are basically done.  No chance of recovery but intra.

To conclude: I don't think, the "priority", "importance' etc. is something
that should be specified in a media independent way.

Regards,
Stephan

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 05 Dec 2005 10:37:48 +0100

Hi Andrea,

Lets split out the MDC discussion from the signalling of layered for a 
while.

> While pushing for MDC we believe to be in the same track as SVC.

AVT's charter is explicit about that this WG is not doing codec or media 
definitions. The charter currently includes these sentences:

   "The group continues to be precluded from work on codecs
   themselves because of overlap with the other standards
   bodies, and because the IETF does not have the ability
   to effectively review new codecs. An exception was made
   for the freeware iLBC codec on a highly experimental basis,
   but acceptance of new codec work is unexpected and subject
   to rechartering."

To me it seems that defining pixel domain transformations suitable for 
MDC in the pixel domain is still highly media specific and requires good 
understanding of video as media plus codecs. I don't see that AVT have 
the expertise necessary for defining these transformations.

This is the reason I made the comment I made during the Vancouver 
meeting. If you have a set of MDC transformation standardized somewhere. 
Then we or rather MMUSIC is the correct venue for defining how these 
transforms and necessary parameters and relations are signalled using SDP.

Cheers

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]

---

From: Andrea Lorenzo VITALI <[email-masked]>
Date: Mon, 05 Dec 2005 15:08:54 +0100

I try to recap your mail and give my comments:

. Middleboxes cannot manipulate RTP streams without knowledge of the 
signaling. This is ok to me, I would dare to say that it is obvious.

. Per-packet signaling is useful if different packets are muxed in the 
same stream (as can be for H.264/RFC 3984) or if streams cannot be 
prioritized (diffserv is not available). This is also ok to me.

In conclusion: I am not talking about media-unaware middleboxes. Instead 
I am talking about a generic method to signal dependency and importance.

If you do not have such generic method, you are forced to implement 
payload specific methods. This was done in RFC 3984 for H.264.

> I could imagine technically valid arguments for more than two bits in SVC.

As you say, defining a payload specific way to signal dependency and 
importance can be a time consuming task. We have to reinvent the wheel 
every time. Also, it may become soon obsoleted as it is tightly bounded 
to the payload characteristics.

Generic signaling would have been better in this case.

Do you agree?

> packet you are basically done.  No chance of recovery but intra.

Almost but not entirely true. Here is the scheme of MPEG-2 scalable 
profile. IQ=inverse quantization, IT=inverse transform, P=prediction, 
UP=upsampling, TS=temporal/spatial switch.

Harmonized hybrid decoder.

Enhance --> IQ1 --> IT --> + -----o-----> high res/quality
                            ^      |
                            |      |
                           TS<-P <-+
                            ^
                            |
                           UP
                         /    \
                        A      B
                        |      |
   Base --> IQ2 --> IT -o-> + -o---o-----> low res/quality
                            ^      |
                            |      |
                            +- P <-+

If the TS switch is in its "temporal" position, enhancement pictures are 
predicted with motion-compensation in the enhancement layer. In this 
case when you loose an enhancement packet, you you get only the base 
quality until the next intra enhancement picture or until the TS switch 
is in its "spatial" position.

If the TS switch is in its "spatial" position, enhancement pictures are 
computed by upsampling pictures in the base layer. Upsampling can be 
done in the error prediction domain (A) or in the pixel domain (B). In 
this case when you loose an enhancement packet, only the current picture 
is affected: you get only the base quality.

> that should be specified in a media independent way.

I tried to explain that a media independent way can exist, and may be 
worth implementing and using.

   STMicroelectronics

---

From: Andrea Lorenzo VITALI <[email-masked]>
Date: Mon, 05 Dec 2005 15:22:57 +0100

> Lets split out the MDC discussion from the signalling of layered for a 

This is indeed useful.

Signaling dependence/importance is very different from defining pre- and 
post-processing techniques to be executed on multimedia data.

> the expertise necessary for defining these transformations.

Ok.

> transforms and necessary parameters and relations are signalled using SDP.

As I tried to explain in the meeting, I'm not defining any new 
transform. There is no need to standardize digital filters. Even if 
implementations may differ substantially, the result of filtering is 
very well defined given the coefficients.

As you point out, I came to you (AVT) just to know how to signal 
necessary parameters (coefficients) and relations (dependence, 
importance) using SDP.

   Andrea

   STMicroelectronics

---

From: "Thomas Schierl" <[email-masked]>
Date: Mon, 5 Dec 2005 16:08:33 +0100

Andrea,

Sure, it would be great having a universal, generic mode for signaling
priority in the transport part as well.
But wouldn't that require a new RTP version including a priority field
within the header?
Or a high level payload format for all sub-payload formats using one generic
header with priority-signaling?

Both alternatives are not the best choice.

The SVC payload format will signal the dependency of layers and the
importance of the RTP payload in a way that media aware middle-boxes will be
able to interpret this information in an easy, fast and efficient way. 
That should be our main goal, not creating an all-in-one solution suitable
for every purpose.

Thomas.

Andrea Lorenzo VITALI wrote on Monday, December 05, 2005 3:09 PM

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 05 Dec 2005 16:45:38 +0100

> very well defined given the coefficients.

But there is a need to standardize this. Just as we have profile and 
levels for individual video codecs someone needs to define which set of 
pre and post operations that are to be implemented. Some may be so 
complex that I have certain limitations to use them. I can't see how you 
simply can define operations without a framework defining what is 
expected to be supported in the implementations and which range of 
parameters are needed. Thus this also requires standardization to select 
the operations, which possible operation points that are needed.

Cheers

Magnus Westerlund

---

From: "Thomas Schierl" <[email-masked]>
Date: Mon, 5 Dec 2005 17:35:48 +0100

Stephan,

It's good to go on with the SVC discussion!

In my opinion the NRI field of H.264 will not be enough for signaling the
layer functionality of SVC on the payload level.
In the H.264 case (base layer) one of the four possible combinations is
already assigned to non-reference slices. In the SVC case, an additional
value is fixed, that is the one for key frames. There could be a way of
reassigning them within payload format, no doubt.
Burt anyway if a middle-box will be able to parse two bits in the first byte
of the header (NRI field of H.264), it also can parse the 6 bits of the
second byte of the payload header (simple priority id). 
If there should be really an instance, which is able to parse the payload
header, then it can also parse a little bit more of the header or a little
bit deeper into the packet's structure.

But that brings us to the point, where to position and/or how to signal the
SVC enhanced priority information within the RTP payload.
The H.264 payload format can't achieve signaling the extended SVC payload
header, since RFC 3984 is based on the one byte NAL Unit header and not on
the extended two or three byte header of SVC.

- One solution could be doing an implicit signaling of the highest priority,

  contained in a packet, using the first NAL Unit's extended header in an 
  Aggregation Packet, but that will not work for FUs and is possibly not 
  suitable for MTAPs. 
- However the single NAL Unit mode could be used, guaranteeing the 
  extended information in each payload header, while leaving the other 
  packet types unchanged.
- Last but not least, we have the way of reassigning the NRI for the payload

  format, but that would forbid using single NAL Units within such a 
  packetization mode, since the values of the SVC standard would not match 
  the values of the payload format.

That will be a part of the discussion for the next months.      

Thomas.

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Stephan Wenger
> Sent: Sunday, December 04, 2005 9:24 PM
> To: 'David R Oran'; 'Andrea Lorenzo VITALI'
> Cc: 'Magnus Westerlund'; [email-masked]; 'Stephen Casner'; 'AVT
> WG'; [email-masked]; 'Colin Perkins'; [email-masked]
> Subject: RE: [AVT] Sigalling for layered media -- do itgeneric or do
> itpayload specific
> 
> Hi David, folks,
> 
> Long email below.  Sorry.
>

---

From: Andrea Lorenzo VITALI <[email-masked]>
Date: Tue, 06 Dec 2005 15:54:18 +0100

> pre and post operations that are to be implemented. Some may be so 

Perfect.

This is exactly what is written in the proposed draft:
draft-vitali-ietf-avt-mdc-lc-00.txt.

> complex that I have certain limitations to use them. I can't see how you 

Complexity is an issue that I raised in the meeting. The proposed 
framework can indeed be simplified.

> the operations, which possible operation points that are needed.

I agree.

   STMicroelectronics

---

From: Colin Perkins <[email-masked]>
Date: Tue, 6 Dec 2005 15:03:11 +0000

On 6 Dec 2005, at 14:54, Andrea Lorenzo VITALI wrote:
>> But there is a need to standardize this. Just as we have profile  
>> and levels for individual video codecs someone needs to define  
>> which set of pre and post operations that are to be implemented.  
>> Some may be so
>
> Perfect.
>

Quite.  And that is where the objection arise, since the AVT charter  
forbids us to develop media coding standards. Our expertise is in  
transport protocols, not media transforms.

Colin

---

From: Andrea Lorenzo VITALI <[email-masked]>
Date: Tue, 06 Dec 2005 16:14:40 +0100

This is not "coding" in any way.

We are not doing source coding (i.e. not doing compression).
We are not doing channel coding (i.e. not doing FEC).

In the proposed framework it is only specified how to "shuffle" the data 
prior coding. This is why it is codec-independent and backward compatible.

E.g. in its simplest scheme, two descriptions are created by separating 
odd and even lines from each picture of a video sequence. Two 
subsequences are created, having half height and the same width. They 
are then coded, transmitted and decoded independently.

Robustness to packet loss comes from the fact that it is unlikely to 
loose the same portion of the same picture in both subsequences.

>>> levels for individual video codecs someone needs to define  which set 
>>> of pre and post operations that are to be implemented.  Some may be so
>>
>>
>> Perfect.
>>

   STMicroelectronics

---

From: [email-masked]
Date: Tue, 6 Dec 2005 16:31:42 +0100 (CET)

Hi Andrea,

we did read your draft and we understand quite well what you want to do. 
Yes, this may not be source "coding" in the sense of an entropy code. 
However, my main objection is that the meat of your proposal lies in the
pixel domain and not in the transport of bytes -- and I keep it up.

Your draft does not fit AVT.  It would reasonably well fit into the work
of MPEG, the JVT, or ITU-T's VCEG-group.  There are the people who can
competently tell you that your proposal is not up to par with traditional
multiple decription and layered coding schemes, and how to modifiy it so
that it becomes competitive.  Maybe they even appreciate the flexibility
of your technology.  But taking the back door in the IETF -- no.

Give it up.  Both chairs have advised you your work is not within AVT
scope.  You got fairly clear advise duringthe AVT meeting in Vancouver. 
What else do you need?

Sorry for the direct language, but your postings start to be annoying --
at least to a short-fused person like me.

Regards,
Stephan

---

From: Andrea Lorenzo VITALI <[email-masked]>
Date: Tue, 06 Dec 2005 16:53:54 +0100

No problem for the direct language. If it is done to speed up conversation.

You see: the proposed piece of technology is in the middle. It is not 
coding, so it doesn't fit MPEG. And it is not AVT stuff, you say.

Please, understand that we are not taking the backdoor. We are seeking 
the way to put the technology in place.

Let's have a constructive discussion.

We want to deploy this technology. What's the right way to put it into 
place in a way that is open to other contributors?

   Andrea

PS I will try to keep my reply as short as possible to match your short 
fuse, right? However, you should allow for discussion.

   STMicroelectronics

---

From: "Stephan Wenger" <[email-masked]>
Date: Tue, 6 Dec 2005 18:46:44 +0200

Hi Andrea,

I understand your dilemma.  Your technology is not fully in the scope of
AVT, and not fully in the scope of other organizations (say: MPEG, for
example).  This has happened before; actually, it's a quite common situation
in standardization.  Two solutions:

a) split it up.  Take the pixel oriented part to MPEG.  They are working on
subjects like this, and have embraced similar technologies (that are note
quite in their main line of thinking) quite more openly than the IETF does.
The IETF comes into play with a traditional payload format and signaling for
it.  The "framework" work needs to be done in MPEG, though.

After thinking it through, I don't believe that VCEG or JVT are options.
Their respective focus is too narrow.  But perhaps the NGN guys would be
interested? I believe your company is somewhat involved in this work.

b) Rally up a sufficient number of supporters and press for re-chartering
the AVT working group, to allow this type of work.  If it's free technology,
the open source community would be a good starting point.  It has happened
before -- once.  Talk to the Vorbis guys.  If this is not free technology,
forget it.  But the fact that you have not made a patent statement no your
technology so far makes me hopeful.

Regards,
Stephan

-----Original Message-----
From: Andrea Lorenzo VITALI [mailto:[email-masked]] 
Sent: Tuesday, 06 December, 2005 17:54
To: [email-masked]
Cc: Magnus Westerlund; Colin Perkins; 'AVT WG'
Subject: Re: [AVT] Pixel domain MDC

   Andrea

   STMicroelectronics