Subject: Re: [AVT] Clock Skew update issue for layered codecs

From: "Ingemar Johansson S" <[email-masked]>
Date: Wed, 10 Dec 2008 08:12:02 +0100

Hi

The idea to use header extensions with NTP time stamps was discussed
earlier 
http://www.ietf.org/mail-archive/web/avt/current/msg10366.html

One of the issues discussed then was that header extensions may be
removed by middle boxes. 
Another issue  may be that RTP stack may not recognize the header
extensions but I believe belongs to the same ball park as the "crappy
RTP stacks" arguments. 
A third argument was that it is difficult/impossible to add several
header extensions but isn't that solved with RFC5285 ?

So how big is this issue with middle boxes ?

Even though I originally liked header extensions more, one thing that
also needs to be considered is channel switching. If timestamp alignment
is used it will be possible to render all layers immediately after a
channel switch. If header extensions are used it will be necessary use
only the base layer until the next package of header extensions for all
layers arrive at the receiver. If header extensions ae transmitted often
then this is probably not a big issue.

Regards
/Ingemar

> ------------------------------
> 
> Message: 6
> Date: Tue, 09 Dec 2008 16:22:59 +0100
> From: Thomas Schierl <[email-masked]>
> Subject: Re: [AVT] Clock Skew update issue for layered codecs
> To: Magnus Westerlund <[email-masked]>
> Cc: Colin Perkins <[email-masked]>, IETF AVT WG <[email-masked]>
> Message-ID: <[email-masked]>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> 
> 
> 
> Magnus Westerlund wrote:
> > Colin Perkins skrev:
> >   
> >> [inline]
> >>
> >> On 3 Dec 2008, at 18:19, Magnus Westerlund wrote:
> >>     
> >>> I wanted to start a separate thread on this particular issue with
> >>> multi-layered codecs.
> >>>
> >>> Lets assume that we have 3 layers produced by the media 
> codec from a
> >>> single media source (microphone). We have some clock skew 
> on the audio
> >>> card sampling clock that will need to be communicated at 
> least now and
> >>> then to keep this under control.
> >>>
> >>> The transmission of RTCP SR will happen at regular 
> intervals in each
> >>> layer but the jittering will make this transmission into a random
> >>> process on the short time scale.
> >>>
> >>> In the below figure each x is an RTCP packet, the y is 
> when the RTCP SR
> >>> information has been updated for the clock skew.
> >>>
> >>>           123456789ABCDE
> >>> A ---x----x----y--y----y
> >>>
> >>> B -x----x--x----y---y---
> >>>
> >>> C --x--x---x--x---y----y
> >>>
> >>> So the big question here is when a receiver can take the 
> clock skew
> >>> compensation into consideration and apply it to the 
> actual playout?
> >>>
> >>> A time 1 all layers have the same synch information. At 
> time 6 layer A
> >>> receives the clock skew update in the RTCP SR. At this 
> point it could
> >>> apply it to this layer. However, to avoid misaligning the 
> audio frames
> >>> the compensation needs to happen at all layers. To me it 
> appears that
> >>> can be done in two ways either wait until you have seen 
> it in all layers
> >>> or do it as soon as you see it in one layer. And the 
> first doesn't work
> >>> as one can't be certain that all layers will provide the 
> same update to
> >>> all layers because of the following situation:
> >>>
> >>>           123456789ABCDE
> >>> A ---x----x----y--z----z
> >>>
> >>> B -x----x--x----y---z---
> >>>
> >>> C --x--x---x--x---z----z
> >>>
> >>> So in this case the C layer never transmit an RTCP packet 
> with the y
> >>> clock skew correction because a new one is needed an 
> indicated by z. As
> >>> this can continue forever there is no way waiting until 
> it has been seen
> >>> in all layers. Thus one must use the latest adjustment.
> >>>       
> >> I'm not sure I understand. For a layered codec, all the layers will
> >> share a common media clock, and will have identical skew 
> relative to the
> >> NTP format wall clock. Why can't the mapping conveyed by 
> RTCP packet z
> >> be used?
> >>     
> >
> > It can as long as it is applied on all layers at the time a 
> first RTCP
> > packet with information z is received. That is one of the 
> points here
> > that I do tried to make. You need to be able to apply the 
> clock skew on
> > all layers.
> >
> > If you try to apply the RTCP synch in each layer 
> individually then you
> > can't correctly map an RTP packet to the common NTP clock. 
> Lets assume
> > that a video frame was captured at NTP time 8. At this 
> point layers A
> > and B have updated their RTP TS to NTP mapping to reflect Y 
> while layer
> > C has not and still use X as the mapping. When resolving 
> the NTP time
> > from the different RTP timestamps in the RTP packet you 
> will arrive at
> > different NTP times and potentially determine that they are not
> > representing the same frame, but different frames.
> >
> >   
> >>> Frankly, I think aligning the timestamp values between 
> the layers to
> >>> simplify the clock skew compensation is the only 
> reasonable solution.
> >>>       
> >> It's one solution, certainly. The other possibility is to 
> ensure that
> >> the mapping from NTP format wall clock to the RTP media 
> clock is sent at
> >> exactly the same time (i.e. identical NTP format 
> timestamps) for each
> >> layer, perhaps by including it as a header extension on 
> RTP data packets.
> >>
> >>     
> >
> > Yes, that is also one possibility, as long as you resolve 
> the issue that
> > you can correctly map the RTP timestamp to the NTP clock 
> even when doing
> > clock skew compensations and do this commonly over all 
> layers that you
> > need to synchronize.
> >   
> This is possible, since such an NTP timestamp in the header extension 
> would be connected to the RTP timestamp in the RTP header. 
> The important 
> point is to mandate that these NTP header extension is sent 
> for the same 
> NTP timestamp in all the sessions of the layered media, 
> otherwise this 
> approach does not help at all.
> 
> -- 
> Thomas Schierl
> --------------
> Fraunhofer HHI
> 
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: <[email-masked]>
Date: Wed, 10 Dec 2008 09:44:43 +0200

Yes, we are circling :-), but in a good manner - getting more (and
important :-) people to think more, and hopefully a converging solution
is coming out. 

BR, YK 

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On 
>Behalf Of ext Ingemar Johansson S
>Sent: 10 December, 2008 09:12
>To: [email-masked]
>Cc: Magnus Westerlund; [email-masked]

---

From: Dave Singer <[email-masked]>
Date: Thu, 11 Dec 2008 09:53:27 +0900

one would surely put the header extensions in
a) using the new RFC
b) on all streams for the same time
c) on those times where tune-in or stream switching could happen
d) as a 'duplicate' of regular sync in RTCP

-- 
David Singer
Multimedia Standards, Apple Inc.
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 12 Dec 2008 13:48:22 +0100

Hi,

With my individual contributor hat on.

> d) as a 'duplicate' of regular sync in RTCP

and e) when clock skew compensation is done in RTCP.

We are definitely going to need to discuss how this is implemented in
RTCP. Due to the asynchronous behavior of RTCP an implementor do need to
take care when handling multi-layer in a completely different way then
single layer streams have needed.

I am still a bit doubtful on the reasons that we choose header
extensions over aligning timestamps for the layers.

RTP Timestamp alignment vs Header extensions:

- Timestamp alignment have 0 overhead compared to 8/12/16 bytes for
header extensions in each packet it is included.

- Timestamp alignment will allow for alignment in each media frame/unit
also across any clock skew adjustments or any changes where header
extension if always included can do the same. But to save overhead
periodic and event driven inclusion is likely, especially for less
bandwidth intensive streams like audio where individual layers may be as
small as 4-8kbps.

- Both fit the RTP model for time synchronization to my knowledge about it.

- Header extensions forces one to think about NTP time in the first
order where timestamp alignment only does this in the second order when
one implements RTCP for all the other reasons, like lip sync between
audio and video.

So I am worried that we are doing something unnecessary complicated here
just because we are scared that some will skip implementing RTCP. My
belief is that the ones that doesn't want to implement RTCP will
continue to do that and soon learn that they do need it anyway. A few
will actually not needed and the RTP framework was never intending to
push that one those.

Thus I want to do a check that I am not missing a vital reason why a
header extension would be a better solution. But to me it doesn't seem
that currently and would recommend that we select an RTP timestamp
alignment solution as the one we document as basis for all layered
codecs that needed it and doesn't have more advanced needs.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: <[email-masked]>
Date: Sat, 13 Dec 2008 11:30:43 +0200

I personally prefer the RTP timestamp alignment solution too, as seen from many earlier discussions. If would be very nice if a concensus can be achieved soon, as we are finalizing the SVC payload format. 

BR, YK 

>Behalf Of ext Magnus Westerlund
>Sent: 12 December, 2008 14:48
>To: Dave Singer
>Cc: Ingemar Johansson S; [email-masked]; [email-masked]

---

From: Colin Perkins <[email-masked]>
Date: Sat, 13 Dec 2008 10:38:38 +0000

On 12 Dec 2008, at 12:48, Magnus Westerlund wrote:
> We are definitely going to need to discuss how this is implemented  
> in RTCP. Due to the asynchronous behavior of RTCP an implementor do  
> need to take care when handling multi-layer in a completely  
> different way then single layer streams have needed.
>
> - Timestamp alignment will allow for alignment in each media frame/ 
> unit also across any clock skew adjustments or any changes where  
> header extension if always included can do the same. But to save  
> overhead periodic and event driven inclusion is likely, especially  
> for less bandwidth intensive streams like audio where individual  
> layers may be as small as 4-8kbps.
>

I disagree that both fit the RTP synchronisation model, for the  
reason you mention below:

> order where timestamp alignment only does this in the second order  
> when one implements RTCP for all the other reasons, like lip sync  
> between audio and video.

But the relation to NTP time is key to the RTP synchronisation model.  
Using an RTP header extension including an NTP timestamp means that  
you use exactly the same synchronisation algorithm no matter whether  
you synchronise using the data provided in the header extension, or  
using RTCP SR information. The only difference is the time taken to  
converge.

Using aligned timestamps changes that: you need additional, out of  
band, information to know that some streams are pre-synchronised  
based on their RTP timestamps, while others need relating to NTP time.

> So I am worried that we are doing something unnecessary complicated  
> here just because we are scared that some will skip implementing  
> RTCP. My belief is that the ones that doesn't want to implement  
> RTCP will continue to do that and soon learn that they do need it  
> anyway. A few will actually not needed and the RTP framework was  
> never intending to push that one those.

I'm pushing it because I'd prefer one synchronisation algorithm to be  
used everywhere, even if we have several different ways of providing  
the data that algorithm needs to work, rather than having two  
different ways of doing synchronisation depending on what is being  
synchronised.

Similarly, I think aligned SSRC values across layers (section 8.3 of  
RFC 3550) are a mistake, again because they take a uniform mechanism,  
and add a special case.

> Thus I want to do a check that I am not missing a vital reason why  
> a header extension would be a better solution. But to me it doesn't  
> seem that currently and would recommend that we select an RTP  
> timestamp alignment solution as the one we document as basis for  
> all layered codecs that needed it and doesn't have more advanced  
> needs.

Defining a myriad of special cases that solve today's problem but  
need to be extended for "more advanced needs" is, IMHO, the way to  
kill a protocol in the long term (e.g. look at the mess that is SDP).  
Using an RTP header extension here generalises the existing  
synchronisation mechanisms, making the current algorithms useful in a  
wider range of circumstances. Aligned timestamps introduce a special  
case of limited utility, slightly simplifying today's  
implementations, but making the protocol as a whole more complex.

-- 
Colin Perkins
http://csperkins.org/

---

From: Dave Singer <[email-masked]>
Date: Mon, 15 Dec 2008 09:01:48 -0800

>and e) when clock skew compensation is done in RTCP.

e) when clock skew compensation is done

(since the same logical information is sent in 
RTCP and header extensions, the same clock 
calculations are applied.  The point of the 
header extensions is that the information can be 
both co-located with tune-in or stream-switching 
points, and can be placed on aligned media 
packets for scalable coding).

I think your text below may reflect a coding 
style -- where RTCP is implemented at arms-length 
from RTP -- which was true in say VIC/VAT but not 
very common today...

---

From: Thomas Schierl <[email-masked]>
Date: Tue, 16 Dec 2008 11:17:18 +0100

Dave Singer wrote:
> (since the same logical information is sent in RTCP and header 
> extensions, the same clock calculations are applied.  The point of the 
> header extensions is that the information can be both co-located with 
> tune-in or stream-switching points, and can be placed on aligned media 
> I think your text below may reflect a coding style -- where RTCP is 
> implemented at arms-length from RTP -- which was true in say VIC/VAT 
> but not very common today...

Let us assume this for a while. Using the currently proposed header 
extension, since it somehow solves the RTP problem of not using the same 
reference clock point for all media sessions, may let people rely for 
some use cases on the header extension only instead of using RTCP at all.

Header extension may solve the weakness of the non-exact RTP 
synchronization, which has to be fixed anyway. And from this point of 
view, I really like this solution. Since even the currently proposed 
header extension may be not 100% aligned if having two different media 
types like video and audio, some one may think about putting more 
synchronization information into one RTP stream. This may be e.g., 
additionally carrying the corresponding RTP timestamps of all the other 
sessions to be synchronized in a session. This may be the only way to do 
it 100% right. One and the same reference clock for all the streams.

-- 
Thomas Schierl
--------------
Fraunhofer HHI

---

From: [email-masked]
Date: Tue, 16 Dec 2008 11:27:03 +0100

I want to notify you, that Fraunhofer HHI will send an IPR statement on 
the header extension discussion thread in the next days.

Best regards,
Thomas Schierl

-- 
______________________________________________
Thomas Schierl, Dipl.-Ing.
Fraunhofer HHI
Senior Research Engineer
Image Processing Dept. / Multimedia Transport
Einsteinufer 37, 10587 Berlin, Germany
PHONE: +49 30 31002 227 , FAX: +49 30 392 7200
WEB: http://ip.hhi.de , SKYPE: thomas.schierl
EMAIL: [email-masked]

---

From: Franceschini Guido <[email-masked]>
Date: Tue, 16 Dec 2008 11:46:19 +0100

Hi all,

Let's assume that we use Header Extension for carrying the information to exactly correlate the layers. Then, having this information represented by the NTP is, let me say, nice but useless.
I mean, any correlation info would be valid, not just the NTP.
In practice indeed, a receiver implementation would just care the NTP carried in the header extension, not those carried in the RTCP. That's why I say that referring the same wallclock in header extension and RTCP is useless: we just need an information element (in the header extension) that is capable of exactly align all layers. E.g., an equally valid data is the "native" timestamp (by native timestamp I mean here the TS before applying the random offset for RTP transmission).

Also, let me remind that:
- NTP in RTCP serve the purpose of *approx* sync between media before *rendering*
- the issue on the table refers to *exact* alignment before *decoding*

Having said that, I observe that a range of solutions is currently on the table:
- TS alignment
- RTP header extension carrying NTP
- RTP header extension carrying other data (e.g.: native TS)

In my opinion, these solutions are all nearly equivalent when considering direct TX-RX connections, since all of them require some specific signaling (explicit or implicit) and some addition on an RTP stack implementation.

The comparison might become more interesting when considering paths including mid-boxes.
Indeed, those mid-boxes would be required to either preserve TS alignment or to propagate RTP Header Extension (and in this case also to possibly inspect and modify their content)

I think some investigation in this field would be helpful.

Best regards
Guido Franceschini

------------------------------------------------------------------
Telecom Italia
Guido FRANCESCHINI
Broadband Solutions and Multimedia Innovation
Via Guglielmo Reiss Romoli, 274 10148 Torino
office   +39 011 2286137
mobile +39 335 7669748

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Thomas Schierl
> Sent: martedì 16 dicembre 2008 11.17
> To: Dave Singer
> Cc: Ingemar Johansson S; Magnus Westerlund; [email-masked];
> [email-masked]
> here

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

Rispetta l'ambiente. Non stampare questa mail se non e' necessario.

www.avoicomunicare.it      Ogni giorno, il tuo luogo di dialogo.
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Colin Perkins <[email-masked]>
Date: Tue, 16 Dec 2008 11:34:23 +0000

On 16 Dec 2008, at 10:46, Franceschini Guido wrote:
> Let's assume that we use Header Extension for carrying the  
> information to exactly correlate the layers. Then, having this  
> information represented by the NTP is, let me say, nice but useless.
> In practice indeed, a receiver implementation would just care the  
> NTP carried in the header extension, not those carried in the RTCP.  
> That's why I say that referring the same wallclock in header  
> extension and RTCP is useless: we just need an information element  
> (in the header extension) that is capable of exactly align all  
> layers. E.g., an equally valid data is the "native" timestamp (by  
> native timestamp I mean here the TS before applying the random  
> offset for RTP transmission).
>

No - they can both give you exact synchronisation. The only  
difference is the pre-synchronisation delay: you need to receive  
several RTCP SR packets before you have enough information for exact  
synchronisation, whereas using NTP-format timestamps in a header  
extension allows you to synchronise as soon as you've received one  
NTP timestamp for each layer.

> In my opinion, these solutions are all nearly equivalent when  
> considering direct TX-RX connections, since all of them require  
> some specific signaling (explicit or implicit) and some addition on  
> an RTP stack implementation.
>
> Indeed, those mid-boxes would be required to either preserve TS  
> alignment or to propagate RTP Header Extension (and in this case  
> also to possibly inspect and modify their content)
>
>>>> RTCP. Due to the asynchronous behavior of RTCP an implementor do  
>>>> need
>> to
>>>> bandwidth intensive streams like audio where individual layers  
>>>> may be
>> as
>>>> small as 4-8kbps.
>>>>
> Questo messaggio e i suoi allegati sono indirizzati esclusivamente  
> alle persone indicate. La diffusione, copia o qualsiasi altra  
> azione derivante dalla conoscenza di queste informazioni sono  
> rigorosamente vietate. Qualora abbiate ricevuto questo documento  
> per errore siete cortesemente pregati di darne immediata  
> comunicazione al mittente e di provvedere alla sua distruzione,  
> Grazie.
>

-- 
Colin Perkins
http://csperkins.org/

---

From: Stefan Döhla <[email-masked]>
Date: Tue, 16 Dec 2008 13:41:23 +0100

Colin,

plese see inline question.

- Stefan

Colin Perkins schrieb:
>> In practice indeed, a receiver implementation would just care the NTP 
>> carried in the header extension, not those carried in the RTCP. 
>> native timestamp I mean here the TS before applying the random offset 
>> for RTP transmission).
>>
> No - they can both give you exact synchronisation. The only difference 
> is the pre-synchronisation delay: you need to receive several RTCP SR 
> packets before you have enough information for exact synchronisation, 
> whereas using NTP-format timestamps in a header extension allows you 
> to synchronise as soon as you've received one NTP timestamp for each 
> layer.
What do you mean by pre-synchronisation delay? I guess you mean that the 
data needs to be filtered to give an *exact* alignment - but then the 
question is, how many SRs do you need until you are sample-accurate 
(which is the coarsest granularity for layered codecs)?
>> considering direct TX-RX connections, since all of them require some 
>> specific signaling (explicit or implicit) and some addition on an RTP 
>> stack implementation.
>>
>> alignment or to propagate RTP Header Extension (and in this case also 
>> to possibly inspect and modify their content)
>>
>>>>> when
>> alle persone indicate. La diffusione, copia o qualsiasi altra azione 
>> derivante dalla conoscenza di queste informazioni sono rigorosamente 
>> vietate. Qualora abbiate ricevuto questo documento per errore siete 
>> cortesemente pregati di darne immediata comunicazione al mittente e 
>> di provvedere alla sua distruzione, Grazie.
>>

-- 
Dipl.-Ing. Stefan Döhla     | Email: [email-masked]
Multimedia Transport        | Phone: +49 (0)9131 776 6042 (!NEW!)
Audio Department            | Fax:   +49 (0)9131 776 6099 (!NEW!)
Fraunhofer IIS              |
Am Wolfmantel 33            |
91058 Erlangen, Germany     | Web:   http://www.iis.fraunhofer.de/amm

---

From: Colin Perkins <[email-masked]>
Date: Tue, 16 Dec 2008 14:31:21 +0000

On 16 Dec 2008, at 12:41, Stefan Döhla wrote:
>>> NTP carried in the header extension, not those carried in the  
>>> RTCP. That's why I say that referring the same wallclock in  
>>> header extension and RTCP is useless: we just need an information  
>>> element (in the header extension) that is capable of exactly  
>>> align all layers. E.g., an equally valid data is the "native"  
>>> timestamp (by native timestamp I mean here the TS before applying  
>>> the random offset for RTP transmission).
>>>
>> several RTCP SR packets before you have enough information for  
>> exact synchronisation, whereas using NTP-format timestamps in a  
>> header extension allows you to synchronise as soon as you've  
>> received one NTP timestamp for each layer.
> What do you mean by pre-synchronisation delay? I guess you mean  
> that the data needs to be filtered to give an *exact* alignment -

Right.

> but then the question is, how many SRs do you need until you are  
> sample-accurate (which is the coarsest granularity for layered  
> codecs)?

That depends on the clock skew and resolution, the network jitter,  
and the RTCP reporting interval. The more data you have, the more  
accurate your estimate of the clock skew, and so the better you can  
synchronise the flows. For lip-sync, you can quickly get a good  
enough estimate, I suspect it's too slow for layered codecs (except  
as a fall-back if the RTP header extensions containing the mapping  
are lost).

Distributing the mapping mapping between media clock and wall-clock  
in an RTP header extension in each layer, at the same instant, gives  
you an exact synchronisation no matter how poor your estimate of the  
skew. That speeds things up, but you use the same synchronisation  
algorithm in both cases.

-- 
Colin Perkins
http://csperkins.org/