Subject: [AVT] Clock Skew update issue for layered codecs

From: Magnus Westerlund <[email-masked]>
Date: Wed, 03 Dec 2008 19:19:38 +0100

Hi,

I wanted to start a separate thread on this particular issue with
multi-layered codecs.

Lets assume that we have 3 layers produced by the media codec from a
single media source (microphone). We have some clock skew on the audio
card sampling clock that will need to be communicated at least now and
then to keep this under control.

The transmission of RTCP SR will happen at regular intervals in each
layer but the jittering will make this transmission into a random
process on the short time scale.

In the below figure each x is an RTCP packet, the y is when the RTCP SR
information has been updated for the clock skew.

          123456789ABCDE
A ---x----x----y--y----y

B -x----x--x----y---y---

C --x--x---x--x---y----y

So the big question here is when a receiver can take the clock skew
compensation into consideration and apply it to the actual playout?

A time 1 all layers have the same synch information. At time 6 layer A
receives the clock skew update in the RTCP SR. At this point it could
apply it to this layer. However, to avoid misaligning the audio frames
the compensation needs to happen at all layers. To me it appears that
can be done in two ways either wait until you have seen it in all layers
or do it as soon as you see it in one layer. And the first doesn't work
as one can't be certain that all layers will provide the same update to
all layers because of the following situation:

          123456789ABCDE
A ---x----x----y--z----z

B -x----x--x----y---z---

C --x--x---x--x---z----z

So in this case the C layer never transmit an RTCP packet with the y
clock skew correction because a new one is needed an indicated by z. As
this can continue forever there is no way waiting until it has been seen
in all layers. Thus one must use the latest adjustment.

If one is relying on the individual layers RTCP SR to align the audio
frames in each layer one suddenly have a issue. Does an RTCP SR packet
in one layer adjust the clock skew or provide an accurate point of
aligning the frames? The only way of getting out of this bind I think
would be to rely very carefully on the provided clocks and really check
if the timestamp values align perfectly. And in cases the clock skew is
exactly one frame length one needs to adjust the compensation to not align.

Frankly, I think aligning the timestamp values between the layers to
simplify the clock skew compensation is the only reasonable solution.

Cheers

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www.ietf.org/mailman/listinfo/avt

---

From: Colin Perkins <[email-masked]>
Date: Tue, 9 Dec 2008 14:38:45 +0000

[inline]

> in all layers. Thus one must use the latest adjustment.

I'm not sure I understand. For a layered codec, all the layers will  
share a common media clock, and will have identical skew relative to  
the NTP format wall clock. Why can't the mapping conveyed by RTCP  
packet z be used?

> simplify the clock skew compensation is the only reasonable solution.

It's one solution, certainly. The other possibility is to ensure that  
the mapping from NTP format wall clock to the RTP media clock is sent  
at exactly the same time (i.e. identical NTP format timestamps) for  
each layer, perhaps by including it as a header extension on RTP data  
packets.

-- 
Colin Perkins
http://csperkins.org/

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 09 Dec 2008 16:05:47 +0100

Colin Perkins skrev:
> share a common media clock, and will have identical skew relative to the
> NTP format wall clock. Why can't the mapping conveyed by RTCP packet z
> be used?

It can as long as it is applied on all layers at the time a first RTCP
packet with information z is received. That is one of the points here
that I do tried to make. You need to be able to apply the clock skew on
all layers.

If you try to apply the RTCP synch in each layer individually then you
can't correctly map an RTP packet to the common NTP clock. Lets assume
that a video frame was captured at NTP time 8. At this point layers A
and B have updated their RTP TS to NTP mapping to reflect Y while layer
C has not and still use X as the mapping. When resolving the NTP time
from the different RTP timestamps in the RTP packet you will arrive at
different NTP times and potentially determine that they are not
representing the same frame, but different frames.

> the mapping from NTP format wall clock to the RTP media clock is sent at
> exactly the same time (i.e. identical NTP format timestamps) for each
> layer, perhaps by including it as a header extension on RTP data packets.
> 

Yes, that is also one possibility, as long as you resolve the issue that
you can correctly map the RTP timestamp to the NTP clock even when doing
clock skew compensations and do this commonly over all layers that you
need to synchronize.

Cheers

Magnus Westerlund

---

From: Thomas Schierl <[email-masked]>
Date: Tue, 09 Dec 2008 16:22:59 +0100

> clock skew compensations and do this commonly over all layers that you
This is possible, since such an NTP timestamp in the header extension 
would be connected to the RTP timestamp in the RTP header. The important 
point is to mandate that these NTP header extension is sent for the same 
NTP timestamp in all the sessions of the layered media, otherwise this 
approach does not help at all.

-- 
Thomas Schierl
--------------
Fraunhofer HHI