Subject: [AVT] Review of draft-brandenburg-avt-rtcp-for-idms-02

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Wed, 24 Nov 2010 11:00:05 -0800

I have several editorial and architectural comments.

Abstract: "watching apart together" . Is this a common term?

Section 1.2: 
First para.: RTCP is supposed to be used whenever RTP is used. So saying they are typically used in conjunction is misleading.

Third para.: Put "respectively" after [RFC3550].

Section 1.3:
Put  a reference for RFC 4566.

Section 4:
First para.: The reference for "Feedback Target" should be 5760 not 3550.

The "shall" requirement in Block Type and Reserved Bits should be "SHALL".

Page 7 (top): The reference for RTP/RTCP is 3550 not 3611.

Payload Type field is said to use the types from 3551. And you assume each type has a certain clock rate. But most applications use dynamic payload types and for them you need to get the clock rate from the SDP itself.

Packet Presented NTP timestamp: I really don't know how you could know this information. In the RTP plane, you can only know when you delivered the packet to higher layer. But that has nothing to do with when it will be decoded/displayed/played/etc. Even if we try to synch the times that a packet is delivered to the higher layer, shall we assume everybody runs the same hardware with the same configuration so that the chances of having all the viewers seeing the same content are somewhat >0?

Beyond that, another question I have is that what happens when a viewer detects that it is running ahead of other viewers. What does he do? Slow down, stop for a while? Similarly, if one is lagging behind, how we do make him go faster to catch up with others (more accurately to the reference point determined by the MSAS)?

Even if we make this work, there could be very funny attacks that even friends can run against each other. In Section 6, you mention a threshold of 10 seconds, but to me even that is very high for most apps.

Finally, when there are multiple temporally correlated streams (audio + video) that are transmitted in separate RTP streams, I guess you run this stuff only for one of them. Correct? 

-acbegen

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 26 Nov 2010 11:44:46 +0100

Hi,

As Ali provided his feedback on this I can provide my high level points
about the proposal and what I think needs fixing if one goes forward
with an IETF version. I think it makes sense to produce a specification
for this.

1. Use the appropriate RTP and RTCP mechanisms. RTCP XR is a good match
for the report about how the reception and playback point are in
relation to NTP from the receiver to the master. However, information on
when a particular RTP TS should be played in relation to the master
clock I don't find RTCP XR a suitable mechanism. I see that a new RTCP
message type for dynamic configuration of playback, or an RTP header
extension seems a better match.

2. There is need to have a architectural discussion around how the
common clock is distributed to the session participants. First of all
different NTP servers have different precision and different path
characteristics. In addition there is in fact different clocks being
distributed on different servers and their lower stratas. For example
here in sweden there is swedish time produced by a governmental agency.
Yes, it has a certain relation to the UTC, and the clocks used for
swedish time do contribute to UTC. But there exact relation can only be
known after the fact.

As one goes down the stratas it is likely that one has bigger
variations. Adding in the path variations it is quite likely that
peoples clock are actually off with more than a few ms (which seems to
be a reasonable precision requirement) even if running NTP. Thus there
might be need to discuss what a suitable architecture are and how to
handle clock sources. Should one run against a common clock, or should
one try to determine what the differences are etc.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Fri, 26 Nov 2010 06:17:10 -0500

> with an IETF version. I think it makes sense to produce a specification
Btw I looked at the etsi spec which does not have anything this draft does not, either. So I don't think the IETF version, if goes forward, should cite etsi, especially if the changes will not be adopted by etsi. But before that there are issues that need to be addressed. 

-acbegen
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www.ietf.org/mailman/listinfo/avt

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Sun, 28 Nov 2010 04:46:01 -0800

> -----Original Message-----
> From: Ishan Vaishnavi [mailto:[email-masked]]
> Sent: Sunday, November 28, 2010 4:44 AM
> To: Ali C. Begen (abegen)
> Cc: [email-masked]; [email-masked]
> Subject: Re: Review of draft-brandenburg-avt-rtcp-for-idms-02
> 
> Hi,
> 
> Please find my replies inline.
> 
> Ishan
> 
> On Nov 24, 2010, at 8:00 PM, Ali C. Begen (abegen) wrote:
> 
> 
> I agree that "watching apart together" is not the best term we could use. In the next version we would probably replace it by
> "non-colocated video watching" or simply "Social TV".

Stick with Social TV.
 
> I am not sure I understand your comment here.

The phrase "typically used in conjunction" is not right here. You have to use RTCP when you use RTP.
  
> 
> 	Payload Type field is said to use the types from 3551. And you assume each type has a certain clock rate. But most
> applications use dynamic payload types and for them you need to get the clock rate from the SDP itself.
> 
> Can you explain this a bit more? What I understand is that I need to get the frame rate of the media I render from the SDP.
> But I am rendering this media anyways so I need to know the frame rate anyways. Am i missing something?

I am not specifically talking about frame rate (you could be using clock rate for many things). When you receive an SDP that uses the dynamic payload types, 3551 will not help you. You have to look in the SDP to figure out what your clock rate for that media stream is.
 
> 	Packet Presented NTP timestamp: I really don't know how you could know this information. In the RTP plane, you can
> only know when you delivered the packet to higher layer. But that has nothing to do with when it will be
> decoded/displayed/played/etc. Even if we try to synch the times that a packet is delivered to the higher layer, shall we
> assume everybody runs the same hardware with the same configuration so that the chances of having all the viewers seeing
> the same content are somewhat >0?
> 
> Interesting question. I think we need to create an assumptions section, where we enlist all our assumptions. All distributed
> synchronization algorithms assume sufficient power in the hardware so that the error in  "jump to 23s in the file" is bounded
> by an acceptable amount (say 100ms).

You may say that but it does not address my primary question. This is not about the hardware power. This is about the application specific things you may access from the RTP plane. E.g., what kind of message or command or whatever in the RTP/RTCP plane will give you access to make the application jump to a certain point in time?

If you wanna program your application in a specific manner to get some hints from the RTP plane, I guess you could do so but that would not be  a standardized app and that would not be a standardized RTP plane. Your draft talks about the RTCP communication between a master and slaves but what happens at the slaves is not specified - which is I would like to see.
 
> 	Beyond that, another question I have is that what happens when a viewer detects that it is running ahead of other
> viewers. What does he do? Slow down, stop for a while? Similarly, if one is lagging behind, how we do make him go faster to
> catch up with others (more accurately to the reference point determined by the MSAS)?
> 
> We had this earlier written in one internal version which explains different synchronization signaling architectures. I think we
> are going to incorporate it in the next version . The default signaling architecture used in the TISPAN document is known as
> Sync Maestro. In this architecture a node(MSAS) which actually does not play out the media itself maintains what can be
> called a "virtual playout node". All reference time is based on the MSAS' perception of the playout at this virtual node. As to
> whether the play out at the users is jumped back, stopped, or the frame rate altered to catch up with the other players is a
> high level decision. Having said all this, the purpose of this document is to only describe the signaling packets, not how they
> are used or implemented.

Sorry then I don't see much benefit in this. See my comments to the previous question. Simply defining the XR format and leaving everything to the implementation will not get us anywhere. Even basic things are not clear how to implement.

Above, you said the hardware should be able "to jump to 23 s in the file": How do you exactly propose to make this happen? And if my player is now at 21 seconds, shall I miss that 2 seconds? What if I don't have that data yet or enough data in my buffer? Should I jump forward/backward every time somebody else wants to synch with me?

I have to admit that the goal behind social tv is nice but thinking more about the issues that will come up in the implementation is hurting my brain. It would help me to see some detailed guidelines. 

As I also mentioned in another email, the ETSI specification does not say anything else that this spec does not already say. So, that is not much help, either.

> 	Finally, when there are multiple temporally correlated streams (audio + video) that are transmitted in separate RTP
> streams, I guess you run this stuff only for one of them. Correct?
> 
> Yes, only one stream is treated as the parent or the primary stream. The other streams need to be locally synchronized  with
> the primary stream.

Then, make this clear in the text.

Cheers,
-acbegen

---

From: Ishan Vaishnavi <[email-masked]>
Date: Sun, 28 Nov 2010 10:44:28 +0100

Hi,

Ishan

> Abstract: "watching apart together" . Is this a common term?

I agree that "watching apart together" is not the best term we could  
use. In the next version we would probably replace it by "non- 
colocated video watching" or simply "Social TV".

>
> Section 1.2:
> First para.: RTCP is supposed to be used whenever RTP is used. So  
> saying they are typically used in conjunction is misleading.

>
> Payload Type field is said to use the types from 3551. And you  
> assume each type has a certain clock rate. But most applications use  
> dynamic payload types and for them you need to get the clock rate  
> from the SDP itself.
>

Can you explain this a bit more? What I understand is that I need to  
get the frame rate of the media I render from the SDP. But I am  
rendering this media anyways so I need to know the frame rate anyways.  
Am i missing something?

> Packet Presented NTP timestamp: I really don't know how you could  
> know this information. In the RTP plane, you can only know when you  
> delivered the packet to higher layer. But that has nothing to do  
> with when it will be decoded/displayed/played/etc. Even if we try to  
> synch the times that a packet is delivered to the higher layer,  
> shall we assume everybody runs the same hardware with the same  
> configuration so that the chances of having all the viewers seeing  

Interesting question. I think we need to create an assumptions  
section, where we enlist all our assumptions. All distributed  
synchronization algorithms assume sufficient power in the hardware so  
that the error in  "jump to 23s in the file" is bounded by an  
acceptable amount (say 100ms).

>
> Beyond that, another question I have is that what happens when a  
> viewer detects that it is running ahead of other viewers. What does  
> he do? Slow down, stop for a while? Similarly, if one is lagging  
> behind, how we do make him go faster to catch up with others (more  
> accurately to the reference point determined by the MSAS)?

We had this earlier written in one internal version which explains  
different synchronization signaling architectures. I think we are  
going to incorporate it in the next version . The default signaling  
architecture used in the TISPAN document is known as Sync Maestro. In  
this architecture a node(MSAS) which actually does not play out the  
media itself maintains what can be called a "virtual playout node".  
All reference time is based on the MSAS' perception of the playout at  
this virtual node. As to whether the play out at the users is jumped  
back, stopped, or the frame rate altered to catch up with the other  
players is a high level decision. Having said all this, the purpose of  
this document is to only describe the signaling packets, not how they  
are used or implemented.

>
> Finally, when there are multiple temporally correlated streams  
> (audio + video) that are transmitted in separate RTP streams, I  
> guess you run this stuff only for one of them. Correct?
>

Yes, only one stream is treated as the parent or the primary stream.  
The other streams need to be locally synchronized  with the primary  
stream.

---

From: Ishan Vaishnavi <[email-masked]>
Date: Sun, 28 Nov 2010 10:58:31 +0100

Hi

> extension seems a better match.

I see your  point. We need to discuss this and see what is the best  
way of moving forward.

> one try to determine what the differences are etc.

NTP has been shown in a number of surveys (including an unpublished  
one I conducted this year myself) to have a network wide dispersion of  
less than 100ms for 90% of the nodes.  This  level of accuracy is  
enough for our purposes. http://www.ntpsurvey.arauc.br/ presents the  
latest published NTP survey results and a comparison to other surveys.  
I do agree that most of these surveys incorporated a majority of level  
2 and 3 nodes and going down levels might increase the dispersion.  
However, my question is how much lower do you think we will need to  
go? level 5, 6 at the most? I don't think the NTP dispersion rises  
that much down a couple of levels.

Ishan

---

From: Peter Musgrave <[email-masked]>
Date: Sun, 28 Nov 2010 16:50:19 -0500

On 2010-11-28, at 4:58 AM, Ishan Vaishnavi wrote:

>> 
> 
> NTP has been shown in a number of surveys (including an unpublished one I conducted this year myself) to have a network wide dispersion of less than 100ms for 90% of the nodes.  This  level of accuracy is enough for our purposes. http://www.ntpsurvey.arauc.br/ presents the latest published NTP survey results and a comparison to other surveys. I do agree that most of these surveys incorporated a majority of level 2 and 3 nodes and going down levels might increase the dispersion. However, my question is how much lower do you think we will need to go? level 5, 6 at the most? I don't think the NTP dispersion rises that much down a couple of levels.

There is another issue here which I have found to be very serious in real systems. NTP updates may be well synchronized - but devices poll NTP infrequently and their clock drift can be very significant. This leads to abrupt jumps when an NTP update does come in - I have seen shifts of several seconds in both directions. I suspect that some more complicated scheme will be needed to ensure global synchronization at a scale of 100ms. (These observation apply to Linux systems about 18 months ago). 

Peter Musgrave

---

From: "Brandenburg, R. (Ray) van" <[email-masked]>
Date: Mon, 29 Nov 2010 10:08:02 +0000

Please see my comments inline.

Ray 

> use. In the next version we would probably replace it by "non-colocated video watching" or simply "Social TV".

Stick with Social TV.

[Ray] Ok.

[Ray] I agree.
  
> 
> 	Payload Type field is said to use the types from 3551. And you assume 
> each type has a certain clock rate. But most applications use dynamic payload types and for them you need to get the clock rate from the SDP itself.
> 

[Ray] I assume you're talking about the reference to 3551 being incorrect in this context? The reference to 3551 is related to the Payload Type indicator, not to the asociated clock rate, which you might get from the SDP. Do you mean that you would like a reference to this fact to be included here?
 
> delivered the packet to higher layer. But that has nothing to do with 
> when it will be decoded/displayed/played/etc. Even if we try to synch 
> the times that a packet is delivered to the higher layer, shall we assume everybody runs the same hardware with the same configuration so that the chances of having all the viewers seeing the same content are somewhat >0?
> 
> synchronization algorithms assume sufficient power in the hardware so that the error in  "jump to 23s in the file" is bounded by an acceptable amount (say 100ms).

[Ray] I'm not completely understanding this discussion. Do you suggest that there is a potential for layer-violation between application layer packet-presented information and RTCP-layer reporting of this?
 
> viewer detects that it is running ahead of other viewers. What does he 
> do? Slow down, stop for a while? Similarly, if one is lagging behind, how we do make him go faster to catch up with others (more accurately to the reference point determined by the MSAS)?
> 
> All reference time is based on the MSAS' perception of the playout at this virtual node. As to whether the play out at the users is jumped back, stopped, or the frame rate altered to catch up with the other players is a high level decision. Having said all this, the purpose of this document is to only describe the signaling packets, not how they are used or implemented.

[Ray] The actual synchronization method used on the client-side is not covered in the current ID since we felt that this ID should only describe the necessary communication method, not the complete synchronization algorithm. Furthermore, the XR block is designed to be agnostic to the synchronization method on the clients. I understand that in order to understand and review the merits of this XR block that it would be useful to have some examples of possible synchronization algorithms. If there is a demand for such a section, we will include this in the next version of the ID.

> 	Finally, when there are multiple temporally correlated streams (audio 
> + video) that are transmitted in separate RTP streams, I guess you run this stuff only for one of them. Correct?
> 
> The other streams need to be locally synchronized  with the primary stream.

[Ray] Ok.
This e-mail and its contents are subject to the DISCLAIMER at http://www.tno.nl/disclaimer/email.html

---

From: "Brandenburg, R. (Ray) van" <[email-masked]>
Date: Mon, 29 Nov 2010 10:17:27 +0000

I agree that for use in IDMS, NTP (or its implementations) is not perfect. 

However, I believe the problem described would be mostly solved if clients were to poll their NTP server each time they are starting a new IDMS session. Assuming that the typical IDMS session takes a few hours at most, the problem of clock drift could be reduced. 

Maybe it would be an idea to change the current text saying "SC's shall be time-synchronized using e.g. NTP", with a section describing the assumptions made by the use of NTP for this XR block.

Ray

-----Original Message-----
From: Peter Musgrave [mailto:[email-masked]] 
Sent: zondag 28 november 2010 22:50
To: Ishan Vaishnavi
Cc: Magnus Westerlund; [email-masked]; [email-masked]
Subject: Re: [AVT] Review of draft-brandenburg-avt-rtcp-for-idms-02

Peter Musgrave

---

From: "Brandenburg, R. (Ray) van" <[email-masked]>
Date: Mon, 29 Nov 2010 10:39:58 +0000

I think it is important to note that the return message (from server to client) is not strictly a 'control' message. It will typically be the exact same message that the 'slowest' client sent to the server, now forwarded to all other clients with the only adjustment being the SPST header. 

In that sense it is a report of the playout delay of the slowest client to all other clients. 

What clients do with this information is up to the clients themselves. They can choose to try to synchronize to this slow/reference client. It is also possible for different clients within the same synch-group to use a different synchronization method (one might immediately pause its playback for several second while another might slow down its playout-rate by inserting additional frames).

Ray

-----Original Message-----
From: Ishan Vaishnavi [mailto:[email-masked]] 
Sent: zondag 28 november 2010 10:59
To: Magnus Westerlund
Cc: [email-masked]; [email-masked]
Subject: Re: [AVT] Review of draft-brandenburg-avt-rtcp-for-idms-02

>
> 1. Use the appropriate RTP and RTCP mechanisms. RTCP XR is a good 
> match for the report about how the reception and playback point are in 
> relation to NTP from the receiver to the master. However, information 
> on when a particular RTP TS should be played in relation to the master 

I see your  point. We need to discuss this and see what is the best way of moving forward.

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 29 Nov 2010 13:14:48 +0100

> that much down a couple of levels.

No, I don't think the strata here is extremely important. What was
really important in your response is what level of synchronization you
expect. Here you state that you consider 100 ms to be good enough. I
think this is highly dependent on what your usage is. If you for example
want to have multiple RTP connected speakers in the ceiling of a store,
or announcement system using VoIP phones, you may easily need at least 1
ms precision. Then you need to really consider what your NTP source is.

So basically some architectural considerations regarding NTP sources and
their implementations. Like running frequent enough updates, etc.

Cheers

Magnus Westerlund

---

From: "Ali C. Begen (abegen)" <[email-masked]>
Date: Mon, 29 Nov 2010 05:10:49 -0800

...
> I am not specifically talking about frame rate (you could be using clock rate for many things). When you receive an SDP that
> uses the dynamic payload types, 3551 will not help you. You have to look in the SDP to figure out what your clock rate for that
> media stream is.
> 
> [Ray] I assume you're talking about the reference to 3551 being incorrect in this context? The reference to 3551 is related to
> the Payload Type indicator, not to the asociated clock rate, which you might get from the SDP. Do you mean that you would
> like a reference to this fact to be included here?

Yes, I don't think you need to refer to anything here. You could be using another RTP profile (say 4585).
 
> > the times that a packet is delivered to the higher layer, shall we assume everybody runs the same hardware with the same
> configuration so that the chances of having all the viewers seeing the same content are somewhat >0?
> >
> > synchronization algorithms assume sufficient power in the hardware so that the error in  "jump to 23s in the file" is
> bounded by an acceptable amount (say 100ms).
> 
> You may say that but it does not address my primary question. This is not about the hardware power. This is about the
> application specific things you may access from the RTP plane. E.g., what kind of message or command or whatever in the
> RTP/RTCP plane will give you access to make the application jump to a certain point in time?
> 
> If you wanna program your application in a specific manner to get some hints from the RTP plane, I guess you could do so but
> that would not be  a standardized app and that would not be a standardized RTP plane. Your draft talks about the RTCP
> communication between a master and slaves but what happens at the slaves is not specified - which is I would like to see.
> 
> [Ray] I'm not completely understanding this discussion. Do you suggest that there is a potential for layer-violation between
> application layer packet-presented information and RTCP-layer reporting of this?

I will not claim whether it is a violation or not at least for now.

My concern is more fundamental. How do you get the app-layer presentation times in your RTP plane? Or, alternatively, how are you planning to change the app-layer presentation times from your RTP plane?
 
> > do? Slow down, stop for a while? Similarly, if one is lagging behind, how we do make him go faster to catch up with others
> (more accurately to the reference point determined by the MSAS)?
> >
> > All reference time is based on the MSAS' perception of the playout at this virtual node. As to whether the play out at the
> users is jumped back, stopped, or the frame rate altered to catch up with the other players is a high level decision. Having
> said all this, the purpose of this document is to only describe the signaling packets, not how they are used or implemented.
> 
> Sorry then I don't see much benefit in this. See my comments to the previous question. Simply defining the XR format and
> leaving everything to the implementation will not get us anywhere. Even basic things are not clear how to implement.
> 
> Above, you said the hardware should be able "to jump to 23 s in the file": How do you exactly propose to make this happen?
> And if my player is now at 21 seconds, shall I miss that 2 seconds? What if I don't have that data yet or enough data in my
> buffer? Should I jump forward/backward every time somebody else wants to synch with me?
> 
> I have to admit that the goal behind social tv is nice but thinking more about the issues that will come up in the
> implementation is hurting my brain. It would help me to see some detailed guidelines.
> 
> [Ray] The actual synchronization method used on the client-side is not covered in the current ID since we felt that this ID

Is it covered in the ETSI doc? I must have missed it if it is.

> should only describe the necessary communication method, not the complete synchronization algorithm. Furthermore, the
> XR block is designed to be agnostic to the synchronization method on the clients. I understand that in order to understand and
> review the merits of this XR block that it would be useful to have some examples of possible synchronization algorithms. If
> there is a demand for such a section, we will include this in the next version of the ID.

Might help.

Cheers, acbegen.

---

From: David R Oran <[email-masked]>
Date: Mon, 29 Nov 2010 09:39:06 -0500

> NTP has been shown in a number of surveys (including an unpublished one I conducted this year myself) to have a network wide dispersion of less than 100ms for 90% of the nodes.  This  level of accu…
Then you are very much restricting the use cases to those in which the audio spaces do not overlap. For example, this won't work for synchronized playout with one TV in the living room and another in the kitchen, because the audio will be audibly (and annoyingly) out of sync. This may also have bad effects in other scenarios where the audio "leaks". For example, if the two locations have a "side channel" audio conference so the watchers can talk about what they are watching, the mis-synchronization will leak back via the audio conference and may badly confuse most echo cancelers.

This is not a comment so much about NTP, but raising the requirement for the protocol extensions you propose to be able not only to synchronize playout within some bound, but also to have distributed knowledge of the bound so that remedial action might be taken (exactly what that is or cold be is an interesting design subject in and of itself.

DaveO.