Subject: Re: [MMUSIC] RTSP: Speed and Scale overview text

From: "Burklin Helmut" <[email-masked]>
Date: Wed, 3 Dec 2008 10:09:08 +0100

Hm, Torbjörn,
I'm a little bit surprised. You say:
"   Speed: RTP time stamps increase with average rate Speed*(RTP clock frequency). 
          No change in relation between media playout speed and original media speed.
          The media is played at the original speed."
I believed that we may use Speed for doing slow playout if, e.g., the server is not supporting the Scale. Or doing fast playout if the server doesn't support scale and the network supports the additional load.
I don't think that we shall impose here to the client at which rate he shall do the playout - this is under his whole responsability. Presentation time stamps indicate whent the content should be presented under nominal conditions - but here we are not under nominal conditions any more.
Note also that we have a special case for MPEG2-TS (RFC2250) where the timestamp is NOT a presentation time stamp.

For the other questions, I think they go beyond the introductory text Magnus had in mind. (though they are valuable: for the first one I think the server SHOULD choose the nearest value - whatever he considers to be nearest.

best regards
Helmut Bürklin

-----Original Message-----
From: Torbjörn Einarsson [mailto:[email-masked]] 
Sent: mercredi 3 décembre 2008 09:46
To: mmusic (E-mail)
Cc: Burklin Helmut; Magnus Westerlund
Subject: RE: [MMUSIC] RTSP: Speed and Scale overview text

Hi,

I agree that speed is conceptually simpler (and more easy to implement if bandwidth increase is allowed), so it may go first.

I think one should put in the work "time" in the definition of Scale, to make the ratio unitless. 

   Speed: The ratio of playback time delivered per unit of wall clock time

   Scale: The ratio of media content time delivered per unit playback time

Furthermore, I think some examples referring to RTP timestamps would make thinks clearer. That text might be put at the end.

Here are some details for RTP, to illustrate how different time lines are influenced.

  For RTP, the playout time is always given by the RTP time stamps. The influence of Speed and Scale is

   Speed: RTP time stamps increase with average rate Speed*(RTP clock frequency). 
          No change in relation between media playout speed and original media speed.
          The media is played at the original speed.

   Scale: RTP time stamps increase with average rate equal to RTP clock frequency.
          The relation between the RTP clock and the original media clock has been changed by a factor Scale.
          The media is played Scale times faster than the original speed.

Another thing is:
* What shall the server answer if it cannot do the scale asked for, but another close-by scale? For example, the client asks for scale 5, but the server can only support 3 and 6. Shall it respond with 6, or send an error message?

A final thing that needs a solution is the bandwidth handling in the case os Speed, or rather if we should have a trade-off between bandwidth and quality.

Shall it be allowed to send data faster than the bandwidth signalled in SDP (or alike)?
For early buffer filling, this is definitely useful. Furthermore, if the bandwidth limitations must be kept, then the server must manipulate the streams, which may be very difficult and lead to undesirable results.
Maybe the client should be able to signal that it can accept adaptation? E.g.
* Speed 3, no-content adaptation
* Speed 3, keep band-width but allow adaptation Increasing the bitrate above the session bandwidth is of course a potential congestion problem, so maybe we should simply state that the server should be careful to avoid congestiosn, e.g. by monitoring RTCP RR?

BR,
Torbjörn
--
Torbjörn Einarsson, MSc PhD
Mgr Media Protocols and Applications
Ericsson Research, Multimedia Technologies
Phone: +46 10 7172346
Mobile: +46 70 673 3424
[email-masked]

Ericsson AB
Färögatan 2-6
S-164 80 Stockholm, Sweden
www.ericsson.com 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Burklin Helmut
Sent: den 2 december 2008 15:24
To: Magnus Westerlund; mmusic (E-mail)
Subject: Re: [MMUSIC] RTSP: Speed and Scale overview text

Hallo,
I think for the layman the text will be easier to understand if you put the simple case, i.e. Speed, first, followed by the more complicated one, i.e. Scale. 
I am also wondering if it would make sense to add to the Speed chapter a sentence like:
"Note that the data rate on the transmission medium is proportional to the Speed factor, e.g. a content delivered with Speed=2 will use twice the data rate as the same content delivered with Speed=1."

And this brings me to another question: what about negative Speed values ? I didn't look up if they are generally prohibited somewhere, but since the text talks about negative scale values, it should at least mention negative speed values - if necessary stating that they are prohibited.

Helmut Bürklin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Magnus Westerlund
Sent: mardi 2 décembre 2008 13:46
To: mmusic (E-mail)
Subject: [MMUSIC] RTSP: Speed and Scale overview text

Hi,

Below is an overview text about Speed and Scale. Trying to ensure that we all agree what Scale and Speed is supposed to do. So please review and see if you think you understand this. If it matches your understanding of the concepts and what need you see of either of the mechanisms. I want to ensure that we are on the same page and do see a need for both before updating the rest of the text regarding these subjects.

Media Delivery Manipulations

The basic playback functionality of RTSP is to request content for a particular range to be delivered to the client in a pace that enables playback as intended by the creator. However, RTSP can also manipulate how this delivery is done to the client in two ways.

   Scale: The ratio of media content delivered per unit playback time.

   Speed: The ratio of playback time delivered per unit of wall clock
          time.

So both affects the media delivery per time unit. However, they are manipulating two independent time scales and the effects are possible to combine.

Scale is used for fast forward or slow motion control as it changes the amount of content timescale that should be played back per time unit.
Scale > 1.0, means fast forward, e.g. Scale=2.0 results in that 2 seconds of content is played back every second of playback. Scale = 1.0 is the default value that is used if no Scale is specified, i.e.
playback at the contents original rate. Scale values between 0 and 1.0 is providing for slow motion. Scale can be negative to allow for reverse playback in either regular pace (Scale = -1.0) or fast backwards (Scale < -1.0) or slow motion backwards (-1.0 < Scale < 0).

In most cases the realization of scale means server side manipulation of the media to ensure that the client can actually play it back. These media manipulation and when they are needed are highly media type dependent. Lets exemplify with two common media types audio and video.

It is very difficult to modify the playback rate of audio. A maximum of 10-30% is possible by changing the pitch-rate of speech. Music goes out of tune if one tries to manipulate the playback rate by resampling it.
This is a well known problem and audio is commonly muted or played back in short segments with skips to keep up with the current playback point.

For video is possible to manipulate the number of frames that is displayed per second. But the rendering capabilities are often limited to certain frame rates. The decoding, handling capabilities and bitrate of received encoded content also limits the number of frames that can be delivered. Therefore when providing fast forward one generally picks a subset of the frames from the original content to be displayed. However, the video encoding methods use will commonly limit the possibilities on which frames that can be chosen and still be decoded by the receiver.

Due to the media restrictions a particular content will commonly be restricted to a limited set of possible scale ratios. To handle this correctly, RTSP has mechanism to indicate the supported Scale ratios for the content. To support aggregated or dynamic content where this may change during the ongoing session and dependent on the location within the content a mechanism for updating the media properties and the current used scale factor exist.

Speed affects how much of the playback timeline that is delivered in a given wall clock period. The default is Speed = 1 which is to deliver at the same rate the media is consumed. Speed > 1 means that the receiver will get content faster than it regularly would consume it. Speed < 1 means that it receives it slower than the regular consumption rate. This mechanism enables two general functionalities. Client side scale operations, i.e. the client receives all the frames and makes the adjustment to the playback locally. The second usage is to control its buffering of media. By specifying a speed over 1.0 the client can build up the amount of playback time it has present in its buffers to a level that is sufficient for its needs.
-- 

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------
_______________________________________________
mmusic mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mmusic
_______________________________________________
mmusic mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mmusic
_______________________________________________
mmusic mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mmusic

---

From: Torbjörn Einarsson <[email-masked]>
Date: Wed, 3 Dec 2008 10:19:21 +0100

You're of course right. We cannot specify the client behaviour. It should be free to do adaptive playout or something similar. The main purpose of the suggested text was to make it clearer what is changed in the different cases. Do you think we should put in any text on RTP at all?

BR,
Torbjörn  

-----Original Message-----
From: Burklin Helmut [mailto:[email-masked]] 
Sent: den 3 december 2008 10:09
To: Torbjörn Einarsson; mmusic (E-mail)
Cc: Magnus Westerlund
Subject: RE: [MMUSIC] RTSP: Speed and Scale overview text

 Hm, Torbjörn,

best regards
Helmut Bürklin

-----Original Message-----
From: Torbjörn Einarsson [mailto:[email-masked]]
Sent: mercredi 3 décembre 2008 09:46
To: mmusic (E-mail)
Cc: Burklin Helmut; Magnus Westerlund
Subject: RE: [MMUSIC] RTSP: Speed and Scale overview text

Hi,

Another thing is:

BR,
Torbjörn

Ericsson AB
Färögatan 2-6

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Burklin Helmut
Sent: den 2 december 2008 15:24
To: Magnus Westerlund; mmusic (E-mail)
Subject: Re: [MMUSIC] RTSP: Speed and Scale overview text

Hallo,

Helmut Bürklin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Magnus Westerlund
Sent: mardi 2 décembre 2008 13:46
To: mmusic (E-mail)
Subject: [MMUSIC] RTSP: Speed and Scale overview text

Hi,

Magnus Westerlund

---

From: <[email-masked]>
Date: Wed, 3 Dec 2008 12:32:35 +0200

Hi,

in order not to overload the usage of Speed, I would propose to define it in relationship to bandwidth only and not time.
This would make it clear that no content adaptation is to be performed by the server in reaction to speed changes and that only transmission scheduling is affected.

By decoupling both concepts, client and server are able to negotiate content adaptation in a more explicit manner. Handling of Speed requests that result in congestion is also made easier by adjusting output bitrate to channel bandwidth and then determining the closest feasible Speed.

Desirable quality adaptation may still be achieved explicitly by switching to a lower/higher bitrate representation of the same content or otherwise.

Input                     Scaled Content                Modified Speed
Bitrate +------------+       Bitrate    +--------------+   Bitrate   +------------+
------->|   Scaler   |----------------->|  Scheduler   |------------>|  Channel   |
        +------------+                  +--------------+             +------------+
Modified Speed = min(Speed,available bandwidth/Scaled Content Bitrate)
Modified Speed Bitrate = Modified Speed * Scaled Content Bitrate  

Regards, 
Imed Bouazizi

-----Original Message-----
From: [email-masked] on behalf of ext Burklin Helmut
Sent: Wed 12/3/2008 11:09 AM
To: Torbjörn Einarsson; mmusic (E-mail)
Cc: Magnus Westerlund
Subject: Re: [MMUSIC] RTSP: Speed and Scale overview text

 Hm, Torbjörn,

best regards
Helmut Bürklin

-----Original Message-----
From: Torbjörn Einarsson [mailto:[email-masked]] 
Sent: mercredi 3 décembre 2008 09:46
To: mmusic (E-mail)
Cc: Burklin Helmut; Magnus Westerlund
Subject: RE: [MMUSIC] RTSP: Speed and Scale overview text

Hi,

Another thing is:

BR,
Torbjörn

Ericsson AB
Färögatan 2-6

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Burklin Helmut
Sent: den 2 december 2008 15:24
To: Magnus Westerlund; mmusic (E-mail)
Subject: Re: [MMUSIC] RTSP: Speed and Scale overview text

Hallo,

Helmut Bürklin

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Magnus Westerlund
Sent: mardi 2 décembre 2008 13:46
To: mmusic (E-mail)
Subject: [MMUSIC] RTSP: Speed and Scale overview text

Hi,

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 03 Dec 2008 11:41:14 +0100

> You're of course right. We cannot specify the client behaviour. It should be free to do adaptive playout or something similar. The main purpose of the suggested text was to make it clearer what is…

I think there will be text on this, but not in this section. The right
spot to put RTP related discussion on Scale and Speed are in the
Appendix where media delivery using RTP is defined.

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 03 Dec 2008 11:55:08 +0100

[email-masked] skrev:
> Hi,
> 
> in order not to overload the usage of Speed, I would propose to define
> it in relationship to bandwidth only and not time.
> This would make it clear that no content adaptation is to be performed
> by the server in reaction to speed changes and that only transmission
> scheduling is affected.
> 
> By decoupling both concepts, client and server are able to negotiate
> content adaptation in a more explicit manner. Handling of Speed requests
> that result in congestion is also made easier by adjusting output
> bitrate to channel bandwidth and then determining the closest feasible
> Speed.
> 
> Desirable quality adaptation may still be achieved explicitly by
> switching to a lower/higher bitrate representation of the same content
> or otherwise.
> 

Imed,

I think the current definition of Speed does managed to have a
definition that isn't overloaded at all. Speed affects the amount of
nominal playback time that is delivered per time. Yes, reducing speed
will have the affect on bandwidth restricted paths to enable higher
quality. But the definition is clear.

Considering your definition of modified speed, that will be a dynamic
value that changes during the delivery as the value for available BW
fluctuates.

I think we are getting to the core of the issues around speed. Namely
what it is to be used for. Opportunistic buffer filling, in which case
your proposal makes total sense, or if one actually like to make a local
scaling operation where strict following the given speed value is more
important.

My personal view is that the client buffer management should be solved
with some solution that is more dynamic than RTSP.

Cheers

Magnus Westerlund

---

From: "Rémi Denis-Courmont" <[email-masked]>
Date: Thu, 4 Dec 2008 11:11:58 +0200

On Wednesday 03 December 2008 12:32:35 ext [email-masked], you wrote:
> Hi,
>
> in order not to overload the usage of Speed, I would propose to define it
> in relationship to bandwidth only and not time. This would make it clear
> that no content adaptation is to be performed by the server in reaction to
> speed changes and that only transmission scheduling is affected.
>
> that result in congestion is also made easier by adjusting output bitrate
> to channel bandwidth and then determining the closest feasible Speed.

Bandwidth is fundamentally tied to time, by its very definition. Decoupling 
them makes no sense to me, and the very idea is utmostly confusing to me.

-- 
Rémi Denis-Courmont
Maemo Software, Nokia Devices R&D
_______________________________________________
mmusic mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mmusic