Subject: [clue] draft-romanow-clue-telepresence-requirements-01: Additional requirement for media, adaptation and media encoding control

From: Magnus Westerlund <[email-masked]>
Date: Tue, 22 Mar 2011 15:42:35 +0100

Hi,

When reviewing draft-romanow-clue-telepresence-requirements-01 I
realized that it has no requirements on that the system supports
adaptation of the media streams. There are two aspects of this I see
that are related but have separate purposes and therefore warrants two
different requirements.

Media Adaptation Requirements:

ReqX: The Telepresence system needs to have congestion control of the
media streams.

ReqX+1: Dynamic Control of sources' encoding of a media stream, in such
aspects as video frame-rate, bit-rate upper limit, temporal spatial
quality trade-off, resolution, Full Intra Request, Picture Loss
Indicator, etc.

Motivation:

Congestion Control is needed to enable Telepresence systems to work also
over network paths that doesn't provide QoS guarantees or when QoS
guarantees fails. This problem is well studied for peer to peer
applications. With an RTP mixer in a multi-party conference session then
this gets more complexed as one needs to consider both what can be
accomplished in the media source to the RTP mixer direction, and then
from the mixer to each of the other participants. This requires some
control of the media encoding in the sources by the RTP mixer to ensure
that the media provided is as useful as possible.

The second requirement is based on the need to enable a conference mixer
to maximize the medias usability for the participants. There are several
aspects here:

- FIR for switching of video streams

- Bit-rate upper limit to adapt media stream to suit mixer to
  participant direction.

- PLI to enable a participant with broken decoder state to get that
  cleaned up.

- Frame-rate to adapt to receiver capabilities or change purpose of a
  media stream.

- Temporal Spatial Trade-off if one wants to control video encoders to
  prefer frame rate over spatial quality or the other way around.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Sat, 26 Mar 2011 12:21:50 -0700

Hi Magnus,

A couple of brief comments inline below. 
Also, since we are meeting this coming week, we could discuss further if it seems like a good idea.

Best regards,
Allyn

> -----Original Message-----
> From: Magnus Westerlund [mailto:[email-masked]]
> Sent: Tuesday, March 22, 2011 7:43 AM
> To: CLUE; Allyn Romanow (allyn)
> Subject: draft-romanow-clue-telepresence-requirements-01: Additional
> requirement for media, adaptation and media encoding control
> 
> Hi,
> 

[AR] I agree this is important to consider, and within the scope of the CLUE working group --however I think it is not within the scope of the CLUE protocol of exchanging information for handling multiple streams, which mostly boils down to layout, spatial relations, in addition to stream characteristics - i. e., saying which streams are where when so an adequate representation can be made at the receiver.  

The CLUE group is chartered to consider additional issues related to telepresence interoperability, and I feel this is where the congestion control might be discussed. What do you think?

> quality trade-off, resolution, Full Intra Request, Picture Loss

[AR] Good catch - This was meant to be covered in the requirements, but clearly was not done adequately. I had thought that Reqmt-1 covered these types of stream characteristics, but it doesn't really now that I see it again.
So, yes I agree with you and will propose some text for including this type of information.  

As for the dynamic aspect, I do thing that is covered in Reqmt-2 Support dynamic behavior of media streams, including:...changes in stream characteristics... 
What do you think?

Thanks for your thoughtful comments-
Allyn

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 29 Mar 2011 09:07:17 +0200

Please see inline.

Allyn Romanow (allyn) skrev 2011-03-26 20:21:
> Hi Magnus,
> 
> A couple of brief comments inline below. Also, since we are meeting
> this coming week, we could discuss further if it seems like a good
> idea.
> 
> 
> Best regards, Allyn
> 
>> -----Original Message----- From: Magnus Westerlund
>> [mailto:[email-masked]] Sent: Tuesday, March 22,
>> 2011 7:43 AM To: CLUE; Allyn Romanow (allyn) Subject:
>> draft-romanow-clue-telepresence-requirements-01: Additional 
>> adaptation of the media streams. There are two aspects of this I
>> see that are related but have separate purposes and therefore
>> warrants two different requirements.
>> 
>> ReqX: The Telepresence system needs to have congestion control of
>> the media streams.
> 
> [AR] I agree this is important to consider, and within the scope of
> the CLUE working group --however I think it is not within the scope
> of the CLUE protocol of exchanging information for handling multiple
> streams, which mostly boils down to layout, spatial relations, in
> addition to stream characteristics - i. e., saying which streams are
> where when so an adequate representation can be made at the receiver.
> 

Well, CLUE does have a media plane, and we clearly have interoperability
questions for it. I think congestion control is one. Clarifications on
how to use RTP sessions seems to be another one. In that part as Clue
develops usages and requirements AVT* can determine what issues that
exist within this space and need clarification.

> 
> The CLUE group is chartered to consider additional issues related to
> telepresence interoperability, and I feel this is where the
> congestion control might be discussed. What do you think?
> 

My preferred model is that CLUE make it clear what requirements and what
usages and topologies that are important and that this is thrown over
the wall to the right AVT* group.

>> 
>> ReqX+1: Dynamic Control of sources' encoding of a media stream, in
>> such aspects as video frame-rate, bit-rate upper limit, temporal
>> spatial quality trade-off, resolution, Full Intra Request, Picture
>> Loss Indicator, etc.
> 
> [AR] Good catch - This was meant to be covered in the requirements,
> but clearly was not done adequately. I had thought that Reqmt-1
> covered these types of stream characteristics, but it doesn't really
> now that I see it again. So, yes I agree with you and will propose
> some text for including this type of information.
> 
> As for the dynamic aspect, I do thing that is covered in Reqmt-2
> Support dynamic behavior of media streams, including:...changes in
> stream characteristics... What do you think?
> 

Adding that example is a great improvement. I wonder however, if some
more expansive list on desired functionality would be good.

Cheers

Magnus Westerlund

---

From: "Allyn Romanow (allyn)" <[email-masked]>
Date: Tue, 29 Mar 2011 05:00:57 -0700

Hi Magnus,
It sounds like we are in agreement on how to handle your point on congestion control, and other AVT related issues.

As for your second point about media stream description not being in the requirements, how is this change to requirement 1?

REQMT-1:   Provide a means of describing the relationship of one stream to another stream.  This includes 		     exchanging appropriate information to enable:
		*	layout to be accomplished, such as relevant spatial ordering information 
		*	Sender and/or receiver capabilities and preferences, such as resolution, bandwidth, video 			frame rate

> Sent: Tuesday, March 29, 2011 12:07 AM
> To: Allyn Romanow (allyn)
> Cc: CLUE
> Subject: Re: draft-romanow-clue-telepresence-requirements-01:
> Additional requirement for media, adaptation and media encoding control
> 
> Please see inline.
>

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 29 Mar 2011 16:22:56 +0200

Allyn Romanow (allyn) skrev 2011-03-29 14:00:
> Hi Magnus, It sounds like we are in agreement on how to handle your
> point on congestion control, and other AVT related issues.
> 
> As for your second point about media stream description not being in
the requirements, how is this change to requirement 1?
> 
> REQMT-1: Provide a means of describing the relationship of one
> stream to another stream. This includes exchanging appropriate information to
> * Sender and/or receiver capabilities and preferences, such as resolution,
    bandwidth, video frame rate

Not that it is not appropriate to have in the negotiation on
capabilities or desired behavior from streams.

However, I was actually desiring a dynamic control of these aspects of
media streams during a session.

Cheers

Magnus Westerlund

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 29 Mar 2011 16:33:54 +0200

Hi Magnus

I agree that we will want some ability to control some aspects of the media
dynamically mid-call.

Though I think is also a need to give the encoder freedom to choose the best
possible mode.  For instance video framerate is often slowed down during
fast motion in order to maintain the best bitrate.  I am not sure that
receiver-driven controls over all parameters is needed.

Similarly, providing feedback on the desired receive video resolution for a
particular is valuable.  But if the same stream is going to multiple
destinations, the encoder should be able to ignore this feedback.

Stephen Botzko

On Tue, Mar 29, 2011 at 4:22 PM, Magnus Westerlund <
[email-masked]> wrote:

> > * Sender and/or receiver capabilities and preferences, such as
> resolution,
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 29 Mar 2011 16:44:04 +0200

Stephen Botzko skrev 2011-03-29 16:33:
> Hi Magnus
> 
> I agree that we will want some ability to control some aspects of the
> media dynamically mid-call.
> 
> Though I think is also a need to give the encoder freedom to choose the
> best possible mode.  For instance video framerate is often slowed down
> during fast motion in order to maintain the best bitrate.  I am not sure
> that receiver-driven controls over all parameters is needed. 
> 
> Similarly, providing feedback on the desired receive video resolution
> for a particular is valuable.  But if the same stream is going to
> multiple destinations, the encoder should be able to ignore this feedback.
> 

Not disagreeing with these comments.

This comes in several flavors. One is expressing preferences for what is
received. There is another case when the mixer will desire to put hard
restrictions on the source encoder to make it suitable for distribution
to more receivers. What is suitable for a particular knob I think
depends on which knob it is and what control its have.

So in centralized mixer topologies there can be a clear difference in
semantics between the client -> mixer direction to the mixer->client
one. For point to point both semantics can be useful depending on what
one tries to achieve.

Cheers

Magnus Westerlund

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 29 Mar 2011 16:50:22 +0200

So perhaps a good way to proceed is for people to propose specific controls
(knobs), and describe their value in the various topologies.

Stephen Botzko

---

From: "Espen Berger (espeberg)" <[email-masked]>
Date: Thu, 31 Mar 2011 11:34:21 +0200

I think it's a useful feature to separate maximum capabilities with preferred capabilities. Examples of max capabilities and preferred capabilities. 

Max-caps 
 * Max-mbps 
 * Bit rate 

Preferences
 * Resolution  
 * fps (Frames pr. second)

A receiver can then request a stream with a maximum bit rate of 2 mb/s (absolute maximum) and hint that it prefer 60 fps. It should give the encoder enough information be configured with high frame rate and lower resolution. The bit rate of 720p60 is comparable to 1080p30, hence an encoder need some help to choose between the two options.  The video decoder should be able to decode any stream that is inside the max-mbps and bit rate limits (I think that is a H.264 spec requirement).  

In a point to point call the receiving endpoint can select, in a multipoint call the multipoint server can aggregate request from the multiple receivers, or in multi cast scenarios the sender can do the aggregation. 

-Espen 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Magnus Westerlund
Sent: 29. mars 2011 16:44
To: Stephen Botzko
Cc: CLUE
Subject: Re: [clue] draft-romanow-clue-telepresence-requirements-01: Additional requirement for media, adaptation and media encoding control

Cheers

Magnus Westerlund