Subject: [clue] MCC switching example

From: "Duckworth, Mark" <[email-masked]>
Date: Thu, 9 Jan 2014 11:23:01 -0800

In framework-13, there is a new example for MCC usage in section 12.3.3.  I think the part about Capture Scene #8 is either incorrect or incomplete.  This is supposed to describe how the MCU can advertise captures representing the non-current-talker endpoints, and allow the receiving endpoint to render them with proper spatial relationships.  For example, I don't see any way for endpoint A (the receiving consumer endpoint) to be able to render captures from scene #8 that come from endpoint B (3 cameras) next to each other.  There is no way for A to know at any point in time which MCCs contain VC4, VC5, and VC6 from endpoint B, so there is no way for A to render them next to each other.

We discussed this general topic at IETF88, but didn't have much time on it.  We can discuss again at the design team meeting on January 14 and the interim meeting.
See presentation http://tools.ietf.org/agenda/88/slides/slides-88-clue-5.pdf.

Some possible approaches are (corresponding to 1-3 on page 15 from the presentation):

1. Use an advertisement like in the framework example, but also need to add ability for receiver to relate the original source of incoming capture encodings at the RTP level to the captures in the CLUE advertisement.

2. Use an advertisement like in the framework example, but the consumer selects the specific sources at any point in time by sending a configure message whenever it wants to change its selection.  Need to add ability for the consumer to know about audio activity so it can use that to inform switching decisions.  Maybe it can just come from audio streams sent from MCU to the consumer. Probably also want same ability to relate the incoming encodings to original captures as in item 1 above.

3. Use a different advertisement, breaking apart Capture Scene #8 into three separate scenes, where each scene has three MCCs with spatial information on the MCCs themselves. This would be three separate scenes practically the same as Scene #7, but with different policy values to indicate they are further down in the MCU's recent speaker list.  The MCU is responsible for maintaining the advertised spatial relation between MCCs when it switches to different sources.

I am most interested in option 2.  But I think we can make all of them work in CLUE with some additions about relating incoming encodings to advertised captures.

I'd like to discuss this more before deciding if/how to update this example in the framework.

Regards,
Mark

---

From: Christian Groves <[email-masked]>
Date: Fri, 10 Jan 2014 12:34:49 +1100

Hello Mark,

Please see my comments below.

Regards, Christian

On 10/01/2014 6:23 AM, Duckworth, Mark wrote:
>
> In framework-13, there is a new example for MCC usage in section 
> 12.3.3.  I think the part about Capture Scene #8 is either incorrect 
> or incomplete.  This is supposed to describe how the MCU can advertise 
> captures representing the non-current-talker endpoints, and allow the 
> receiving endpoint to render them with proper spatial relationships. 
> For example, I don't see any way for endpoint A (the receiving 
> consumer endpoint) to be able to render captures from scene #8 that 
> come from endpoint B (3 cameras) next to each other. There is no way 
> for A to know at any point in time which MCCs contain VC4, VC5, and 
> VC6 from endpoint B, so there is no way for A to render them next to 
> each other.
>
[CNG] I agree that capture scene 8 doesn't describe what you want above. 
However I believe that it is valid for describing what is intended i.e. 
The MCU can also supply nine media streams showing the active and 
previous eight speakers. Its displaying the captures of the active 
speaker not the whole speaker site. The site switching aspect is 
described in Capture Scene #7.

I also agree that knowing exactly what capture is in the MCC capture 
encoding at a particular point of time is an issue.

> We discussed this general topic at IETF88, but didn't have much time 
> on it.  We can discuss again at the design team meeting on January 14 
> and the interim meeting.
>
[CNG] Unfortunately i'll be away for the Jan14 meeting.

> See presentation 
> http://tools.ietf.org/agenda/88/slides/slides-88-clue-5.pdf.
>
> 1. Use an advertisement like in the framework example, but also need 
> to add ability for receiver to relate the original source of incoming 
> capture encodings at the RTP level to the captures in the CLUE 
> advertisement.
>
> 2. Use an advertisement like in the framework example, but the 
> consumer selects the specific sources at any point in time by sending 
> a configure message whenever it wants to change its selection.  Need 
> to add ability for the consumer to know about audio activity so it can 
> use that to inform switching decisions.  Maybe it can just come from 
> audio streams sent from MCU to the consumer. Probably also want same 
> ability to relate the incoming encodings to original captures as in 
> item 1 above.
>
> 3. Use a different advertisement, breaking apart Capture Scene #8 into 
> three separate scenes, where each scene has three MCCs with spatial 
> information on the MCCs themselves. This would be three separate 
> scenes practically the same as Scene #7, but with different policy 
> values to indicate they are further down in the MCU's recent speaker 
> list.  The MCU is responsible for maintaining the advertised spatial 
> relation between MCCs when it switches to different sources.
>
> I am most interested in option 2.  But I think we can make all of them 
> work in CLUE with some additions about relating incoming encodings to 
> advertised captures.
>
[CNG] My feeling is that 1) would be the most robust general approach. 
Knowing the source capture at the RTP would be faster and minimise any 
sync issues between CLUE and RTP and the provider/consumer. I think a 
pragmatic approach to the use case would be 3.
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 10 Jan 2014 13:32:44 -0500

I pretty much agree with Christian on this.
Specifically I think (1) is the preferred approach.

Jonathan has frequently commented on the issues around dynamically 
arranging streams - that it may be more important to keep specific 
sources in a fixed location as long as they are displayed, rather than 
moving them around to get them into optimal relationships every time the 
speaker changes. Ultimately that is up to the client to decide, if it 
has the information to do so. The advertiser can't do as good a job 
because it lacks sufficient knowledge of the client's situation.

I'm somewhat dubious that (3) will be very successful. But if an 
advertiser wants to do that it can.

	Thanks,
	Paul

---

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 10 Jan 2014 12:35:27 -0800

I'm responding to both Paul and Christian, inline.
Mark

> -----Original Message-----
> From: clue [mailto:[email-masked]] On Behalf Of Paul Kyzivat
> Sent: Friday, January 10, 2014 1:33 PM
> To: [email-masked]
> Subject: Re: [clue] MCC switching example
> 
> arranging streams - that it may be more important to keep specific sources in
> a fixed location as long as they are displayed, rather than moving them
> around to get them into optimal relationships every time the speaker
> changes. Ultimately that is up to the client to decide, if it has the information
> to do so. The advertiser can't do as good a job because it lacks sufficient
> knowledge of the client's situation.

[Duckworth, Mark] I agree, and that's why I prefer option (2).  I'm concerned about option (1) because the consumer has to be prepared to handle stream switches (completely controlled by the provider) at any time.  So the consumer can be receiving several MCCs, but the original capture contained in the MCC can switch at any time.  So at one moment VC4, VC5, VC6 can be in MCC5, MCC6, MCC7.  Then the next moment they can appear in MCC8, MCC9, MCC10 (just as an example).  If the consumer wants to render them in the same spot, it has to either re-map the incoming encodings to different decoders, or re-map decoder outputs to different locations in its local composition.  It seems to me this would be hard to do in practice, considering all the possible permutations of how source captures can appear in the MCCs, and trying to do it in a nice clean way as it appears to the human user, without pauses or other disruptions in the rendering.

[Duckworth, Mark] In general, with option (1), I think the consumer has enough information to know all the spatial relations of all the captures it is receiving, but I'm concerned about trying to implement changing rendering decisions quickly enough for a good user experience.

[Duckworth, Mark] With option (2), the consumer is in control so it knows what to expect in each MCC as switches occur, and it tailors it to its own needs.

> I'm somewhat dubious that (3) will be very successful. But if an advertiser
> wants to do that it can.
> 
> 	Thanks,
> 	Paul
> 
> 
> 
> >> or incomplete.  This is supposed to describe how the MCU can
> >> advertise captures representing the non-current-talker endpoints, and
> >> allow the receiving endpoint to render them with proper spatial
> relationships.

[Duckworth, Mark] The framework example isn't clear enough in describing the use case for these 9 streams, so we have different assumptions about the desired behavior.  I'll propose something to make this more clear.

> >> Some possible approaches are (corresponding to 1-3 on page 15 from
> >> the
> >> presentation):
> >>
> >> to add ability for the consumer to know about audio activity so it
> >> can use that to inform switching decisions.  Maybe it can just come
> >> from audio streams sent from MCU to the consumer. Probably also want
> >> same ability to relate the incoming encodings to original captures as
> >> in item 1 above.
> >>
> >> 3. Use a different advertisement, breaking apart Capture Scene #8
> >> into three separate scenes, where each scene has three MCCs with
> >> spatial information on the MCCs themselves. This would be three
> >> separate scenes practically the same as Scene #7, but with different
> >> policy values to indicate they are further down in the MCU's recent
> >> speaker list.  The MCU is responsible for maintaining the advertised
> >> spatial relation between MCCs when it switches to different sources.
> >>
> >> I am most interested in option 2.  But I think we can make all of
> >> them work in CLUE with some additions about relating incoming
> >> encodings to advertised captures.
> >>

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 13 Jan 2014 09:39:50 -0500

> [Duckworth, Mark] With option (2), the consumer is in control so it knows what to expect in each MCC as switches occur, and it tailors it to its own needs.

I'd like to hear from Jonathan on this, but since he hasn't yet 
commented I'll try to channel him:

Approach (2) is likely to have sufficient delays that when trying to get 
the "current speaker" video you are likely to end up with "previous 
speaker" video.

(Note that I personally have no information to support this - it's hearsay.)

	Thanks,
	Paul