Subject: [clue] multipoint switching, MCC, and spatial information

From: "Duckworth, Mark" <[email-masked]>
Date: Fri, 17 Jan 2014 15:44:21 -0800

This is follow-up to the topic "[clue] MCC switching example", and to the design team discussion Jan 14.  I'm interested in what others have to say about how approach #1 should work, regarding spatial information and the consumer's ability to render streams together that belong together.

Referring to my framework issue slides<http://trac.tools.ietf.org/wg/clue/trac/raw-attachment/wiki/Design-Team/Framework_Issues_140113.pptx>, pages 8 - 12.  The group had interest mainly in approach #1, where the MCU makes all the switching decisions.  But we didn't have a common understanding about how spatial information was to be conveyed, so the consumer can render multiple streams together when they are spatially related.

On slide 9, for approach 1, it says "MCCs don't have spatial attributes".  But other people in the discussion thought the MCCs should have spatial attributes.  Jonathan said the MCCs could have spatial information like this (Jonathan, correct me if I don't remember right):
MCC1, MCC2, MCC3: left, center, and right of the full scene, like the image on page 6 for VC1-3.
MCC4, MCC5, MCC6: spatial information that indicates they are located on bottom part of MCC1, and much smaller than MCC1, like on page 6 for VC4-6.
MCC7, MCC8, MCC9: spatial information that indicates they are located on bottom part of MCC2, and much smaller than MCC2, like on page 6 for VC7-9.

While this should work for that specific case, I don't think it works for the very similar case in framework section 12.3.3 which has these two alternative rendering layouts for the same streams (use mono-spaced font to see this):

   +---+---+---+ +-------------+ +-------------+ +-------------+
   |   |   |   | |             | |             | |             |
   +---+---+---+ |             | |             | |             |
   |   |   |   | |             | |             | |             |
   +---+---+---+ |             | |             | |             |
   |   |   |   | |             | |             | |             |
   +---+---+---+ +-------------+ +-------------+ +-------------+

  +-------------+ +-------------+ +-------------+
  |             | |             | |             |
  |             | |             | |             |
  |             | |             | |             |
  | +-+ +-+ +-+ | | +-+ +-+ +-+ | | +-+ +-+ +-+ |
  | +-+ +-+ +-+ | | +-+ +-+ +-+ | | +-+ +-+ +-+ |
  +-------------+ +-------------+ +-------------+

If we generalize this approach to support alternatives like this, I think we end up with approach number 3 instead of approach 1.

We also talked about the consumer should have the ability to associate incoming encoded packet streams with a particular individual MC from the advertisement.  This would give the consumer the spatial information of the particular MC currently switched in to the MCC.  My concern about this way of getting spatial information is that it means the renderer has to re-map decoder output to different display locations on the fly without any advance notice.

Regards,
Mark

---

From: Christer Holmberg <[email-masked]>
Date: Sat, 18 Jan 2014 07:35:43 +0000

Hi,

I'm in China, but I'd call in if my schedule permits.

Regards,

Christer

Lähettäjä: clue [mailto:[email-masked]] Puolesta Duckworth, Mark
Lähetetty: 18. tammikuuta 2014 1:44
Vastaanottaja: [email-masked]
Aihe: [clue] multipoint switching, MCC, and spatial information

Regards,
Mark

---

From: Paul Kyzivat <[email-masked]>
Date: Sat, 18 Jan 2014 14:41:05 -0500

Mark,

My thinking is that the advertiser MAY include spatial coordinates on 
the MCCs if it wishes. If so, then they are virtual coordinates for that 
capture in the context of the scene containing the MCC. How the 
advertiser arrives at these coordinates is its business. It might look 
at the spatial relationships of the source captures to do this, or not.

A simple consumer can simply treat an MCC like any other capture, and 
make all of its decisions about how to render it based on the MCC 
coordinates. This might be sub-optimal, but its easy.

A more sophisticated consumer can look at the sources of the MCC 
captures, and their coordinates in their scenes, and use that to make 
its decisions about how to render. It can do that statically (at the 
time of Configure) without regard to which of the alternatives is 
switched in at any one time. (This is somewhat sub-optimal.)

Or the most sophisticated consumer can see what it is receiving in each 
RTP packet, map this back to a particular source capture in the 
advertisement, and then use the spatial coordinates from that to make 
rendering decisions. (We need to figure out how that mapping an be done.)

The advertiser doesn't need to know which way the consumer is doing it. 
But the information provided in the advertisement constrains what the 
consumer is able to do.

	Thanks,
	Paul

On 1/17/14 6:44 PM, Duckworth, Mark wrote:
> This is follow-up to the topic “[clue] MCC switching example”, and to
> the design team discussion Jan 14.  I’m interested in what others have
> to say about how approach #1 should work, regarding spatial information
> and the consumer’s ability to render streams together that belong together.
>
> Referring to my framework issue slides
> <http://trac.tools.ietf.org/wg/clue/trac/raw-attachment/wiki/Design-Team/Framework_Issues_140113.pptx>,
> pages 8 – 12.  The group had interest mainly in approach #1, where the
> MCU makes all the switching decisions.  But we didn’t have a common
> understanding about how spatial information was to be conveyed, so the
> consumer can render multiple streams together when they are spatially
> related.
>
> On slide 9, for approach 1, it says “MCCs don’t have spatial
> attributes”.  But other people in the discussion thought the MCCs should
> have spatial attributes.  Jonathan said the MCCs could have spatial
> information like this (Jonathan, correct me if I don’t remember right):
>
> MCC4, MCC5, MCC6: spatial information that indicates they are located on
> bottom part of MCC1, and much smaller than MCC1, like on page 6 for VC4-6.
>
> MCC7, MCC8, MCC9: spatial information that indicates they are located on
> bottom part of MCC2, and much smaller than MCC2, like on page 6 for VC7-9.
>
> While this should work for that specific case, I don’t think it works
> for the very similar case in framework section 12.3.3 which has these
> two alternative rendering layouts for the same streams (use mono-spaced
> font to see this):
>
> If we generalize this approach to support alternatives like this, I
> think we end up with approach number 3 instead of approach 1.
>
> We also talked about the consumer should have the ability to associate
> incoming encoded packet streams with a particular individual MC from the
> advertisement.  This would give the consumer the spatial information of
> the particular MC currently switched in to the MCC.  My concern about
> this way of getting spatial information is that it means the renderer
> has to re-map decoder output to different display locations on the fly
> without any advance notice.
>
> Regards,
>
> Mark
>
>
>
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: Christian Groves <[email-masked]>
Date: Mon, 20 Jan 2014 14:37:36 +1100

Hello Mark,

I agree with Paul's comments.

Please see my additional comments below.

Regards, Christian

On 19/01/2014 6:41 AM, Paul Kyzivat wrote:
> the MCCs if it wishes. If so, then they are virtual coordinates for 
> that capture in the context of the scene containing the MCC. How the 
> Or the most sophisticated consumer can see what it is receiving in 
> each RTP packet, map this back to a particular source capture in the 
> The advertiser doesn't need to know which way the consumer is doing 
> it. But the information provided in the advertisement constrains what 
> the consumer is able to do.
>
>     Thanks,
>     Paul
>
[CNG] Yes MCCs could have spatial information in this context.
>> two alternative rendering layouts for the same streams (use mono-spaced
[CNG] Table 22 in section 12.3.3 of the framework has spatial 
information for MCC1,MCC2,MCC3 (incorrectly labelled as left,left,left 
it should be left, centre, right). This would be related to the "large" 
boxes above. I don't think there's an issue with this.

There are a number of ways that the "small" boxes could be advertised.

The first is as per table 23 of the framework. Each of the MCCs 
(MMC4-MCC12) could be given a virtual spatial area. i.e. each MCC stream 
has its own space. i.e. MCC4(top left), MCC5(top centre), MCC6(top 
right, etc.) It's then up to the consumer to decide how to render it. 
The Consumer could do either of the above approaches.

A second approach is a per table 24 of the framework. If MCC13 was in 
CaptureScene8 it could use spatial information of the constituent MCCs 
to determine the position in the composed single stream. Unless the 
Consumer had the smarts to "decompose" the media stream then it would 
only be able to the top approach.

A third approach would be for the Provider to Advertise 3 streams for 
the 9 tiles (this isn't shown in the framework), e.g.
         +=======================+=================================+
         | Capture Scene #       | Description=Output3stream       |
         +-----------------------|---------------------------------+
         | MCC4(VC4,VC5,VC6,VC7, | MaxCaptures=1                   |
         |   VC8,VC9,VC10,VC11,  | Policy=SoundLevel:0             |
         |   VC12,VC13,VC14,VC15)| SpatialInfo=Top Left            |
         |                       |                                 |
         | MCC5(VC4,VC5,VC6,VC7, | MaxCaptures=1                   |
         |   VC8,VC9,VC10,VC11,  | Policy=SoundLevel:1             |
         |   VC12,VC13,VC14,VC15)| SpatialInfo=Top Middle          |
         |                       |                                 |
                     to                           to               |
         |                       |                                 |
         | MCC12(VC4,VC5,VC6,VC7,| MaxCaptures=1                   |
         |   VC8,VC9,VC10,VC11,  | Policy=SoundLevel:8             |
         |   VC12,VC13,VC14,VC15)| SpatialInfo=Bottom Right        |
         |                       |                                 |
         | MCC13(MCC4,MCC5,MCC6  | MaxCaptures=3                   |
         |                       | EncodingGroup=1                 |
         |                       | SpatialInfo=Top                 |
         | MCC14(MCC7,MCC8,MCC9  | MaxCaptures=3                   |
         |                       | EncodingGroup=1                 |
         |                       | SpatialInfo=Middle              |
         | MCC15(MCC10,MCC11,    | MaxCaptures=3                   |
         |         MCC12)        | EncodingGroup=1                 |
         |                       | SpatialInfo=Bottom              |
         | CSE(MCC13,MCC14,MCC15)|                                 |
         +=======================+=================================+

In the above case the Provider Advertises 3 encodings MCC13 containing 
the three most active speakers, MCC14 containing the next three most 
active etc. Note that the MCC13-15 are static, always maintaining the 
spatial co-ordinates. Because the Consumer receives 3 streams it could 
display these 3 streams in either the 9 tiles approach or the PiP approach.

As Paul said, the information received from the Advertiser constrains 
what the Consumer is able to do.

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 20 Jan 2014 13:04:51 -0500

> [CNG] Yes MCCs could have spatial information in this context.

OK, is *this* what you have been talking about Christian?

I agree this is *possible*. But it will take more specification for 
consumers to make sense of this in a consistent way.

Specifically, our spatial info describes a quadrilateral in a plane in 
three-space. In the above there is no mention of the third dimension. Do 
you assume that MCC4 and MCC7 fall within the planar area of MCC1? If 
so, then what should the stacking order be when composing them?

Or do you assume that they will be planar areas that are "nearer" to the 
capture point than MC1, providing a hint about the stacking order?

	Thanks,
	Paul

---

From: "Duckworth, Mark" <[email-masked]>
Date: Mon, 20 Jan 2014 14:09:53 -0800

Paul,

I agree with what you wrote.
Now I'm trying to figure out which type of consumer (simple, more sophisticated, or most sophisticated) is necessary in order to implement the scenario in the section 12.3.3 example.  I was thinking it was only the most sophisticated kind, but I was hoping you all had a better idea and could show me how it works with a less sophisticated consumer.  In the design team discussion last week, I got the sense that some people thought the simple consumer could do it, for both of the layout options in the diagram below, but I still don't see how.

Mark

> -----Original Message-----
> From: clue [mailto:[email-masked]] On Behalf Of Paul Kyzivat
> Sent: Saturday, January 18, 2014 2:41 PM
> To: [email-masked]
> Subject: Re: [clue] multipoint switching, MCC, and spatial information
> 
> Mark,
> 
> My thinking is that the advertiser MAY include spatial coordinates on the
> MCCs if it wishes. If so, then they are virtual coordinates for that capture in
> the context of the scene containing the MCC. How the advertiser arrives at
> these coordinates is its business. It might look at the spatial relationships of
> the source captures to do this, or not.
> 
> A simple consumer can simply treat an MCC like any other capture, and make
> all of its decisions about how to render it based on the MCC coordinates. This
> might be sub-optimal, but its easy.
> 
> A more sophisticated consumer can look at the sources of the MCC captures,
> and their coordinates in their scenes, and use that to make its decisions
> about how to render. It can do that statically (at the time of Configure)
> without regard to which of the alternatives is switched in at any one time.
> (This is somewhat sub-optimal.)
> 
> Or the most sophisticated consumer can see what it is receiving in each RTP
> packet, map this back to a particular source capture in the advertisement,
> and then use the spatial coordinates from that to make rendering decisions.
> (We need to figure out how that mapping an be done.)
> 
> > This is follow-up to the topic "[clue] MCC switching example", and to
> > the design team discussion Jan 14.  I'm interested in what others have
> > to say about how approach #1 should work, regarding spatial
> > information and the consumer's ability to render streams together that
> belong together.
> >
> > <http://trac.tools.ietf.org/wg/clue/trac/raw-attachment/wiki/Design-Te
> > am/Framework_Issues_140113.pptx>, pages 8 - 12.  The group had
> > interest mainly in approach #1, where the MCU makes all the switching
> > decisions.  But we didn't have a common understanding about how
> > spatial information was to be conveyed, so the consumer can render
> > multiple streams together when they are spatially related.
> >
> > On slide 9, for approach 1, it says "MCCs don't have spatial
> > attributes".  But other people in the discussion thought the MCCs
> > should have spatial attributes.  Jonathan said the MCCs could have
> > spatial information like this (Jonathan, correct me if I don't remember
> right):
> >
> > MCC4, MCC5, MCC6: spatial information that indicates they are located
> > on bottom part of MCC1, and much smaller than MCC1, like on page 6 for
> VC4-6.
> >
> > MCC7, MCC8, MCC9: spatial information that indicates they are located
> > on bottom part of MCC2, and much smaller than MCC2, like on page 6 for
> VC7-9.
> >
> > While this should work for that specific case, I don't think it works
> > two alternative rendering layouts for the same streams (use
> > mono-spaced font to see this):
> >
> > incoming encoded packet streams with a particular individual MC from
> > the advertisement.  This would give the consumer the spatial
> > information of the particular MC currently switched in to the MCC.  My
> > concern about this way of getting spatial information is that it means
> > the renderer has to re-map decoder output to different display
> > locations on the fly without any advance notice.
> >
> > Regards,
> >
> > Mark
> >
> >
> >

---

From: "Duckworth, Mark" <[email-masked]>
Date: Mon, 20 Jan 2014 14:16:01 -0800

Christian,

Your first approach for the way the "small boxes" could be advertised sounds to me like the advertiser is expecting the consumer to be able to separate the top, middle, and bottom rows within the 3x3 grid scene into three separate areas for rendering.  This would be better done by using three separate scenes in the first place, not just one scene.  Three separate scenes is an example of my approach #3 from the slides.

I'm still focusing on the media-switching variety of MCU, so I am looking for solution that doesn't involve composition in the MCU.  But I do agree your composition examples make sense if the MCU is capable of it.

Mark

> From: clue [mailto:[email-masked]] On Behalf Of Christian Groves
> Sent: Sunday, January 19, 2014 10:38 PM
> To: [email-masked]
> >> the design team discussion Jan 14.  I'm interested in what others
> >> have to say about how approach #1 should work, regarding spatial
> >> information and the consumer's ability to render streams together
> >> that belong together.
> >>
> >> pages 8 - 12.  The group had interest mainly in approach #1, where
> >> the MCU makes all the switching decisions.  But we didn't have a
> >> common understanding about how spatial information was to be
> >> conveyed, so the consumer can render multiple streams together when
> >> they are spatially related.
> >>
> >> on bottom part of MCC1, and much smaller than MCC1, like on page 6
> >> for VC4-6.
> >>
> >> on bottom part of MCC2, and much smaller than MCC2, like on page 6
> >> for VC7-9.
> [CNG] Table 22 in section 12.3.3 of the framework has spatial information for
> MCC1,MCC2,MCC3 (incorrectly labelled as left,left,left it should be left,
> centre, right). This would be related to the "large"
> has its own space. i.e. MCC4(top left), MCC5(top centre), MCC6(top right,
> etc.) It's then up to the consumer to decide how to render it.
> CaptureScene8 it could use spatial information of the constituent MCCs to
> determine the position in the composed single stream. Unless the Consumer
> had the smarts to "decompose" the media stream then it would only be able
> to the top approach.
> 
> A third approach would be for the Provider to Advertise 3 streams for the 9
> tiles (this isn't shown in the framework), e.g.
> 
> In the above case the Provider Advertises 3 encodings MCC13 containing the
> three most active speakers, MCC14 containing the next three most active
> etc. Note that the MCC13-15 are static, always maintaining the spatial co-
> ordinates. Because the Consumer receives 3 streams it could display these 3
> streams in either the 9 tiles approach or the PiP approach.
> 
> As Paul said, the information received from the Advertiser constrains what
> the Consumer is able to do.
> 
> 
> >>

---

From: Christian Groves <[email-masked]>
Date: Tue, 21 Jan 2014 10:19:28 +1100

Hello Mark,

Please see below.

Regards, Christian

> Your first approach for the way the "small boxes" could be advertised sounds to me like the advertiser is expecting the consumer to be able to separate the top, middle, and bottom rows within the 3…
[CNG] In my first case I don't think the Provider assumes that three 
separate areas are used. Its simply suggesting a 3x3 grid, 9 tile scene. 
How the Consumer uses the information is up to it. The consumer could 
equally take the left, middle, right columns Captures. By having these 
in the same Scene it indicates that there is a spatial relation between 
the Captures. If you had multiple scenes then any spatial relationship 
is lost. If the Provider didn't want to indicate any spatial 
relationship between the MCC (i.e. each tile) then they could also be in 
separate CaptureScenes.

---

From: Christian Groves <[email-masked]>
Date: Tue, 21 Jan 2014 10:37:03 +1100

Hello Paul,

Please see below.

Regards, Christian

> OK, is *this* what you have been talking about Christian?
[CNG] Partly, MCCs can be sources to other MCCs (like individual 
Captures) and thus could have spatial information associated with them.
> consumers to make sense of this in a consistent way.
[CNG] I'm happy to work on that.
> three-space. In the above there is no mention of the third dimension. 
> Do you assume that MCC4 and MCC7 fall within the planar area of MCC1? 
> If so, then what should the stacking order be when composing them?
[CNG] I'm taking the above case as a composition so I think its valid to 
assume that they are on the same plane. With regards to stacking order I 
hadn't really thought about it because I assumed the same plane. Being a 
composed image I didn't think order was important as the images would 
basically be merged, i.e. if you cut out the smaller area you'd be left 
with a hole.

In a multiple encoding case if you wanted to imply stacking I guess an 
Advertiser could use a plane parallel to the "main scene". In this way 
it could indicate a parallel plane in front or behind. If it was the 
same plane then the Consumer could decide which way to display it. 
However this aspect seems to be a general issue with Spatial information 
in captures. Not just for MCCs.

>
> Or do you assume that they will be planar areas that are "nearer" to 
> the capture point than MC1, providing a hint about the stacking order?
>
>     Thanks,
>     Paul

---

From: "Roni Even" <[email-masked]>
Date: Tue, 21 Jan 2014 16:28:35 +0200

Paul,
I think that spatial information in 3 dimensional when y is back to front
Roni

> Sent: 20 January, 2014 8:05 PM
> To: [email-masked]
> >>> MCC4, MCC5, MCC6: spatial information that indicates they are
> >>> located on bottom part of MCC1, and much smaller than MCC1, like on
> >>> page 6 for VC4-6.
> >>>
> >>> MCC7, MCC8, MCC9: spatial information that indicates they are
> >>> located on bottom part of MCC2, and much smaller than MCC2, like on
> >>> page 6 for VC7-9.
consumers to
> make sense of this in a consistent way.
> 
three-space.
> In the above there is no mention of the third dimension. Do you assume
that
> MCC4 and MCC7 fall within the planar area of MCC1? If so, then what should
> the stacking order be when composing them?
>