Subject: [MMUSIC] More thoughts on multiple media streams in a single m= line

From: Justin Uberti <[email-masked]>
Date: Sun, 3 Feb 2013 23:45:44 -0800

Sorry for the short lead time, but here's a quick read for the plane.

Peter Thatcher and I have put together a writeup of our thoughts on the
advantages of putting multiple media streams in a single m= line.

http://www.ietf.org/id/draft-pthatcher-mmusic-many-sources-00.txt

We look forward to discussing in more detail at the interim.

---

From: Adam Roach <[email-masked]>
Date: Tue, 05 Feb 2013 10:25:47 -0500

> Sorry for the short lead time, but here's a quick read for the plane.

I addressed several of the assertions made in this draft in my previous 
response (e.g. the need for glare resolution regardless of which 
solution we select). Some additional thoughts:

>     o  Adding or removing a source requires sending a full list of all
>        sources, which doesn't scale well to 10s or 100s of sources,
>        especially when they are added or removed frequently.

Realistically, this isn't how multiparty conferences are handled. It's 
unrealistic to think that my DSL connection is going to be sucking down 
hundreds of video streams just because I'm in a video conference with 
hundreds of people.

In practice, the way existing conferencing systems handle this is to 
send a small, manageable number of streams to each participant, changing 
them out as necessary to have a reasonable user experience (either under 
the control of the user, its own algorithms, or a combination of the two).

Which is to say: even with hundreds of source streams being fed into a 
conference system, you're really looking at a realistic upper bound on 
m= sections of a dozen or two, simply because you can't reasonably 
expect recipients to receive more than that many streams simultaneously.

And really, if you're imagining hundreds of media streams flowing 
towards an endpoint, any arguments about the bits consumed by SDP go 
well beyond disingenuous.

>     o  Because this solution requires BUNDLE, which is still not widely
>        supported, it can be deployed in fewer situations

This is completely out of left field: the alternative is MMT, which is 
even less well developed than BUNDLE.

>     4.  The RFC5576-based approach has already been in use for many years
>         in a major video conferencing system (Google+ Hangouts), and has
>         worked very well.  We feel that we have searched the "unknown"
>         space and solved all of the major issues.

I think this actually does a good job of illustrating your blind spot: 
you're working within RTCWEB, but developing for a single, very precise 
use case. For example, this kind of experience doesn't translate well 
into the challenges that might arise from systems that are dealing with 
radically different user experiences. Google-hangout-type uses are 
certainly important, but they're not the only thing we're designing for. 
You need to think about systems with pre-encoded (stored) media; systems 
with radical imbalances in the processing power and device capabilities 
between the endpoints; systems with different stream control models, 
etc, etc.

>         On the other hand, the multi m=
>         section approach has never been tried by anyone, that we are
>         aware of

Do a web search for "Marconi ViPr."

/a

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Tue, 5 Feb 2013 16:19:50 +0000

I think we are all in agreement that for a large conference (e.g. 100+ audio/video streams), we cannot require that each media stream have its own m=section, nor can we require that each media stream's SSRC is signaled in SDP. For the sake of example, let's assume a centralized conference server is able to send up to 10 video streams, where each video stream may be any of the 100 video streams at various points in time (e.g. active speakers switched in/out, specific sources requested, etc.), I think there are three realistic alternatives:

1) Use 10 m=sections to define the ten video streams, no multiplexing required
2) Use 1 m=section to define  a video stream, and indicate that up to 10 video streams are multiplexed within it
3) Use 'n' m=sections to define the video streams, where 1<=n<=10

We already have all we need to do (1). There is a desire to improve on (1).
If we define some additional syntax/semantics that enable (2), people can choose to use (2) or (3) depending on works best for their scenario. There have already been some drafts submitted along these lines, I suggest to take a closer look at those and improve on them. Sounds reasonable?

Charles

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Adam Roach
> Sent: Tuesday, February 05, 2013 10:26 AM
> To: Justin Uberti
> Cc: [email-masked]
> Subject: Re: [MMUSIC] More thoughts on multiple media streams in a single
> m= line
> 
> response (e.g. the need for glare resolution regardless of which solution we
> select). Some additional thoughts:
> 
> 
> 
> In practice, the way existing conferencing systems handle this is to send a
> small, manageable number of streams to each participant, changing them
> out as necessary to have a reasonable user experience (either under the
> control of the user, its own algorithms, or a combination of the two).
> 
> conference system, you're really looking at a realistic upper bound on m=
> sections of a dozen or two, simply because you can't reasonably expect
> recipients to receive more than that many streams simultaneously.
> 
> And really, if you're imagining hundreds of media streams flowing towards
> an endpoint, any arguments about the bits consumed by SDP go well
> beyond disingenuous.
> 
> 
> 
> This is completely out of left field: the alternative is MMT, which is even
> less well developed than BUNDLE.
> 
> 
> 
> I think this actually does a good job of illustrating your blind spot: you're
> working within RTCWEB, but developing for a single, very precise use case.
> For example, this kind of experience doesn't translate well into the
> challenges that might arise from systems that are dealing with radically
> different user experiences. Google-hangout-type uses are certainly
> important, but they're not the only thing we're designing for. You need to
> think about systems with pre-encoded (stored) media; systems with radical
> imbalances in the processing power and device capabilities between the
> endpoints; systems with different stream control models, etc, etc.
> 
> 
>

---

From: "Roni Even" <[email-masked]>
Date: Tue, 5 Feb 2013 19:57:47 +0200

Charles,
Even for 1 you need maxssrc, you assume that one m-line means one RTP
stream. 
Roni

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Charles Eckel (eckelcu)
Sent: 05 February, 2013 6:20 PM
To: Adam Roach; Justin Uberti
Cc: [email-masked]
Subject: Re: [MMUSIC] More thoughts on multiple media streams in a single m=
line

I think we are all in agreement that for a large conference (e.g. 100+
audio/video streams), we cannot require that each media stream have its own
m=section, nor can we require that each media stream's SSRC is signaled in
SDP. For the sake of example, let's assume a centralized conference server
is able to send up to 10 video streams, where each video stream may be any
of the 100 video streams at various points in time (e.g. active speakers
switched in/out, specific sources requested, etc.), I think there are three
realistic alternatives:

1) Use 10 m=sections to define the ten video streams, no multiplexing
required
2) Use 1 m=section to define  a video stream, and indicate that up to 10
video streams are multiplexed within it
3) Use 'n' m=sections to define the video streams, where 1<=n<=10

We already have all we need to do (1). There is a desire to improve on (1).
If we define some additional syntax/semantics that enable (2), people can
choose to use (2) or (3) depending on works best for their scenario. There
have already been some drafts submitted along these lines, I suggest to take
a closer look at those and improve on them. Sounds reasonable?

Charles

> Subject: Re: [MMUSIC] More thoughts on multiple media streams in a 
> single m= line
> 
> I addressed several of the assertions made in this draft in my 
> previous response (e.g. the need for glare resolution regardless of 
> which solution we select). Some additional thoughts:
> 
> 
> 
> unrealistic to think that my DSL connection is going to be sucking 
> down hundreds of video streams just because I'm in a video conference 
> with hundreds of people.
> 
> send a small, manageable number of streams to each participant, 
> changing them out as necessary to have a reasonable user experience 
> (either under the control of the user, its own algorithms, or a
combination of the two).
> you're working within RTCWEB, but developing for a single, very precise
use case.
> challenges that might arise from systems that are dealing with 
> certainly important, but they're not the only thing we're designing 
> for. You need to think about systems with pre-encoded (stored) media; 
> systems with radical imbalances in the processing power and device 
> capabilities between the endpoints; systems with different stream control
models, etc, etc.
> Do a web search for "Marconi ViPr."

_______________________________________________
mmusic mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/mmusic

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Tue, 5 Feb 2013 19:43:57 +0000

True, max-ssrc or something that provides similar functionality, would be useful in all three.

Cheers,
Charles

> From: Roni Even [mailto:[email-masked]]
> Sent: Tuesday, February 05, 2013 12:58 PM
> To: Charles Eckel (eckelcu); 'Adam Roach'; 'Justin Uberti'
> Cc: [email-masked]
> Subject: RE: [MMUSIC] More thoughts on multiple media streams in a single
> m= line
> 
> Charles,
> Behalf Of
> m=

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 08 Feb 2013 21:56:44 +0100

> down hundreds of video streams just because I'm in a video conference 

(challenging assumptions)
why is it realistic to assume that you're on a DSL line?
In Kansas City, a hundred video streams each doing 100 Kbits/s or so may 
be a completely realistic way of doing conference-participation 
thumbnails; it's only 10 Mbits, which isn't that much on a gigabit link.

(with today's codecs, CPU may be a more limiting factor. It's the DSL 
part I reacted to.)

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 14 Feb 2013 15:00:31 +0100

> expect recipients to receive more than that many streams simultaneously.

Hi,

I definitely want to comment on this from an RTP usage perspective that
might be very relevant to remember. Even with a centralized conference
solution there is two very distinct solutions in RTP how to transmit a
session where you have total of 100 participants and you receive 5 video
streams.

A) Central point with conceptual sources
In this case the central entity creates conceptual RTP streams that
carries all the media intended for a specific usage, for example main
video, thumbnail one, thumbnail two, thumbnail three, thumbnail four.

B) Central point doing source projection
In this case there exist 100 potential RTP streams and SSRC from the
central point down to the receiver. However, at any point no more than
five will actually have RTP packets. But as activity changes and which
is most relevant for the receiver the SSRCs in use will switch.

For a solution where each SSRC requires its own m= line A) is definitely
preferable as it will not require renegotiating the properties for these
five conceptual streams. However, B will require a large number of m=
lines and dynamic updates as participants comes and go.

IF one had a single m= line per RTP session and media type, the amount
of updates/renegotiation depends on the amount of meta data about
streams that you have put in the SDP. They solutions A) and B) can be
identical from an SDP perspective.

The above two cases I am quite certain we did discuss in the CLUE
interim meeting in Stockholm when discussing topologies and also the
relation ship between RTP streams (SSRC) and their origin, i.e. media
source/capture.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM
----------------------------------------------------------------------
Ericsson AB                | Phone  +46 10 7148287
Färögatan 6                | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden| mailto: [email-masked]
----------------------------------------------------------------------

---

From: "Roni Even" <[email-masked]>
Date: Sat, 16 Feb 2013 08:41:54 +0200

Hi,
I fully agree with Magnus cases and I think they should be addressed. 
At CLUE when working on RTP mapping we addressed this used cases by having
multiple RTP streams in a single m-line
Roni Even

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Magnus Westerlund
Sent: 14 February, 2013 4:01 PM
To: Adam Roach
Cc: [email-masked]
Subject: Re: [MMUSIC] More thoughts on multiple media streams in a single m=
line

combination of the two).

Hi,

A) Central point with conceptual sources In this case the central entity
creates conceptual RTP streams that carries all the media intended for a
specific usage, for example main video, thumbnail one, thumbnail two,
thumbnail three, thumbnail four.

B) Central point doing source projection In this case there exist 100
potential RTP streams and SSRC from the central point down to the receiver.
However, at any point no more than five will actually have RTP packets. But
as activity changes and which is most relevant for the receiver the SSRCs in
use will switch.

For a solution where each SSRC requires its own m= line A) is definitely
preferable as it will not require renegotiating the properties for these
five conceptual streams. However, B will require a large number of m= lines
and dynamic updates as participants comes and go.

IF one had a single m= line per RTP session and media type, the amount of
updates/renegotiation depends on the amount of meta data about streams that
you have put in the SDP. They solutions A) and B) can be identical from an
SDP perspective.

The above two cases I am quite certain we did discuss in the CLUE interim
meeting in Stockholm when discussing topologies and also the relation ship
between RTP streams (SSRC) and their origin, i.e. media source/capture.

Cheers

Magnus Westerlund

---

From: Bernard Aboba <[email-masked]>
Date: Sat, 16 Feb 2013 12:53:18 -0800

Comments below. 

> lines and dynamic updates as participants comes and go.

[BA] I think there are advantages to A even if multiple SSRCs are declared within a m-line.  If the "main" video and "thumbnails" change SSRCs, even with multiple SSRCs on an m-line, you'd still need to renegotiate unless SVC is being used, right? 
 
> If one had a single m= line per RTP session and media type, the amount
> streams that you have put in the SDP. The solutions A) and B) can be

[BA] That's not obvious to me. Even if you have a single m=line per RTP session and media type, 
you can still have individual a=ssrc lines within a m= line.  In case B, as participants come and go,
aren't dynamic updates still required?  Also, instead of many m=lines, you'll have many a=ssrc
lines. 

> relationship between RTP streams (SSRC) and their origin, i.e. media
> source/capture.

[BA] Were there any specific requirements that arose in the CLUE discussion?  In general, given some of the context issues, it would be helpful to understand whether we're engineering a solution for RTCWEB use cases, CLUE use cases, or both.