Subject: Re: [clue] Multiple Streams - RTP muxing?

From: "Elwell, John" <[email-masked]>
Date: Tue, 1 Feb 2011 12:30:41 +0100

I really think we should have some sort of understanding on scalability requirements here. I don't think this is covered in either of the two drafts to date, but I may have missed it.

John 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Charles Eckel (eckelcu)
> Sent: 31 January 2011 15:50
> To: Stephen Botzko; Magnus Westerlund
> Cc: CLUE
> Subject: Re: [clue] Multiple Streams - RTP muxing?
> 
> I was thinking (2) as well; however, one concern I have is 
> scalability. As the number of screens increases from 3 or so 
> to say 10 or 100, does the set of pros and cons change. I 
> would hate to have 100 m-lines, but 100 streams each with its 
> own SSRC multiplexed in a single RTP session may be 
> programmatic as well.
> I am hoping those more familiar with the complexities and 
> limitations of RTP stream processing can chime in. Will we 
> have no choice but to break it into multiple RTP sessions?
> 
> Thanks,
> Charles
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Stephen Botzko
> > Sent: Monday, January 31, 2011 4:52 AM
> > To: Magnus Westerlund
> > Cc: CLUE
> > Subject: Re: [clue] Multiple Streams - RTP muxing?
> > 
> > Hi
> > >>>
> > 1. One RTP session, with one SSRC that contains the 
> encoding of multiple
> > different original flows through either transcoding, or a new
> > multiplexing mechanism?
> > 
> > 2. One RTP session, with one SSRC per original video stream.
> > >>>
> > 
> > I was thinking (2).  (one session for audio and one for 
> video).  I'd be open to (1) if it did not
> > require transcoding.  However, I am not convinced a totally 
> new mechanism is needed.
> > 
> > Though it hasn't come up yet, I was also wondering about 
> having all video and audio streams from the
> > same CLUE endpoint use the same clock offset, in order to 
> simplify lip-sync management with multiple
> > audio and multiple video streams.  Not sure about that, but 
> I do see a lot of complexity in
> > determining the proper delays.
> > 
> > Stephen Botzko
> > 
> > 
> > 
> > On Mon, Jan 31, 2011 at 7:41 AM, Magnus Westerlund 
> <[email-masked]> wrote:
> > 
> > 
> > 	Hi,
> > 
> > 	(AS AVTCore Chair)
> > 
> > 	I want to inject into this discussion a request to be a 
> bit more clear
> > 	on what people are considering. Because RTP has several 
> multiplexing
> > 	points and some may even want to go furhter than the 
> existing ones. So
> > 	can everyone please clarify what they are talking about.
> > 
> > 	Please keep in mind that RTP has certain point of where 
> things are
> > 	intended to separated. This also affects the 
> possibility to use existing
> > 	RTP tools.
> > 
> > 	So a primer in RTP terminology:
> > 
> > 	RTP Session: One SSRC space, normally determined based 
> on transport
> > 	level ports. Each RTP session normally servers one 
> media type and one
> > 	purpose. Using different media types (audio and video) 
> will creates
> > 	issues for some tools and SDP based signalling.
> > 
> > 	SSRC, a single media stream source. Multiple SSRC can 
> be used in the
> > 	same RTP session.
> > 
> > 	RTP payload type, identifies the encoding of the RTP 
> payload. Thus
> > 	possibly identifying video screen sizes etc.
> > 
> > 	So I will start with asking Peter to clarify the below 
> statement:
> > 
> > 	Peter Musgrave skrev 2011-01-29 08:51:
> > 
> > 	>
> > 	> I think the notion of a single, multiplexed video 
> stream for a TP endpoint makes a lot of
> > sense in the context of CLUE.
> > 
> > 
> > 	I can interpret this as meaning either of:
> > 
> > 	1. One RTP session, with one SSRC that contains the 
> encoding of multiple
> > 	different original flows through either transcoding, or a new
> > 	multiplexing mechanism?
> > 
> > 	2. One RTP session, with one SSRC per original video stream.
> > 
> > 	So please be clear on what you are discussing and 
> suggesting here.
> > 
> > 	Cheers
> > 
> > 	Magnus Westerlund
> > 
> > 	
> ----------------------------------------------------------------------
> > 	Multimedia Technologies, Ericsson Research EAB/TVM
> > 	
> ----------------------------------------------------------------------
> > 	Ericsson AB                | Phone  +46 10 7148287
> > 	Färögatan 6                | Mobile +46 73 0949079
> > 	SE-164 80 Stockholm, Sweden| mailto: 
> [email-masked]
> > 	
> ----------------------------------------------------------------------
> > 
> > 	_______________________________________________
> > 	clue mailing list
> > 	[email-masked]
> > 	https://www.ietf.org/mailman/listinfo/clue
> > 
> > 
> 
> _______________________________________________
> clue mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/clue
>

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 1 Feb 2011 06:55:41 -0500

The use cases and the problem statement drafts were not intended to be
requirements documents.  I agree that scalability needs to be discussed, and
that use cases for large scale conferences are needed.

Existing telepresence systems I am aware of receive/decode one video stream
per display.

However, it is quite practical to send one video stream per rendered image
(instead of using video transcoding), especially if the sender either
simulcasts multiple resolutions or uses a layered video codec like SVC.
This can greatly increase the number of streams that are received at each
endpoint, and also makes the received stream mix much more dynamic. One
benefit is that transcoding/transrating delay is at least reduced and in
many cases eliminated.

Both approaches were listed in the problem statement draft last year (in the
multipoint section).  Personally I favor a solution that accommodates both.

Stephen Botzko

On Tue, Feb 1, 2011 at 6:30 AM, Elwell, John <
[email-masked]> wrote:

> I really think we should have some sort of understanding on scalability
> requirements here. I don't think this is covered in either of the two drafts
> to date, but I may have missed it.
>
> John
>

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 1 Feb 2011 13:51:02 +0100

aravind sethuraman skrev 2011-01-31 20:03:
> Hi all,
> if we chose a single RTP session for ALL video and AUDIO, how will we
> signal which SSRC conforms to which screen etc? Not via RTCP like TIP does.

I just want to point out that signalling this with a single m= line in
SDP is impossible. As the general media type is part of the m= line one
can't have both RTP payload types of both audio and video type in the
same m= line.

When it comes to RTP RFC 3550 discusses this type of multiplexing and
has the following to say on the issue (from section 5.2 of RFC 3550):

   For efficient protocol processing, the number of multiplexing points
   should be minimized, as described in the integrated layer processing
   design principle [10].  In RTP, multiplexing is provided by the
   destination transport address (network address and port number) which
   is different for each RTP session.  For example, in a teleconference
   composed of audio and video media encoded separately, each medium
   SHOULD be carried in a separate RTP session with its own destination
   transport address.

   Separate audio and video streams SHOULD NOT be carried in a single
   RTP session and demultiplexed based on the payload type or SSRC
   fields.  Interleaving packets with different RTP media types but
   using the same SSRC would introduce several problems:

   1. If, say, two audio streams shared the same RTP session and the
      same SSRC value, and one were to change encodings and thus acquire
      a different RTP payload type, there would be no general way of
      identifying which stream had changed encodings.

   2. An SSRC is defined to identify a single timing and sequence number
      space.  Interleaving multiple payload types would require
      different timing spaces if the media clock rates differ and would
      require different sequence number spaces to tell which payload
      type suffered packet loss.

   3. The RTCP sender and receiver reports (see Section 6.4) can only
      describe one timing and sequence number space per SSRC and do not
      carry a payload type field.

   4. An RTP mixer would not be able to combine interleaved streams of
      incompatible media into one stream.

   5. Carrying multiple media in one RTP session precludes: the use of
      different network paths or network resource allocations if
      appropriate; reception of a subset of the media if desired, for
      example just audio if video would exceed the available bandwidth;
      and receiver implementations that use separate processes for the
      different media, whereas using separate RTP sessions permits
      either single- or multiple-process implementations.

   Using a different SSRC for each medium but sending them in the same
   RTP session would avoid the first three problems but not the last
   two.

   On the other hand, multiplexing multiple related sources of the same
   medium in one RTP session using different SSRC values is the norm for
   multicast sessions.  The problems listed above don't apply: an RTP
   mixer can combine multiple audio sources, for example, and the same
   treatment is applicable for all of them.  It may also be appropriate
   to multiplex streams of the same medium using different SSRC values
   in other scenarios where the last two problems do not apply.

As can be seen if one puts audio and video in the same RTP session but
with unique SSRCs for each stream one still do need to take care.

Issue 4, can be solved by explicit signalling, but makes the solution
more fragile in cases with dynamic memberships. If you don't have the
inforamtion about the stream mixing decisions are more difficult.

Issue 5, I think becomes important as soon as we talk about a
heterogeneous set of session participants. A case that are part of the
charter to consider and solve.

In general I think this discussion should up level for a while. Lets
start discussing the general architecture and use cases and what
requirements that puts on the different nodes in the architecture. Then
we can return to this and see what requirements we have and how they
matches the capabilities of the protocols we intended to solve it with.

Cheers

Magnus Westerlund

SE-164 80 Stockholm, Sweden| mailto: [email-masked]

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 1 Feb 2011 08:34:17 -0500

As the general media type is part of the m= line one
can't have both RTP payload types of both audio and video type in the
same m= line.
>>>There might be some confusion here, I didn't hear anyone propose
multiplexing both audio and video in the same session.  I believe the
multiplex idea is to have two sessions, one for audio and one for video.
This is what existing video systems that employ multiplexing do.

In general I think this discussion should up level for a while.
Probably a good idea.  This will come back, there are a lot of folks who
build telepresence systems who favor SSRC multiplexing.

Stephen Botzko

---

From: Alex Eleftheriadis <[email-masked]>
Date: Tue, 1 Feb 2011 09:37:51 -0500

On Feb 1, 2011, at 3:34 PM, Stephen Botzko wrote:

>>>There might be some confusion here, I didn't hear anyone propose multiplexing both audio and video in the same session.  I believe the multiplex idea is to have two sessions, one for audio and one for video.  This is what existing video systems that employ multiplexing do.

Yes, I don't think anyone intentionally suggested that the two (audio and video) are muxed.

Probably a good idea.  This will come back, there are a lot of folks who build telepresence systems who favor SSRC multiplexing.

Certainly (and this goes beyond pure telepresence). We will need some additional signaling tools. Check out also draft-lennox-mmusic-sdp-source-selection-02, if you haven't already. For future discussion.

Incidentally, since the iPad was mentioned, you can see a demonstration of iPad running RTP (de)muxing of 3 H.264 streams in http://bit.ly/fMTP6j (posted in August 2010). This is not a "demo"; this is our regular SDK running. The complexity argument is really moot, in view of all the other things that happen in a server or endpoint.

--Alex

Stephen Botzko

On Tue, Feb 1, 2011 at 7:51 AM, Magnus Westerlund <[email-masked]<mailto:[email-masked]>> wrote:

Cheers

Magnus Westerlund

SE-164 80 Stockholm, Sweden| mailto: [email-masked]<mailto:[email-masked]>

_______________________________________________
clue mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/clue

---

From: "Elwell, John" <[email-masked]>
Date: Tue, 1 Feb 2011 16:34:31 +0100

> From: Stephen Botzko [mailto:[email-masked]] 
> Sent: 01 February 2011 11:56
> To: Elwell, John
> Cc: Charles Eckel (eckelcu); Magnus Westerlund; CLUE
> The use cases and the problem statement drafts were not 
> intended to be requirements documents.  I agree that 
> scalability needs to be discussed, and that use cases for 
> large scale conferences are needed.
[JRE] That's what I thought. So perhaps we do need to start assembling requirements, although I am not suggesting that should prevent us having these discussions on solution.

> 
> Existing telepresence systems I am aware of receive/decode 
> one video stream per display.   
> 
> However, it is quite practical to send one video stream per 
> rendered image (instead of using video transcoding), 
> especially if the sender either simulcasts multiple 
> resolutions or uses a layered video codec like SVC.  This can 
> greatly increase the number of streams that are received at 
> each endpoint, and also makes the received stream mix much 
> more dynamic. One benefit is that transcoding/transrating 
> delay is at least reduced and in many cases eliminated.
[JRE] You mention the mix being much more dynamic. So if we were to go for multiplexing at the transport level, presumably there is then the question of whether changes are negotiated at the RTP level (likely to be quicker) or at the SDP level.

John

> 
> Both approaches were listed in the problem statement draft 
> last year (in the multipoint section).  Personally I favor a 
> solution that accommodates both.
> 
> Stephen Botzko
> 
> 
> On Tue, Feb 1, 2011 at 6:30 AM, Elwell, John 
> <[email-masked]> wrote:
> 
> 
> 	I really think we should have some sort of 
> understanding on scalability requirements here. I don't think 
> this is covered in either of the two drafts to date, but I 
> may have missed it.
> 	
> 	John
> 	
>

---

From: <[email-masked]>
Date: Tue, 1 Feb 2011 18:22:41 +0100

If we use a single RTP session for all video streams (or audio streams) and stream-specific properties (e.g. stream position or image format) have to be signaled, then I guess RFC5576 will come into play....  Right?

/Bruno

De : [email-masked] [mailto:[email-masked]] De la part de Alex Eleftheriadis
Envoyé : mardi 1 février 2011 15:38
À : Stephen Botzko
Cc : CLUE
Objet : Re: [clue] Multiple Streams - RTP muxing?

On Feb 1, 2011, at 3:34 PM, Stephen Botzko wrote:

	

--Alex

Stephen Botzko

On Tue, Feb 1, 2011 at 7:51 AM, Magnus Westerlund <[email-masked]> wrote:

Cheers

Magnus Westerlund

---

From: "Henry Lum" <[email-masked]>
Date: Tue, 1 Feb 2011 09:43:08 -0800

I agree that SSRC multiplexing makes a lot of sense for this use case.
Trying to match m= lines even with the same number of cameras on both
endpoints can be ambiguous; how do you know which cameras to pair up? 

Henry

> As mentioned previously, this simplifies NAT/firewall traversal
> substantially.  It also makes asymmetric cases clearer -- if I have
> three cameras and you have two, it's very unclear to me how my m=
lines
> should map to your m= lines if we wanted to use the single-source-per-
> session model, whereas for SSRC multiplexing it's straightforward.

-------------------------------------------------------------------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail and any files attached may contain confidential and proprietary information of Alcatel-Lucent and/or its affiliated entities. Access by the intended recipient only is authorized. Any liability arising from any party acting, or refraining from acting, on any information contained in this e-mail is hereby excluded. If you are not the intended recipient, please notify the sender immediately, destroy the original transmission and its attachments and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Copyright in this e-mail and any attachments belongs to Alcatel-Lucent and/or its affiliated entities.

---

From: Peter Musgrave <[email-masked]>
Date: Wed, 2 Feb 2011 05:57:36 +0000

Great reference. I think with a few additions for CLUE specific needs this is a very strong candidate for having a muxed audio and separate muxed video signalled in SDP. 

Peter Musgrave

On 2011-02-01, at 2:37 PM, Alex Eleftheriadis wrote:

> I