Subject: [MMUSIC] Proposal for what bundle should say about demux

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Thu, 23 May 2013 18:02:37 +0000

Here's is my proposal for roughly what the bundle draft should say about this demux topic 

Application will decide which packet processing pipeline to pass an given RTP/RTCP packet to based on what the application knows:

1) If future RFCs define new things (like RTP header extension), that explicitly specify the mapping, check if that future RFC is in use and if so then use that to form the mapping 

2) If the PT type is uniquely identifies a mapping, use that to form the mapping

3) If application knows the SSRC the other side will use, use that to form the mapping 

4) if there is no way to know which pipeline to pass it too, the packet MAY be discarded or the application MAY decide to buffer it until the mapping is known 

This is trivial to implement. It meets the requirements for Plan A, Plan B, UCIF, CLUE, and so on. 

We could swap the order of step 2 and 3, My thinking for this order was the only time it made any difference the order was if the PT were unique and indicated a different mapping than the SSRC. The only way this could happen is with a SSRC collision so the PT is the one that would be correct not the SSRC. If someone feels strongly the order of steps 2 and 3 should be the opposite way around, I can live with that.

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 23 May 2013 14:39:33 -0400

Cullen,

I just posted a separate, related, message on associating packets to 
m-lines.

IMO they are complementary, and somewhat layered:

First, the packet should be matched to the m-line.
The m-line then provides an bunch of related info:
- mapping from PT to codec
- bandwidth info
- per-ssrc info
- labels that can be bound to application specific data
- ...

All of that info is then available to help associate the packet to a 
processing pipeline. Then the algorithms you talk about kick in.

Of course that is the hypothetical layering. It is subject to 
optimization. Generally, once one packet has been bound to a processing 
pipeline then the SSRC from that can be remembered, and used to shortcut 
the mapping for future packets that have the same SSRC.

	Thanks,
	Paul

On 5/23/13 2:02 PM, Cullen Jennings (fluffy) wrote:
> _______________________________________________
> mmusic mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mmusic
>

---

From: Emil Ivov <[email-masked]>
Date: Thu, 23 May 2013 23:49:40 +0300

Hey all,

So first of all, I'd like to say that I like the approach and think this
is the way to go.

I have just one concern here. I already mentioned it in the call but I
am not quite sure what the answer was. So here I go:

> 1) If future RFCs define new things (like RTP header extension), that explicitly specify the mapping, check if that future RFC is in use and if so then use that to form the mapping 

So my problem with having the above as number one is that while it would
work perfectly with protocols like XMPP that have good discovery
support, it would be a problem for SDP O/A based ones like SIP and WebRTC.

The reason is that even when demuxing RFCs like that exist, an offerer
won't know if the remote party supports them. They would hence always
need to construct an offer where a fallback to another mechanism could
be possible (like for example falling back to uniquely allocated PTs).

Now, if the offerer does indeed know in advance that the fancy demuxing
RFC will be supported, and only then, they would be able to rely on it
and be careless about PT allocation.

I believe we've built bundle port allocation and trickle ICE (and a
number of other O/A mechanisms) around the same model.

Emil

-- 
https://jitsi.org

---

From: Martin Thomson <[email-masked]>
Date: Thu, 23 May 2013 13:57:49 -0700

On 23 May 2013 11:39, Paul Kyzivat <[email-masked]> wrote:
> Generally, once one packet has been bound to a processing pipeline then the
> SSRC from that can be remembered, and used to shortcut the mapping for
> future packets that have the same SSRC.

This is actually the key.  The pipeline should be attached to the
SSRC.  How that attachment is initially formed is what I think that
Cullen is talking about.

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 23 May 2013 17:34:16 -0400

> SSRC.  How that attachment is initially formed is what I think that

Well, I said *generally*, not *always*.

For instance, we have a clue case where we can have two clue switched 
captures, one representing the video for loudest speaker, and one for 
the 2nd loudest speaker. A single SSRC can move from one of those 
captures to the other. Its the capture (identified by an rtp header 
extension) that selects the processing pipeline.

Or, you can view it as layered. Where the ssrc maps to an early stage of 
the pipeline (selected by ssrc) that does the codec processing, and then 
a different stage of the pipeline (selected by capture id from the 
header extension) that switches the decoded output to a particular 
rendering device.

	Thanks,
	Paul

---

From: "Mo Zanaty (mzanaty)" <[email-masked]>
Date: Thu, 23 May 2013 21:48:08 +0000

I agree with this, including the order of 2 and 3.

I was initially averse to PT demux, for RTP purist reasons (since it is intended to convey codecs not sources) and practical reasons (limited space of 32-96 code points, depending on your interop risk tolerance, for all combinations of codecs and sources). But I have warmed up to it based on a realization that I (and maybe others) missed.

When using rtcp-mux, there must *always* be an initial step:
0) PT demux to classify RTP vs. RTCP
...before you can look at RTP header extensions in 1 (no X bit in RTCP headers)
...before you can look at SSRCs in 3 (different SSRC offsets in RTP vs. RTCP headers)

Given step 0, it is hard to argue against PT demux from a RTP purist view. The practical problem of the limited PT space remains. But for simple cases, like a single audio and video stream with a reasonable number of codec formats, I see no reason to disallow or discourage this, since step 2 can be folded into step 0. For more complex cases, with more sources and codec formats, FEC/RTX, scalable/simulcast layers, screen sharing, multiple cameras, etc., it quickly becomes impractical or impossible. In that sense, Plan A may work best for simple cases, while Plan B may work best for complex cases. But both could still share these common demux rules.

Mo

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 23 May 2013 17:59:13 -0400

> and be careless about PT allocation.

It will work for CLUE, because CLUE won't start using it on the first 
O/A. It will only do so after it knows that it makes sense. And in clue 
it can potentially take precedence over PT. *In theory* we could have 
one switched capture that sometimes gets an H.264 RTP stream, and other 
times gets a VP8 RTP stream. (I don't know if that is likely to happen 
in practice. Jonathan might have more to say about that.)

IMO the "before the O/A completes" issues are separate from the "after 
the O/A completes" issues. I'm still not convinced that the "before the 
O/A completes" issues are important. (We have reliable 183 responses to 
deal with it.) But I'm ok with Cullen using PT to solve that problem if 
he thinks he needs to.

I'm more concerned with having a well defined way of associating packets 
with m-lines in the bundle after the O/A is complete.

	Thanks,
	Paul

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 23 May 2013 18:07:15 -0400 (EDT)

> From: Paul Kyzivat <[email-masked]>
> 
> Cullen,
> 

I think Cullen is addressing both layers together, but his proposal is
best looked at by splitting it into two parts, one that addresses the
first layer and one that addresses the second layer.

The reason that splitting the layers is useful is that the second
layer problem has been solved -- or solved in multiple ways -- but it
is a problem somebody already has to solve, and any legacy situation
has solved it in some particular way.  So we only have to solve the
first layer problem.

As I read it, Cullen proposes that the first layer demultiplexing by

1) looking for an RTP extension header, or RTCP SDES extension
information, that tells which m= line the packet is allocated to

2) seeing if the payload type identifies a unique m= line

3) seeing if the SSRC is specified by some attribute in an an m= line

The second layer demultiplexing is close to the "A vs. B" problem, how
do you differentiate packets for the application processing streams,
which is closely tied to which application processing streams can
share the same m= line, and how that sharing is signaled.

Dale

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Thu, 23 May 2013 22:22:33 +0000

> number of other O/A mechanisms) around the same model.

What I had in mind was something along the lines of the following 

A is sending an offer that forks to B and C. A and B support the new mechanisms but C does not. A has no idea if the things it is talking to support the new thing. 

A constructs an offer that in the SDP has an indication it can receive the header extension. Perhaps something like "a=new-hdr-ext=123456". The new header extension allows one to embed an identifer in the RTP packets and this SDP indicates that the identifier should be 123456 for RTP with this mapping. 

A sends the offer and it arrives at both B and C. C does not understand it and just ignores it. B understands it and inserts a TP header extension with the identifier 123456. 

Imagine the RTP packets from B and C arrive at A before any SDP answer. A looks at the RTP packet from B, and see the header exertion and reads out the identifier 123456 and now knows the mapping. A looks at the RTP packet from C and does not see the RTP header extension so it carries on to try other things. 

So yes, as you point out, for the packets from C where it can't use PT, it needs to fall back to SSRC which means A can't really process the packets from C until the answer arrives. But that is fine - if you are using something that does not support the new stuff, the behavior you get is the best we can do with the old stuff yet at the same time you get the faster behavior with things that do the new stuff. C is as fast as it can be and B is as fast as it can be. 

That seems like the type of design we want.

---

From: Martin Thomson <[email-masked]>
Date: Thu, 23 May 2013 17:17:23 -0700

On 23 May 2013 14:34, Paul Kyzivat <[email-masked]> wrote:
> Or, you can view it as layered. Where the ssrc maps to an early stage of the
> pipeline (selected by ssrc) that does the codec processing, and then a
> different stage of the pipeline (selected by capture id from the header
> extension) that switches the decoded output to a particular rendering
> device.

That is what I would do for that case.  At the layer I'm concerned
about, this is still the same stream.  The fact that someone
downstream might use other information to wiggle that around doesn't
concern me greatly.

---

From: Emil Ivov <[email-masked]>
Date: Fri, 24 May 2013 10:08:00 +0300

Hey Cullen,

On 24.05.13, 01:22, Cullen Jennings (fluffy) wrote:
> <snip>
> A is sending an offer that forks to B and C. A and B support the new
> mechanisms but C does not. A has no idea if the things it is talking
> to support the new thing.

Exactly.

> A constructs an offer that in the SDP has an indication it can
> receive the header extension. Perhaps something like
> "a=new-hdr-ext=123456". The new header extension allows one to embed
> an identifer in the RTP packets and this SDP indicates that the
> identifier should be 123456 for RTP with this mapping.

OK.

> A sends the offer and it arrives at both B and C. C does not
> understand it and just ignores it. B understands it and inserts a TP
> header extension with the identifier 123456.
> 
> Imagine the RTP packets from B and C arrive at A before any SDP
> answer. A looks at the RTP packet from B, and see the header exertion
> and reads out the identifier 123456 and now knows the mapping. A
> looks at the RTP packet from C and does not see the RTP header
> extension so it carries on to try other things.

That's it. This is the point that I am talking about. In order for A to
be able to fall back to something, it needs to have attributed PTs such
that they could be used for demuxing.

So in other words: when constructing your offer, whatever you do, you
MUST make sure that your PTs would be enough for you to match to m=
lines, UNLESS you are certain that the other party would support some
other mechanism for doing the same.

Does this make sense?

> So yes, as you point out, for the packets from C where it can't use
> PT, it needs to fall back to SSRC 

SSRC is also a non-guaranteed approach because you don't know if C is
going to pre-announce them in SDP.

Emil

-- 
https://jitsi.org

---

From: Emil Ivov <[email-masked]>
Date: Fri, 24 May 2013 10:17:56 +0300

> deal with it.) But I'm ok with Cullen using PT to solve that problem if 

I wasn't referring to "before the O/A completes" situations. If your
peer does not support fancy-RFC m-line matching nor does it provide
SSRCs, you have to be able to fall back to PT demuxing.

Therefore, all your offers have to accommodate for that possibility
unless you know that the offer is going to someone who also supports
other mechanisms.

Emil

-- 
https://jitsi.org

---

From: Kevin Dempsey <[email-masked]>
Date: Fri, 24 May 2013 09:40:16 +0100

Isn't the onus on the answerer to ensure the demux is possible, since it is
accepting the bundle. Therefore if it doesn't support signalling SSRCs or
an offerred 'fancy-RFC m-line matching' method, it must only send PTs that
are tied to a single m-line.

---

From: Emil Ivov <[email-masked]>
Date: Fri, 24 May 2013 11:59:04 +0300

On 24.05.13, 11:40, Kevin Dempsey wrote:
> Isn't the onus on the answerer to ensure the demux is possible, since it
> is accepting the bundle. Therefore if it doesn't support signalling
> SSRCs or an offerred 'fancy-RFC m-line matching' method, it must only
> send PTs that are tied to a single m-line.

PTs are defined by the offerer. The answerer cannot change them for the
offerer-bound direction (even if it could request different values for
the packets that it, the answerer, is going to receive).

Emil

> 
> 
> On 24 May 2013 08:17, Emil Ivov <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     [email-masked] <mailto:[email-masked]>

-- 
https://jitsi.org

---

From: Kevin Dempsey <[email-masked]>
Date: Fri, 24 May 2013 10:02:55 +0100

I didn't say anything about changing the PTs. The answerer decides what it
codecs it will use from the ones listed in the offer. If it doesn't support
any other mechanism to allow demux then it must choose codecs that have PTs
on only one m-line, or not accept the BUNDLE.

---

From: Emil Ivov <[email-masked]>
Date: Fri, 24 May 2013 12:14:51 +0300

On 24.05.13, 12:02, Kevin Dempsey wrote:
> I didn't say anything about changing the PTs.

Sorry, I misunderstood.

> The answerer decides what
> it codecs it will use from the ones listed in the offer. If it doesn't
> support any other mechanism to allow demux then it must choose codecs
> that have PTs on only one m-line, or not accept the BUNDLE.

Matching packets to m= lines is the concern of the receiving party. So
is the mechanism that it uses. What you are suggesting is for the
answerer to basically say: "It seems to me that you won't be able to
demux these two payloads so I will not send them to you because I know
better".

In the same time the offerer may be planning on using a demux technique
that the answerer simply isn't aware of. For example (and this is really
just to make a point, so please don't assume I am suggesting something
like this in general) if 101 is mapped to both telephone-event and VP8,
demuxing can be easily made without any other knowledge of SSRC or
header extensions.

Cheers,
Emil

> 
> 
> On 24 May 2013 09:59, Emil Ivov <[email-masked]
>     <mailto:[email-masked]>
>     > <mailto:[email-masked] <mailto:[email-masked]>>> wrote:
>     >
>     <mailto:[email-masked] <mailto:[email-masked]>>

-- 
https://jitsi.org

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 24 May 2013 10:16:19 -0400

> demuxing can be easily made without any other knowledge of SSRC or

ISTM that after the O/A exchange both sides need to have a common 
understanding that their communications will be understandable.

If the combination of the offer and answer aren't sufficient for each 
side to encode what it sends in a way that it knows can be demuxed 
according in accord with the O/A, then that offer should be incorrect.

IMO injecting "magic" into this isn't helpful.

	Thanks,
	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 24 May 2013 10:23:34 -0400

> downstream might use other information to wiggle that around doesn't

But it has an impact on what is signaled.

If I intend to use capture-id to associate with an m-line then I don't 
need to signal the SSRCs. The SSRCs may still be used to demux to a 
decoder, but for that they don't need to be known in advance.

This is important for plan A.

	Thanks,
	Paul

---

From: Emil Ivov <[email-masked]>
Date: Fri, 24 May 2013 17:40:25 +0300

> understanding that their communications will be understandable.

I assume this means you agree that an offerer has to make sure at least one
of the demuxing mechanisms it supports will be understood by the answerer.

Emil

--sent from my mobile
>>>      >     > deal with it.) But I'm ok with Cullen using PT to solve
that
>>>      >     problem if

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 24 May 2013 10:57:32 -0400

On 5/24/13 10:40 AM, Emil Ivov wrote:
> On May 24, 2013 5:16 PM, "Paul Kyzivat" <[email-masked]
> <mailto:[email-masked]>> wrote:
>  >
> I assume this means you agree that an offerer has to make sure at least
> one of the demuxing mechanisms it supports will be understood by the
> answerer.

No. But the offerer had better ensure that there is some way to answer 
that meets the requirements.

Then it becomes a requirement on the answerer to answer in a way that 
makes the communication unambiguous. If it can't do that while bundling 
then it must remove the bundle, or remove some m-lines from the bundle, 
or reject some of the m-lines in the bundle, so that the resullting 
answer constructs an unambiguous communication.

If the intent is to rely on "magic" then that can be achieved by using 
fewer m-lines in the bundle, in a way that dumps more streams/flows into 
the same m-line. So this is really only an issue for plan A.

	Thanks,
	Paul

---

From: Emil Ivov <[email-masked]>
Date: Fri, 24 May 2013 18:03:53 +0300

Paul,

This looks very much like what I suggested initially in my response to
Cullen, so I am not sure why you sound as if you are disagreeing.

Emil

--sent from my mobile
On May 24, 2013 5:57 PM, "Paul Kyzivat" <[email-masked]> wrote:

>> <mailto:[email-masked]>**> wrote:
>>  >
> then it must remove the bundle, or remove some m-lines from the bundle, or
> reject some of the m-lines in the bundle, so that the resullting answer
> constructs an unambiguous communication.
>

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 24 May 2013 11:24:01 -0400

> Cullen, so I am not sure why you sound as if you are disagreeing.

Looking back, there is some similarity, but still some difference.

IIUC you are saying that the if the offerer doesn't know the 
capabilities of the answerer then it must construct the offer so that 
any possible answer that accepts the bundle will allow proper 
association to the correct m-line. And I am not saying that.

I am saying that the offer may indeed depend on the answer supporting 
some mechanism specified in the offer in order to make the association 
unique. If the answerer doesn't recognize that mechanism (and so ignores 
it) it should be able then to recognize that the answer it would 
construct will not provide uniqueness. It then needs to fix that, by 
refusing the bundle, or refusing some of the m-lines, or whatever.

	Thanks,
	Paul

> Emil
>
> --sent from my mobile
>
> On May 24, 2013 5:57 PM, "Paul Kyzivat" <[email-masked]
>         <mailto:[email-masked]>
>         <mailto:[email-masked] <mailto:[email-masked]>>__>
>         wrote:
>           >
>     No. But the offerer had better ensure that there is some way to
>     answer that meets the requirements.
>
>     Then it becomes a requirement on the answerer to answer in a way
>     that makes the communication unambiguous. If it can't do that while
>     bundling then it must remove the bundle, or remove some m-lines from
>     the bundle, or reject some of the m-lines in the bundle, so that the
>     resullting answer constructs an unambiguous communication.
>
>     If the intent is to rely on "magic" then that can be achieved by
>     using fewer m-lines in the bundle, in a way that dumps more
>     streams/flows into the same m-line. So this is really only an issue
>     for plan A.
>
>              Thanks,
>              Paul
>

---

From: Colin Perkins <[email-masked]>
Date: Mon, 27 May 2013 13:41:35 +0100

I'm not sure I agree.

As I said in my previous message to the list, if we are agreed that the m= lines in a BUNDLE group form a single RTP session, then I believe we need unique payload types across all m= lines. In this case, BUNDLE can simply say that regular RTP source demultiplexing based on the SSRC has to be performed, then the payload type can be used to match sources to m= lines for those applications that care about doing so. 

If we're not agreed that the m= lines in a BUNDLE group form a single RTP session, then we have a lot more to discuss...

Colin

-- 
Colin Perkins
http://csperkins.org/

---

From: Christer Holmberg <[email-masked]>
Date: Mon, 27 May 2013 12:48:57 +0000

Hi,

> I'm not sure I agree.
>
> As I said in my previous message to the list, if we are agreed that the m= lines in a BUNDLE group form a single RTP session, then I believe we
> need unique payload types across all m= lines. In this case, BUNDLE can simply say that regular RTP source demultiplexing based on the SSRC 
> has to be performed, then the payload type can be used to match sources to m= lines for those applications that care about doing so. 
>

Keep in mind that the single-RTP-session has been an assumption for a long time already - it was not something we agreed upon last week :)

Regards,

Christer

-- 
Colin Perkins
http://csperkins.org/

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Mon, 27 May 2013 13:15:09 +0000

On May 23, 2013, at 3:34 PM, Paul Kyzivat <[email-masked]> wrote:

> 
> For instance, we have a clue case where we can have two clue switched captures, one representing the video for loudest speaker, and one for the 2nd loudest speaker. A single SSRC can move from one of those captures to the other. Its the capture (identified by an rtp header extension) that selects the processing pipeline.

Note my proposal put the RTP header extension ahead of looking at SSRC to meet needs of CLUE and also possible stuff from UCIF.

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Mon, 27 May 2013 13:15:18 +0000

Great - sounds like we agree this algorithm will work.

---

From: Colin Perkins <[email-masked]>
Date: Mon, 27 May 2013 14:57:23 +0100

I don't agree with the phrasing about "packet processing pipelines", but can't tell if this is a terminology disagreement or a more fundamental disconnect. The way I see the demultiplexing logically working is:

                    |
                    | packets
    +--             v
    |           +----------+
    |           |UDP socket|
 A  |           +----------+
    |        RTP ||  |  |
    |   and RTCP ||  |  +------> SCTP
    |            ||  +---------> STUN/ICE
    +--          ||
    +--          ||
    |       split by SSRC
    |       ||   ||   ||
    |       ||   ||   ||
 B  |      +--+ +--+ +--+
    |      |PB| |PB| |PB| Playout buffer, process RTCP, FEC, etc.
    |      +--+ +--+ +--+
    +--      |   |     |
    +--      |  /      |
    |        +---+     |
    |         /  |     |
 C  |      +--+ +--+ +--+
    |      |CR| |CR| |CR| Codecs and rendering
    |      +--+ +--+ +--+
    +--

If your algorithm is for demultiplexing at layer C, i.e., to figure out what codec and rendering pipeline to use, then I think we're in agreement apart from terminology. 

For layer B, I believe the SSRC is the right thing to use to demultiplex, and fits with RTP and RTCP. This is where RTCP is processed, playout de-jitter buffering happens, FEC is processed, NACKs are sent, etc. It's logically independent of the decoding and rendering process since you can start filling your de-jitter buffer for an SSRC before you figure out if/how/where you're going to render that SSRC. 

For layer A, there was a clear-cut way of doing this with RTP, RTCP, and STUN. I haven't looked at SCTP enough to know how that affects things. I do think it's a logically separate issue, and should be documented separately to BUNDLE though, since it the same issues arise with non-bundled sessions.

An implementation might merge these together, of course, but to avoid confusion the standards should be clear what level they're considering.

Colin

-- 
Colin Perkins
http://csperkins.org/

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 27 May 2013 13:31:28 -0400

Colin,

(I guess you know I am naive about RTP, so take this with a grain of salt.)

In the below, you first split by SSRC. Then later you map to codec. That 
seems to suggest that each SSRC might have a unique mapping from PT to 
codec. Did you intend that?

I also don't understand below how you expect unknown SSRCs to be 
handled. ISTM that the flow you show could result in establishing a 
bunch of state for each received SSRC, whether that SSRC is of interest 
or not.

In some cases it may be that specific SSRCs are expected, and others can 
be ignored (or whatever). In other cases the recipient may not know what 
SSRCs to expect, but may have some other way of deciding which it wants 
to process.

That was the point of my thread about associating packets to m-lines. If 
you can at least associate a packet with one of the bundled m-lines, 
even if it is by a means other than SSRC, then it becomes easier to 
decide what to do with it. For instance, I think Cullen intends that in 
plan A you could associate packets with m-lines via the PT, and then for 
each m-line process only packets for the most recently seen SSRC. (So 
when a new SSRC is received, drop all state for the prior SSRC and 
establish a new state for the new one.)

The same approach can work using a header extension identifying a clue 
capture encoding to map to an m-line. When doing that, it won't be 
necessary for the PT to be unique across m-lines.

	Thanks,
	Paul

---

From: Colin Perkins <[email-masked]>
Date: Mon, 27 May 2013 19:08:12 +0100

Paul,

On 27 May 2013, at 18:31, Paul Kyzivat wrote:
> In the below, you first split by SSRC. Then later you map to codec. That seems to suggest that each SSRC might have a unique mapping from PT to codec. Did you intend that?

That's not how RTP has worked historically. The mapping from payload type to payload format has always been done on a per-RTP session basis, with a static mapping for some types, and a dynamic binding via SDP for others.

> I also don't understand below how you expect unknown SSRCs to be handled. ISTM that the flow you show could result in establishing a bunch of state for each received SSRC, whether that SSRC is of interest or not.

If you receive RTP and/or RTCP packets you need to keep some state for that SSRC. At least, you need to record the SSRC as in-use so you can avoid collisions, and you need to send RTCP reports for that SSRC. The state requirements for this are small (tens of bytes per SSRC). If you care about decoding and rendering the source there is more state, of course, but you only need that if you're rendering their media.

You also need to be slightly smart to prevent DoS attacks if the other side decides to send you a stream of packets with a different, random, SSRC in each, but that's just basic robustness, and there are plenty of other ways to DoS a poorly engineered receiver at the RTP layer without playing tricks with SSRCs.

> In some cases it may be that specific SSRCs are expected, and others can be ignored (or whatever). In other cases the recipient may not know what SSRCs to expect, but may have some other way of deciding which it wants to process.

Sure - if you know what SSRCs you're expecting, you can pre-populate some of the state at the RTP level, and tell that layer that unknown SSRCs can be discarded. None of that changes the basic approach to de-multiplexing.

> That was the point of my thread about associating packets to m-lines. If you can at least associate a packet with one of the bundled m-lines, even if it is by a means other than SSRC, then it becomes easier to decide what to do with it. For instance, I think Cullen intends that in plan A you could associate packets with m-lines via the PT, and then for each m-line process only packets for the most recently seen SSRC. (So when a new SSRC is received, drop all state for the prior SSRC and establish a new state for the new one.)

I would suggest instead: de-multiplex into sources based on the SSRC and keep state for those according to the RTP timing rules. If you want to associate sources with m= lines via the PT, and determine what to render based on most recently active, that works fine, and works with all the various RTP and RTCP extensions (unlike de-multiplexing sources based on the PT), and gives you many more possible sources since the SSRC space is larger than the PT space.

> The same approach can work using a header extension identifying a clue capture encoding to map to an m-line. When doing that, it won't be necessary for the PT to be unique across m-lines.

The combination of an RTP Header Extension and/or RTCP SDES packet to identify an m= line would also work, and associates with the stream based on the SSRC. You don't need to use the PT to demultiplex for this.

Colin

-- 
Colin Perkins
http://csperkins.org/

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 27 May 2013 15:08:26 -0400

> That's not how RTP has worked historically. The mapping from payload type to payload format has always been done on a per-RTP session basis, with a static mapping for some types, and a dynamic bind…

Sure, I understand that. There was no way to describe that in SDP, so of 
course it wasn't done that way.

The introduction of bundling makes it possible to describe more 
possibilities. So that it becomes necessary to decide which ones will 
now be valid and which will not.

I asked because it wasn't clear. I think things can be made to work 
either way. If this is to be illegal, then the rules for forming bundles 
should say so.

> The combination of an RTP Header Extension and/or RTCP SDES packet to identify an m= line would also work, and associates with the stream based on the SSRC. You don't need to use the PT to demultip…

One of my points is that if SSRC and/or header extension value is known 
in advance, then putting that in the SDP and using it to associate to an 
m-line can relax the requirement that PTs be unique per-m-line. That 
makes it possible to support more m-lines in a bundle. This could be 
advantageous for plan A.

	Thanks,
	Paul

---

From: Colin Perkins <[email-masked]>
Date: Mon, 27 May 2013 20:16:07 +0100

On 27 May 2013, at 20:08, Paul Kyzivat wrote:
> Sure, I understand that. There was no way to describe that in SDP, so of course it wasn't done that way.
> 
> The introduction of bundling makes it possible to describe more possibilities. So that it becomes necessary to decide which ones will now be valid and which will not.
> 
> I asked because it wasn't clear. I think things can be made to work either way. If this is to be illegal, then the rules for forming bundles should say so.

Without a wholesale revision to RTP and the signalling, I believe this should be illegal (i.e., you MUST NOT map the same payload type to two different payload formats in a single bundle group). 

> One of my points is that if SSRC and/or header extension value is known in advance, then putting that in the SDP and using it to associate to an m-line can relax the requirement that PTs be unique per-m-line. That makes it possible to support more m-lines in a bundle. This could be advantageous for plan A.

I don't see any problem using the a=ssrc: or RTP header extension based methods to determine what m= line an RTP source maps to. I don't think you need to relax the requirement that PTs be unique per m= line to do this. 

As I said in the other thread, how you decide how to render an RTP source is separate to how you demultiplex RTP sources. Most of this confusion is coming from trying to combine the two operations.

-- 
Colin Perkins
http://csperkins.org/

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 27 May 2013 15:37:07 -0400

Colin,

On 5/27/13 3:16 PM, Colin Perkins wrote:
> I don't see any problem using the a=ssrc: or RTP header extension based methods to determine what m= line an RTP source maps to.

> I don't think you need to relax the requirement that PTs be unique per m= line to do this.

I agree there is no *need* to relax the PT uniqueness requirement in 
order to map the packets (or sources if you prefer) to m-lines.

But (at least with plan A) there is a concern that there will be 
insufficient PTs to keep them unique per-m-line.

What I am saying is that if you do the mapping to m-line *before* the 
mapping from PT to payload format, then you *can* relax the PT 
uniqueness requirement.

This isn't violating precedent, because there is no precedent. Neither 
the problem nor the solution are present without bundling.

	Thanks,
	Paul

---

From: Colin Perkins <[email-masked]>
Date: Mon, 27 May 2013 21:12:46 +0100

On 27 May 2013, at 20:37, Paul Kyzivat wrote:
...
> I agree there is no *need* to relax the PT uniqueness requirement in order to map the packets (or sources if you prefer) to m-lines.
> 
> But (at least with plan A) there is a concern that there will be insufficient PTs to keep them unique per-m-line.
> 
> What I am saying is that if you do the mapping to m-line *before* the mapping from PT to payload format, then you *can* relax the PT uniqueness requirement.

That would mean using signalled SSRC values to map from an RTP source to an m= line. Once you've done that, though, it's not clear that you need that many unique payload types (since if you're using the same payload format in two m= lines you can use the same payload type in both, when using a signalled SSRC to distinguish the m= lines). 

> This isn't violating precedent, because there is no precedent. Neither the problem nor the solution are present without bundling.

There's precedent in that the RTP specifications assume the mapping is fixed in a single RTP session. 

-- 
Colin Perkins
http://csperkins.org/

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 27 May 2013 17:23:22 -0400

> That would mean using signalled SSRC values to map from an RTP source to an m= line. Once you've done that, though, it's not clear that you need that many unique payload types (since if you're usin…

Yes, that may be true. If that is sufficient for people its ok with me.
But it does still mean, in response to Christer, that the same PT might 
be in multiple m-lines in the bundle.

> There's precedent in that the RTP specifications assume the mapping is fixed in a single RTP session.

There is also precedent that you don't have audio and video in the same 
RTP session. We are going to be breaking *some* precedents. Its just a 
matter of which ones are sacred.

	Thanks,
	Paul

---

From: Mary Barnes <[email-masked]>
Date: Tue, 28 May 2013 12:36:35 -0500

> Note my proposal put the RTP header extension ahead of looking at SSRC to meet needs of CLUE and also possible stuff from UCIF.
[MB] Cullen, when you are talking about UCIF, you are explicitly
talking about SVC - is that correct?  [/MB]

---

From: "Cullen Jennings (fluffy)" <[email-masked]>
Date: Thu, 30 May 2013 19:06:43 +0000

Colin, 

I've been deliberately merging layers A, B, C all into one in my discussion of this and focusing on the media stack code which has to deal with all those layers. The reasons I have been doing that is people just seemed to confused about what all the parts of RTP are such as RTP sessions. But most people seemed to understand what bits in the various packets they could look at what and what data had to end up at the right spot in the end. 

At a high level, I would draw the layers a little differently than you in that many of the implementation I work with do the jitter buffer after the data is decoded. This allows the implementations to do better packet loss concealment and play rate games where they speed up or slow down playback very slightly to smoothly adapt the jitter buffer depth. 

But regardless of the details of the order, the jitter buffer needs to look at the SSRC and if the SSRC has changed, it probably needs to retrain the jitter buffer. 

Lets talk a bit more about your diagram and what happens when Alice creates and Offer with that does not have use bundle and has two audio codecs. When an RTP packets #1 gets sent to Alice and it arrives before the answer, somewhere in layer B/C, something needs to look at the PT value to decide what codec this goes to. Alice knows nothing about what SSRC the other side is using. Now lets say that packet #2 arrives (again before answer) and packet #2 is on the same port, same PT, but has a different SSRC. None of the implementations I work with for audio would allocate a new PB or CR. Instead they would pass packet #2 throughout the same codec instance that packet #1 had used and expect things like the jitter buffer to note the SSRC change and act accordingly. For DSP based audio systems where the actual codec instances are loaded into the DSP and have a reserved set or resources allocated for them, this is a very common design. 

There is probably more variety in video applications but many conferences bridges act this way. 

So I have no problem if things split on SSRC and whatever processing that can be done knowing only that is done. That probably includes RTCP, FEC, RTX , who knows. But before the packed can be decoded by the codec, something needs to consider the PT and other things such as port when not doing bundle to make this work. Keep in mind things like PLI are much more likely to be sent at the layer near the codec than at layer B so layers B and C are not as cleanly separated as one might wish. 

That's a long way of saying, I was talking about the aggregate demux in media stack that include whatever happened at layers B and C. If layer B only used SSRC that's fine with me. I am trying to say what happens in the combination of layer B/C put together.  

What terminology should I be using to not cause confusion ?