Subject: Re: [MMUSIC] Remove msid-semantic

From: Harald Alvestrand <[email-masked]>
Date: Wed, 15 Apr 2015 15:41:38 +0200

On 03/31/2015 06:32 PM, Martin Thomson wrote:
> On 31 March 2015 at 02:07, Magnus Westerlund
> <[email-masked]> wrote:
>> Just to be clear here and ensure that I haven't missed anything. RTP
>> usage requires all WebRTC endpoint to provide synchronization
>> information when sending. However, it is up to the receiver to decide if
>> they want to synchronize the playout. Thus this LS group would only be
>> there to indicate a strong preference by the sender that the receiver
>> should/shall playout in synchronization?
>
> Correct, the information in RTP will make it possible to synchronize
> tracks, but whether that is acted upon will ultimately depend on
> whether the tracks are rendered in the same stream.  I think that the
> intent with the LS groups is to make this clear, not just to a WebRTC
> receiver, but to legacy receivers as well.
OK, here is how the section on handing unsignalled tracks looks when 
I've removed msid-semantic.

Note that depending on exactly how we do early media, this may cause 
trouble in delivering it.

3.1.  Handling of non-signalled tracks

    Entities that do not use msid will not send msid.  This means that
    there will be some incoming RTP packets that the recipient has no
    predefined MediaStream id value for.

    Note that this handling is triggered by incoming RTP packets, not by
    SDP negotiation.

    When MSID is used, the only time this can happen is when, at a time
    subsequent to the initial negotiation, a negotiation is performed
    where the answerer adds a MediaStreamTrack to an already established
    connection and starts sending data before the answer is received by
    the offerer.  For initial negotiation, packets won't flow until the
    ICE candidates and fingerprints have been exchanged, so this is not
    an issue.

    The recipient of those packets will perform the following steps:

    o  When RTP packets are initially received, it will create an
       appropriate MediaStreamTrack based on the type of the media
       (carried in PayloadType), and use the mid RTP attribute (if
       present) to associate the RTP packets with a specific media
       section.  If the connection is not in the RTCSignalingState
       "stable", it will wait at this point.

    o  When the connection is in the RTCSignalingState "stable", it will
       look at the relevant media section to find the msid attribute.

    o  If there is an msid attribute, it will use that attribute to
       populate the "id" field of the MediaStreamTrack and associated
       MediaStreams, as described above.

    o  If there is no msid attribute, the identifier of the
       MediaStreamTrack will be set to a randomly generated string, and
       it will be signalled as being part of a MediaStream with the
       WebIDL "label" attribute set to "Non-WebRTC stream".

    o  After deciding on the "id" field to be applied to the
       MediaStreamTrack, the track will be signalled to the user.

    The process above may involve a considerable amount of buffering
    before the stable state is entered, If the implementation wishes to
    limit this buffering, it MUST send a "mediadiscarded" event to the
    user, as described in [????].

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 21 Apr 2015 11:31:31 +0200

Hi Harald,

>    user, as described in [????].

I think there are two aspects of the buffering here that may need a bit
more discussion. So one buffers to ensure that one have a good media
encoding entry point. However, in most cases one can't really play out
all the buffered media, unless one can play it at faster than real-time.
Video fairly simple, audio a bit trickier, especially if one likes to
catch up quickly with the minimal delay point. Thus, one either miss
media or may end up with a significant time delay that take time to
drain. Catching up is also resource demanding as one must be able to
decode faster than real-time.

Based on this, I wonder if you there need to be clearer recommendations
what to do. For example if I discard up until the last receiver media
entry point available, should I then send mediadiscarded event? Is the
application happier if I play from start and then spend 15 seconds
catching up? How do we know which behaviour an application prefer here?

Maybe the issue and possible strategies should be highlighted, and
further note that application may want to ensure that signalling has
concluded before sending media.

I am not happy to have to go into detail here, but it also looks like
this will be one pile of inconsistent behaviour between implementations.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Services, Media and Network features, Ericsson Research EAB/TXM
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Färögatan 6                 | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Martin Thomson <[email-masked]>
Date: Tue, 21 Apr 2015 10:05:37 -0700

On 21 April 2015 at 02:31, Magnus Westerlund
<[email-masked]> wrote:
> However, in most cases one can't really play out

I never imagined buffering being any more than a few milliseconds or
tens of milliseconds.  That might just be a case of allowing the audio
jitter buffer to fill.

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 22 Apr 2015 14:15:14 +0200

> tens of milliseconds.  That might just be a case of allowing the audio

Well, that is fine for audio codecs which can generate good quality
audio from any frame. But, for a video codec with its dependency on
intra frames, then not buffering, in case the signalling arrives after
the media, then discarding just the first packet will result in quality
degradation or extra delay.  The actual situation will be the same as a
packet loss situation (without possibility to repair) and recovery form
it, most likely through a FIR request.

To be clear this only occurs in the situation where the signalling for a
new track arrives after the actual delivery.

Cheers

Magnus Westerlund

---

From: Harald Alvestrand <[email-masked]>
Date: Wed, 22 Apr 2015 14:29:03 +0200

> new track arrives after the actual delivery.

There's some simplification possible ... if the codec uses some variant
of pyramid encoding, where not all frames are required in order to
preserve the reference chain, all frames that aren't depended on can be
thrown away until the decision to show or not show can be made. When the
decision to show is made, just the frames required to have a proper
reference for the next incoming frame need to be encoded.

Probably increases reasonable hold times from milliseconds to seconds,
but not to minutes.....

---

From: Martin Thomson <[email-masked]>
Date: Wed, 22 Apr 2015 10:13:02 -0700

> Probably increases reasonable hold times from milliseconds to seconds,

Yep, if you can decode properly, holding the iframe - or a functional
equivalent - could make the hold time indefinite.

---

From: Harald Alvestrand <[email-masked]>
Date: Thu, 23 Apr 2015 11:36:30 +0200

> tens of milliseconds.  That might just be a case of allowing the audio

The inter-frame time of 30 fps video is 33 ms, so if you buffer 3
frames, you have 100 ms.

We're always in a hurry, but not THAT much of a hurry....