Subject: Re: Ability To Synchronize Streams

From: John Segers <[email-masked]>
Date: Mon, 1 Nov 1999 14:16:33 +0100

Actually, there was an oversight.  In the model that came out of
Santiago, all streams coming from one user went into the same
Termination.  The streams within one Termination were synchronised by
default.  In the discussion following the Santiago meeting, we changed
the model for multimedia.  We do not have a mechanism to specify that
streams on different Terminations have to be synchronised.

Specifically, if there is an RTP video stream on one Termination and an
RTP audio stream on another, the protocol currently cannot specify that
they have to be synchronised.  We can either add a synchronisation
mechanism or mandate that streams that have to be synchronised must
"pass through" one Termination.

I would say that a synchronisation mechanism is best done at the context
level because it involves multiple Terminations and because I can't
think of a case where you would want to streams to be synchronised for
one user, but not for the other.

John

> Tom-PT Taylor wrote:
>
> The thought that streams are all synchronized by default may be the
> answer.  I confess I had forgotten.  This is something the multimedia
> specialists will have to think about.
>
>      -----Original Message-----
>      From:   Chip Sharp [SMTP:[email-masked]]
>      Sent:   Thursday, October 28, 1999 11:41 AM
>      To:     Taylor, Tom-PT [CAR:B904-I:EXCH];
>      [email-masked]
>      Subject:        Re: Ability To Synchronize Streams
>
>      It would definitely be easier to put it off.
>
>      One of the main reasons streams were introduced were for
>      multimedia which
>      requires at least lip-synch.
>
>      My memory of Santiago was that streams in a context were synched
>      by
>      definition.  I thought that was the main reason for streams.
>      Otherwise, we
>      could get rid of streams and just have each media stream in its
>      own context
>      with a (synch=ContextID) parameter indicating the contexts that
>      should be
>      synchronized.
>
>      Chip
>
>      At 10:09 AM 10/28/99 -0400, Tom-PT Taylor wrote:
>      >This is to remind you of an oversight arising out of our
>      multimedia
>      >discussions last spring.  We need a way to indicate which
>      streams are to be
>      >synchronized (e.g. synchronization of voice with video) on exit
>      from the MG.
>      >My proposal was that such synchronization would be indicated at
>      each
>      >termination by including a "synch=StreamId" property within one
>      stream
>      >descriptor to point at another stream to which it should be
>      bound.  On
>      >second thought, it may be that we actually want to do this at
>      the context
>      >level, which means that the whole thing should be deferred for
>      version 2
>      >like the bridging context property has been.
>      >
>      >Comments?
>      >
>      >Tom Taylor
>      >Advisor -- IPConnect Standards
>      >E-mail: [email-masked] (internally, Tom-PT Taylor)
>      >Phone and FAX: +1 613 736 0961
>
>      Support NetAid!  http://www.netaid.org
>      --------------------------------------------------
>      Chip Sharp                 Consulting Engineering
>      Cisco Systems              Telco Bio-region
>      Reality - Love it or Leave it.
>      --------------------------------------------------

--
John Segers                                  email: [email-masked]
Lucent Technologies                                        Room HE 306
Dept. Forward Looking Work                      phone: +31 35 687 4724
P.O. Box 18, 1270 AA  Huizen                      fax: +31 35 687 5954

---

From: Tom-PT Taylor <[email-masked]>
Date: Tue, 23 Nov 1999 08:54:59 -0500

Steady on: this isn't your action item.  The monomedia vs. multimedia
context debate was decided in Santiago, and I'm not about to reopen it.  We
definitely have more work to do on multimedia, but I can't see us getting
along without streams.  They are essential in any instance where multiple
media types are multiplexed over the same bearer.

Synchronization only applies in the outgoing direction.  So the answer to
your first question below is that streams 1 and 2 are synchronized leaving
A, streams 1 and 3 are synchronized leaving B, and streams 2 and 3 are
synchronized leaving C.

Going back to basics, it's clear that we need some way to identify which
outgoing media flows must be synchronized with each other.  One way is as
suggested by John: implicitly synchronize everything passing out of a given
termination.  Anything you don't want synchronized gets put onto separate
terminations.  Synchronization across different transports can be managed if
you use RTP to tie everything together.

I think there's another answer: a context-level indication of which streams
are synchronized to each other.  But it does assume that RTP or the
equivalent is there to keep everything in step whether the streams pass
through the same termination or different ones.  I'd be interested in other
people's views on this.

> From: Rosen, Brian [SMTP:[email-masked]]
> Sent: Tuesday, November 23, 1999 8:05 AM
> To:   [email-masked]
> If I have 3 terminations in a context.
>         TerminationA in on streams 1 & 2
>         TerminationB is on streams 1 & 3
>         TerminationC is on streams 2 & 3
> Are all three streams synchronized?
>
> What is the difference in functionality between two contexts
> each with two terminations, one stream per context, and
> one context with 4 terminations, 2 terminations on one stream
> and two on the other?
>
> In fact, what prevents me from making an MGC that puts every
> termination in the same context and uses streams to differentiate
> which gets connected to which?  If I allow streamID on signals and events,
> this would actually work pretty well, wouldn't it?
>
> What fails to work if I never put two streams in the same
> context other than sync?  Oh, I know, decomposed H.320 gateways
> with the mux.
>
> I actually have thought through this very question for really useful
> decomposed gateways like - your desktop PC with audio and
> video (and T.120).  It every way, it's much cleaner and easier
> to never mix the audio context with the video context as long as
> you can sync (well, tearing down the call takes two actions
> instead of one).
>
> Just what is it that streams do?
>
> I think we should drop all further work on streams.  They are
> broken (maybe "not completely worked out" is a nicer way to
> say that).  I'd much prefer flushing any reference to streams,
> because I think that when we really figure out how to do
> multimedia correctly, the current text will get in the way,
> and as we have recently been reminded, we are stuck with
> backwards compatibility forever.
>
> We can totally fix sync, and streams will still be broken.
>
> Brian
>
> > From: John Segers [mailto:[email-masked]]
> > Sent: Tuesday, November 23, 1999 3:48 AM
> > To: [email-masked]
> > Chip,
> > Apparently you have a different understanding of the Santiago
> > model than
> > I do.  At the Santiago meeting Tom suggested that a Termination would
> > represent a user (or at least the streams to/from a user for one
> > session).  The idea was that synchronization would be specified among
> > streams (voice synchronized with video, but not with data).  I don't
> > remember that we resolved the question whether it would be done at the
> > Context level or the Termination level.
> >
> > The discussions after Santiago led to a rejection of this
> > notion that a
> > Termination contains the streams to/from one user.  My
> > understanding was
> > that we have to at least allow multiple streams to be on different
> > Terminations.  It seems that we don't have to allow multiple,
> > synchronized streams to/from one user on different
> > Terminations (except
> > when they are multiplexed on to several physical Terminations).
> >
> > Nevertheless, if we want to consider multimedia with audio, video and
> > data, we cannot simply assume all streams in the Context to be
> > synchronized.  If I have a session with three streams (audio,
> > video-speaker, video-document-camera), I don't necessarily lipsync all
> > three.  I think we need to be able to specify which streams to
> > synchronize.
> >
> > Given the time frame and the more important matters (events,
> > signals!),
> > I propose that we assume for now that 1) streams to be
> > synchronized will
> > be on one Termination, and 2) all streams on the same Termination are
> > synchronized.  (I have to admit that I don't know if 2) makes
> > sense if a
> > termination uses different types of transport for different streams).
> >
> > John
> >
> > Chip Sharp wrote:
> > >
> > > I agree.  In the RTP case, most likely both streams will go
> > to the same IP
> > > address with a different port.
> > >
> > > I had assumed out of Santiago that streams in a context
> > were synchronized
> > > by default.  Since contexts are an artificial construct anyway this
> > > shouldn't be too hard.
> > >
> > > Are you suggesting that we have a synchronization parameter
> > for contexts
> > > similar to the topology parameter?
> > >
> > > BTW,  SDP should be able to provide a session description
> > for both audio
> > > and video from the same RTP termination.  That would be one
> > way of doing it
> > > for the SDP encoding.  I don't know about the binary version.
> > >
> > > Chip
> > >
> > > At 02:48 PM 11/3/99 +0100, John Segers wrote:
> > > >Tom,
> > > >
> > > >What you say is correct, but does not address the problem
> > I referred to.
> > > >
> > > >Consider an audio/video call.  On the IP side, there would be two
> > > >Terminations for the one call: one for audio and one for video.
> > > >Synchronization of the streams on these two Terminations could be
> > > >desired (lip synch).  I don't see how a rule like the one
> > you mention
> > > >would achieve this.
> > > >
> > > >John
> > > >
> > > > > Tom-PT Taylor wrote:
> > > > >
> > > > > Streams on different terminations are correlated by
> > StreamID: StreamID
> > > > > is context-wide.  In the case you describe where there
> > is video on one
> > > > > termination and audio on the other, there is no
> > synchronization to be
> > > > > done -- neither termination handles both streams.  Or
> > maybe you mean
> > > > > to say that one termination is receiveOnly for audio
> > while the other
> > > > > termination is receiveOnly for video.  Same thing -- no need for
> > > > > synchronization until at least one termination changes
> > to a mode where
> > > > > it is sending out both streams.  At that point, the
> > rule that says
> > > > > that all streams in a context are synchronized to the
> > extent this is
> > > > > meaningful assures the required synchronization.
> > > > >
> > > > >      From:   John Segers [SMTP:[email-masked]]
> > > > >      Sent:   Monday, November 01, 1999 8:17 AM
> > > > >      To:     [email-masked]
> > > > >      default.  In the discussion following the Santiago
> > meeting, we
> > > > >      changed
> > > > >      the model for multimedia.  We do not have a
> > mechanism to specify
> > > > >      that
> > > > >      Specifically, if there is an RTP video stream on
> > one Termination
> > > > >      and an
> > > > >      mechanism or mandate that streams that have to be
> > synchronised
> > > > >      must
> > > > >      I would say that a synchronisation mechanism is
> > best done at the
> > > > >      context
> > > > >      level because it involves multiple Terminations
> > and because I
> > > > >      can't
> > > > >      > The thought that streams are all synchronized by
> > default may be
> > > > >      the
> > > > >      > answer.  I confess I had forgotten.  This is
> > something the
> > > > >      multimedia
> > > > >      >      My memory of Santiago was that streams in a
> > context were
> > > > >      synched
> > > > >      >      by
> > > > >      >      could get rid of streams and just have each
> > media stream
> > > > >      in its
> > > > >      >      own context
> > > > >      >      with a (synch=ContextID) parameter
> > indicating the contexts
> > > > >      that
> > > > >      >      should be
> > > > >      >      synchronized.
> > > > >      >
> > > > >      >      Chip
> > > > >      >
> > > > >      >      >synchronized (e.g. synchronization of
> > voice with video)
> > > > >      on exit
> > > > >      >      from the MG.
> > > > >      >      >termination by including a
> > "synch=StreamId" property
> > > > >      within one
> > > > >      >      stream
> > > > >      >      >descriptor to point at another stream to
> > which it should
> > > > >      be
> > > > >      >      bound.  On
> > > > >      >      >second thought, it may be that we actually
> > want to do
> > > > >      this at
> > > > >      >      the context
> > > > >      >      >E-mail: [email-masked]
> > (internally, Tom-PT
> > > > >      Taylor)
> > > > >      Lucent Technologies
> >         Room
> > > > >      HE 306
> > > > >      Dept. Forward Looking Work
> > phone: +31 35 687
> > > > >      4724
> > > > >      P.O. Box 18, 1270 AA  Huizen
> > fax: +31 35 687
> > > > >      5954
> > > >
> > > >--
> > > >John Segers                                  email:
> > [email-masked]
> > > >Lucent Technologies
> > Room HE 306

---

From: "Rosen, Brian" <[email-masked]>
Date: Tue, 23 Nov 1999 11:35:27 -0500

I don't know how to fix all of this.  What I think is that
there are more questions than answers.  I think we have to
spend quite a lot of time working out how real multimedia
gateways work, and how the MGC wants to control them.
Use cases would be really nice.  Examples would be very
helpful.  Understanding interactions between one MGC
controlling multiple MGs that had different media for the
same call should be considered.

Doing this after we have hacked things like topology, sync,
who gets a streamID, etc, etc is asking for serious
compromises on good solutions to preserve backwards
compatibility with bad solutions.

So, I'm asking for a moratorium on multimedia.  There is NO
burning need to specify sync at this time.  Whatever the answer
is, it can be decided when we actually go through and define
how to do multimedia.  Doing it now is not helpful,
there is not enough spec to do any multimedia gateway,
half fixing this problem doesn't solve that.  You risk
doing it wrong, and binding your hands when you figure
out what right is.

In this particular case you have to:
        Define exactly what the MG does when it is asked
          to sync two streams, considering the range of
          network types, etc.  It's not all RTP issues.
        Define what kinds of things CAN be synced (what does
                sync of "data" mean?
        Define what happens if the MG can't maintain sync
        Define things like sync tolerance
        Decide if you need controls on the above, properties
                to report capabilities for such, etc.
Dealing with the syntax of how to specify sync is not the only problem.

Much better to be silent on the matter, unless you define the
behavior exactly.

Brian

> From: John Segers [mailto:[email-masked]]
> Sent: Tuesday, November 23, 1999 9:03 AM
> To: Rosen, Brian
> Cc: [email-masked]
> Brian,
> I don't know what the intention of your message is, but I
> find it highly
> confusing.  See my comments below.
>
> John
>
> "Rosen, Brian" wrote:
> >
> According to my proposal, the outgoing streams on TeminationA are
> synchronized, as are those on TerminationB and TerminationC.  If you
> implement synchronization by buffering outgoing packets per
> termination
> based on RTP time stamp, do you call the three streams then
> synchronized?
>
> Difference in how you tear down the call.  Contexts give you a way to
> group terminations that for some reason are associated.  If you are
> asking about differences in functionality when not considering this
> grouping, the answer is that there are no differences.
>
> You'll have a hell of a lot of topology information to describe and
> tearing down a multipoint session requires larger messages.
>
> And you want to rule those out forever, I guess.
>
> >
> Hmm.  I gather that you think that multimedia should be handled by
> multiple contexts, right?  Let's not reopen that discussion now, we've
> spent enough bandwidth on it already.
>
> I haven't seen an argument convincing me that streams are
> broken.  Until
> I see one, I suggest we leave them unchanged.
>
> > Brian
> >
> > > > > >      RTP audio stream on another, the protocol
> currently cannot
> > > > > >      specify that
> > > > > >      >      definition.  I thought that was the
> main reason for
> > > > > >      streams.
> > > > > >      >      Otherwise, we
> > > > > >      >      >My proposal was that such
> synchronization would be
> > > > > >      indicated at
> > > > > >      >      each
> > > > > >      >      >level, which means that the whole
> thing should be
> > > > > >      deferred for
> > > > > >      >      version 2
> > > Room HE 306
>  Room HE 306