Subject: [Slim] Correlating media streams / roles for media streams

From: Paul Kyzivat <[email-masked]>
Date: Sat, 11 Jul 2015 17:22:02 -0400

I have a feeling there is still a missing part to this puzzle. I bring 
it up without having yet formed an opinion on how to solve it, or 
whether slim is the right place to do so.

In some cases there is a need to coordinate the content in multiple 
streams, in varying ways.

One case is already mentioned in the -negotiating- draft: when it is 
desired for the content of a video stream to represent the speaker in an 
audio stream.

Another one hasn't been discussed yet: it ought to be possible to 
initiate a session with a relay/translation service with the expectation 
that the service will relay/translate between two groups of streams, 
each group representing one party in a conversation. There are many 
variations on this. E.g.,

- two audio streams, each with a different natural language. The desire
   is for translation in both directions between them.

- one group has an audio stream with English, the other has audio with
   English and video with ASL. The desire is for translation in both
   directions between spoken English and ASL.

It *may* be that the desired relationship between the streams can be 
inferred from the media types, directionality, and humaninlang 
attributes. But I think this starts to become ambiguous in any but the 
most simple cases. For instance, in the ASL example above, which audio 
"belongs" to the video stream? And, should the audio be "passed through" 
with the translation mixed in, or not? As we add more modalities it will 
get more complex.

I think we could start by trying to define heuristics for figuring out 
what to do. Beyond that, we could come up with additional SDP syntax. Or 
perhaps an independent channel, such as used in CLUE, could be used.

	Thanks,
	Paul

---

From: Eric Burger <[email-masked]>
Date: Sun, 12 Jul 2015 10:10:48 -0400

I think this can be even more complex. Granted, my experience in simultaneous translation is 25 years old, but at the time, the European Commission had a rule that one could not have more than two serial simultaneous translations in a row. The constraint was providing simultaneous translations in (what today) is 24 languages, yet doing it with a finite number of interpreters. Scheduling was an ‘interesting’ task given the serial translation limitation.

I bring this up because one can envision someone from Malta visiting Latvia. Finding someone who speaks both Maltese and Latvian is unlikely, but finding one person who speaks one of Maltese and Italian or English, and someone who speaks one of Latvian and Italian or English is pretty good. If it is a bad day, you might be able to find someone who speaks Maltese and Arabic, someone who speaks Arabic and German, and someone who speaks German and Latvian.

I get the sense we are only looking at bilateral translations (e.g., English - ASL, Vietnamese - English, French - RTT, etc.). Given the likelihood of serial interpretations in language-dense regions of the world (Europe, India, and the US come immediately to mind), are we ready for it?

> On Jul 11, 2015, at 5:22 PM, Paul Kyzivat <[email-masked]> wrote:
> 
> I have a feeling there is still a missing part to this puzzle. I bring it up without having yet formed an opinion on how to solve it, or whether slim is the right place to do so.
> 
> In some cases there is a need to coordinate the content in multiple streams, in varying ways.
> 
> One case is already mentioned in the -negotiating- draft: when it is desired for the content of a video stream to represent the speaker in an audio stream.
> 
> Another one hasn't been discussed yet: it ought to be possible to initiate a session with a relay/translation service with the expectation that the service will relay/translate between two groups of streams, each group representing one party in a conversation. There are many variations on this. E.g.,
> 
> It *may* be that the desired relationship between the streams can be inferred from the media types, directionality, and humaninlang attributes. But I think this starts to become ambiguous in any but the most simple cases. For instance, in the ASL example above, which audio "belongs" to the video stream? And, should the audio be "passed through" with the translation mixed in, or not? As we add more modalities it will get more complex.
> 
> I think we could start by trying to define heuristics for figuring out what to do. Beyond that, we could come up with additional SDP syntax. Or perhaps an independent channel, such as used in CLUE, could be used.
> 
> 	Thanks,
> 	Paul
> 
> _______________________________________________
> SLIM mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/slim

---

From: Brian Rosen <[email-masked]>
Date: Sun, 12 Jul 2015 14:41:32 -0400

I think the mechanism only needs to label the desires of the participants.  An intermediary can work out serial translation by B2BUA and modifying the language requests.  You can’t “escape” an SDP block, so the media has to come back to the B2BUA before sending it to the second translator, but that isn’t a big deal.  It wouldn’t have to decode/encode.

Brian

> On Jul 12, 2015, at 10:10 AM, Eric Burger <[email-masked]> wrote:
>

---

From: Paul Kyzivat <[email-masked]>
Date: Sun, 12 Jul 2015 15:30:09 -0400

> I think the mechanism only needs to label the desires of the participants.  An intermediary can work out serial translation by B2BUA and modifying the language requests.  You can’t “escape” an SDP…

I agree. The algorithms could be "interesting" for choosing combinations 
of translators based on a particular pool of resources. But those need 
not be a subject for standardization.

But how to say "I want streams A+B translated to X+Y" does seem to be 
something we need.

	Thanks,
	Paul

---

From: Gunnar Hellström <[email-masked]>
Date: Sun, 12 Jul 2015 23:44:18 +0200

It seems that this requirement relates to the fuzzy line in the service 
indication section of the requirement chapter I just proposed.

There we have this vague requirement about the languages parameter a 
service shall declare or receive.
"Languages shall be media/modality/language indications in pairs showing 
the capabilities of the service."

  That language needs to be sharpened up to correspond to Paul's view 
that the translation service indication must specify input stream and 
output stream.

Would it be sufficient to change the requirements spec line to:

""Languages" shall be stream/media/modality/language indications in 
pairs showing the capabilities of the service."

?

Next issue is how that can be coded.

/Gunnar

Paul Kyzivat skrev den 2015-07-12 21:30:
>> I think the mechanism only needs to label the desires of the 
>> participants.  An intermediary can work out serial translation by 
>> B2BUA and modifying the language requests.  You can’t “escape” an SDP 
>> block, so the media has to come back to the B2BUA before sending it 
>> to the second translator, but that isn’t a big deal.  It wouldn’t 
>> have to decode/encode.
>
> I agree. The algorithms could be "interesting" for choosing 
> combinations of translators based on a particular pool of resources. 
> But those need not be a subject for standardization.
>
>>> I think this can be even more complex. Granted, my experience in 
>>> simultaneous translation is 25 years old, but at the time, the 
>>> European Commission had a rule that one could not have more than two 
>>> serial simultaneous translations in a row. The constraint was 
>>> providing simultaneous translations in (what today) is 24 languages, 
>>> yet doing it with a finite number of interpreters. Scheduling was an 
>>> ‘interesting’ task given the serial translation limitation.
>>>
>>> I bring this up because one can envision someone from Malta visiting 
>>> Latvia. Finding someone who speaks both Maltese and Latvian is 
>>> unlikely, but finding one person who speaks one of Maltese and 
>>> Italian or English, and someone who speaks one of Latvian and 
>>> Italian or English is pretty good. If it is a bad day, you might be 
>>> able to find someone who speaks Maltese and Arabic, someone who 
>>> speaks Arabic and German, and someone who speaks German and Latvian.
>>>
>>> I get the sense we are only looking at bilateral translations (e.g., 
>>> English - ASL, Vietnamese - English, French - RTT, etc.). Given the 
>>> likelihood of serial interpretations in language-dense regions of 
>>> the world (Europe, India, and the US come immediately to mind), are 
>>> we ready for it?
>>>
>>>> I have a feeling there is still a missing part to this puzzle. I 
>>>> bring it up without having yet formed an opinion on how to solve 
>>>> it, or whether slim is the right place to do so.
>>>>
>>>> One case is already mentioned in the -negotiating- draft: when it 
>>>> is desired for the content of a video stream to represent the 
>>>> speaker in an audio stream.
>>>>
>>>> initiate a session with a relay/translation service with the 
>>>> expectation that the service will relay/translate between two 
>>>> groups of streams, each group representing one party in a 
>>>> conversation. There are many variations on this. E.g.,
>>>>
>>>> It *may* be that the desired relationship between the streams can 
>>>> be inferred from the media types, directionality, and humaninlang 
>>>> attributes. But I think this starts to become ambiguous in any but 
>>>> the most simple cases. For instance, in the ASL example above, 
>>>> which audio "belongs" to the video stream? And, should the audio be 
>>>> "passed through" with the translation mixed in, or not? As we add 
>>>> more modalities it will get more complex.
>>>>
>>>> I think we could start by trying to define heuristics for figuring 
>>>> out what to do. Beyond that, we could come up with additional SDP 
>>>> syntax. Or perhaps an independent channel, such as used in CLUE, 
>>>> could be used.
>>>>
>>>>     Thanks,
>>>>     Paul
>>>>

-- 
-----------------------------------------
Gunnar Hellström
Omnitor
[email-masked]
+46 708 204 288

---

From: Paul Kyzivat <[email-masked]>
Date: Sun, 12 Jul 2015 18:10:13 -0400

> pairs showing the capabilities of the service."

ISTM that *normal* endpoints/users aren't intended to provide any 
translation/relay service. So they don't map inputs to outputs.

OTOH, relays provide this mapping. They also have the same need as other 
endpoints to express needs/capabilities of individual streams.

Perhaps it would help to adopt some of the terminology from the 
grouping-taxonomy draft to talk about this. I haven't gotten that into 
my head yet, so it will take some study to find proper terms. But I 
expect they are there.

This is only a problem if you want to invite a relay using an offer and 
count of the offer to convey all the needs to the relay. It isn't needed 
if a MiTM matches an offer and answer, selects a suitable relay, and 
inserts it in the call while configuring the relay using *private* means.

But inviting the relay that way fits well with the PSAP inviting the 
relay for emergency calls. And it has some attractive properties in the 
more general case.

	Thanks,
	Paul

---

From: Paul Kyzivat <[email-masked]>
Date: Fri, 17 Jul 2015 11:27:48 -0400

I've been thinking more about how to indicate this correlation. There 
are some pieces of SDP machinery that could be adapted for this.

My first thought was the grouping framework (RFC5888). With a new group 
type it could be used to tie together the input and output streams for a 
relay. But that doesn't say which is the input and which is the output.

My next thought was the media loopback mechanism (RFC6849). But it only 
covers literal loopback, and does it on a single m-line.

But it would be possible to:
- define a new grouping type - loopback;
- reuse the a=loopback-source and a=loopback-miror from 6849
   to define the roles of the two ends.

E.g.,

offer:
   a=group:loopback:a1 v1
   m=audio:...
   a=mid:a1
   a=loopback-source
   m=video:...
   a=mid:v1
   a=loopback-source

answer:
   a=group:loopback:a1 v1
   m=audio:...
   a=mid:a1
   a=loopback-mirror
   m=video:...
   a=mid:v1
   a=loopback-mirror

In the above, relaying is symmetric. Data sent from source to mirror on 
any stream in the group is reflected back on the other stream of the 
group. If you wanted it to be asymmetric you could constrain it by 
making one stream sendonly and the other recvonly.

Note that this can be applied to any instance of 3pcc transcoding 
(RFC4117), not just a human relay service like VRS.

And I think it can work with groups that have more than two streams, 
such as audio+video+RTT.

These are just preliminary thoughts. I'm interested to hear what others 
think.

	Thanks,
	Paul

---

From: Gunnar Hellström <[email-masked]>
Date: Sat, 18 Jul 2015 22:07:05 +0200

There is an interesting difference between relay services and language 
translation services that infuences the requiremets on the functions you 
describe:

Relay services act between different modalities, and it is acceptable 
that the output from the relay in the medium it produces is either the 
only source in that medium that reaches the other party, or is mixed 
with the source.

Example: video relay, taking sign language input in video, producing 
spoken output in audio, and taking spoken input in audio producing sign 
language output in video. Real-time text might be used occasionally by 
any party to spell out something or type it for easier memorizing.
The medium that the relay service takes as input may also be received by 
theparty not producing it, making it realistic to use a regular 
multimedia three-party-bridge for the relay service.
The functionality can be worked out from language/modality preferences 
of the users.
The basic video relay primary user has high preference for receiving 
sign language and producing sign language.
The basic video relay speech user has high preference for receiving 
speech and producing speech.
Matching these, it is apparent that a service with the functionality of 
the traditional video relay functionality is needed.

Language translators on the other hand is taking as input the same 
medium as it produces, and the receiving part only wants to receive the 
output from the language translator in the medium that the translator 
acts on.

Example:  Spoken language interpreter between French and English. Taking 
spoken French as input replacing it with spoken English, and taking 
spoken English as input and replacing it with spoken French. For this 
service it is not feasible to just arrange a regular three-party bridged 
call, because then both the original speech and the translated speech 
will be heard mixed to the other user, and that is not desirable in most 
language translation situations.
This can also be worked out from SLIM language and modality indications 
from the users, but the resulting invocation of the translating service 
need to be a with a bridge with two audio streams between the bridge and 
the translating service. This can be done by using two coordinated 
calls, or one call with multiple streams of the language-carrying media.

I realize that this influences the interface between the bridge and the 
translating service to contain two media streams of the same kind, but 
it does not influence  the interfaces between the users and the bridge.
The task of correlating streames and roles would be between the bridge 
and the language translation service. I can see that done with the SLIM 
indications if they are inserted in SDP media descriptions as planned, 
but  there is a need for an out-of-band application agreement that they 
mean that translation is required. I guess that it is this out-of-band 
agreement you want to replace with signaling.

Paul Kyzivat skrev den 2015-07-17 17:27:
> My first thought was the grouping framework (RFC5888). With a new 
> group type it could be used to tie together the input and output 
> streams for a relay. But that doesn't say which is the input and which 
> is the output.
>
> My next thought was the media loopback mechanism (RFC6849). But it 
> only covers literal loopback, and does it on a single m-line.
The words and the intentions from RFC 6849 are misleading. Only the 
coarse concept is reusable
> In the above, relaying is symmetric. Data sent from source to mirror 
> on any stream in the group is reflected back on the other stream of 
> the group. 
GH: But to me it looks as if the media are only used one-way. In both 
translating services and relay services, the normal mode is two-way 
transmission of media. E.g.  two way sign language vs two-way speech. 
source and mirror seem to have the intention to be used as 
"send-original" and "send-translated" indications, and that is only half 
the required use.

GH: I am not totally sure about what use case you want to solve. Is it 
requesting suitable services from a relay or translating service 
connected by means of a bridge in a session between two users?

GH: With "reflected back", I assume you mean "sent after translation".

> If you wanted it to be asymmetric you could constrain it by making one 
> stream sendonly and the other recvonly.
GH:  Requesting translation to one direction is usually not the same as 
restricting media transmission to just one direction. The SLIM 
indications seem more appropriate.
> These are just preliminary thoughts. I'm interested to hear what 
> others think.
GH: It it true that this is not for the endpoints to use, but only for a 
point where the need for relay or translation is evaluated and the 
assisting service invoked?

/Gunnar

---

From: Eric Burger <[email-masked]>
Date: Sun, 19 Jul 2015 13:43:20 -0400

What about business meetings? SLIM is for more than just emergency services. I can sort of follow French, so hearing the French before translation into English is of value to me.

> On Jul 18, 2015, at 4:07 PM, Gunnar Hellström <[email-masked]> wrote:
> 
> Example:  Spoken language interpreter between French and English. Taking spoken French as input replacing it with spoken English, and taking spoken English as input and replacing it with spoken French. For this service it is not feasible to just arrange a regular three-party bridged call, because then both the original speech and the translated speech will be heard mixed to the other user, and that is not desirable in most language translation situations.
> This can also be worked out from SLIM language and modality indications from the users, but the resulting invocation of the translating service need to be a with a bridge with two audio streams between the bridge and the translating service. This can be done by using two coordinated calls, or one call with multiple streams of the language-carrying media.

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 20 Jul 2015 19:22:24 +0900

> What about business meetings? SLIM is for more than just emergency services. I can sort of follow French, so hearing the French before translation into English is of value to me.

I have also been at events where I listened to both channels to be able 
to catch translation errors and be able to intervene if they created a 
misunderstanding.

(I would never have been able to do simultaneous translation as well as 
the professional interpreter, but I was fluent in both languages and 
much more familiar with the subject matter.)

Regards,   Martin.

---

From: Brian Rosen <[email-masked]>
Date: Mon, 20 Jul 2015 13:00:18 +0200

So, I think that our original intent was to label what was desired, and not to specify the signaling that would be used to create the connections.  There are many, maybe even most cases, where we don’t need a signaling specification to get interoperability.  Proxies can and sometimes do route on what is contained in the SDP.  

There are some cases where we might need some more specification for signaling.  Paul has suggested some ideas for the case where a proxy or B2BUA needs to create a translation where the result is a loop back (translated media comes back to where it is originally sent from).  While the SDP addresses are easily specified, it isn’t clear how to signal the request to translate.  As long as proxies can route on SDP, we can get most of what we want from  labeling what is desired.  Even the case where you want to invoke a translation service based on the non-overlapping set of options in the offer/answer can be solved simply because the translation service knows that all calls it gets at specified URIs are requests for translation.  

I suggest that we keep the charter limited to labeling for now, finish that work and then recharter for any signaling use cases we come up with.  The labeling is really useful, and I’m not sure what other signaling mechanisms we will need beyond that loopback case.

Brian

> On Jul 20, 2015, at 12:22 PM, Martin J. Dürst <[email-masked]> wrote:
> 
> I have also been at events where I listened to both channels to be able to catch translation errors and be able to intervene if they created a misunderstanding.
> 
> (I would never have been able to do simultaneous translation as well as the professional interpreter, but I was fluent in both languages and much more familiar with the subject matter.)
> 
> Regards,   Martin.
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/slim <https://www.ietf.org/mailman/listinfo/slim>