Subject: [MMUSIC] Open source SIP/WebRTC real-time text integration

From: Lorenzo Miniero <[email-masked]>
Date: Wed, 18 Dec 2019 12:04:51 +0100

Hi all,

as I anticipated in another post, I've recently started working with
real-time text: mostly for fun (I've always been intrigued by the
protocol), but also because I wanted to get it working with WebRTC,
especially in light of its importance for the upcoming NG emergency
services. Considering Christer's recent efforts on standardizing how to
use RTT on WebRTC, I thought I'd share this here, so that it can
hopefully provide a testing framework: it is not complete, and doesn't
adhere 100% to the specification yet (more on that in a minute), but I
think it might be helpful as a running code reference anyway.

Specifically, I've integrated support for real-time text in our open
source Janus WebRTC server. Janus is a modular component, and one of
the plugins implements a SIP gateway: this is where I implemented
support for T.140 and red. The way it works is "straightforward":

	1. Any time we get an incoming SIP call on the SIP side with an
	m=text m-line, we translate that to an m=application, so that
	we can negotiate data channels on the WebRTC side.

	2. At the same time, if the WebRTC peer negotiates an
	m=application line, we turn that into an m=text section to
	negotiate real-time text on the SIP side.

	3. Then, if we receive T.140 packets from SIP, we relay them
	via data channels using binary data (we use an Uint8Array in
	the web application); if we receive RED packets, we parse it in
	order to get the T.140 blocks we need, and relay those via data
	channels (since the draft explains redundancy is not needed,
	thanks to SCTP).

	4. When we get T.140 blocks via data channels (which again the
	web application sends as an Uint8Array), we either send them as
	they are in RTP packets (if RED was not negotiated), or we
	create a RED RTP packet with redundant info on previously sent
	packets (if RED was negotiated instead).

This is the effort in a nutshell, and from my simple tests it seems to
be working as expected so far. You can find a more comprehensive
description of the whole effort in this blog post:

https://www.meetecho.com/blog/realtime-text-sip-and-webrtc/

The Janus branch supporting real-time text, instead, is here:

https://github.com/meetecho/janus-gateway/pull/1898

As explained in the blog post, the effort is not complete, and there
are some things that are either missing, or need to be improved, namely:

	1. We're not using the dcmap and dcsa attributes on the WebRTC
	side yet: the reason is that browsers don't support them, at
	the moment, and so putting them there may result in the SDP
	being rejected and the session broken. This means that,
	currently, we use a default label for exchanging T.140 blocks
	with the server.

	2. While we support RED, we're not using the redundant info for
	packets we receive yet, and we're not properly handling packet
	loss or out of order packets yet either. This is something we
	plan to work on in the future, as I was more interested in
	getting the specification in general to work first.

	3. On the client side in WebRTC, we're not doing any buffering,
	meaning we send every character as soon as we type it, which is
	clearly suboptimal. Again, something we plan to fix later on,
	either in the web application (buffer there), or on the server
	side (buffer incoming T.140 blocks there, before crafting RTP
	packets).

I hope this will be considered useful, and I'm looking forward to keep
on working on this as the specifications moves forward. If you have any
questions or doubts, please don't hesitate to ask; besides, I'll be in
Vancouver for the next IETF, so in case you want to talk about it in
person there, see a demo, or make some interoperability tests, I'll be
glad to do that as well.

Thanks,
Lorenzo

-- 
I'm getting older but, unlike whisky, I'm not getting any better
https://twitter.com/elminiero

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 18 Dec 2019 13:58:06 +0000

Hi Lorenzo,

It's super nice to hear that you are implementing RTT over data channel! Feel free to keep us up to data as the work progress.

As far as browsers rejecting non-supported attributes is concerned, I think that is an issue that should be raised in W3C. As a temporary solution, could the JS app simply not remove them before passing the SDP to the browser?

Regards,

Christer

On 18/12/2019, 13.05, "mmusic on behalf of Lorenzo Miniero" <[email-masked] on behalf of [email-masked]> wrote:

    Hi all,

    

    

    https://protect2.fireeye.com/v1/url?k=c2a19457-9e284e10-c2a1d4cc-0cc47ad93c18-c1355b6e18f62d8a&q=1&e=78bf43a6-cc22-44d8-bbaf-2cb644177cfa&u=https%3A%2F%2Fwww.meetecho.com%2Fblog%2Frealtime-text-sip-and-webrtc%2F

    https://protect2.fireeye.com/v1/url?k=ce273934-92aee373-ce2779af-0cc47ad93c18-c8ae4c94a2ec0095&q=1&e=78bf43a6-cc22-44d8-bbaf-2cb644177cfa&u=https%3A%2F%2Fgithub.com%2Fmeetecho%2Fjanus-gateway%2Fpull%2F1898

    

    
    Thanks,
    Lorenzo

    _______________________________________________
    mmusic mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/mmusic

---

From: Lorenzo Miniero <[email-masked]>
Date: Wed, 18 Dec 2019 15:48:07 +0100

On Wed, 18 Dec 2019 13:58:06 +0000
Christer Holmberg <[email-masked]> wrote:

> Hi Lorenzo,
> 
> It's super nice to hear that you are implementing RTT over data
> channel! Feel free to keep us up to data as the work progress.
> 

Hi Christer, will do for sure!

> As far as browsers rejecting non-supported attributes is concerned, I
> think that is an issue that should be raised in W3C. As a temporary
> solution, could the JS app simply not remove them before passing the
> SDP to the browser?
> 

Actually I should clarify that it's what I *thought* would happen, as
I've seen it happening in the past in some cases. Anyway, I just tried
injecting some of those attributes in the SDP Janus sends to the
browser, i.e.:

	a=dcmap:1 label="JanusDataChannel";subprotocol="t140"
	a=dcsa:1 hlang-send:en
	a=dcsa:1 hlang-recv:en

and to my pleasant surprise Chrome didn't complain, so I was wrong in
that regard. It doesn't do anything with those attributes either,
though, which probably means they're completely ignored at the moment.
There's no trace of those in the answer, for instance, which means at
the moment the web application would have to munge the SDP to add those
values manually, something I try to avoid whenever possible.

That said, since the SIP endpoint I'm using for testing doesn't provide
any hint on what should be added there (e.g., in terms of supported
languages), my server wouldn't be able to come up with something to
negotiate there anyway. For labels, since this first integration only
supports 1-1 calls, I thought it safe to rely on the default label that
is opened when data channels are negotiated with Janus (the
"JanusDataChannel" you see listed in the snippet above).

Lorenzo

> Regards,
> 
> Christer
> 
> 
> 
> On 18/12/2019, 13.05, "mmusic on behalf of Lorenzo Miniero"
> <[email-masked] on behalf of [email-masked]> wrote:
> 
>     Hi all,
>     
>     as I anticipated in another post, I've recently started working
> with real-time text: mostly for fun (I've always been intrigued by the
>     protocol), but also because I wanted to get it working with
> WebRTC, especially in light of its importance for the upcoming NG
> emergency services. Considering Christer's recent efforts on
> standardizing how to use RTT on WebRTC, I thought I'd share this
> here, so that it can hopefully provide a testing framework: it is not
> complete, and doesn't adhere 100% to the specification yet (more on
> that in a minute), but I think it might be helpful as a running code
> reference anyway. 
>     Specifically, I've integrated support for real-time text in our
> open source Janus WebRTC server. Janus is a modular component, and
> one of the plugins implements a SIP gateway: this is where I
> implemented support for T.140 and red. The way it works is
> "straightforward": 
>     	1. Any time we get an incoming SIP call on the SIP side
> with an m=text m-line, we translate that to an m=application, so that
>     	3. Then, if we receive T.140 packets from SIP, we relay
> them via data channels using binary data (we use an Uint8Array in
>     	the web application); if we receive RED packets, we parse
> it in order to get the T.140 blocks we need, and relay those via data
>     	channels (since the draft explains redundancy is not
> needed, thanks to SCTP).
>     
>     	4. When we get T.140 blocks via data channels (which
> again the web application sends as an Uint8Array), we either send
> them as they are in RTP packets (if RED was not negotiated), or we
>     	create a RED RTP packet with redundant info on previously
> sent packets (if RED was negotiated instead).
>     
>     This is the effort in a nutshell, and from my simple tests it
> seems to be working as expected so far. You can find a more
> comprehensive description of the whole effort in this blog post:
>     
>     As explained in the blog post, the effort is not complete, and
> there are some things that are either missing, or need to be
> improved, namely: 
>     	1. We're not using the dcmap and dcsa attributes on the
> WebRTC side yet: the reason is that browsers don't support them, at
>     	the moment, and so putting them there may result in the
> SDP being rejected and the session broken. This means that,
>     	currently, we use a default label for exchanging T.140
> blocks with the server.
>     
>     	2. While we support RED, we're not using the redundant
> info for packets we receive yet, and we're not properly handling
> packet loss or out of order packets yet either. This is something we
>     	3. On the client side in WebRTC, we're not doing any
> buffering, meaning we send every character as soon as we type it,
> which is clearly suboptimal. Again, something we plan to fix later on,
>     	either in the web application (buffer there), or on the
> server side (buffer incoming T.140 blocks there, before crafting RTP
>     	packets).
>     
>     I hope this will be considered useful, and I'm looking forward to
> keep on working on this as the specifications moves forward. If you
> have any questions or doubts, please don't hesitate to ask; besides,
> I'll be in Vancouver for the next IETF, so in case you want to talk
> about it in person there, see a demo, or make some interoperability
> tests, I'll be glad to do that as well.
>     
>     Thanks,
>     Lorenzo
>     
>     --

---

From: Christer Holmberg <[email-masked]>
Date: Wed, 18 Dec 2019 14:54:42 +0000

Hi,

    > that regard.

That's good news!

 > It doesn't do anything with those attributes either,

Sure, that is expected.

>    "JanusDataChannel" you see listed in the snippet above).
  
That's fine - the T.140 data channel usage does not specify a label value, only the sub-protocol value ("t140").

Regards,

Christer