Subject: [Wish] Lecturing and WebRTC

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 09 Mar 2021 17:10:06 +0100

Hi,

At today's meeting, I made a few comments about the difficulties we
encountered at the Université de Paris (formerly Paris-Diderot) when
trying to teach during lockdown.  I was asked to give a few details on the
list.

Every lecture is different, but I have found that our online activities
can be roughly categorised in 4 groups:

1. Recorded lectures

   The lecturer records a video that they put on a web server.  No
   protocol beyond HTTP and the HTML video element is required.

2. Large online lectures

   The lecturer is streaming their lecture to 60 to 400 students.  The
   students interrupt to ask questions over textual chat.

3. Small online lectures

   A dozen students in a videoconference

4. Practicals

   Students distributed in groups of 2 to 4 people, with the lecturer
   moving from group to group.

It seems to me that (2) is almost within the scope of the WISH working
group.  Until now, we've been mostly using Zoom, BigBlueButton and Galène
(https://galene.org) for this use case (for some reason, Jitsi doesn't
seem popular); a WISH protocol would be a welcome addition to this kind of
software, especially if native WISH clients are readily available.

We encountered the following requirements which might or might not be
similar to the requirements that the other participants in this WG are
familiar with.

1. Multiple streams

Most lecturers like to stream multiple media at the same time.  I, for
one, tend to stream no less than three video streams:

  - my webcam;
  - my slides (PDF);
  - a virtual blackboard (I use a painting program, while a lot of my
    colleagues use dedicated virtual blackboard software).

Note that the different streams might have different lifetimes: the
lecturer might start with just their slides, then switch their virtual
blackboard on when they get asked an interesting question.

2. Firewalls

A lot of undergrad networks were built in the 1990s, and while the
hardware and software have been rotated multiple times, the firewall
policies are still adapted to Windows 95.  In extreme cases, the networks
are not connected to the Internet at all, and all traffic to the Internet
must go through a manually configured HTTP proxy (Squid, usually).  In
some cases, there's a transparent HTTP proxy.  It is sometimes possible to
get an (IP, port) whitelisted, it is generally not possible to get a whole
IP whitelisted.

As to the WiFi networks available to the students, they usually allow
traffic to just a few outgoing ports.  In practice, I have found the
firewalls to be configured to comply with the minimum requirements of the
Eduroam service definition -- please see page 32 of

  https://www.eduroam.org/wp-content/uploads/2016/05/GN3-12-192_eduroam-policy-service-definition_ver28_26072012.pdf

To me, this implies the requirement that either TCP TURN or ICE-TCP be
supported.

3. NAT

A lot of students use 4G or the university WiFi, and are therefore behind
multiple layers of NAT.  While these NAT devices are fine for checking
wikipedia, their behaviour wrt. UDP tends to be erratic.  For example,
I have seen a single media session roam over 5 different externally
visible IP addresses (!) while the associated WebSocket remained stable.

This, to me, implies the requirement that media transports be
renegotiated, using either an ICE restart or some other mechanism.

Regards,

-- Juliusz

---

From: Lorenzo Miniero <[email-masked]>
Date: Tue, 9 Mar 2021 17:37:57 +0100

On Tue, 09 Mar 2021 17:10:06 +0100
Juliusz Chroboczek <[email-masked]> wrote:

> trying to teach during lockdown.  I was asked to give a few details
> on the list.
> 

Hi Juliusz,

thanks for this very interesting overview and insight! I've added a
couple of notes inline.

> Every lecture is different, but I have found that our online
> activities can be roughly categorised in 4 groups:
> 
> 1. Recorded lectures
> 
> group.  Until now, we've been mostly using Zoom, BigBlueButton and
> Galène (https://galene.org) for this use case (for some reason, Jitsi
> doesn't seem popular); a WISH protocol would be a welcome addition to
> this kind of software, especially if native WISH clients are readily
> available.
> 

Oh, I didn't know you were the author of Galene! Sean DuBois, the author
of Pion, shared the slides for a presentation you did on Galene on
Twitter a few days ago, which as a WebRTC server developer myself I
found quite interesting to read: I'll have to see if the video of the
presentation is available somewhere.

> blackboard on when they get asked an interesting question.

This last point is quite important, I think, and it also popped to my
mind while discussions on streams ordinality were going on during the
session. Being able to support multiple streams at the same time is
indeed desirable (Sergio made a compelling example there, with the
multiple languages), but the different lifetimes for different streams
assumes support for renegotiations. This means being able to update a
session after it has been started (however that will happen), and
something that can't be done on the client side alone (e.g., with a
replaceTrack). I'm not sure this is envisaged in Adam's proposal,
unless it should become part of the extensible mechanisms that have to
be defined? Assuming we want support for ICE restarts, some form of
renegotiations will need to be supported anyway (at the very least, a
new offer/answer round), but adding/removing media may go beyond the
simpler scope of this effort.

In this case, it seems simpler to push streams which are supposed to
have different lifetimes as separate "broadcasts", that the application
layer can tie together, even though this means losing the ability of
having everything on the same PeerConnection. I don't have an answer or
an opinion here: just pointing out that it looks like renegotiations
will need to be discussed.

> policies are still adapted to Windows 95.  In extreme cases, the
> networks are not connected to the Internet at all, and all traffic to
> the Internet must go through a manually configured HTTP proxy (Squid,
> usually).  In some cases, there's a transparent HTTP proxy.  It is
> sometimes possible to get an (IP, port) whitelisted, it is generally
> not possible to get a whole IP whitelisted.
> 
> firewalls to be configured to comply with the minimum requirements of
> the Eduroam service definition -- please see page 32 of
> 

I agree, and I don't think the spec will prevent you from doing that. I
think the main point discussed today was whether or not, as a WISH
client, you should be supposed to advertise those candidates, either
via trickle or as part of your SDP offer. Even if they're not, if your
WebRTC server is publicly reachable then it should still be able to make
use of them nevertheless, by discovering them via prflx candidates. 

> 3. NAT
> 
> A lot of students use 4G or the university WiFi, and are therefore
> behind multiple layers of NAT.  While these NAT devices are fine for
> checking wikipedia, their behaviour wrt. UDP tends to be erratic.
> For example, I have seen a single media session roam over 5 different
> externally visible IP addresses (!) while the associated WebSocket
> remained stable.
> 

Thanks,
Lorenzo

-- 
I'm getting older but, unlike whisky, I'm not getting any better
https://twitter.com/elminiero

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 09 Mar 2021 17:45:51 +0100

Oh, I forget.

Some students follow lectures over 4G, and have a limited amount of data
traffic per month.  These students want to receive audio, slides and
blackboard, but don't want to spend money on viewing the lecturer's face,
no matter how handsome.

This, to me, implies the need for labelling of the individual streams or
tracks, and the ability for the client to select which tracks it receives.

This might or might not be related to simulcast.

-- Juliusz

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 09 Mar 2021 17:58:25 +0100

Hi Lorenzo,

May I have an autograph, please?

> shared the slides for a presentation you did on Galene [...]  I'll have
> to see if the video of the presentation is available somewhere.

The slides are here:

  https://galene.org/galene-20210222.pdf

The raw video footage is here, but it's a mess (400MB tarfile, one
unseekable WebM per stream):

  https://galene.org/galene-20210222.tar

> but the different lifetimes for different streams assumes support for
> renegotiations [...] In this case, it seems simpler to push streams
> which are supposed to have different lifetimes as separate "broadcasts",

Yes, you could carry each stream within a separate session.  But it does
still imply the need to renegotiate something, either the set of streams
or the set of sessions.

> this means losing the ability of having everything on the same PeerConnection

You lose a few UDP ports, and exchange a few extra packets with your STUN
server.  Am I missing anything?

> I think the main point discussed today was whether or not, as a WISH
> client, you should be supposed to advertise [relayed] candidates, either

I'm not sure I follow.  Suppose you're on an Eduroam network, so the only
outgoing traffic allowed is TCP/443 and TCP/1194.  You run your signalling
server on 443, your TURN server on 1194.

Since the client cannot reach any UDP ports, how is it going to learn
about the TURN server on port 1194 if it's not signalled explicitly?

-- Juliusz

---

From: Lorenzo Miniero <[email-masked]>
Date: Tue, 9 Mar 2021 18:14:42 +0100

Hi Juliusz,

a few more notes inline.

On Tue, 09 Mar 2021 17:58:25 +0100

> May I have an autograph, please?

:-)

> > shared the slides for a presentation you did on Galene [...]  I'll
> > have to see if the video of the presentation is available
> > somewhere.  
> 
> The slides are here:
> 

Thanks! Unseekable works for me, I plan to watch it all :-)

> > but the different lifetimes for different streams assumes support
> > for renegotiations [...] In this case, it seems simpler to push
> > streams which are supposed to have different lifetimes as separate
> > "broadcasts",  
> 
> Yes, you could carry each stream within a separate session.  But it
> does still imply the need to renegotiate something, either the set of
> streams or the set of sessions.
> 

Yes, in this case I was talking about renegotiations on a specific
ongoing session: if you're creating a new one to start slides after you
already started your webcam, this is not touching other sessions that
existed already, but will be a completely separate one. As such, an
easier problem to solve than actually updating an existing session.

> You lose a few UDP ports, and exchange a few extra packets with your
> STUN server.  Am I missing anything?
> 

There may be more resources used on both client and server side in
general, but yes, more ports needed and additional overhead would be
the main "issues".

> > client, you should be supposed to advertise [relayed] candidates,
> > either via trickle or as part of your SDP offer. Even if they're
> > not, if your WebRTC server is publicly reachable then it should
> > still be able to make use of them nevertheless, by discovering them
> > via prflx candidates.  
> 
> I'm not sure I follow.  Suppose you're on an Eduroam network, so the
> only outgoing traffic allowed is TCP/443 and TCP/1194.  You run your
> signalling server on 443, your TURN server on 1194.
> 

No matter how you talk to the TURN server (TCP or TLS), the TURN server
itself will normally talk to the WebRTC server via UDP (once you tell
it which address to interact with, which you'll know from the SDP
answer in this case), acting as a media relay for you. As such,
assuming the server is publicly reachable, your client doesn't need to
do any UDP, it just needs to talk to the TURN server somehow: even if
you don't advertise those relay candidates, the TURN server will send
connectivity checks to the WebRTC server on your behalf, and since the
credentials in the STUN message are correct, the server will know that
the address it just received the check from is a potential candidate
for communication. These candidates discovered at runtime are called
"peer reflexive candidates", and are what would make the session work
even without any candidate in SDP/trickle.

Lorenzo

---

From: Adam Roach <[email-masked]>
Date: Tue, 9 Mar 2021 11:15:33 -0600

> will need to be discussed.

This is actually closely related to a number of the use cases that I 
have in mind, where I want to be able to tie multiple sessions together, 
both for the kinds of scenarios that Juliusz lays out, as well as 
sessions originating from entirely different endpoints. From that 
perspective, having a way to associate different sessions with each 
other somehow seems far more flexible (this is in large part why the 
"conferencing" analogy comes to mind when thinking about extensions). I 
was going to leave this off as something we can define in a future 
extension; but if we're trending towards including it as base 
functionality, I don't have any objections to adding it in.

/a

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Tue, 9 Mar 2021 18:17:38 +0100

> about the TURN server on port 1194 if it's not signalled explicitly?

That is client (let's call it broadcaster/publisher/encoder better to avoid
confusion with the viewing clients) configuration and will never be
exchanged via signaling.

Media server will just receive a new ICE request coming from the turn
server allocated candidate, no need to do a ICE restart in this case if the
network on the media server side is stable (which I would make it a
requirement).

Best regards
Sergio

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 09 Mar 2021 18:36:51 +0100

> I was going to leave this off as something we can define in a future
> functionality

I'm rather fond of concrete use cases, so I try to think about it in the
following terms: will I be able to usefully broadcast a lecture over WISH?
If the feature is just convenience, it can go into an extension; if the
lack of the feature prevents me from doing my broadcast, then I'd like it
to be mandatory.

If the lecturer adds a stream to their lecture, and some WISH clients
don't support the functionality, then I'm pretty much stuck, amn't I?

-- Juliusz

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Tue, 9 Mar 2021 18:53:07 +0100

> don't support the functionality, then I'm pretty much stuck, amn't I?

Take node that tipical wish publishers are going to be tools like ffmpeg or
OBS,  I doubt they will provide the flexibility you are looking for.

I dont mind labeling the content of the different tracks as per the sdp
attribute, but I would be against renegotiation/dinamically adding removing
tracks.

Anyway, as said before, this case could be implemented by assigning one url
to each stream (which could be even send from different devices) and
managed at service level.

Best regards
Sergio

---

From: Adam Roach <[email-masked]>
Date: Tue, 9 Mar 2021 11:53:31 -0600

> don't support the functionality, then I'm pretty much stuck, amn't I?

Your described use case includes three streams, which is one more than I 
would have imagined necessary. From that perspective, the chance that 
the broadcast client of your choice will support all the streams that 
you might want seems a bit slim -- for example, I can't imagine OBS 
including support for sending three video streams in the near future, 
since it's fundamentally different than how it's architected today.

On the other hand, if we take Sergio's proposed direction -- where the 
ingest network provides you, as a broadcaster, with multiple endpoints 
-- then it provides you the ability to fire up multiple broadcast 
clients, on different devices if you want, and to direct them to the 
different endpoints as you see fit. Even better, this saves us from the 
difficulty of having to map each video stream in the SDP to its intended 
semantics: you, as the broadcaster, get to hook your whiteboard up to 
the whiteboard endpoint, your camera up to the camera endpoint, and your 
slides up to the slide endpoint.

To make this concrete, for the lecturer service you've described, I 
would imagine that each lecturer is given three URLs. Yours would look 
something like this:

https://example.org/jch/whiteboard

https://example.org/jch/camera

https://example.org/jch/slides

Then you could start up three broadcasting clients, each configured with 
the correct of these three URLs.

Even better: assuming a certain level of success for WISH, maybe you 
have a hardware whiteboard that supports WISH output, and maybe your 
preferred slide generation tool (e.g., OpenOffice, Powerpoint, Keynote) 
lets you do a "share via WISH" option. If we divide things up how Sergio 
has described, then you can just put the correct URL in the correct 
thing, and they all go to the right place, without having to somehow 
coordinate through the tool that you're using for your camera broadcast.

The more I think about it, the more appealing this simplified approach 
is to me.

/a

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 09 Mar 2021 19:03:00 +0100

> OBS,  I doubt they will provide the flexibility you are looking for.

I think the requirement is on the receiver.  What will the protocol allow
my implementation to publish?

> Anyway, as said before, this case could be implemented by assigning one url to
> each stream (which could be even send from different devices) and managed at
> service level.

How do you envision interoperability between independently developed clients?

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 09 Mar 2021 19:04:20 +0100

> To make this concrete, for the lecturer service you've described, I would
> imagine that each lecturer is given three URLs. Yours would look something
> like this:

> https://example.org/jch/slides

How would the client discover the URLs?

---

From: Lorenzo Miniero <[email-masked]>
Date: Tue, 9 Mar 2021 19:15:56 +0100

On Tue, 09 Mar 2021 19:03:00 +0100
Juliusz Chroboczek <[email-masked]> wrote:

> > Take node that tipical wish publishers are going to be tools like
> > ffmpeg or OBS,  I doubt they will provide the flexibility you are
> > looking for.  
> 
> I think the requirement is on the receiver.  What will the protocol
> allow my implementation to publish?
> 

But receivers are actually out of scope, here. WISH will just specify
how a media publisher can send media to a server via WebRTC. How you
then distribute the media is entirely up to you (it may not even be via
WebRTC, e.g., if you're transcoding to use an HLS CDN). In your case,
WISH basically provides an alternative way by which the teacher can
send media to your Galene media server: how you distribute the streams
to students via WebRTC is up to you (you may even decide to pack them
all together into a single PeerConnection yourself, for instance).

> > Anyway, as said before, this case could be implemented by assigning
> > one url to each stream (which could be even send from different
> > devices) and managed at service level.  
> 

In this case, the only interoperability that really matters is between
those that will write the publishing clients, i.e., the endpoints
encoding and sending the media (OBS, FFmpeg, GStreamer, or a web
application), and those that will write the media servers receiving the
media (Galene, Janus, Medooze, or whatever else).

If you design your WISH server to expose three different endpoints for
the media streams you want to publish, then I think Adam's point is
that you can use any WISH client to push the media you want to the
endpoint you want, knowing that as a media server/service you'll have
the info you need to properly map them to the "right thing" when
distributing it.

Lorenzo

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 09 Mar 2021 19:22:12 +0100

> how a media publisher can send media to a server via WebRTC.

Ah... that's the bit that I was missing — I was under the assumption that
we're doing the server⟶receiver bit.  If it's just an interoperable way to
plug OBS into a media server, then I'm fine with a much more static
design.

---

From: Adam Roach <[email-masked]>
Date: Tue, 9 Mar 2021 12:26:57 -0600

> How would the client discover the URLs?

These aren't discovered. They're provided to you by the broadcast 
network, in the same way that RTMP URLs are provided to broadcasters today.

/a

---

From: Juliusz Chroboczek <[email-masked]>
Date: Tue, 09 Mar 2021 21:29:22 +0100

> These aren't discovered. They're provided to you by the broadcast network,
> in the same way that RTMP URLs are provided to broadcasters today.

Thanks for the explanation.  Most of the issues in my initial mail turn
out to be irrelevant for what I know understand is described in the WISH
charter.  Thanks for the explanations, and sorry for wasting everybody's time.

It wasn't clear to me that a "broadcasting tool" is a sender-side piece of
software, I was assuming that a "broadcasting tool" is the software used
to broadcast media to the receivers.  It is of course clear if you look at
the list of examples given in the charter.

-- Juliusz