Subject: [V3] High level comments on the RIPT charter and scope

From: Magnus Westerlund <[email-masked]>
Date: Fri, 27 Mar 2020 14:23:59 +0000

Hi,

So this is my personal opinion and based on my long work with RTP. 

Yes, there are clearly interest in chartering some work in IETF. Howver, I think
there is significant diversity in what people are interested in actually
happening. 

Clearly there is the signalling protocol for seession/call establishment and
control between peering provider nodes. Lets call this the outbound part. The
main issue I see in this area is the quetion of feature creap. It would be good
to figure out what core functionality is really needed and actually get that
described in a charter. A Question is if conference applications impacts this
scope or not. 

The next part is the inbound problem, i.e. having end-user devices/application
being able to register to their provider as being ready to receive/answer calls.
This may actually be sufficient similar to the outbound part that they can be
done in the same WG. 

However, an interesting question for both of the above is if the solution is
required to be able to be implemented in a browser, at least the client part. I
think the signalling is fairly straight forward to be able to implement in JS as
long as we ignore the media. 

The third big part is the media transport. I think we need to strongly consider
if this can be broken out in to it seperate WG. There are several aspects here.
First of the split in expertise between the media transport and the signalling
folks. The next aspect is the resuability of the media transport, that anyway
need to be define as individual reusable protocol component that is not
dependent on the signalling. I don't see a huge problem with being able to
initiate a media transport channel into the relevant server node using an
initial HTTP request. I am thinking WebSocket like establishment of the media
transport channel here. And possibly there will be need for a fallback to HTTP
only, but also that should be further discussed. 

I am also very worried that if the model for the media transport is only point
to point calls then we get something that will become obsolete or at least need
hacks immediately. This thing need to support centralized conferences with
cascaded or meshed central media nodes. So I think it is cruical that one define
the usage model with switched central nodes and have that in mind so that one
can actually support the applications needed. I think by defining a strict model
one can actually avoid a lot of feature creep in the media transport. 

We also have use cases that has practical total overlapp, namely real-time
sensor streaming services, where cameras, and other frequently sampling sensors
send data of real-time control loop nature. There is also the media industries
need for content contribution, i.e. how to get camera's media into the
production centers. These are similar enough and need of standardized protocols.
SRT in Dispatch is only one aspect of this. What we don't need to take on for
this is the one-to-many distribution only. So broadcast and multicast can be
declared out of scope. 

I think E2E media secruity need to be defined from day one. It will be a central
part of these systems even if in the legacy interconnect there will be the
termination in gateways. Also media processing nodes can be included but the
inclusion of them in the trust will need to be explicit in this model. 

People have been voicing that we don't need things to work as well as RTP. I
think in fact that a dedicated real-time media over QUIC (with datagram) can
actually achieve better than RTP performance by using a mix of reliable and
semi-reliable transport primitives on applciation data unit level, i.e. audio
and video frames. 

The next media transport issue that was barely discussed in the BOF is the
legacy (RTP) interoperate question. This is going to come back in the scope
discussion and do needs at least a direction. 

Coming back to my initial statement about diversity, I think there at least two
communities that are interested in this. And I think we actually need to attempt
to make relevant protocol work for both these communities. For the moment the
most outspoken people in this effort have been the signalling people. The media
transport crowd have been waiting for their turn until QUIC to matured
sufficiently to be a realistic start. But as people know some discussion have
happened. However we are this point now I think where work can start in ernest
also on this topic. 

So can we seriously consider writing two different charter that together enable
the intended use cases? 

Cheers

Magnus Westerlund 

----------------------------------------------------------------------
Networks, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 27 Mar 2020 10:10:58 -0600

> On Mar 27, 2020, at 8:23 AM, Magnus Westerlund <magnus.westerlund=[email-masked]> wrote:
> 

I think it does make sense to support some meta data on a media flow for centralized conferencing support. What semantic level information do you think is needed in the media ?

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 27 Mar 2020 10:19:07 -0600

> if this can be broken out in to it seperate WG.

I think the WG we are talking about here is the WG to do media of H3. So we can talk about other things being in or out of it but that is the core thing. 

I would like to point out that one of the design properties that has been making this easier is that media is self describing. What I mean by that is that if a device receives media that is one of the supported formats and within in the limits of that device supports, the media has all the information the device needs to decode it. No out of band information needed. That is key to making this easier to deal with HA and scalability on cloud side. I think that is a good property for modern designs. We have been going that way for a long time with things like opus and it works out well. 

> here are several aspects here.
> folks. 

I’m not sure what you mean by signalling but the the singling here was mostly start media , stop media. It is not call routing, forking etc that is done by SIP.

---

From: Cullen Jennings <[email-masked]>
Date: Fri, 27 Mar 2020 10:23:46 -0600

> discussion and do needs at least a direction. 

In WebRTC, we decided to be able to gateway SIP to WebRTC without processing media. That turned out to be a major limitation to how WebRTC worked. That is good and we have WebRTC but for this, I am storngly of the view we should not have that design constraint. 

I think it is fine to require a media gateway to go from RIPT to RTP. Sure that should be pretty easy - one of the major use cases for this work will likely involve building SBCs that talk RIPT on one side and SIP on the other. Pretty sure that a bunch of other people are thinking about the same. 

That sound right to you are or are you arguing for something else ?

---

From: Justin Uberti <[email-masked]>
Date: Fri, 27 Mar 2020 09:28:15 -0700

I think Magnus is trying to establish the conference topology, which has
ramifications on how we want to handle things like RTCP.

One strawman could be to focus on RTCP-terminating SFUs, i.e., cementing
the notion that RIPT media flows are indeed point-to-point (i.e.,
client-to-SFU, as opposed to client-to-client1, client2, client3).

On Fri, Mar 27, 2020 at 9:11 AM Cullen Jennings <[email-masked]> wrote:

>
>
> On Mar 27, 2020, at 8:23 AM, Magnus Westerlund <
> magnus.westerlund=[email-masked]> wrote:
>
> I think it does make sense to support some meta data on a media flow for
> centralized conferencing support. What semantic level information do you
> think is needed in the media ?
>
>
>
> --
> V3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v3
>

---

From: Justin Uberti <[email-masked]>
Date: Fri, 27 Mar 2020 13:22:07 -0700

It's hard to imagine that we can avoid this if we are in fact going to use
H3 as a substrate.

I do think it would be useful to ensure the RIPT->RTP gateway can be
stateless and transparent (i.e., you can transform a RIPT frame into a RTP
packet with reasonable fidelity), so that said gateway is all you need in
order for existing services to participate in the RIPT ecosystem.

On Fri, Mar 27, 2020 at 9:32 AM Cullen Jennings <[email-masked]> wrote:

> In WebRTC, we decided to be able to gateway SIP to WebRTC without
> processing media. That turned out to be a major limitation to how WebRTC
> worked. That is good and we have WebRTC but for this, I am storngly of the
> view we should not have that design constraint.
>
> I think it is fine to require a media gateway to go from RIPT to RTP. Sure
> that should be pretty easy - one of the major use cases for this work will
> likely involve building SBCs that talk RIPT on one side and SIP on the
> other. Pretty sure that a bunch of other people are thinking about the
> same.
>

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 27 Mar 2020 22:35:39 +0000

Hi,

So yes, I think there are the aspect of the topology as well as how one handle
the media related controll messages, i.e. the equivalent of RTCP. For example
things like FIR, PAUSE/Resume. 

So, if one with RTCP terminating middleboxes means that the ID spaces are soley
locally that could be a choice. I will note that in some cases one still need to
have the complete set of media sources from all end-points in a conference be
identified at a specific point to point leg. Even if only a subset is currently
transmitted over this leg. There are both benefits and downsides I think of
using a local namespace and having each node translate these.

I will note that one may still need to have end-to-end identies on media ADUs if
one are going to have any end-to-end security. 

Cheers

Magnus Westerlund

On Fri, 2020-03-27 at 09:28 -0700, Justin Uberti wrote:
> One strawman could be to focus on RTCP-terminating SFUs, i.e., cementing the
> notion that RIPT media flows are indeed point-to-point (i..e., client-to-SFU,
> as opposed to client-to-client1, client2, client3).
> 
-- 
Cheers

Magnus Westerlund

---

From: Colin Perkins <[email-masked]>
Date: Sun, 29 Mar 2020 23:05:43 +0100

It may be a reasonable goal to aim to re-use RTP payload formats as the media packetisation within the new transport. That is, different headers and transport, but format the media content the same way.

Since payload format packetisation already all have registered media types this would make the media content easy to identify, and save a lot of work in re-defining packetisation schemes.

Colin

> On 27 Mar 2020, at 20:22, Justin Uberti <juberti=[email-masked]> wrote:
> 
> It's hard to imagine that we can avoid this if we are in fact going to use H3 as a substrate. 
> 
> I do think it would be useful to ensure the RIPT->RTP gateway can be stateless and transparent (i.e., you can transform a RIPT frame into a RTP packet with reasonable fidelity), so that said gateway is all you need in order for existing services to participate in the RIPT ecosystem.
> 
> On Fri, Mar 27, 2020 at 9:32 AM Cullen Jennings <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
>> On Mar 27, 2020, at 8:23 AM, Magnus Westerlund <magnus.westerlund=[email-masked] <mailto:magnus.westerlund=[email-masked]>> wrote:
>> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/v3 <https://www.ietf.org/mailman/listinfo/v3>
> -- 
> V3 mailing list
> [email-masked]

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 30 Mar 2020 13:16:44 +0000

Hi,

(Still as invidual)

Attempting to answer Cullen's question I think there are significant points in
trying to actually enabling the utilization of the advantages this redefinition
can enable. For example to me it appears that one should ensure that the media
transport becomes more ADU focused rather than packet focused. With 1080p or 4K
video in good quality the number of 1400 bytes packets needed to carry a single
frame are more in the range of hundreds than in single digit. I think one should
try to ensure that this work as well as possible. Thus semi-reliable head of
line blocking free transport of whole ADUs becomes of more interest. 

I think there is a point to re-use the RTP payload format. If one has such
support in the end-point one can use the RTP payload fragmentation when needed
for interconnect with RTP systems and not use it if one can rely on QUIC+RIPT
primitives. Reusing RTP payload formats also have the advantage of avoiding to
have to redefine all the codecs of interest.

Cheers

Magnus Westerlund

On Sun, 2020-03-29 at 23:05 +0100, Colin Perkins wrote:
> It may be a reasonable goal to aim to re-use RTP payload formats as the media
> packetisation within the new transport. That is, different headers and
> transport, but format the media content the same way.
> 
> Since payload format packetisation already all have registered media types
> this would make the media content easy to identify, and save a lot of work in
> re-defining packetisation schemes.
> 
> Colin
> 
> 
> 
> > On 27 Mar 2020, at 20:22, Justin Uberti <juberti=[email-masked]
> > > wrote:
> > 
-- 
Cheers

Magnus Westerlund

---

From: Joerg Ott <[email-masked]>
Date: Mon, 30 Mar 2020 15:23:27 +0200

Hi,

one should add that the payload formats are mostly not overly complex
and their design is defined by the needs of the codecs -- plus mapping
to an unreliable packet transport.  The codecs don't change and the
unreliability still holds with QUIC underneath.  So, there would seem
to be little gain to begin with in attempting to redo all of this.

Cheers,
Jörg

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 30 Mar 2020 07:42:29 -0600

> On Mar 29, 2020, at 4:05 PM, Colin Perkins <[email-masked]> wrote:
> 

+1 on reusing the RTP payload formats. As Joerg points out they are a good design for what is needed here.

---

From: Magnus Westerlund <[email-masked]>
Date: Mon, 30 Mar 2020 13:44:26 +0000

On Fri, 2020-03-27 at 10:19 -0600, Cullen Jennings wrote:
> I think the WG we are talking about here is the WG to do media of H3. So we
> can talk about other things being in or out of it but that is the core thing. 
> 
> I would like to point out that one of the design properties that has been
> making this easier is that media is self describing. What I mean by that is
> that if a device receives media that is one of the supported formats and
> within in the limits of that device supports, the media has all the
> information the device needs to decode it. No out of band information needed.
> That is key to making this easier to deal with HA and scalability on cloud
> side. I think that is a good property for modern designs. We have been going
> that way for a long time with things like opus and it works out well. 

So, let me ask a question here. How important is it be capable to transmitt
individual media ADU or even RTP payloads in a single HTTP request that ripples
through the system independent compared to have a system that can in a 0-rtt
fashion (re-)establish an point to point by initiating an HTTP request? 

Treating each media ADU as its own request potentially being routed in a non-
sticky way will induce significant jitter would it not? The drafts talks about
sticky routing, and I did notice some comments about that assumption already.
Thus, from my perspective it appears that the high-performance target would be
to establish an point to point quic based transport with the relevant server
node that handles media forwarding. If that fails have a mechanism that can
rapidly re-establish that connection with what ever node that is now the
appointed one for the given call/conference. 

Then I do understand the desire to have a fallback, and maybe that is plain HTTP
requests, but let that be the fallback and not the primary mechanism. Becasue I
think that would unnecessary limit the potential of this system. 

> I’m not sure what you mean by signalling but the the singling here was mostly
> start media , stop media. It is not call routing, forking etc that is done by
> SIP. 

Yes, but even session establishment and control is still signalling from my
perspective. 

Cheers

Magnus Westerlund

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 30 Mar 2020 07:55:47 -0600

On the stateless or not … I’ve tried implementing this a few different ways and I think what we want is a sort of mostly stateless with frequent sate updates for the parts that are not stateless. Let me explain ….

First keep in mind there is nothing stateless about the QUIC connection. Lots of state for crypto. It also seems like it would be a good thing to allow state for compression. So we are already out of a fully stateless design before we even start. 

So when we say RIPT can map to RTP, we really mean RTP + RTCP as that is needed. In the RTCP we need things like the NTP timestamp. I don’t think we want to send the full NTP timestamp in ever RIPT packet which probably rules out a fully stateless design. In WebRTC we took the approach of putting some data in RTP header extensions and sending them not ever packet but relatively frequently like every second or every IFrame.  I think we should do the same thing here - we should have some stuff that is sent frequently enough that if you loose all state until it is resent, it is  not a huge deal. 

This would allow us to do things like send the info to map to NTP time once a second not in every packet. When you look at what is in a basic RTP packet, I think it is easy to have that 1:1 mapping of all that data to something in the RIPT packet but when you start looking at the RTCP data or some of the RTP header extensions, I think that needs to all map to stuff in RIPT but probably not something that has to be every RIPT packet. 

Hope that makes sense. That said, I think this still is an extremely mostly 1:1 and simple 20 lines of code to map between RTP/RTCP and RIPT but I think we will end up with a bit of state that if frequently refreshed and is no worse than loosing an IFrame if you loose that state. 

> On Mar 27, 2020, at 2:22 PM, Justin Uberti <juberti=[email-masked]> wrote:
>

---

From: Cullen Jennings <[email-masked]>
Date: Mon, 30 Mar 2020 08:15:13 -0600

I’m thinking of this as 100% client / server.  Sure you can build a conferences on top of that but the RIPT layer would not even need to have a clue about the overall conference media topology. The server can do things like pause / resume media FIR etc but it all just two parties involved - the client and the server. Assuming a server that maps media is one of the things that make this so much easier than an RTP multicast conferences. 

> On Mar 27, 2020, at 4:35 PM, Magnus Westerlund <magnus.westerlund=[email-masked]> wrote:
> 
> Hi,
>

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Mon, 30 Mar 2020 09:37:07 -0500

For what it's worth,

> Yes, but even session establishment and control is still signalling from my

I've certainly been thinking of "everything except media payload" as
signaling in my own comments. So, close to Magnus here, IIUC.

If we don't have a common understanding of this (and it sounds like we
might not), that would be a good thing to talk about during the charter
development.

Best,

Spencer

---

From: Justin Uberti <[email-masked]>
Date: Mon, 30 Mar 2020 09:56:34 -0700

On Mon, Mar 30, 2020 at 6:55 AM Cullen Jennings <[email-masked]> wrote:

>
> On the stateless or not … I’ve tried implementing this a few different
> ways and I think what we want is a sort of mostly stateless with frequent
> sate updates for the parts that are not stateless. Let me explain ….
>
> First keep in mind there is nothing stateless about the QUIC connection.
> Lots of state for crypto. It also seems like it would be a good thing to
> allow state for compression. So we are already out of a fully stateless
> design before we even start.
>

Sure, there is lots of state for crypto. But this will typically be handled
by a frontend server at the edge of the cloud infra, so we should focus on
the RTP-specific parts.

>
> So when we say RIPT can map to RTP, we really mean RTP + RTCP as that is
> needed. In the RTCP we need things like the NTP timestamp. I don’t think we
> want to send the full NTP timestamp in ever RIPT packet which probably
> rules out a fully stateless design. In WebRTC we took the approach of
> putting some data in RTP header extensions and sending them not ever packet
> but relatively frequently like every second or every IFrame.  I think we
> should do the same thing here - we should have some stuff that is sent
> frequently enough that if you loose all state until it is resent, it is
>  not a huge deal.
>
> This would allow us to do things like send the info to map to NTP time
> once a second not in every packet. When you look at what is in a basic RTP
> packet, I think it is easy to have that 1:1 mapping of all that data to
> something in the RIPT packet but when you start looking at the RTCP data or
> some of the RTP header extensions, I think that needs to all map to stuff
> in RIPT but probably not something that has to be every RIPT packet.
>
> Hope that makes sense. That said, I think this still is an extremely
> mostly 1:1 and simple 20 lines of code to map between RTP/RTCP and RIPT but
> I think we will end up with a bit of state that if frequently refreshed and
> is no worse than loosing an IFrame if you loose that state.
>

I can get on board with this. As you say, there already is some state with
RTP/RTCP in terms of the timestamp mapping. The main thing is that this
needs to be loss-tolerant (which makes compression somewhat harder) and
self-healing, e.g., on a gateway failover.

But yeah, it would be nice to not have to send the ~same 12 bytes + 0xBEDE
in every packet. (And this is why I think we can reach parity or better
with RTP efficiency.)

>
>
> On Mar 27, 2020, at 2:22 PM, Justin Uberti <
> juberti=[email-masked]> wrote:
>
>> I think it is fine to require a media gateway to go from RIPT to RTP.
>> Sure that should be pretty easy - one of the major use cases for this work
>> will likely involve building SBCs that talk RIPT on one side and SIP on the