Subject: [Moq] Current State of Live Media Ingest Protocols

From: Ying Yin <[email-masked]>
Date: Tue, 22 Mar 2022 15:15:37 -0400

Hi MOQ,

To complement Luke's doc on video distribution protocols, here's a doc as
an attempt to summarize on the Current State of Live Media Ingest Protocols
<https://docs.google.com/document/d/1qOBOLAMhsCHsDZQ_qLEKjcQVoBfuM7FAQLqSUxB3jZI/edit?usp=sharing>
(with
contributions from James and Kirill Pugin). I'll also be giving a short
presentation tomorrow during the BoF about live media ingest challenges.

Thanks,
Ying

---

From: Alex Converse <[email-masked]>
Date: Wed, 23 Mar 2022 12:21:52 -0700

Hey Ying,

Thanks for sharing your thoughts about the current state of these
ingest protocols. I’ve done various degrees of experimentation with
these protocols at Twitch and thought I might share some of my
thoughts and experiences.

== RTMP ==

I disagree a little bit about the characterization of RTMP being frame
based. RTMP is based on chunks where a chunk is not a multi-frame
segment but an envelope of bytes that when re-assembled make up a
frame.

I think this distinction matters because the purpose of these chunks
seems to be to enable prioritization. In particular the chunkstream
format allows for pausing sending a video frame to send something
higher priority (like audio) before resuming that frame. It also
allows for abandoning a frame that has been partially transmitted.

The more I’ve dug into it, the more I’ve started to think that RTMP
chunk stream is pretty close to optimal for the constraints of having
a single TCP connection and being willing to keep it open for the
entirety of a broadcast session.

As far as I can tell, Flash Player really did use chunks for
prioritization. However the most popular RTMP senders (librtmp and OBS
Studio) all seem to set chunk size to the maximum message size and do
not engage in any chunkstream level operations.

Ideally an RTMP replacement would make it easier for a sender to
engage in prioritization. I think this ties nicely into the
constraints of TCP.

>From where I’m sitting the biggest problem with TCP congestion control
in the context of RTMP as a contribution protocol is that you are at
the mercy of the host OS for TCP. The OS provides the TCP stack and
you get what it provides. There is little room to iterate on
congestion control or congestion detection, and it’s extremely
unlikely that a consumer device (phone, desktop, laptop) will let the
application make these decisions, or even have the OS itself ship
updates to TCP in a timely fashion.

With regard to codecs, the media layer of RTMP (FLV tags) are
insufficiently extensible. They might have been suitable for a closed
Flash ecosystem, but small consecutive integers in narrow bitfields
were obsolete when RTMP first launched. Ideally an RTMP replacement
not just supports modern codecs, but is extensible enough to add new
codecs without revising the protocol and to allow vendors to
experiment with new protocols without the risk of codec id collisions.

=== SRT and RIST ===

In my experience SRT and RIST both have a lot of the same problems.
They primarily act as a tunnel for linear MPEG Transport Stream. They
both nominally allow other payloads, but lack a great interop story
for them.

The most popular implementations of both ask integrators to set a
buffer size in terms of wall time based on connection characteristics,
without having sent or received a single packet. In practice this
seems mean that users on high quality connections get stuck with
oversized buffers.

SRT and RIST are certainly capable of sub-second latency in and of
themselves, but eat quite a bit of the overall latency budget if the
end-to-end latency target is sub-second, especially if transcoding is
in the mix.

SRT and RIST generally act as a media (media type importance)
oblivious tunnel. The RIST and SRT protocols don’t know anything about
what kind of data is in a lost packet, so it’s not clear how to
prioritize retransmissions beyond FIFO and tunnel deadline minus RTT.
If a big group of packets are lost at once, a receiver cannot ask for
audio first or not bother asking for the tail of a GOP. A sufficiently
smart sender could potentially keep an index of packet types, but I
don’t know of any that actually do.

The fixed buffer size for a linearly muxed input and content agnostic
tunneling also limits the ability to take advantage of any differences
in down stream buffering. It’s conceivable that when feeding a
transcoder engaging in video frame reordering (e.g. b-frames, or even
long rate control lookahead) one might be willing to wait longer on
missing audio than video.

Using QUIC streams for media of different priorities and having a
sender and receiver understand the relation between media dependence
of these streams (like having video streams start on IDR) presents an
opportunity for big improvement over the constraints of RIST and SRT.

== CMAF over SRT ==

There seems to be some discussion about using CMAF over SRT in the
comments on the Google Doc, but it seems to suffer from the same
stream multiplex linearization problems as TS.

It also seems like it would need an additional framing layer to
correct for unrecovered loss. For TS in SRT, SRT datagrams always
begin on TS packet boundaries, and the TS packet headers allow the
decoder to find frame boundaries. It’s not clear to me how CMAF over
SRT deals with unrecovered loss without an additional framing layer.

I’d love to read more from the SRT folks about how CMAF-over-SRT works
and how it handles these problems.

On Tue, Mar 22, 2022 at 12:16 PM Ying Yin
<yingyin=[email-masked]> wrote:
>
> Hi MOQ,
>
> To complement Luke's doc on video distribution protocols, here's a doc as an attempt to summarize on the Current State of Live Media Ingest Protocols (with contributions from James and Kirill Pugin). I'll also be giving a short presentation tomorrow during the BoF about live media ingest challenges.
>
> Thanks,
> Ying
> --
> Moq mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/moq

---

From: Maxim Sharabayko <[email-masked]>
Date: Thu, 24 Mar 2022 12:59:54 +0000

Hey Alex,

Thank you for sharing your thoughts and experiences! And thank you for a great talk on SRT back in 2019! 😊

On RTMP I believe I saw some activities on reviving it. As if Veriskope has taken over the RTMP server and protocol, also offering the SDK.
Not sure how things are going there.
There is also a variation RTMP called RTMFP, which is RTMP over UDP. But it feels like they’ve missed the momentum.

> They primarily act as a tunnel for linear MPEG Transport Stream.
> They both nominally allow other payloads, but lack a great interop story for them.

RIST is RTP-based, and it allows what RTP allows. IT is a compilation of RTP-related RFCs covering different SRT functionalities, but at the same time bringing extra complexities in. For example, RIST simple profile is RTP + RTCP, each requiring a separate UDP port number.
RIST Main Profile pulls GRE (RFC 1701) in, multiplexing RTP and RTCP into GRE, and thus losing compatibility with legacy hardware understanding pure RTP.
I may be wrong, but RIST feels like a small subset of what WebRTC has to offer.

SRT is content agnostic thus allowing any type of payload.
The main reason why MPEG-2 TS over SRT is largely used nowadays is that MPEG-2 TS is the industry-standard among broadcasters.
For them transferring existing workflows on top of SRT is just as simple as switching from sending MPEG-2 TS over UDP to sending MPEG-2 TS over SRT. This simplicity also comes from the SRT API, which tries to follow the Berkley Sockets API.

In fact, SRT does not put limits onto the payload. You should think of it as of UDP with ARQ and some fancy staff on top, and with a certain awareness about timestamps and the end-to-end latency constraints. And most of that (except for ARQ) is optional and can be disabled.

> The most popular implementations of both ask integrators to set a buffer size
> in terms of wall time based on connection characteristics,
> without having sent or received a single packet. In practice this seems mean
> that users on high quality connections get stuck with oversized buffers.

A fair point. There are however certain reasons for it. This topic is raised quite often in the SRT working group.
The size of the receiver buffer is configured in bytes and is considered a receiver-side setting only (at least as of now). However, during the transmission SRT sender is aware about the available space in the receiver’s buffer.
SRT asks the user to configure the receiver buffer size and the maximum expected bitrate of the stream multiplied by the target SRT buffering latency (delay).
Please refer to the SRT Configuration Guidelines here: https://github.com/Haivision/srt/blob/master/docs/API/configuration-guidelines.md

SRT currently does not negotiate the receiver buffer size at the moment (SRT itself does not block improvements of this behavior in the future versions though).
Expecting certain maximum allowed latency and maximum allowed bitrate, SRT receiver is expected to pre-configure it.
The main caveat is that SRT sender can  negotiate a higher latency than the receiver expects. The responsibility to track this via SRT connection callbacks is currently on the application. It can e.g. reject  this kind of connections, or increase the receiver buffer size from the listener callback before the connection is finally accepted.
But once the connection is established, buffer sizes can no longer be changed. This constraint is applied on purpose to not overuse the memory in run time when a lot of simultaneous transmissions is happening over different SRT connections on the same device.

Thus we come to the point, when if the sending application is aware about its maximum sending rate, it can negotiate it with the receiving application via e.g. SRT Stream ID, and properly configure buffer sizes at the connection time.

Still I agree it would be nicer and easier for the end user to pull this inside of SRT. This improvement is in our backlog. There is also another extremely cool and related improvement backlogged, that would offer something mind blowing!
Once we add some feature to SRT, it is adopted quite promptly, while deprecation is extremely hard. Therefore, unfortunately, it takes time.

> SRT and RIST are certainly capable of sub-second latency in and of themselves,
> but eat quite a bit of the overall latency budget if the
> end-to-end latency target is sub-second, especially if transcoding is in the mix.

Could you please elaborate on this? I am not sure I understood. Transuding certainly has its own latency. Furthermore, general case MPEG-2 TS demultiplexing applies additional buffering. If SRT with its Timestamp-Based Packet Delivery is used, the buffering requirements of demultiplexers can be significantly reduced.

> SRT and RIST generally act as a media (media type importance)oblivious tunnel.
> The RIST and SRT protocols don’t know anything about what kind of data is in a lost packet,
> so it’s not clear how to prioritize retransmissions beyond FIFO and tunnel deadline minus RTT.

Another direct hit on our backlist! :)
We did a small research among SRT alliance members some time ago. Generally speaking, this feature can be added. However we found out, that supporting priorities in SRT-ready solutions would require a certain R&D effort from SRT integrators. In live media contribution and distribution use cases priorities would arguably not provide a decent improvement is QoS given the latency-aware Too-Late packet drop mechanism is in play anyway (but likely makes sense in the deliver-to-the-end-user use case).
In the most of the cases it makes sense to prioritize an I frame (or some reference frame P/B frame). At the same time, this I-frame usually takes the most share in the stream. In case of congestion within the end-to-end latency constraints it is very likely you would not be able to recover loss on time. Furthermore, even an I frame have certain priorities inside :)
Losing a portion in the very beginning of the bitstream (MB closer to the top-left corner in raster order) would ruin all other macroblocks that follow in the slice.

That is definitely a separate topic of discussion. As a former video codec developer, I am extremely interested in this discussion and would love to hear your thoughts!

> missing audio than video.

Waiting longer for a video decoder and demultiplexer means rebuffering, and stalls in the picture. Imagine this happening on the contribution side during a live event, affecting everything and everyone down the pipeline, including all end users.
The key concept of SRT in the live streaming configuration is to not allow such situations, and transmit the content within the configured latency constrains.
The size of the buffer does not have a direct influence on the buffering delay. The latency does. The latency is configured and fixed during the handshake. Later on the goal is to keep the end-to-end latency the same even when RTT changes. Still, if the buffer is too small to hold latency x bitrate, the receiver would not accept those packets until it gains some space.
A brief overview of the SRT Latency concept can be found here:  https://srtlab.github.io/srt-cookbook/protocol/tsbpd/latency/
A more elaborate description can be found in the SRT RFC draft https://datatracker.ietf.org/doc/html/draft-sharabayko-srt-01.

I think I should pause here, and maybe reply on the rest in a separate e-mail unless I’ve already bored you with all these details L

From: Moq <[email-masked]> on behalf of Alex Converse <[email-masked]>
Date: Wednesday, 23. March 2022 at 20:52
To: Ying Yin <yingyin=[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [Moq] Current State of Live Media Ingest Protocols
Hey Ying,

== RTMP ==

=== SRT and RIST ===

== CMAF over SRT ==

***[EXTERNAL] This message comes from an external organization. Exercise caution when opening attachments or clicking links, especially from unknown senders.***