Subject: Re: [Moq] Current State of Live Media Ingest Protocols

From: [email-masked] (Michael C. Thornburgh)
Date: Thu, 31 Mar 2022 17:51:40 -0700 (PDT)

Hi Maxim, Alex, list. Apologies: I had to copy the included message from the
list archive. I wasn't subscribed so I wasn't able to make a proper reply.

I have more details on RTMFP and RTMP.

Maxim said:

> There is also a variation RTMP called RTMFP, which is RTMP over UDP. But it feels like they’ve missed the momentum.

RTMFP is a general purpose transport protocol over UDP, similar to (but
predating) QUIC. It's described in RFC 7016, and before that it was presented
in the TSV Area meeting at IETF 77 [1]. It's designed for real-time and P2P
communication.

RFC 7425 describes how to transport RTMP over RTMFP, for both client-server
and 1-to-1 P2P.

There have been several open source implementations of RTMFP over the years,
but all of the ones I've found have been incomplete _and_ incorrect. To fix
this, I wrote a new open source implementation [2]. It includes a relay/proxy
tool ("test/tcrelay") that can translate between RTMFP and RTMP, so you can
use RTMFP with your existing RTMP systems. In addition to giving real-time
treatment to messages (prioritization, deadlines, optional delay-based
congestion detection), [2] has some experimental features beyond RFC 7016,
like support for ECN.

Alex said:

> With regard to codecs, the media layer of RTMP (FLV tags) are
> insufficiently extensible. They might have been suitable for a closed
> Flash ecosystem, but small consecutive integers in narrow bitfields
> were obsolete when RTMP first launched.

RTMP's media layer is intrinsically tied to the structure of the RTMP chunk
stream and FLV. When transporting RTMP messages according to RFC 7425, RTMFP
has the same codec limitations. However, RTMFP isn't limited to RTMP messages,
and can transport different message formats and accommodates permissionless
innovation. See for example [3].

Regarding RTMP, Alex also said:

> However the most popular RTMP senders (librtmp and OBS
> Studio) all seem to set chunk size to the maximum message size and do
> not engage in any chunkstream level operations.

I've seen that OBS will drop video frames (but not audio) if they've been
queued for too long, but in general the open source RTMP implementations I've
found don't do the "Real-Time" part of "Real-Time Messaging Protocol" at all.
Last time I checked, none of them take advantage of TCP_NOTSENT_LOWAT (of
course, if using RTMP just for framing with no real-time treatment, there's
no reason to).

Note the "tcrelay" tool in [2] can also relay RTMP ↔︎ RTMP. It uses
TCP_NOTSENT_LOWAT to make transmission decisions as late as possible, and
applies real-time treatment to video and audio (prioritizing audio over video
and dropping frames if they're past deadline, including abandoning in-progress
frames if needed). Try it with your favorite RTMP system!

Possibly related to the lack of any real-time treatment in other open source
RTMP senders, I recently discovered that VLC really doesn't like it if there
are missing frames when playing an RTMP stream. I haven't tried GStreamer or
ffmpeg yet to see what they do when frames have been dropped.

-michael thornburgh

  [1]: https://www.ietf.org/proceedings/10mar/slides/tsvarea-1.pdf
  [2]: https://github.com/zenomt/rtmfp-cpp
  [3]: http://zenomt.com/ns/rtmfp#media

> Maxim Sharabayko <[email-masked]> Thu, 24 March 2022 13:00 UTC
> 
> Hey Alex,
> 
> Thank you for sharing your thoughts and experiences! And thank you for a great talk on SRT back in 2019! 😊
> 
> On RTMP I believe I saw some activities on reviving it. As if Veriskope has taken over the RTMP server and protocol, also offering the SDK.
> Not sure how things are going there.
> There is also a variation RTMP called RTMFP, which is RTMP over UDP. But it feels like they’ve missed the momentum.
> 
> > In my experience SRT and RIST both have a lot of the same problems.
> > They primarily act as a tunnel for linear MPEG Transport Stream.
> > They both nominally allow other payloads, but lack a great interop story for them.
> 
> RIST is RTP-based, and it allows what RTP allows. IT is a compilation of RTP-related RFCs covering different SRT functionalities, but at the same time bringing extra complexities in. For example, RIST simple profile is RTP + RTCP, each requiring a separate UDP port number.
> RIST Main Profile pulls GRE (RFC 1701) in, multiplexing RTP and RTCP into GRE, and thus losing compatibility with legacy hardware understanding pure RTP.
> I may be wrong, but RIST feels like a small subset of what WebRTC has to offer.
> 
> SRT is content agnostic thus allowing any type of payload.
> The main reason why MPEG-2 TS over SRT is largely used nowadays is that MPEG-2 TS is the industry-standard among broadcasters.
> For them transferring existing workflows on top of SRT is just as simple as switching from sending MPEG-2 TS over UDP to sending MPEG-2 TS over SRT. This simplicity also comes from the SRT API, which tries to follow the Berkley Sockets API.
> 
> In fact, SRT does not put limits onto the payload. You should think of it as of UDP with ARQ and some fancy staff on top, and with a certain awareness about timestamps and the end-to-end latency constraints. And most of that (except for ARQ) is optional and can be disabled.
> 
> > The most popular implementations of both ask integrators to set a buffer size
> > in terms of wall time based on connection characteristics,
> > without having sent or received a single packet. In practice this seems mean
> > that users on high quality connections get stuck with oversized buffers.
> 
> A fair point. There are however certain reasons for it. This topic is raised quite often in the SRT working group.
> The size of the receiver buffer is configured in bytes and is considered a receiver-side setting only (at least as of now). However, during the transmission SRT sender is aware about the available space in the receiver’s buffer.
> SRT asks the user to configure the receiver buffer size and the maximum expected bitrate of the stream multiplied by the target SRT buffering latency (delay).
> Please refer to the SRT Configuration Guidelines here: https://github.com/Haivision/srt/blob/master/docs/API/configuration-guidelines.md
> 
> SRT currently does not negotiate the receiver buffer size at the moment (SRT itself does not block improvements of this behavior in the future versions though).
> Expecting certain maximum allowed latency and maximum allowed bitrate, SRT receiver is expected to pre-configure it.
> The main caveat is that SRT sender can  negotiate a higher latency than the receiver expects. The responsibility to track this via SRT connection callbacks is currently on the application. It can e.g. reject  this kind of connections, or increase the receiver buffer size from the listener callback before the connection is finally accepted.
> But once the connection is established, buffer sizes can no longer be changed. This constraint is applied on purpose to not overuse the memory in run time when a lot of simultaneous transmissions is happening over different SRT connections on the same device.
> 
> Thus we come to the point, when if the sending application is aware about its maximum sending rate, it can negotiate it with the receiving application via e.g. SRT Stream ID, and properly configure buffer sizes at the connection time.
> 
> Still I agree it would be nicer and easier for the end user to pull this inside of SRT. This improvement is in our backlog. There is also another extremely cool and related improvement backlogged, that would offer something mind blowing!
> Once we add some feature to SRT, it is adopted quite promptly, while deprecation is extremely hard. Therefore, unfortunately, it takes time.
> 
> > SRT and RIST are certainly capable of sub-second latency in and of themselves,
> > but eat quite a bit of the overall latency budget if the
> > end-to-end latency target is sub-second, especially if transcoding is in the mix.
> 
> Could you please elaborate on this? I am not sure I understood. Transuding certainly has its own latency. Furthermore, general case MPEG-2 TS demultiplexing applies additional buffering. If SRT with its Timestamp-Based Packet Delivery is used, the buffering requirements of demultiplexers can be significantly reduced.
> 
> 
> > SRT and RIST generally act as a media (media type importance)oblivious tunnel.
> > The RIST and SRT protocols don’t know anything about what kind of data is in a lost packet,
> > so it’s not clear how to prioritize retransmissions beyond FIFO and tunnel deadline minus RTT.
> > If a big group of packets are lost at once, a receiver cannot ask for
> > audio first or not bother asking for the tail of a GOP. A sufficiently
> > smart sender could potentially keep an index of packet types, but I
> > don’t know of any that actually do.
> 
> Another direct hit on our backlist! :)
> We did a small research among SRT alliance members some time ago. Generally speaking, this feature can be added. However we found out, that supporting priorities in SRT-ready solutions would require a certain R&D effort from SRT integrators. In live media contribution and distribution use cases priorities would arguably not provide a decent improvement is QoS given the latency-aware Too-Late packet drop mechanism is in play anyway (but likely makes sense in the deliver-to-the-end-user use case).
> In the most of the cases it makes sense to prioritize an I frame (or some reference frame P/B frame). At the same time, this I-frame usually takes the most share in the stream. In case of congestion within the end-to-end latency constraints it is very likely you would not be able to recover loss on time. Furthermore, even an I frame have certain priorities inside :)
> Losing a portion in the very beginning of the bitstream (MB closer to the top-left corner in raster order) would ruin all other macroblocks that follow in the slice.
> 
> That is definitely a separate topic of discussion. As a former video codec developer, I am extremely interested in this discussion and would love to hear your thoughts!
> 
> > The fixed buffer size for a linearly muxed input and content agnostic
> > tunneling also limits the ability to take advantage of any differences
> > in down stream buffering. It’s conceivable that when feeding a
> > transcoder engaging in video frame reordering (e.g. b-frames, or even
> > long rate control lookahead) one might be willing to wait longer on
> > missing audio than video.
> 
> Waiting longer for a video decoder and demultiplexer means rebuffering, and stalls in the picture. Imagine this happening on the contribution side during a live event, affecting everything and everyone down the pipeline, including all end users.
> The key concept of SRT in the live streaming configuration is to not allow such situations, and transmit the content within the configured latency constrains.
> The size of the buffer does not have a direct influence on the buffering delay. The latency does. The latency is configured and fixed during the handshake. Later on the goal is to keep the end-to-end latency the same even when RTT changes. Still, if the buffer is too small to hold latency x bitrate, the receiver would not accept those packets until it gains some space.
> A brief overview of the SRT Latency concept can be found here:  https://srtlab.github.io/srt-cookbook/protocol/tsbpd/latency/
> A more elaborate description can be found in the SRT RFC draft https://datatracker.ietf.org/doc/html/draft-sharabayko-srt-01.
> 
> 
> I think I should pause here, and maybe reply on the rest in a separate e-mail unless I’ve already bored you with all these details L
> 
> 
> From: Moq <[email-masked]> on behalf of Alex Converse <[email-masked]>
> Date: Wednesday, 23. March 2022 at 20:52
> To: Ying Yin <yingyin=[email-masked]>
> Cc: [email-masked] <[email-masked]>
> Subject: Re: [Moq] Current State of Live Media Ingest Protocols
> Hey Ying,
> 
> Thanks for sharing your thoughts about the current state of these
> ingest protocols. I’ve done various degrees of experimentation with
> these protocols at Twitch and thought I might share some of my
> thoughts and experiences.
> 
> == RTMP ==
> 
> I disagree a little bit about the characterization of RTMP being frame
> based. RTMP is based on chunks where a chunk is not a multi-frame
> segment but an envelope of bytes that when re-assembled make up a
> frame.
> 
> I think this distinction matters because the purpose of these chunks
> seems to be to enable prioritization. In particular the chunkstream
> format allows for pausing sending a video frame to send something
> higher priority (like audio) before resuming that frame. It also
> allows for abandoning a frame that has been partially transmitted.
> 
> The more I’ve dug into it, the more I’ve started to think that RTMP
> chunk stream is pretty close to optimal for the constraints of having
> a single TCP connection and being willing to keep it open for the
> entirety of a broadcast session.
> 
> As far as I can tell, Flash Player really did use chunks for
> prioritization. However the most popular RTMP senders (librtmp and OBS
> Studio) all seem to set chunk size to the maximum message size and do
> not engage in any chunkstream level operations.
> 
> Ideally an RTMP replacement would make it easier for a sender to
> engage in prioritization. I think this ties nicely into the
> constraints of TCP.
> 
> >From where I’m sitting the biggest problem with TCP congestion control
> in the context of RTMP as a contribution protocol is that you are at
> the mercy of the host OS for TCP. The OS provides the TCP stack and
> you get what it provides. There is little room to iterate on
> congestion control or congestion detection, and it’s extremely
> unlikely that a consumer device (phone, desktop, laptop) will let the
> application make these decisions, or even have the OS itself ship
> updates to TCP in a timely fashion.
> 
> With regard to codecs, the media layer of RTMP (FLV tags) are
> insufficiently extensible. They might have been suitable for a closed
> Flash ecosystem, but small consecutive integers in narrow bitfields
> were obsolete when RTMP first launched. Ideally an RTMP replacement
> not just supports modern codecs, but is extensible enough to add new
> codecs without revising the protocol and to allow vendors to
> experiment with new protocols without the risk of codec id collisions.
> 
> === SRT and RIST ===
> 
> In my experience SRT and RIST both have a lot of the same problems.
> They primarily act as a tunnel for linear MPEG Transport Stream. They
> both nominally allow other payloads, but lack a great interop story
> for them.
> 
> The most popular implementations of both ask integrators to set a
> buffer size in terms of wall time based on connection characteristics,
> without having sent or received a single packet. In practice this
> seems mean that users on high quality connections get stuck with
> oversized buffers.
> 
> SRT and RIST are certainly capable of sub-second latency in and of
> themselves, but eat quite a bit of the overall latency budget if the
> end-to-end latency target is sub-second, especially if transcoding is
> in the mix.
> 
> SRT and RIST generally act as a media (media type importance)
> oblivious tunnel. The RIST and SRT protocols don’t know anything about
> what kind of data is in a lost packet, so it’s not clear how to
> prioritize retransmissions beyond FIFO and tunnel deadline minus RTT.
> If a big group of packets are lost at once, a receiver cannot ask for
> audio first or not bother asking for the tail of a GOP. A sufficiently
> smart sender could potentially keep an index of packet types, but I
> don’t know of any that actually do.
> 
> The fixed buffer size for a linearly muxed input and content agnostic
> tunneling also limits the ability to take advantage of any differences
> in down stream buffering. It’s conceivable that when feeding a
> transcoder engaging in video frame reordering (e.g. b-frames, or even
> long rate control lookahead) one might be willing to wait longer on
> missing audio than video.
> 
> Using QUIC streams for media of different priorities and having a
> sender and receiver understand the relation between media dependence
> of these streams (like having video streams start on IDR) presents an
> opportunity for big improvement over the constraints of RIST and SRT.
> 
> == CMAF over SRT ==
> 
> There seems to be some discussion about using CMAF over SRT in the
> comments on the Google Doc, but it seems to suffer from the same
> stream multiplex linearization problems as TS.
> 
> It also seems like it would need an additional framing layer to
> correct for unrecovered loss. For TS in SRT, SRT datagrams always
> begin on TS packet boundaries, and the TS packet headers allow the
> decoder to find frame boundaries. It’s not clear to me how CMAF over
> SRT deals with unrecovered loss without an additional framing layer.
> 
> I’d love to read more from the SRT folks about how CMAF-over-SRT works
> and how it handles these problems.
> 
> On Tue, Mar 22, 2022 at 12:16 PM Ying Yin
> <yingyin=[email-masked]> wrote:
> >
> > Hi MOQ,
> >
> > To complement Luke's doc on video distribution protocols, here's a doc as an attempt to summarize on the Current State of Live Media Ingest Protocols (with contributions from James and Kirill Pugin). I'll also be giving a short presentation tomorrow during the BoF about live media ingest challenges.
> >
> > Thanks,
> > Ying
> > --
> > Moq mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/moq
> 
> --
> Moq mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/moq
>