Subject: Re: [AVTCORE] Video over QUIC

From: Bernard Aboba <[email-masked]>
Date: Fri, 24 Sep 2021 17:35:10 -0700

(AVTCORE Chair hat off)

Over the last few months, there has been considerable progress toward
implementation of WebCodecs and WebTransport in Chromium.  This means that
as of M94, it is possible to implement use cases that have been under
discussion, such as video ingest over QUIC.

The WebCodecs API, which provides low-level access to encode and decode
functionality in the browser, is available by default in Chromium M94, so
that from this point forward browser applications can access the API
without flags or an Origin Trial token.

The WebCodecs implementation includes support for encoding of temporal
scalability in VP8, VP9 and H.264, as well as decoding.  AV1 encoding is
currently not supported, but AV1 decode is.

For VP8, VP9 and H.264 functionality includes configuring the encoder for
temporal scalability using the scalabilityMode notation documented in
WebRTC-SVC.  It is also possible to encode an input stream with multiple
encoders so as to support simulcast.

The WebCodecs API and sample code are available here:
https://w3c.github.io/webcodecs/

WebTransport over HTTP/3 has been in Origin Trial in Chromium since M91:
https://www.chromestatus.com/feature/4854144902889472

There has also been progress made toward development of Webtransport Test
servers.  For example, a WPT WebTransport test server is here:
https://github.com/web-platform-tests/wpt/tree/master/tools/webtransport

A sample application illustrating the API (which differs somewhat from the
published specification) is available here:
https://webrtc.internaut.com/wt/

All of this progress has both good and challenging aspects.  The good news
is that it is now possible not just to talk about potential protocol
proposals, but to implement them in Javascript, including advanced
functionality.

The challenging aspect is that the learning curve for developers is higher,
because it is necessary to understand WebCodecs and WebTransport APIs as
well as the WHATWG Streams paradigm that enables the chaining together of
media pipelines in an elegant (and easy to read) manner.

IMHO, we are making some progress in illustrating how to hook together
MediaStreamTracks, VideoFrame Streams, Effects, WebCodecs Encode/Decode and
Serialization/Deserialization with WHATWG Streams pipeTo/pipeThrough.

Which is to say that there is now an opportunity for the intrepid (and
patient!) to bring some of the ideas we have been discussing to fruition
without having to endure extraordinary frustration.

If you're interested in learning more (and perhaps helping out with a
future interop/hackathon session), please contact me.