Subject: Prioritizing QUIC DATAGRAMs (was: Re: [Masque] Prioritizing HTTP DATAGRAMs)

From: Lucas Pardue <[email-masked]>
Date: Tue, 22 Jun 2021 15:58:41 +0100

Hey Spencer,

Spinning out a new thread that I think is most suited to the QUIC WG.

On Tue, Jun 22, 2021 at 2:57 PM Spencer Dawkins at IETF <
[email-masked]> wrote:

> Hi, Lucas,
>
> On Tue, Jun 22, 2021 at 8:40 AM Lucas Pardue <[email-masked]>
> wrote:
>
>> Hello HTTP and MASQUE,
>>
>> Over the last couple of months, the question about prioritization with
>> respect to HTTP DATAGRAMs has come up first in MASQUE issue # 46 [1] and
>> then HTTP issue #1550 [2], which was also discussed during the recent HTTP
>> interim.
>>
>> Extensible priorities is pretty far along it's journey, which has so far
>> been focused on HTTP message content (and CONNECT tunnel data, see PR #1544
>> [3]). The scheme fulfills the needs of the base HTTP/2 and HTTP/3
>> specifications, and so far hasn't considered extensions. Extensible
>> priorities acts as a replacement for HTTP/2's prioritization scheme, while
>> being the only known scheme defined for HTTP/3. However, there is nothing
>> to prevent alternate schemes being defined or used in the future (although
>> we hope the need for that can be avoided by the extensibility here).
>>
>> Endpoints that send DATAGRAM flows concurrently with other flows or
>> streams have to make scheduling decisions. Therefore, the question about
>> how to prioritize them, and to communicate that via signals, is a good one.
>> However, currently the editors of draft-ietf-masque-h3-datagram and
>> draft-ietf-httpbis-priority (disclosure: I am co-editor on both) feel that
>> linking these two drafts directly is not the best approach for either.
>>
>> On draft-ietf-masque-h3-datagram issue #46 [1], we resolved the
>> discussion by adding text to say that prioritization of HTTP/3 datagrams is
>> not defined by the document.
>>
>> For draft-ietf-masque-h3-datagram issue #1550 [2], the proposed
>> resolution is PR #1559 [4]. The PR adds a clear statement that the document
>> is focused on HTTP content and CONNECT tunnel data. It also makes clear
>> that extensions like DATAGRAM can also use the scheme but punts that to
>> their court.
>>
>> Kazuho and I are seeking some feedback for PR #1559 [4] before landing
>> it. We appreciate that this leaves a gap for DATAGRAM priorities,
>> especially since DATAGRAM says nothing. But the thought process is that
>> another Internet-Draft could fill this gap. This would create an indirect
>> relationship that would allow documents to progress independently. I'm
>> planning to start a draft soon and have it ready by IETF 111. Which WG it
>> should belong to is probably another matter for debate.
>>
>
> What follows is certainly off-topic for HTTPbis, and probably for MASQUE
> as well, but if it's worth talking about, you'd know better where we should
> talk about it.
>
> I believe that
> https://datatracker.ietf.org/doc/draft-hurst-quic-rtp-tunnelling/ was
> bumping up against the desire to use QUIC datagrams for tunneling and the
> recognition that we don't have an agreed-upon way to multiplex (and
> demultiplex) datagrams that are carried in the same QUIC connection.
>
> Do you see any connection between prioritizing datagrams and
> multiplexing/demultiplexing datagrams?
>

Speaking personally. Yes there is a connection. But it only really matters
where there is competition for resources, which typically manifests when
things occur concurrently.

draft-ietf-quic-datagram calls these entities and says [1]

If the application protocol supports the coexistence of multiple
entities using datagrams inside a single QUIC connection, it may need
a mechanism to allow demultiplexing between them.

This really means the problem of defining the demultiplexing is left
as a job to the application,
if it needs it. And I think that is the correct choice because, as
we're finding out in
HTTP/3 DATAGRAM, we're evolving our understanding of demultiplexing requirements
within the context and constraints of HTTP.

I don't know if draft-ietf-quic-datagram needs to say anything
specific about prioritization.
Maybe it could crib some test from RFC 9000 Section 2.3 [2], so that
people builiding
DATAGRAM-based applications are aware of the considerations.

Cheers,

Lucas

[1] https://datatracker.ietf.org/doc/html/draft-ietf-quic-datagram#section-5
[2] https://www.rfc-editor.org/rfc/rfc9000.html#section-2.3

>
> Best,
>
> Spencer
>
>
>> Cheers
>> Lucas
>> Wearing co-editor hat for HTTP/3 DATAGRAM and Extensible priorities
>>
>>
>> [1]
>> https://github.com/ietf-wg-masque/draft-ietf-masque-h3-datagram/issues/46
>> [2] https://github.com/httpwg/http-extensions/issues/1550
>> [3] https://github.com/httpwg/http-extensions/pull/1544
>> [4] https://github.com/httpwg/http-extensions/pull/1559
>> --
>> Masque mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/masque
>>
>

---

From: Samuel Hurst <[email-masked]>
Date: Wed, 23 Jun 2021 09:36:55 +0100

On 22/06/2021 15:58, Lucas Pardue wrote:
> On Tue, Jun 22, 2021 at 2:57 PM Spencer Dawkins at IETF
> <[email-masked] <mailto:[email-masked]>>
> wrote:
>     I believe
>     that https://datatracker.ietf.org/doc/draft-hurst-quic-rtp-tunnelling/
>     was bumping up against the desire to use QUIC datagrams for
>     tunneling and the recognition that we don't have an agreed-upon way
>     to multiplex (and demultiplex) datagrams that are carried in the
>     same QUIC connection. 
> 
> Speaking personally. Yes there is a connection. But it only really
> matters where there is competition for resources, which typically
> manifests when things occur concurrently.
> 
> This really means the problem of defining the demultiplexing is left as
> a job to the application,
> if it needs it. And I think that is the correct choice because, as we're
> finding out in

The primary issue that I can see with this is that it potentially leads
to the inability to use DATAGRAMs for multiple application protocols in
an interoperable way. While it's fully possible for one application to
de/multiplex it's own traffic on the DATAGRAM channel, multiple
applications sharing the same tunnel might have different (and
incompatible) ideas on how to use an identifier that multiplexes
traffic, or may use different mechanisms entirely.

It's this danger of lack of interoperability that I don't like. I don't
like the idea of having to write lots of application notes saying "you
can do A but not with B, but B and C can coexist", which leads to
applications exploding when someone does something unexpected with
option D down the line, that I didn't foresee.

Of course, I don't know exactly how much call there is for doing this.
For example, with regards to my RTP Tunnelling draft [1] that Spencer
linked to above I haven't encountered a need to run something alongside
RTP/RTCP in DATAGRAMs yet, but that's not to say that it isn't a
possibility.

> I don't know if draft-ietf-quic-datagram needs to say anything specific
> about prioritization.

I'm certainly interested in some form of prioritisation for my RTP
Tunnelling draft [1], as protocols like RTP run in real-time and other
things getting in the way can easily cause poor quality of service. This
could be by making the application use longer receive buffers than
necessarily to ensure smooth audio and video playback at the receiver,
or random pauses and glitches in the stream.

Best regards,
-Sam

[1]: https://datatracker.ietf.org/doc/draft-hurst-quic-rtp-tunnelling/

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 23 Jun 2021 11:53:06 +0100

Hey Sam,

> traffic, or may use different mechanisms entirely.

Right. But having an identifier in the transport doesn't help much here.
Streams have such an identifier (split into 4 types) and don't solve the
problem you describe IIUC.

To give an example, an application mapping like HTTP/3 makes use of 3
types, using the up to the entirety of the values permitted. There isn't
currently a way for multiple completely independent HTTP/3 connections to
share a QUIC connection. HTTP/3 connection coalescing achieves sharing to
some degree, it relies on the demultiplexing occurring in the HTTP
messages. Similarly, CONNECT and CONNECT-UDP methods use HTTP to signal the
intent of streams or HTTP/3 datagrams, and applications can build context
state around this to decide how to dispatch received data.

> RTP/RTCP in DATAGRAMs yet, but that's not to say that it isn't a

The challenge is that if you want to mix usage intents for a shared
transport resource, you probably need some way to signal that. Since QUIC
delegates using streams and DATAGRAMs to upper layers applications, it's
very tricky to design anything at the lower layer to accommodate this.

One approach to allow general purpose indiscriminate resource sharing would
be to design a virtualization layer that sits just below actual mappings.
Such a thing would fool upper layers into thinking they had access to
native QUIC. The layer of indirection would probably require a "hypervisor"
to manage things properly. MASQUE is vaguely an instantion of this design
that relies on HTTP. WebTransport is vaguely another example, we used to
have a QuicTransport protocol but pivoted away from it back to HTTP.

The potential solution space is large. However, I'm not sure the problem
space you describe is broad enough to activate building something else. It
would be interesting to hear otherwise. At the end of the day, there needs
to be strong motivation for building complexity and/or runtime overhead in
order to share QUIC connection resources. Otherwise, isn't it just easier
to use multiple separate connections?

> or random pauses and glitches in the stream.

Makes sense. Would you design that into your RTP tunneling mapping?

Cheers
Lucas

---

From: Patrick Meenan <[email-masked]>
Date: Wed, 23 Jun 2021 09:10:58 -0400

Using multiple connections should be strictly worse than streams sharing a
connection, otherwise we probably have a gap that needs to be filled.
Multiple connections kicks the can down to the congestion control
algorithms for each of the separate connections to play nicely with each
other and hopefully result in a decent experience. With a single connection
we should have better knowledge over the overall transport and make better
decisions.

That said, I'd hate for us to end up going down the same road that we did
for HTTP/2 and designing a complex scheme for something without practical
experience deploying it first. Is there enough flexibility in the
extensions to allow for an entity that controls both ends of the tunnel to
do experimentation before evolving a standard (on the tunnel side of
things)?

There are a lot of moving parts that likely need specs and I'm not sure if
this is the right place to sort through all of them but this is sort of the
mental model I have of the problem space:

Client Application (A) <===> LAN Aggregation (B) <------------------------>
WAN Aggregation (C) <======> Remote Application (D)

A & B can be:
- within the same application (browser using data reduction proxy or IP
anonymization).
- on the same device (software VPN, Cloudflare Warp, etc)
- on the same LAN (hardware VPN, ISP internal tunneling, eero->some Amazon
service, etc)

I'm assuming C <=> D is plain IP on the Internet with no assumptions of the
software at D and no expectations that D will be able to influence the
prioritization from it's side directly (maybe by application-specific info
sent to A to adjust from the client side).

I think most of this discussion is around the B <-> C connection where the
traffic is tunneled over QUIC (and that's usually the constrained link
anyway). It's assuming B has some knowledge of how to prioritize the
tunneled traffic. In a lot of early cases, the same company likely controls
B & C and should be able to experiment. There are cases where the B/C
tunnel should use a standard (allowing Chrome to select IP anonymization
services from multiple providers for example) but if we have prioritization
as an optional extension that interops well, that case could fall back to
unprioritized until we have enough field experience to see what works well
before standardizing it.

Where it feels like we have a gap is between A & B. Within an application
we can build proprietary interfaces but once you're at a machine or network
level I don't think there's a SOCKS-level equivalent for QOS so it mostly
comes down to inferring which isn't great.

"Prioritization" might be a bit overloaded when we move beyond simple
request/response streams. Game data, RTC, video streaming have different
characteristics where latency/jitter can be critical up to a minimum
threshold but you might not want to give the stream highest priority for it
to do whatever it wants with (a 720p RTC video stream while allowing a
parallel 4K video stream and software download is more useful than a 4K RTC
video stream with the video rebuffering for example). I don't know if that
level of specificity would help more than basic prioritization schemes
though (except starvation is more of a problem in the tunneld case).

To me, it feels like where we are today is roughly:
- make sure MASQUE allows for extensions for someone who controls A, B & C
to experiment with prioritization
- make sure it is interoperable with a C endpoint that has not implemented
QOS/Prioritization
- wait for experience and field data from experiments with MASQUE
QOS/prioritization before standardizing something

On Wed, Jun 23, 2021 at 7:05 AM Lucas Pardue <[email-masked]>
wrote:

> One approach to allow general purpose indiscriminate resource sharing
> would be to design a virtualization layer that sits just below actual
> mappings. Such a thing would fool upper layers into thinking they had
> access to native QUIC. The layer of indirection would probably require a
> "hypervisor" to manage things properly. MASQUE is vaguely an instantion of
> this design that relies on HTTP. WebTransport is vaguely another example,
> we used to have a QuicTransport protocol but pivoted away from it back to
> HTTP.
>

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 23 Jun 2021 16:34:05 +0200

> On 23 Jun 2021, at 15.10, Patrick Meenan <[email-masked]> wrote:
> 
> Using multiple connections should be strictly worse than streams sharing a connection, otherwise we probably have a gap that needs to be filled. Multiple connections kicks the can down to the congestion control algorithms for each of the separate connections to play nicely with each other and hopefully result in a decent experience. With a single connection we should have better knowledge over the overall transport and make better decisions.

For what it’s worth I me mentioned this as a potential downside when the protocol was drafted but consensus were, as I recall, that the protocol shouldn’t be designed around overlapping protocols, but that any specific application protocols were free to document how they would interoperate.

Giving up on datagram IDs simplifies a lot and provides application protocols with more freedom and less overhead than forcing a specific ID system. As Lucas pointed out, even if you have IDs there it is not trivial for multiple application protocols to co-exist. For example, an application protocol might  use all even ID’s for game audio, and all uneven IDs for active game state and controller data. Then someone decides to use that protocol to run communication with a remote controlled wheel chair protocol. But that protocol assigns even ID’s to left wheel speed, and uneven ID’s to right wheel speed. And so on and so forth.

Instead you would design an app interop header for all datagrams. For example if the high bit of the first byte in a datagram is set, the first byte codes for the sub app protocol, and the following data is then sub app specific. If the high bit is not set, the 7 following bits is a sequence number modulo 128. Just as an example. Then app protocols can refer to this meta protocol as a means to coordinate.

As Lucas also pointed out, stream IDs also do not work with multiple sub app protocols, at least not in all cases due to implicit context assigned to certain stream IDs. Note that stream IDs are not necessarily handed out in arbitrary order by the transport layer. This might be the default, but app protocols can require more precise control from the transport layer, which of course limits the number of usable stacks for that application.

Mikkel

---

From: Roberto Peon <[email-masked]>
Date: Wed, 23 Jun 2021 16:15:41 +0000

Protocol mechanism-wise, Mux/Demux requires framing.
A normalized framing scheme leads to normalized and/or interoperable mux/demux.

Similarly, routing and prioritizing both require an address space, and having it be normalized leads to interop.

The above is probably completely obvious, but mentioning it again just in case..
This is all that is required on the protocol side to do whatever routing/muxing/prioritization.

The problem, as it was with HTTP2, is that what the protocol allows won't matter if the APIs available to applications don't expose it.
Please don’t learn the wrong lesson w.r.t. the HTTP2 priority stuff—the complexity is and was relevant, yes, but we should be far more worried about the API side.

-=R

From: QUIC <[email-masked]> on behalf of Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wednesday, June 23, 2021 at 7:34 AM
To: Patrick Meenan <[email-masked]>
Cc: MASQUE <[email-masked]>, Samuel Hurst <[email-masked]>, Spencer Dawkins at IETF <[email-masked]>, HTTP Working Group <[email-masked]>, Lucas Pardue <[email-masked]>, David Schinazi <[email-masked]>, QUIC WG <[email-masked]>, Kazuho Oku <[email-masked]>
Subject: Re: Prioritizing QUIC DATAGRAMs (was: Re: [Masque] Prioritizing HTTP DATAGRAMs)

On 23 Jun 2021, at 15.10, Patrick Meenan <[email-masked]<mailto:[email-masked]>> wrote:

Mikkel

---

From: Martin Duke <[email-masked]>
Date: Wed, 23 Jun 2021 09:43:56 -0700

Lucas,

15 months ago, I brought up DATAGRAM priorities as a reason to put the
stream ID in QUIC Datagrams:
https://github.com/quicwg/datagram/issues/6. I was happy with where this
ended up: that QUIC should present a stream-based datagram API to
applications for prioritization purposes, but that there was no reason for
the stream ID to go out over the wire.

Alternatively, this could be done on a per-datagram basis.

Granted, this is not currently in the corresponding PR:
https://github.com/quicwg/datagram/pull/20/files
I'll review accordingly.

ISTM that in H3 use cases the datagram is associated with a QUIC stream,
and it would be straightforward to use httpbis-priority to negotiate a
priority that then applies to datagrams associated with that stream.

When using QUIC over CONNECT-UDP you have streams on top of streams, and H3
priorities are negotiated both with the origin server and the proxy at
different levels of hierarchy.

Does this answer your question? Or have I totally missed the point?

Martin

On Wed, Jun 23, 2021 at 7:34 AM Mikkel Fahnøe Jørgensen <[email-masked]>
wrote:

> For what it’s worth I me mentioned this as a potential downside when the
> protocol was drafted but consensus were, as I recall, that the protocol
> shouldn’t be designed around overlapping protocols, but that any specific
> application protocols were free to document how they would interoperate.
>
> Giving up on datagram IDs simplifies a lot and provides application
> protocols with more freedom and less overhead than forcing a specific ID
> system. As Lucas pointed out, even if you have IDs there it is not trivial
> for multiple application protocols to co-exist. For example, an application
> protocol might  use all even ID’s for game audio, and all uneven IDs for
> active game state and controller data. Then someone decides to use that
> protocol to run communication with a remote controlled wheel chair
> protocol. But that protocol assigns even ID’s to left wheel speed, and
> uneven ID’s to right wheel speed. And so on and so forth.
>
> Instead you would design an app interop header for all datagrams. For
> example if the high bit of the first byte in a datagram is set, the first
> byte codes for the sub app protocol, and the following data is then sub app
> specific. If the high bit is not set, the 7 following bits is a sequence
> number modulo 128. Just as an example. Then app protocols can refer to this
> meta protocol as a means to coordinate.
>
> As Lucas also pointed out, stream IDs also do not work with multiple sub
> app protocols, at least not in all cases due to implicit context assigned
> to certain stream IDs. Note that stream IDs are not necessarily handed out
> in arbitrary order by the transport layer. This might be the default, but
> app protocols can require more precise control from the transport layer,
> which of course limits the number of usable stacks for that application.
>
> Mikkel
> --
> Masque mailing list
> [email-masked]

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 23 Jun 2021 20:52:43 +0100

Hi Martin,

> https://github.com/quicwg/datagram/pull/20/files

Thanks for the reminder.

> priority that then applies to datagrams associated with that stream.

It's easy to say this. The difficulty comes, as an implementer, with
knowing what to actually do with the information. Concrete example, if a
client signals "incremental false" does a server send all streams as FIFO
and then all DATAGRAMS as FIFO, or does it look at DATAGRAM flow creation
order (expect the flow IDs are not strictly ordered as Mikkel points out).
There's a lot of unknowns here and I don't think it is the best use of
folks time on these 2 drafts spinning tires trying to square that circle.

HTTP/3 is shipping without any priorities and the world isn't crumbling.
Prioritization signals are just a fraction of the information that server
implementers take as input.

Servers are going to have to make similar scheduling choices for DATAGRAMs.
That will ultimately depend on how they are being used. IMO neither the
priorities draft nor the HTTP/3 DATAGRAM draft are good venues to consider
this.

> When using QUIC over CONNECT-UDP you have streams on top of streams, and
> H3 priorities are negotiated both with the origin server and the proxy at

Not sure. The question is really, are folks happy to punt detailed
discussion about DATAGRAM multiplexing scheduling and prioritization
signalling to some new I-D. Are you?

Cheers,
Lucas

---

From: Martin Duke <[email-masked]>
Date: Wed, 23 Jun 2021 13:13:31 -0700

Lucas,

Assuming that we have the necessary language about QUIC APIs, as I've just
proposed, to enable something in future, then I personally have no use
cases or strong objection to punting. It is somewhat annoying to have a
draft for priorities, datagrams, and then priorities-with-datagrams, but if
writing the last bit will be a struggle (which I have no informed opinion
on) deferring it is probably the right decision.

order

No, I think the DATAGRAMs correspond to a stream and they go with the
STREAM frames in with priority. Whether one delivers STREAM or DATAGRAM
first within a particular stream maybe doesn't need tight specification; if
we do, I'd say STREAM (if for no other reason than to deliver the headers)
and then DATAGRAM, unless STREAM is being flow-controlled. As datagram
flows don't have a 1-to-1 mapping with requests and responses, I think
streams are the correct abstraction for datagram priority, not flows. If
there are multiple Flow-IDs associated with the same stream, they all get
grouped together for priority purposes.

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 23 Jun 2021 21:24:46 +0100

Hey Martin,

> grouped together for priority purposes.

I think this model breaks down entirely for something like WebTransport
where the session is created with a single stream and then there'll
potentially be multiple datagram flows in either direction all vying for a
slice of the pie.

Cheers,
Lucas

---

From: Martin Duke <[email-masked]>
Date: Wed, 23 Jun 2021 14:01:16 -0700

This is an argument against punting, I think. If there are additional
semantics required to specify priorities on a sub-stream basis, it will be
somewhat more painful to have this be an extension than something native.

Which is not to say we shouldn't punt.

On Wed, Jun 23, 2021, 13:25 Lucas Pardue <[email-masked]> wrote:

>> Assuming that we have the necessary language about QUIC APIs, as I've
>> just proposed, to enable something in future, then I personally have no use

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 23 Jun 2021 22:47:09 +0100

Hey Martin,

> Which is not to say we shouldn't punt.

I don't follow this argument, sorry. Extensibile priorities is not base to
HTTP/3, it could be substituted by something entirely different (although I
kind of hope not). HTTP/3 DATAGRAM is an extension, built on top of the
QUIC transport DATAGRAM extension. WebTransport is a new thing built on top
of these extensions. What does native mean in your definition? I really
think it's better that things wishing to use Extensible priorities take on
board the responsibility of defining that, using the extension mechanisms
that it provides. Otherwise, we're bottlenecking the progress that people
can make for the initial use cases that these things were built towards.

Cheers,
Lucas

---

From: Martin Duke <[email-masked]>
Date: Wed, 23 Jun 2021 15:06:37 -0700

Lucas,

Sorry, I was too brief.

In my straw man where priorities always had stream scope, it would require
h3 datagram (an extension to an extension) and priority header (an
independent extension). That’s a lot of failure cases to design for.

If we want priority granularity below the stream, then it’s an extension to
the priority extension — yet another failure case. This is by no means
unsurmountable but does make it harder to code, specify, and reason about.

Whereas if anyone who implements h3 priorities correctly inherently
understands “flow priorities” or whatever, because it’s in the same draft
or normatively connected, that’s less complexity.

This is all rather abstract for an argument I’m not really making….