Subject: [Masque] Can H3 DATAGRAMs be more self-describing?

From: Alan Frindell <[email-masked]>
Date: Wed, 10 Mar 2021 01:33:37 +0000

My observation about DATAGRAM Flow-ID’s is that they are meant to encode multiple pieces of datagram metadata including:

1. The type or format of the datagram payload
2. The session/flow or “thing” this datagram belongs to
3. Other negotiated options (a form of compression)

Contrast this with HTTP/3 streams, which carry all of this information *on the wire*.

Consider an HTTP/3 implementation that supports many different extensions that each use different kinds of H3 DATAGRAMs.  The way it has to work is with the “flow ID allocation service” allocating Flow IDs and a corresponding map to use when sending and receiving datagrams.

But if the datagrams self-described, no such map is necessary.  Instead, you decode the type of the datagram, and a type-specific decoder decodes the specific format of the datagram to discover things like which session/stream/flow this is associated with (if any).  Compression schemes that overload the metadata to encode option bits are still possible on a per-extension basis.

I think self-describing datagrams are more consistent with other design choices we’ve made in HTTP/3.

My concrete proposal is to remove Flow-ID entirely.  Each H3 DATAGRAM begins with a varint Datagram Type.  The remainder of the datagram is type specific.

CONNECT-UDP could build on this in two ways:

  1.  Define a single datagram type, containing the CONNECT-UDP session ID and option fields, followed by payload
  2.  Define a datagram type for each option flavor it wants to support, containing the session ID followed by payload

Assuming CONNECT-UDP gets a low numbered datagram type, this scheme increases the per datagram overhead by one byte.  If this is really a dealbreaker, we could reduce the number of valid bits of a datagram type, and let extensions use the remaining bits to encode their session IDs in that field.

-Alan

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 9 Mar 2021 20:48:16 -0500

A type header sounds reasonable enough to me, but I'm surprised at the idea
of delegating the ownership question to the type-specific decoder.  Do you
have a use case for datagram frames that aren't bound to a particular
stream?  If not, it would seem more natural to make the datagram-stream
binding universal.

On Tue, Mar 9, 2021 at 8:33 PM Alan Frindell <afrind=[email-masked]>
wrote:

> My observation about DATAGRAM Flow-ID’s is that they are meant to encode
> multiple pieces of datagram metadata including:
>
> Consider an HTTP/3 implementation that supports many different extensions
> that each use different kinds of H3 DATAGRAMs.  The way it has to work is
> with the “flow ID allocation service” allocating Flow IDs and a
> corresponding map to use when sending and receiving datagrams.
>
>
>
> But if the datagrams self-described, no such map is necessary.  Instead,
> you decode the type of the datagram, and a type-specific decoder decodes
> the specific format of the datagram to discover things like which
> session/stream/flow this is associated with (if any).  Compression schemes
> that overload the metadata to encode option bits are still possible on a
> per-extension basis.
>
>
>
> My concrete proposal is to remove Flow-ID entirely.  Each H3 DATAGRAM
> begins with a varint Datagram Type.  The remainder of the datagram is type
> specific.
>
>
>
>    1. Define a single datagram type, containing the CONNECT-UDP session
>    ID and option fields, followed by payload
>    2. Define a datagram type for each option flavor it wants to support,
>    containing the session ID followed by payload
>
>
>
> Assuming CONNECT-UDP gets a low numbered datagram type, this scheme
> increases the per datagram overhead by one byte.  If this is really a
> dealbreaker, we could reduce the number of valid bits of a datagram type,
> and let extensions use the remaining bits to encode their session IDs in
> that field.
>
>
>
> -Alan
> --
> Masque mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/masque
>

---

From: Alan Frindell <[email-masked]>
Date: Wed, 10 Mar 2021 02:08:34 +0000

> A type header sounds reasonable enough to me, but I'm surprised at the idea of delegating the ownership question to the type-specific decoder.  Do you have a use case for datagram frames that aren't bound to a particular stream?  If not, it would seem more natural to make the datagram-stream binding universal.

I don’t have a specific use case for this but I can imagine an extension wanting to transmit connection-specific rather than stream specific-information in a datagram.  Further, I think different extensions might have different definitions of how they want to specify their session identifiers.

-Alan

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 10 Mar 2021 02:24:01 +0000

> begins with a varint Datagram Type.  The remainder of the datagram is type

One benefit I've noticed with the current design approach is that it makes
it quite easy to support in my H3 library code. I don't need any special
handling of mapping or logic there. I just read a QUIC datagram, scan the
Flow ID and dispatch the datagram to whoever cares about that flow. In this
model a lot of the "business logic" is left to the upper layers that have a
lot more context of what's happening.

The suggested design here sounds like defining a concrete H3 frame that can
travel in DATAGRANS. Encoding more information in the frame means: a
possible lack of general purpose multiplexing, and more smarts needed in my
library. Neither of these things are deal breakers. But if the H3 layer
can't use any of the additional information in the frame, then I'm not sure
why it should be there.

Swapping the order of your session ID and frame type around (session first,
then type), I could delegate the handling of anything after the session ID
to other code if I chose.

But levelling up a bit, if people are truly saying that a general purpose
solution to demultiplexing datagrams in H3 is not required, a key premise
of this spec is invalid. That seems unfortunate at this stage.

Cheers
Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 10 Mar 2021 02:35:14 +0000

On Wed, 10 Mar 2021, 02:08 Alan Frindell, <afrind=[email-masked]>
wrote:

>  I don’t have a specific use case for this but I can imagine an extension
> wanting to transmit connection-specific rather than stream
> specific-information in a datagram.
>
What HTTP semantic allows connection-level application data?

Further, I think different extensions might have different definitions of
> how they want to specify their session identifiers.
>
They could use something other than H3_DRAM. A need to mix both types would
be difficult though. YAGNI?

Cheers
Lucas

---

From: Alan Frindell <[email-masked]>
Date: Wed, 10 Mar 2021 03:43:13 +0000

> But levelling up a bit, if people are truly saying that a general purpose solution to demultiplexing datagrams in H3 is not required, a key premise of this spec is invalid. That seems unfortunate at this stage.

I didn’t mean to break general purpose demultiplexing, but I seem to have. Since both Lucas and Ben indicated that every H3 datagram has to be associated with some stream, then why not make the datagram begin with a stream ID?

Flow-ID as a meta-concept binding streams, types and options is really what bothers me about the draft.  I think the use cases for having multiple Flow-IDs map to the same stream can be handled by adding a type, or with datagram specific fields.  Is there a compelling use case for a single Flow-ID mapping to multiple streams?  If not, we’re left with a 1:1 mapping between streams and sessions, and no need for an additional ID space.

-Alan

---

From: Mike Bishop <[email-masked]>
Date: Thu, 11 Mar 2021 15:07:10 +0000

We had hypothetical discussions about sending certain frames from the control stream as DATAGRAMs.  Not HTTP semantics, but mapping-layer data.

Obviously, that could be done with something other than the H3_Datagram draft, but then you wade into the negotiation swamp if the client supports both.

From: Masque <[email-masked]> On Behalf Of Lucas Pardue
Sent: Tuesday, March 9, 2021 9:35 PM
To: Alan Frindell <afrind=[email-masked]>
Cc: Ben Schwartz <bemasc=[email-masked]>; MASQUE <[email-masked]>
Subject: Re: [Masque] Can H3 DATAGRAMs be more self-describing?

On Wed, 10 Mar 2021, 02:08 Alan Frindell, <afrind=[email-masked]<mailto:[email-masked]>> wrote:
 I don’t have a specific use case for this but I can imagine an extension wanting to transmit connection-specific rather than stream specific-information in a datagram.
What HTTP semantic allows connection-level application data?

Further, I think different extensions might have different definitions of how they want to specify their session identifiers.
They could use something other than H3_DRAM. A need to mix both types would be difficult though. YAGNI?

Cheers
Lucas

---

From: Ben Schwartz <[email-masked]>
Date: Thu, 11 Mar 2021 14:47:54 -0500

I agree with Alan: I would prefer to have Flow IDs be stream IDs, or
otherwise be managed within the transport itself.  Placing stream IDs (a
connection-level construct) in an HTTP header seems to violate an HTTP/3
requirement (
https://quicwg.org/base-drafts/draft-ietf-quic-http.html#section-4.1.1):
"An endpoint MUST NOT generate an HTTP/3 field section containing
connection-specific fields; any message containing connection-specific
fields MUST be treated as malformed".

I don't see a problem with a type byte, but I can't think of a need for
it.  Even control streams have stream IDs.

On Thu, Mar 11, 2021 at 10:07 AM Mike Bishop <[email-masked]> wrote:

> We had hypothetical discussions about sending certain frames from the
> control stream as DATAGRAMs.  Not HTTP semantics, but mapping-layer data.
>
>
>
> Obviously, that could be done with something other than the H3_Datagram
> draft, but then you wade into the negotiation swamp if the client supports
> both.
>
>
>
> *From:* Masque <[email-masked]> *On Behalf Of *Lucas Pardue
> *Sent:* Tuesday, March 9, 2021 9:35 PM
> *To:* Alan Frindell <afrind=[email-masked]>
> *Cc:* Ben Schwartz <bemasc=[email-masked]>; MASQUE <
> [email-masked]>
> *Subject:* Re: [Masque] Can H3 DATAGRAMs be more self-describing?
>
>
>
>
>

---

From: Tommy Pauly <[email-masked]>
Date: Thu, 11 Mar 2021 11:58:34 -0800

There is a way to do this that negotiates the DATAGRAM flow metadata as HTTP/3 and HTTP/2 frames, not headers. I think David is exploring that direction, and I’d support that.

However, as discussed in the meeting, I object to the marker that the HTTP application would use to identify the meaning for various DATAGRAMs being synonymous with a QUIC transport stream ID. The transport has no understanding of this marker, and the meaning of different categories of datagrams belongs to the application.

Tommy

> On Mar 11, 2021, at 11:47 AM, Ben Schwartz <bemasc=[email-masked]> wrote:
> 
> I agree with Alan: I would prefer to have Flow IDs be stream IDs, or otherwise be managed within the transport itself.  Placing stream IDs (a connection-level construct) in an HTTP header seems to violate an HTTP/3 requirement (https://quicwg.org/base-drafts/draft-ietf-quic-http.html#section-4.1.1 <https://quicwg.org/base-drafts/draft-ietf-quic-http.html#section-4.1.1>): "An endpoint MUST NOT generate an HTTP/3 field section containing connection-specific fields; any message containing connection-specific fields MUST be treated as malformed".
> 
> I don't see a problem with a type byte, but I can't think of a need for it.  Even control streams have stream IDs.
> 
> On Thu, Mar 11, 2021 at 10:07 AM Mike Bishop <[email-masked] <mailto:[email-masked]>> wrote:
> From: Masque <[email-masked] <mailto:[email-masked]>> On Behalf Of Lucas Pardue
> To: Alan Frindell <afrind=[email-masked] <mailto:[email-masked]>>
> Cc: Ben Schwartz <bemasc=[email-masked] <mailto:[email-masked]>>; MASQUE <[email-masked] <mailto:[email-masked]>>
> On Wed, 10 Mar 2021, 02:08 Alan Frindell, <afrind=[email-masked] <mailto:[email-masked]>> wrote:
>

---

From: Alex Chernyakhovsky <[email-masked]>
Date: Thu, 11 Mar 2021 14:58:35 -0500

Why is that requirement written that way? "Like HTTP/2, HTTP/3 does not use
the Connection header field to indicate connection-specific fields; in this
protocol, connection-specific metadata is conveyed by other means" is the
sentence preceding it, which implies it's just a layering concern. I do
think that the existing QUIC language around treating a message with
connection-specific fields as malformed is too strong and an unnecessary
requirement.

That said, if we want to still have assignable datagrams flow IDs, that
just means we need to negotiate it outside of the HTTP/3 connection and
through some mechanism like settings frames. That doesn't feel like an
improvement to me.

Sincerely,
-Alex

---

From: Ian Swett <[email-masked]>
Date: Thu, 11 Mar 2021 15:32:18 -0500

I've been reading and re-reading the relevant drafts in the last few days,
and I'm unconvinced we need to associate multiple datagram flow IDs with
one stream.  I'll note that this is a separate decision than whether the
stream ID and the flow ID are identical and I think conflating those two
decisions is confusing matters.

One can achieve the same functionality with a single datagram flow per
stream as multiple, so the question to me becomes "Does using multiple
datagram flows per stream create better designs"?

I'll keep reading and thinking about this in the next week, but I haven't
seen strong evidence yet.

On Tue, Mar 9, 2021 at 10:43 PM Alan Frindell <afrind=
[email-masked]> wrote:

> > But levelling up a bit, if people are truly saying that a general
> purpose solution to demultiplexing datagrams in H3 is not required, a key
> premise of this spec is invalid. That seems unfortunate at this stage.
>
>
>
> I didn’t mean to break general purpose demultiplexing, but I seem to have.
> Since both Lucas and Ben indicated that every H3 datagram has to be
> associated with some stream, then why not make the datagram begin with a
> stream ID?
>

One complexity that arises from allowing multiple Flow IDs per stream that
I don't remember being discussed at the meeting is the header format.

>
> Flow-ID as a meta-concept binding streams, types and options is really
> what bothers me about the draft.  I think the use cases for having multiple
> Flow-IDs map to the same stream can be handled by adding a type, or with
> datagram specific fields.  Is there a compelling use case for a single
> Flow-ID mapping to multiple streams?  If not, we’re left with a 1:1 mapping
> between streams and sessions, and no need for an additional ID space.
>

I filed an issue(#33
<https://github.com/ietf-wg-masque/draft-ietf-masque-h3-datagram/issues/33>)
about multiple streams associated with a single Flow-ID, since I think it'd
be a challenge for most proxies to support.

---

From: Victor Vasiliev <[email-masked]>
Date: Fri, 12 Mar 2021 18:34:41 -0500

I agree with the header complexity concern here.  We don't have a fully
featured structured header parser in our library, and I am not
convinced the 20 lines of parsing code that we currently use is actually
compliant with every single requirement of RFC 8941.  Rewriting flow IDs on
proxying would probably require a fully featured serializer too.

If we only allow one flow ID per stream, I think that would still let the
CONNECT-QUIC draft associate multiple streams with a single flow, and make
rewriting flow IDs much easier.

On Thu, Mar 11, 2021 at 3:32 PM Ian Swett <ianswett=
[email-masked]> wrote:

>> I didn’t mean to break general purpose demultiplexing, but I seem to
>> have. Since both Lucas and Ben indicated that every H3 datagram has to be

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 12 Mar 2021 23:50:33 +0000

There's a lot of good thoughts in this thread. I might buy that the
multiplicity afforded by the document currently is too complex, I need to
think on that more and form a response.

But to respond to Victor's point:

> proxying would probably require a fully featured serializer too.

Sorry, I don't buy that. The IETF should be strongly encouraging new
headers to be defined using Structured Fields so that implementations just
leverage existing parsers or serializers. Before the H3-Datagram-Flow
header field became an sf-list, it was an sf-integer. Implementations
should read and write the field as a Structured Header and not hand roll
things (however tempting it might be).

Cheers,
Lucas

---

From: Victor Vasiliev <[email-masked]>
Date: Fri, 12 Mar 2021 19:29:04 -0500

sf-integer is easier to parse, store and serialize than sf-list.

Regardless, I think all of this just convinces me that headers are a wrong
layer for this (especially given its per-connection nature), and we should
add a new frame type instead.

---

From: Luke Curley <[email-masked]>
Date: Fri, 12 Mar 2021 16:39:00 -0800

Yeah, I agree that the current mechanism is far too complex.

I don't think there needs to be explicit support for multiple flows per
CONNECT-UDP session. Let the application encode the flow ID at the start of
the datagram if needed. I respect the desire to save a byte in these
situations, but as others have pointed out there are issues with the header
approach.

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 13 Mar 2021 00:56:24 +0000

> situations, but as others have pointed out there are issues with the header

I'm keen to see what a proposal for a Frame-based approach looks like. I
hope it addresses the concerns that people have but I'll say that based on
my experiences, decoupling HTTP-version specific features from HTTP
messages introduces a whole layer of other complexity. Secondary certs [1]
springs to mind.

To give an example in frames-based world, for a server to validate a
CONNECT-UDP request, you're going to require it to depend on a additional
HTTP/3 frames that come after HEADERS and, probably in an implementation
specific way, require coupling the datagram flow to the HTTP request. So
some of the present problems are just shifted from the spec into
implementations. This isn't awful, it is probably cleaner from a certain
design perspective, but the alternative design doesn't come for free.

---

From: Luke Curley <[email-masked]>
Date: Fri, 12 Mar 2021 18:32:56 -0800

Maybe a practical example would aid the discussion.

Traditionally, a RTP <https://tools.ietf.org/html/rfc3550> "connection"
consists of two ports: one for RTP (media packets) and one for RTCP
(control packets). We need a way to distinguish between these two "flows"
In order to tunnel RTP traffic over MASQUE. This example might actually be
extremely relevant in the future if WebRTC opts to migrate from SRTP to
QUIC.

*Datagram-Flow-Id*
The proposed approach is to send a CONNECT-UDP request with a
Datagram-Flow-Id header with two flow IDs labeled for RTP and RTCP.

If we extend the presented ECN example to RTP, this header would also
include a list of all capabilities (ex. supported RTP extensions, codecs?).
I definitely think that would be a mistake. The application can use another
header if it wants to negotiate functionality.

A simple mapping of labels to flow IDs would be better: rtp=0,rtcp=4.
This does mean that the flows are static which is fine for RTP, but may not
work for every application. And like others have mentioned, this goes
against the HTTP/3 specification because it specifies connection-level
functionality at the request-level.

*WebTransport*
The WebTransport approach is to use the stream ID of the CONNECT request as
the datagram flow ID. Effectively there is a single flow per CONNECT.

RTP and RTCP packets can be differentiated using RFC5761
<https://tools.ietf.org/html/rfc5761> so multiple flows weren't even
necessary. But supposing that wasn't possible, then I would indicate the
packet type with the first byte of the datagram payload. In the RTP
example, the first byte is the version (currently 2), so we could even
overload that to avoid wasting a byte per datagram.

Another minor benefit of stream ID == flow ID is that the server can
associate incoming datagrams with a pending request. If I receive a
datagram with an unknown flow id 32, then I could buffer it and count it
towards any limits for the anticipated CONNECT request on stream 32.

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 13 Mar 2021 03:11:21 +0000

> functionality at the request-level.

That's an interesting example. If you could only establish one datagram
flow per HTTP request, then would you need two requests to establish a
meaningful RTP application context? This means the fate of flows cannot be
easily shared because there is no way to express their relationship.

> *WebTransport*
> The WebTransport approach is to use the stream ID of the CONNECT request
> as the datagram flow ID. Effectively there is a single flow per CONNECT.
>

How prevalent is RTP and RTCP port sharing? I doubt HTTP/3 datagram is
going to influence how people already deploy RTP.

> towards any limits for the anticipated CONNECT request on stream 32.

I don't see how there is much difference to the current design. The flow ID
is just a number. Whatever way it is chosen, there is always a chance that
the DATAGRAM frame arrives before the STREAM frame. Whether the ID is the
same value as the stream ID, or a value sent on the stream (in the header)
seems of little importance. An H3 DATAGRAM that is not bound to any
application semantic cannot be handled.

The one difference that using stream IDs might make when it comes to
linking flow lifetime to stream limits and lifetime. I want to stress that
sharing the same value as a stream ID does not mean H3 DATAGRAMS are part
of the stream, they are operating at completely different layers with
different restrictions. For example, this type of design has to decide
whether receiving an H3 DATAGRAM with a flow ID greater than MAX_STREAMS is
a connection error or not. The wrinkle is that the flow ID is not a
property of the transport, and stream IDs are owned by the transport which
might not exposed them freely.

Cheers,
Lucas

---

From: Tommy Pauly <[email-masked]>
Date: Sat, 13 Mar 2021 10:33:05 -0800

> On Mar 12, 2021, at 7:11 PM, Lucas Pardue <[email-masked]> wrote:
> 
> 
> 
> On Sat, Mar 13, 2021 at 2:33 AM Luke Curley <[email-masked] <mailto:[email-masked]>> wrote:
> Traditionally, a RTP <https://tools.ietf.org/html/rfc3550> "connection" consists of two ports: one for RTP (media packets) and one for RTCP (control packets). We need a way to distinguish between these two "flows" In order to tunnel RTP traffic over MASQUE. This example might actually be extremely relevant in the future if WebRTC opts to migrate from SRTP to QUIC.
> 
> Datagram-Flow-Id
> The proposed approach is to send a CONNECT-UDP request with a Datagram-Flow-Id header with two flow IDs labeled for RTP and RTCP. 
> 
> If we extend the presented ECN example to RTP, this header would also include a list of all capabilities (ex. supported RTP extensions, codecs?). I definitely think that would be a mistake. The application can use another header if it wants to negotiate functionality.
> 
> A simple mapping of labels to flow IDs would be better: rtp=0,rtcp=4. This does mean that the flows are static which is fine for RTP, but may not work for every application. And like others have mentioned, this goes against the HTTP/3 specification because it specifies connection-level functionality at the request-level.
> 
> That's an interesting example. If you could only establish one datagram flow per HTTP request, then would you need two requests to establish a meaningful RTP application context? This means the fate of flows cannot be easily shared because there is no way to express their relationship.
> 
> 
> WebTransport
> The WebTransport approach is to use the stream ID of the CONNECT request as the datagram flow ID. Effectively there is a single flow per CONNECT.
> 
> RTP and RTCP packets can be differentiated using RFC5761 <https://tools.ietf.org/html/rfc5761> so multiple flows weren't even necessary. But supposing that wasn't possible, then I would indicate the packet type with the first byte of the datagram payload. In the RTP example, the first byte is the version (currently 2), so we could even overload that to avoid wasting a byte per datagram.
> 
> How prevalent is RTP and RTCP port sharing? I doubt HTTP/3 datagram is going to influence how people already deploy RTP.
> 
> 
> Another minor benefit of stream ID == flow ID is that the server can associate incoming datagrams with a pending request. If I receive a datagram with an unknown flow id 32, then I could buffer it and count it towards any limits for the anticipated CONNECT request on stream 32.
> 
> I don't see how there is much difference to the current design. The flow ID is just a number. Whatever way it is chosen, there is always a chance that the DATAGRAM frame arrives before the STREAM frame. Whether the ID is the same value as the stream ID, or a value sent on the stream (in the header) seems of little importance. An H3 DATAGRAM that is not bound to any application semantic cannot be handled. 

+1 to this. Allowing the values to be 1-to-1 for a given application use case is always possible. That will certainly be a common case. That doesn’t mean that the actual integer on the wire must be the same as the encoding that the transport layer uses for a stream that happened to carry the request that established this flow ID.

> 
> The one difference that using stream IDs might make when it comes to linking flow lifetime to stream limits and lifetime. I want to stress that sharing the same value as a stream ID does not mean H3 DATAGRAMS are part of the stream, they are operating at completely different layers with different restrictions. For example, this type of design has to decide whether receiving an H3 DATAGRAM with a flow ID greater than MAX_STREAMS is a connection error or not. The wrinkle is that the flow ID is not a property of the transport, and stream IDs are owned by the transport which might not exposed them freely.

Exactly. We lose nothing by allowing flow IDs to be assigned and defined by the layer to which they below (the HTTP application layer)—you can always make them 1-to-1 with streams. However, trying to force there to be a dependency between the layers adds requirements on implementations to unnecessarily coordinate across layers, reduces the ability for the protocol to be extended.

Best,
Tommy

> 
> Cheers,
> Lucas
> -- 
> Masque mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/masque <https://www.ietf.org/mailman/listinfo/masque>