Subject: [Masque] Why aren't capsules frames?

From: "Martin Thomson" <[email-masked]>
Date: Thu, 15 Jul 2021 11:30:47 +1000

Looking at the design in draft-ietf-masque-h3-datagram-03 I don't see why each capsule type couldn't be registered as a frame type.  This would save an additional layer of framing, which is more efficient.  Given the application to tunneling, that efficiency could really mean something.

Separately, the extensibility scheme is not justified, purely on complexity grounds alone.  The things that are included could be mandatory fields of frames/capsules. Additional extensions could use additional frame types, the main cost there being an imposition on HTTP-layer routing decisions.

---

From: David Schinazi <[email-masked]>
Date: Wed, 14 Jul 2021 18:36:41 -0700

Hi Martin,

The motivation for creating a CAPSULE frame is extensibility in the
presence of
intermediaries. It allows you to define new capsule types and only require
updating
the HTTP client and server without having to modify any HTTP intermediary.
As
we've seen many times, protocols that require updating more than just the
endpoints
are much less likely to succeed. More practically, in a modern cloud
deployment
context, it's common for application developers to control the stack on the
client and
on the backend without being in control of the frontend. If the frontend
adds HTTP
Datagram support for the sake of WebTransport, now all backend customers can
start deploying other MASQUE protocols. Switching to a design where each
capsule
is its own HTTP/3 frame would prevent that.

David

On Wed, Jul 14, 2021 at 6:31 PM Martin Thomson <[email-masked]> wrote:

> Looking at the design in draft-ietf-masque-h3-datagram-03 I don't see why
> each capsule type couldn't be registered as a frame type.  This would save
> an additional layer of framing, which is more efficient.  Given the
> application to tunneling, that efficiency could really mean something.
>
> Separately, the extensibility scheme is not justified, purely on
> complexity grounds alone.  The things that are included could be mandatory
> fields of frames/capsules. Additional extensions could use additional frame
> types, the main cost there being an imposition on HTTP-layer routing
> decisions.
>
> --
> Masque mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/masque
>

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 16 Jul 2021 15:02:36 +1000

Hi David,

Thanks for your response. 

I'm not seeing how this middlebox requirement justifies the design.

An intermediary doesn't even need to understand any of your capsules in order to pass datagrams, because those are sent with stream identifiers.  That stream identifier is sufficient for passing any newly defined stuff to the right backend.

An intermediary doesn't even really need datagram contexts in light of this design, which makes me inclined to repeat the same suggestion I made for DATAGRAM: drop that field too.  The MASQUE encapsulation might benefit from a field there, but that might be advanced usage and not something an intermediary unaware of MASQUE would be able to do anything with.

Cheers,
Martin

On Thu, Jul 15, 2021, at 11:36, David Schinazi wrote:
> intermediaries. It allows you to define new capsule types and only 
> require updating
> the HTTP client and server without having to modify any HTTP 
> intermediary. As
> we've seen many times, protocols that require updating more than just 
> the endpoints
> context, it's common for application developers to control the stack on 
> the client and
> on the backend without being in control of the frontend. If the 
> frontend adds HTTP
> start deploying other MASQUE protocols. Switching to a design where 
> each capsule

---

From: Ben Schwartz <[email-masked]>
Date: Fri, 16 Jul 2021 11:10:05 -0400

> I'm not seeing how this middlebox requirement justifies the design.

To see the need for CAPSULE, I would recommend looking at
https://datatracker.ietf.org/doc/html/draft-cms-masque-connect-ip-01#section-6.
In short, CONNECT-IP needs a lot of end-to-end, reliable RPC stuff.  (This
is in addition to datagram context creation in CONNECT-UDP and CONNECT-IP,
which is similar E2E RPC.)  We don't want a new frame type for every
CONNECT-IP RPC message, and every future thing along those lines.

We could do all this RPC in the request/response body instead (i.e. in DATA
frames), but that would create nasty buffering and interleaving issues for
H3->H2 conversion intermediaries.

An intermediary doesn't even need to understand any of your capsules in
> order to pass datagrams, because those are sent with stream identifiers.
> That stream identifier is sufficient for passing any newly defined stuff to
> the right backend.
>

Yes.  That's why the capsule types aren't frame types: they're of no use to
intermediaries.

An intermediary doesn't even really need datagram contexts in light of this
> design, which makes me inclined to repeat the same suggestion I made for
> DATAGRAM: drop that field too.  The MASQUE encapsulation might benefit from
> a field there, but that might be advanced usage and not something an
> intermediary unaware of MASQUE would be able to do anything with.

Yes, datagram contexts are not for use by intermediaries.  To an
intermediary, they're just part of the opaque DATAGRAM payload.  Perhaps we
can adjust the documentation in some way to emphasize that.

---

From: David Schinazi <[email-masked]>
Date: Fri, 16 Jul 2021 09:36:27 -0700

> That stream identifier is sufficient for passing any newly defined stuff to

You're correct that the intermediary doesn't need to understand capsules,
but it needs to forward capsules for things to work. Since HTTP/3 frames
are defined as hop-by-hop constructs, not end-to-end (in the HTTP sense
where the ends are user agent and backend), an intermediary won't (and
shouldn't) blindly copy frames it receives on one connection to the
other, because some new extension frames might not want this property.
Therefore we need something that the intermediary will blindly forward, and
that's exactly what CAPSULE is.

> intermediary unaware of MASQUE would be able to do anything with.

You're right that the intermediary doesn't need to know about contexts.
That's why this field is opaque to intermediaries. We shouldn't drop that
field altogether though, because it's useful end-to-end. There's an
appendix with examples in the HTTP Datagram draft which hopefully shows why
this is a useful feature.

David

Cheers,
> > > Looking at the design in draft-ietf-masque-h3-datagram-03 I don't see
> why each capsule type couldn't be registered as a frame type.  This would
> save an additional layer of framing, which is more efficient.  Given the

---

From: Mike Bishop <[email-masked]>
Date: Fri, 23 Jul 2021 13:44:46 +0000

I was just writing up my feedback and had the same initial reaction on both fronts.  CAPSULE frames seem largely duplicative of HTTP/2 and HTTP/3 frame types at first glance.

However, they differ in one essential way that perhaps makes them worth adding:  Unlike HTTP/[2..] frames, which are hop-by-hop, CAPSULE frames are explicitly end-to-end.  While new HTTP/3 frames could be defined with a requirement that they be relayed unmodified by intermediaries, that requirement would have to be levied (and implemented) separately for each future frame type that needs it.  The benefit of CAPSULE seems to be that it can be implemented once and work with future end-to-end extensions without new dev work on intermediaries.

In that light, it almost seems like it belongs as a separate extension to HTTP/3 which is used by H3 datagrams, similar to the QUIC DATAGRAM frame.  This is a utility that could prove useful beyond the definition of datagrams.  (Though I'm also sympathetic to not wanting to further subdivide this work.)

As to the extensibility, I think there are ways to simplify it, but the intended model is still slightly unclear to me.  The REGISTER_DATAGRAM_CONTEXT/_NOCONTEXT and CLOSE_DATAGRAM_CONTEXT frames capsules contain an ID and an array of extensions, which are TLV objects.  Extensions seem to serve two different purposes - they both identify the protocol being used within the context, and they carry configuration about that context.  However, I'm not totally clear on that, so perhaps I'm misinterpreting.  There's a registry of extensions, and certain extensions are required in or even exclusive to certain capsule types.  This structure mystifies me a bit.

*                 If an extension is required to identify how to parse the incoming datagram, it makes sense to have a registry of those to ensure there's not a collision.  It does not make sense to allow more than one of those to be declared on a context.  Consistent with the principle that it should be impossible to encode something illegal, why not have a single Type field in the REGISTER_* capsules, and the type points into a registry?

*                 Once you know the protocol being spoken within a particular datagram context, any configuration that protocol uses doesn't need a shared registry with other protocols.  The TLV array of configuration - or perhaps merely a blob that fills the remainder of the capsule, and the protocol can define the contents to be a TLV array if it wants - can have an interpretation entirely local to the protocol being spoken.

*                 CLOSE_CODE is both exclusive to and required in the CLOSE_DATAGRAM_CONTEXT capsule.  Why is it not simply a field of that capsule?

*                 The same could be said for DETAILS, though its usefulness might extend beyond CLOSE_DC.

-----Original Message-----
From: Masque <[email-masked]> On Behalf Of Martin Thomson
Sent: Wednesday, July 14, 2021 9:31 PM
To: [email-masked]
Subject: [Masque] Why aren't capsules frames?

--

Masque mailing list

[email-masked]<mailto:[email-masked]>

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 26 Jul 2021 10:40:22 +1000

On Fri, Jul 23, 2021, at 23:44, Mike Bishop wrote:
> However, they differ in one essential way that perhaps makes them worth 
> adding:  Unlike HTTP/[2..] frames, which are hop-by-hop, CAPSULE frames 
> are explicitly end-to-end.  

Yeah, I might have said this elsewhere already (I have in webtrans for sure), but I don't think that a new mechanism is a good idea.  An earlier design was better because it used an existing end-to-end construct: the message contents.  Intermediaries don't need to understand how to forward those.

---

From: David Schinazi <[email-masked]>
Date: Sun, 25 Jul 2021 18:00:32 -0700

Hi Martin, I'm still not completely sure what you're proposing: are you
suggesting we keep the concept of capsules, but send them on the DATA
stream (as a sequence of TLVs) instead of in a new frame type? And then
CONNECT-UDP and WebTrans would say "my DATA stream is a stream of capsules,
see draft-ietf-masque-h3-datagram for details"? In other words revert to
Stream Chunks. Or are you proposing something different?

Thanks,
David

On Sun, Jul 25, 2021 at 5:40 PM Martin Thomson <[email-masked]> wrote:

> Yeah, I might have said this elsewhere already (I have in webtrans for
> sure), but I don't think that a new mechanism is a good idea.  An earlier
> design was better because it used an existing end-to-end construct: the
> message contents.  Intermediaries don't need to understand how to forward
> those.
>
> --
> Masque mailing list
> [email-masked]

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 26 Jul 2021 11:08:00 +1000

On Mon, Jul 26, 2021, at 11:00, David Schinazi wrote:
> CONNECT-UDP and WebTrans would say "my DATA stream is a stream of 
> capsules, see draft-ietf-masque-h3-datagram for details"? In other 
> words revert to Stream Chunks. Or are you proposing something different?

Pretty much.

---

From: David Schinazi <[email-masked]>
Date: Sun, 25 Jul 2021 18:22:25 -0700

Ah, thanks for clarifying.

The rationale for a new frame was the fact that some intermediaries need to
parse some capsules, (we refer to those as "transparent to intermediaries"
vs "opaque to intermediaries" in draft-ietf-masque-h3-datagram).

For example: the DATAGRAM capsule needs to be transparent. Let's assume we
have the following topology: <client --- intermediary --- server ---
target> and the client is issuing a CONNECT-UDP request to the server
(asking the server to then forward UDP to the target). Let's assume the
<client --- intermediary> leg uses HTTP/3 with the QUIC datagram extension
enabled, and that the <intermediary --- server> leg uses HTTP/2. If the
client sends a UDP payload in a QUIC DATAGRAM frame to the intermediary,
then the intermediary will need to send that to the server in a stream
chunk over HTTP/2. For the intermediary to be able to do that, it needs to
parse the lengths for the entire stream of stream chunks so it knows where
it can insert a new datagram chunk. So it can't just forward those blindly.

Because of this, stream chunks require the intermediary to parse the
message contents (i.e. DATA stream). We had landed on a new capsule frame
to simplify things.

Does this make sense?
David

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 26 Jul 2021 11:44:47 +1000

On Mon, Jul 26, 2021, at 11:22, David Schinazi wrote:
> message contents (i.e. DATA stream). We had landed on a new capsule 
> frame to simplify things.
> 
> Does this make sense?

It makes sense, but it doesn't mean that I agree with the assessment :)

You are optimizing for intermediaries that want to split flows (from a single stream to stream + DATAGRAM) or recombine them (stream + DATAGRAM into a single stream).  But those are precisely the intermediaries for which the additional work is already necessary.

Intermediaries that are not modified (the server in your example perhaps), will need to be modified in order to receive capsules too.  Whereas DATA frames/message content can just be passed.

---

From: David Schinazi <[email-masked]>
Date: Sun, 25 Jul 2021 19:06:19 -0700

On Sun, Jul 25, 2021 at 6:45 PM Martin Thomson <[email-masked]> wrote:

> You are optimizing for intermediaries that want to split flows (from a
> single stream to stream + DATAGRAM) or recombine them (stream + DATAGRAM
> into a single stream).  But those are precisely the intermediaries for
> which the additional work is already necessary.
>

The way I see it, I think I'm optimizing for the most prevalent
intermediaries out there: frontends that speak HTTP/3 to the client and an
older version over TCP to the backend. And FWIW I don't expect this
situation to change, the added CPU cost of QUIC will go down but still
remain non-zero, which will make it worth it on the WAN but not necessarily
for intra-datacenter traffic.

Intermediaries that are not modified (the server in your example perhaps),
> will need to be modified in order to receive capsules too.  Whereas DATA
> frames/message content can just be passed.
>

The server (by which I mean the one that responds to CONNECT-UDP and opens
UDP sockets) will need to be modified no matter what to support the new
method/functionality. The only intermediary that could go unmodified with
the DATA stream approach would be one that doesn't speak HTTP/3 and is not
the server. I'm not aware of anyone who is both deploying those and
interested in supporting CONNECT-UDP (nor CONNECT-IP or WebTransport).

All in all, both solutions would work with very similar practical
properties but different architectural/theoretical properties. Sounds like
a recipe for a good old fashioned IETF bikeshed. Let's have it during the
MASQUE session in 21 hours :-)

David