Subject: SETTINGS_ENABLE_CONNECT_PROTOCOL and new protocols

From: "Martin Thomson" <[email-masked]>
Date: Tue, 02 Nov 2021 17:41:33 +1100

SETTINGS_ENABLE_CONNECT_PROTOCOL is defined as enabling the :protocol pseudo-header field on CONNECT.  The implication being that any protocol is OK once this setting is negotiated.

In reviewing WebTransport proposals, I realized that this is probably a mistake.  There, Victor (I think) realized that you need to negotiate the use of :protocol AND the use of the specific protocol.  Otherwise, you might attempt to use extended CONNECT when the server does not support some of the extra fancy stuff that is in WebTransport.

That leads me to ask:

Should this setting be defined as **extended CONNECT for WebSockets** rather than extended CONNECT more generally?

A generic thing might be useful if the purpose of the exchange is to negotiate for use of the bidirectional channel that is established by CONNECT.  That's all WebSockets does.  Maybe that's enough for a bunch of different cases.

A generic thing is not sufficient if there are other things going on in the protocol as a result.  WebTransport needs additional negotiation as it uses datagrams (and sometimes other streams).  Datagrams are the main thing here, but I don't know if we should limit our thinking to that, especially if we might want to consider partial reliability or other transport-level features.

The proposed solution for WebTransport is to provide a setting that covers the whole package needed by WebTransport.  Is that a better pattern in general?  Should we retcon SETTINGS_ENABLE_CONNECT_PROTOCOL to SETTINGS_ENABLE_WEBSOCKETS to follow this pattern?  Or, is this only a problem that "special" protocols need to concern themselves with?  Those that need more than just the "CONNECT stream".  

I struggle with the view that datagrams might be the only new and special thing.  For one, they are not in the current WebTransport for HTTP/3 proposal.  Also, it seems like with MASQUE and WebTransport, all uses of extended CONNECT require some special handling.

Now, you might say that these "special" protocols might just need to separately negotiate the features they use, like datagrams or non-request streams.  That might work, but these protocols are not all the same.  Support for one does not immediately imply support for all others, even if they use the same basic protocol features.  For instance, the datagrams they exchange cannot be generically handled without understanding the details of the protocol.

That leads me to think that maybe we need to treat this as a whole-stack thing.  A setting per :protocol value is pretty unambiguous.  What do others think?  Should we update the setting definition in RFC 8441?

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 2 Nov 2021 08:35:43 +0100

Hi Martin,

On Tue, Nov 02, 2021 at 05:41:33PM +1100, Martin Thomson wrote:
> SETTINGS_ENABLE_CONNECT_PROTOCOL is defined as enabling the :protocol
> pseudo-header field on CONNECT.  The implication being that any protocol is
> OK once this setting is negotiated.
> 
> In reviewing WebTransport proposals, I realized that this is probably a
> mistake.  There, Victor (I think) realized that you need to negotiate the use
> of :protocol AND the use of the specific protocol.  Otherwise, you might
> attempt to use extended CONNECT when the server does not support some of the
> extra fancy stuff that is in WebTransport.
> 
> That leads me to ask:
> 

This is a good question indeed. In haproxy after figuring that we were
incorrectly reusing some backend H2 connections that do not support WS
and that we ought to include the knowledge of support of extended CONNECT
in the choice of connection, we also noted that there will be an issue if
later another protocol wants to use the extended CONNECT because a single
boolean will not let us know which protocol the server supports. For now
we're considering that all this is WS but I'm already anticipating fun
things in the future.

> A generic thing might be useful if the purpose of the exchange is to
> negotiate for use of the bidirectional channel that is established by
> CONNECT.  That's all WebSockets does.  Maybe that's enough for a bunch of
> different cases.

Yep, we'd need something similar to ALPN in fact, because that's exactly
what it is in reality, while we're negotiating the ability to establish
a tunnel we don't know what protocol the application layer supports.

> A generic thing is not sufficient if there are other things going on in the
> protocol as a result.  WebTransport needs additional negotiation as it uses
> datagrams (and sometimes other streams).  Datagrams are the main thing here,
> but I don't know if we should limit our thinking to that, especially if we
> might want to consider partial reliability or other transport-level features.
> 
> The proposed solution for WebTransport is to provide a setting that covers
> the whole package needed by WebTransport.  Is that a better pattern in
> general?  Should we retcon SETTINGS_ENABLE_CONNECT_PROTOCOL to
> SETTINGS_ENABLE_WEBSOCKETS to follow this pattern?

We risk to quickly get out of settings ID this way.

> Or, is this only a
> problem that "special" protocols need to concern themselves with?  Those that
> need more than just the "CONNECT stream".  

I suspect that we'd need something that more resembles a negotiation.
But settings can be sent at once in both directions. But we could state
that servers supporting some protocol extensions ought to wait for the
client's settings frame before advertising their support for a requested
application-layer protocol. Or we could have another setting back to
indicate which one was chosen. The only thing is that we only have 32 bits
to encode plenty of protocols and this would require some registration.

Or maybe we could have a sequence like this one, where when a client
advertises support for such an extended CONNECT, a new frame of type
ALPN advertises the list of protocols supported over CONNECT:

   client  --------> SETTINGS_ENABLE_APPLICATION_PROTOCOL -------->
           <----- "ALPN" frame indicating supported protocols <---- server

But even with that I wouldn't be surprised that some want to advertise
support for nothing in return (empty string maybe) and others would
advertise support for everything (plain transparent tunnels).

> I struggle with the view that datagrams might be the only new and special
> thing.  For one, they are not in the current WebTransport for HTTP/3
> proposal.  Also, it seems like with MASQUE and WebTransport, all uses of
> extended CONNECT require some special handling.

CONNECT has always required special handling, since HTTP/1 anyway. It's
pure protocol encapsulation, and each time we think we've covered it all
we discover yet another layer on top of it that needs to be known upfront
or be negotiated.

> Now, you might say that these "special" protocols might just need to
> separately negotiate the features they use, like datagrams or non-request
> streams.  That might work, but these protocols are not all the same.  Support
> for one does not immediately imply support for all others, even if they use
> the same basic protocol features.  For instance, the datagrams they exchange
> cannot be generically handled without understanding the details of the
> protocol.

And it's not always trivial when you want to deal with idle connections,
sometimes you would like to know which ones support what before deciding
to use them.

> That leads me to think that maybe we need to treat this as a whole-stack
> thing.  A setting per :protocol value is pretty unambiguous.  What do others
> think?  Should we update the setting definition in RFC 8441?

It is unambiguous but could be bothersome to manage and retrofit into
existing components each time we create a new one. At least having a
setting with a protocol ID would be better but then we could end up
with fun stuff like tunnels advertising all 2^32 possibilities, which
might be stupid as well.

But I do think that it deserves a little bit of discussion to see what
could be done. I wouldn't be surprised if we end up with a new setting
that broadly covers :protocol with ext CONNECT but mandates advertising
protocol support, and that the current one only goes deprecated.

Willy

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 2 Nov 2021 13:56:17 +0100

Caveat: I have not written a Websocket h2 implementation and just
read up a bit about WebTransport. So many grains of salt here...

> Am 02.11.2021 um 08:35 schrieb Willy Tarreau <[email-masked]>:
> 
> Hi Martin,
> 
>> [...]

I think it's beneficial to differentiate between a HTTP capability, like
Extended CONNECT, and something like "supported protocols". Because if
a specific protocol is supported or not depends on where the CONNECT
is really going to. E.g. one has to do it in order to learn if a 
protocol is supported or not.

SETTINGS_ENABLE_CONNECT_PROTOCOL says: "I, the server know about
H2 CONNECT and :protocol and my responses will be in accordance to
its definition." It does NOT say: "I support websockets".

hop-by-hop vs. end-to-end.

As Willy noted about HAProxy, some backend connections will support
it, some won't. The client needs to really say where it wants to go,
e.g. open the stream to find out. But without the CONNECT capability,
there is no use in trying.

Now, if WebTransport requires some additional capabilities, *on
the server it talks to* those seem to be what SETTINGS are for.

On scanning <https://www.ietf.org/archive/id/draft-ietf-webtrans-http2-02.html>
I was so far not able to detect those. All WT_ frames seems to happen
inside the H2 STREAM's DATA, but maybe I got that wrong.

Kind Regards,
Stefan

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 2 Nov 2021 10:10:18 -0400

I think :protocol should be analogous to Content-Type.  Whether a given
:protocol value is supported should be a property of the resource, not the
origin, and certainly not the server.  Otherwise, it will be very difficult
to roll out support for new :protocols.

As an extreme example, consider a CDN that serves many origins in a single
certificate (as is common).  The backend servers for these different
origins might have wildly different sets of supported :protocols, but they
will all share a single HTTP/2 connection to the client.  There is no
SETTINGS value that the CDN could send to explain which :protocols are
supported, as it depends on the origin of each request.

Similar considerations apply for a single origin that offers distinct
services at different paths.

In my view, the problem here is WebTransport's use of :protocol.  I think
this use is very special and weird: it is a request header  on a single
request that modifies the entire connection.  This breaks the whole HTTP
layering model.  I would prefer to spell this some other way (e.g. add
"multistream representations" to HTTP), but failing that, I think we should
understand that WebTransport is Super Weird and its use of :protocol is the
least of it.

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 03 Nov 2021 11:20:28 +1100

On Wed, Nov 3, 2021, at 01:10, Ben Schwartz wrote:
> In my view, the problem here is WebTransport's use of :protocol.  I 
> think this use is very special and weird:

I started here also.  But MASQUE is similarly weird and it seems like we have more weird cases than simple ones.  Are we making a decision that is right for just one example?

Stefan might have the answer we want.  The generic capability is just the ability to negotiate the use of the bidirectional channel for use by a different protocol.  That might be enough for both websockets and the HTTP/2 mapping of WebTransport.  It's only when we need extra capabilities that we need another negotiation.

What bothers me still is that this is only part of the puzzle.  As Dragana noted, support for a given :protocol at a connection level is not assured.  So while a resource might support the :protocol, they might not support that over the current type of connection.  That means that there are cases where the connection supports the ability to negotiate :protocols in general, the resource supports this specific :protocol, and the combination of the two doesn't.  

For example, my server might support WebTransport, so it declares support for :protocol with SETTINGS_ENABLE_CONNECT_PROTOCOL. However, it doesn't support WebSockets over anything but an upgraded HTTP/1.1 connection.  I don't think that you can just say "well implementing WebSockets for h2 is trivial if you already understand :protocol".  It's not trivial.

This is why I asked the question.  Is a generic negotiation really the best idea here?

---

From: David Schinazi <[email-masked]>
Date: Tue, 2 Nov 2021 18:03:04 -0700

I think there's some interesting nuance in the dichotomy between hop-by-hop
vs end-to-end properties of HTTP. Unfortunately Section 7 of RFC 8441
roughly says "don't use extended CONNECT across intermediaries". However,
as discussed on the MASQUE mailing list, there is real interest in using
CONNECT-UDP across intermediaries. And CONNECT-UDP relies on extended
CONNECT, so we need to make that work.

Pseudo-headers are a hop-by-hop wire-format property of HTTP/2 and 3.
Sending a header that contains a colon but isn't one of the RFC7540
pseudo-headers makes your request malformed. In my understanding,
SETTINGS_ENABLE_CONNECT_PROTOCOL was introduced to make the :protocol
pseudo-header no longer malformed. Conceptually, it's a hop-by-hop way of
saying "I won't treat your request as malformed". It doesn't provide any
guidance as to whether the sender of the setting actually supports any
protocol, or whether it'll reject such requests or not.

In contrast, whether any given protocol is supported is an end-to-end
property. That's because, for protocols that only use the request stream,
the intermediary can blindly shovel bytes back and forth and support
protocols it's not aware of. In a world where everything is TCP, that's all
we need - we're done. If the intermediary supports extended CONNECT, it can
happily forward the stream without having to know the protocol.
Unfortunately, both MASQUE and WebTransport throw some wrenches in those
gears. The first one is QUIC DATAGRAM frames: now the intermediary needs to
also know how to forward those, and that's where the H3_DATAGRAM setting
comes in. A setting is a good fit here because QUIC DATAGRAM frames are a
per-hop concept. WebTransport-over-h3 throws even more wrenches in the
gears, because it allows for custom uses of QUIC streams which aren't
within HTTP semantics, and again that's where we introduce
the ENABLE_WEBTRANSPORT setting. Again those streams are per-hop.

Conceptually, I think defining a new extended CONNECT protocol is very
similar to defining a new HTTP method. HTTP methods are end-to-end: when we
introduce a new HTTP method, we don't need a setting because HTTP semantics
already tell intermediaries that they can forward methods that they don't
understand.

Based on this train of thought, I think it makes more sense to have
settings for specific features (i.e. QUIC DATAGRAM frame when ALPN=h3) then
for whole-stack protocols, because features are hop-by-hop whereas
whole-stack is end-to-end.

I'm not sure it's really possible to solve the "as a client I want to know
all the feature set of this server" because the first intermediary might
know what all its backends support.

David

On Mon, Nov 1, 2021 at 11:45 PM Martin Thomson <[email-masked]> wrote:

> mistake.  There, Victor (I think) realized that you need to negotiate the
> use of :protocol AND the use of the specific protocol.  Otherwise, you
> might attempt to use extended CONNECT when the server does not support some
> of the extra fancy stuff that is in WebTransport.
>
> That leads me to ask:
>
> Should this setting be defined as **extended CONNECT for WebSockets**
> rather than extended CONNECT more generally?
>
> A generic thing is not sufficient if there are other things going on in
> the protocol as a result.  WebTransport needs additional negotiation as it
> uses datagrams (and sometimes other streams).  Datagrams are the main thing
> here, but I don't know if we should limit our thinking to that, especially
> if we might want to consider partial reliability or other transport-level
> features.
>
> SETTINGS_ENABLE_WEBSOCKETS to follow this pattern?  Or, is this only a
> problem that "special" protocols need to concern themselves with?  Those
> that need more than just the "CONNECT stream".
>
> streams.  That might work, but these protocols are not all the same.
> Support for one does not immediately imply support for all others, even if
> they use the same basic protocol features.  For instance, the datagrams
> they exchange cannot be generically handled without understanding the
> details of the protocol.
>
> thing.  A setting per :protocol value is pretty unambiguous.  What do
> others think?  Should we update the setting definition in RFC 8441?
>
>
>

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 03 Nov 2021 14:38:33 +1100

On Wed, Nov 3, 2021, at 12:03, David Schinazi wrote:
> I'm not sure it's really possible to solve the "as a client I want to 
> know all the feature set of this server" because the first intermediary 
> might know what all its backends support.

That's a nice philosophy.  That seems like the cleanest outcome.  However, it doesn't give me an answer to the dilemma from before:

> That means that there are cases where the connection supports the ability to negotiate :protocols in general, the resource supports this specific :protocol, and the combination of the two doesn't.

I'm aware of cases where this happens already and we jump through all sorts of hoops as a result.  Are you suggesting that we just live with this situation because it is unavoidable?

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Nov 2021 04:54:28 +0100

On Wed, Nov 03, 2021 at 11:20:28AM +1100, Martin Thomson wrote:
> For example, my server might support WebTransport, so it declares support for
> :protocol with SETTINGS_ENABLE_CONNECT_PROTOCOL. However, it doesn't support
> WebSockets over anything but an upgraded HTTP/1.1 connection.  I don't think
> that you can just say "well implementing WebSockets for h2 is trivial if you
> already understand :protocol".  It's not trivial.
> 

That's why I suggest that we very likely need something like ALPN to
negotiate the upper level protocols (and I think that those transported
over CONNECT are the same as those we may transport over TLS as in both
cases these are raw tunnels).

But probably that we ought to see this differently: we could imagine
that in order to establish a tunnel with CONNECT + :protocol, we ought
to send the 200 back only once established from end-to-end, and plan
to indicate in return whether or not the requested protocol could
successfully be established or not. After all, in HTTP/1.1 it was
recommended not to push the payload before seeing the 200, and it seems
to me that we in fact have everything available here to perform some
adjustments. It's just that instead of negotiating we ought to attempt
to connect and get prepared to fall back. In case of failure we could
imagine sending an optional list of supported protocols in a header
(even :protocol or something else). It can be slightly harder for some
components but much more reliable than asking on a connection for
end-to-end support that will ultimately fail because the whole chain
was not validated. 

With such a minor adjustment we could state that instead of asking for
a single protocol in ":protocol", we could pass a list of protocols
(or ALPN-compatible tokens?) and that in case of successful connection
the accepted protocol would be returned (":protocol as well?").

Just my two cents,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Nov 2021 07:35:23 +0100

On Wed, Nov 03, 2021 at 02:38:33PM +1100, Martin Thomson wrote:
> That's a nice philosophy.  That seems like the cleanest outcome.

I agree with this as well, and that's also why I retract my very first
proposal of seeing the server advertise what protocol it supports.

>  However, it doesn't give me an answer to the dilemma from before:
> 
> > That means that there are cases where the connection supports the ability
> > to negotiate :protocols in general, the resource supports this specific
> > :protocol, and the combination of the two doesn't.
> 
> I'm aware of cases where this happens already and we jump through all sorts
> of hoops as a result.  Are you suggesting that we just live with this
> situation because it is unavoidable?

If we consider that :protocol is hop-by-hop and that 200 is end-to-end,
then I think we should be able to address most if not all use cases. By
this I mean that each intermediary in the chain is responsible for
negotiating a tunnel (or any other form) with its next hop for the
requested protocol, and only ought to respond after being certain that
it was properly set up, and that in case of failure the first element
in the chain aware of the trouble and able to fall back on other methods
(e.g. HTTP/1 for WebSocket) ought to try this before responding.

It's not the easiest to implement (and at least requires to hold the
data before the 200) but I don't think we can do much more without
knowing upfront if some backend servers will support the protocol or
not.
A CDN front node could very well advertise support for extended CONNECT
and deal with it by itself depending on what the origin servers support.

I think the main issue in RFC8441 is that it doesn't speak about failures
to set up the tunnel, only the successful case is documented, because it
immediately remaps to RFC6455 as it was initially intended to deal with
WebSocket only, and uses WebSocket's header-based signaling. We'd need
to indicate what to return when :protocol is not supported (status code
such as 501?  RST with a new error code?). This is what a client or
intermediary needs to know if it ought to retry, fall back, or fail.

But I do think that having the ability to support multiple protocols,
and letting the server choose, or signal in response which alternatives
it suggests could be useful for the long term.

Willy

---

From: Stefan Eissing <[email-masked]>
Date: Wed, 3 Nov 2021 10:00:59 +0100

> Am 03.11.2021 um 07:35 schrieb Willy Tarreau <[email-masked]>:
> 

+1

I'd prefer a response instead of RST. 501 seems good.

> it suggests could be useful for the long term.

Error causes and/or available protocols as response headers. If more than
one protocol is allowed in negotiation, return in a 200 the protocol 
switched to (header or content type?).

Imagine there being a version 1 and 2 of WebTransfer one day.

Kind Regards,
Stefan

---

From: Stefan Eissing <[email-masked]>
Date: Wed, 3 Nov 2021 10:21:49 +0100

> Am 03.11.2021 um 10:00 schrieb Stefan Eissing <[email-masked]>:
> 
> 

Thinking about this some more, the H1 WebSocket's idea to
verify the protocol negotiation via its Web-Sec-Key was
a good idea. The client should be able to identify a
successful negotiation of the end-to-end protocol
established.

- Stefan

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Nov 2021 11:14:05 +0100

> > I'd prefer a response instead of RST. 501 seems good.

I prefer a response as well since it can be forwarded end-to-end,
and in this case it may even convey a header announcing a possible
list of supported protocols, alternate URL to use or whatever we'll
think about when needed.

> successful negotiation of the end-to-end protocol

Sure but that's limited to websocket, and by then it was designed this
way to be absolutely certain that we were talking to a truly websocket-
aware server (since at this time nobody implemented the upgrade and the
fears of dealing with random responses were rightfully high). Over H2
there isn't this problem.

Willy

---

From: David Schinazi <[email-masked]>
Date: Wed, 3 Nov 2021 10:05:31 -0700

On Tue, Nov 2, 2021 at 8:38 PM Martin Thomson <[email-masked]> wrote:

> That's a nice philosophy.  That seems like the cleanest outcome.  However,
> it doesn't give me an answer to the dilemma from before:
>
> > That means that there are cases where the connection supports the
> ability to negotiate :protocols in general, the resource supports this
> specific :protocol, and the combination of the two doesn't.
>
> I'm aware of cases where this happens already and we jump through all
> sorts of hoops as a result.  Are you suggesting that we just live with this

One potential solution to this dilemma is to find a way to communicate to
the client what is supported (e.g. settings as you mentioned). Another
solution would be to have the client try and have a way to communicate the
reason for failure. There's discussion of a new status code in this thread,
and I like that. In general I much prefer the "try what you want to do and
failover if you fail" approach over the "ask for permission first" approach
because the latter introduces time-to-check-to-time-of-use issues.

David

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Nov 2021 18:13:43 +0100

On Wed, Nov 03, 2021 at 10:05:31AM -0700, David Schinazi wrote:
> In general I much prefer the "try what you want to do and

Yeah, that's exactly the reason we went into that direction for
early-data :-)

It can sometimes be difficult to deal with on intermediaries, but with
a CONNECT you're not supposed to start sending your data anyway so I
think that retrying on a failure is much less difficult in this case.

Willy

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 3 Nov 2021 17:17:41 +0000

Dare I say that it sounds like we are reinventing RFC 2774[1]?

Section 7 defines the 510 Not Extended status code:

   The policy for accessing the resource has not been met in the
   request.  The server should send back all the information necessary
   for the client to issue an extended request. It is outside the scope
   of this specification to specify how the extensions inform the
   client.

   If the 510 response contains information about extensions that were
   not present in the initial request then the client MAY repeat the
   request if it has reason to believe it can fulfill the extension
   policy by modifying the request according to the information provided
   in the 510 response. Otherwise the client MAY present any entity
   included in the 510 response to the user, since that entity may
   include relevant diagnostic information.

[1] - https://datatracker.ietf.org/doc/html/rfc2774

---

From: Ilari Liusvaara <[email-masked]>
Date: Wed, 3 Nov 2021 20:23:20 +0200

> there is no use in trying.

Or worse yet, there might be chained reverse proxy, and as result some
resources support it, and some do not... On the same backend from the
reverse proxy perspective.

> inside the H2 STREAM's DATA, but maybe I got that wrong.

It is not just question if WebTransport is special (apparently no), but
Websockets is special, due to extended negotiation (to ensure server
actually supports it) it does. Specifically, if translating Websockets
from HTTP/2 to HTTP/1.1 backend, the reverse proxy must synthethize a
fake key (even if just always fixed one).

-Ilari

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 4 Nov 2021 06:47:04 +0100

Hi Lucas,

> [1] - https://datatracker.ietf.org/doc/html/rfc2774

I could be nit-picking here but I don't see a failure to connect using
a specific protocol as a failure to reach an extension. We do use the
extended CONNECT because it was already advertised as available, it's
only the application-level protocol that's not available. Hence I tend
to think that 501 is very close (no protocol stack is implemented for
the requested protocol name). However I would see 510 as perfectly
suited for an extended CONNECT sent over a connection that did not
advertise support for it in the first place (except that here it's
done using H2 after negotiating a setting).

Willy