Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 15 Oct 2020 23:44:14 +0000

HI Lucas,

RFC7231 defines CONNECT originally like this:

“The CONNECT method requests that the recipient establish a tunnel to
   the destination origin server identified by the request-target and,
   if successful, thereafter restrict its behavior to blind forwarding
   of packets, in both directions, until the tunnel is closed.”

So I would interpret that the connection is not really a HTTP connection anymore after it has concluded the CONNECT. Again in HTTP/2 this did work because of multiplexing but in HTTP/3 is would work again and effectively maybe be the more flexible solution.

Mirja

From: Lucas Pardue <[email-masked]>
Date: Thursday, 15. October 2020 at 19:35
To: Alex Chernyakhovsky <[email-masked]>
Cc: Mirja Kuehlewind <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

Hey Mirja,

I'm against allowing unframed bytes on request streams. It limits extensibility (as pointed out by Alex) and introduces complexity to conventional HTTP/3 server implementations. HTTP desync attacks are something that framing protects against, let's not introduce risk for the sake of optimization.

The good news is that DATA frames can span QUIC packets. So if you're ok to take the hit once, you can send a very-long DATA frame and just keep appending data to it.

Cheers
Lucas

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 15 Oct 2020 23:51:22 +0000

Damn, missing „not“ below… meant to say that you need the HTTP framing for multiplexing in h2 but you don’t need it for that purpose in h3..

From: Masque <[email-masked]> on behalf of Mirja Kuehlewind <mirja.kuehlewind=[email-masked]>
Date: Friday, 16. October 2020 at 01:44
To: Lucas Pardue <[email-masked]>, Alex Chernyakhovsky <[email-masked]>
Cc: "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

HI Lucas,

Mirja

Hey Mirja,

Cheers
Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 16 Oct 2020 01:28:29 +0100

Is the overhead really that bad though?

A common case for CONNECT will be to tunnel TLS, so if you assume 16Kbyte
records the frame overhead of 3 bytes comes out at ~0.02%.

If you have TCP quarks that are smaller than DATA frames it's a different
story. But the solution there is to have a large DATA frame and write into
it several times. I'd hope that HTTP/3 implementations can offer streaming
frame consumption. Buffering is going to cause all sorts of issues.

Extension frames seem like a solid alternative. I know of at least one
other proposal for a tighter coupling of DATA-like frame to STREAM.

Cheers
Lucas

Cheers

On Fri, 16 Oct 2020, 00:51 Mirja Kuehlewind, <[email-masked]>
wrote:

> Damn, missing „not“ below… meant to say that you need the HTTP framing for
> multiplexing in h2 but you don’t need it for that purpose in h3..
>
>
>
> *From: *Masque <[email-masked]> on behalf of Mirja Kuehlewind
> <mirja.kuehlewind=[email-masked]>
> *Date: *Friday, 16. October 2020 at 01:44
> *To: *Lucas Pardue <[email-masked]>, Alex Chernyakhovsky <
> [email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
> *Subject: *Re: [Masque] HTTP DATA frames for HTTP CONNECT?
>
>
>
> HI Lucas,
>
>
>
> So I would interpret that the connection is not really a HTTP connection
> anymore after it has concluded the CONNECT. Again in HTTP/2 this did work
> because of multiplexing but in HTTP/3 is would work again and effectively
> maybe be the more flexible solution.
>
>
>
> Mirja
>
>
>
>
>
> *From: *Lucas Pardue <[email-masked]>
> *Date: *Thursday, 15. October 2020 at 19:35
> *To: *Alex Chernyakhovsky <[email-masked]>
> *Cc: *Mirja Kuehlewind <[email-masked]>, "[email-masked]"
> <[email-masked]>
> *Subject: *Re: [Masque] HTTP DATA frames for HTTP CONNECT?
>
>
>
> Hey Mirja,
>
>
>
> I'm against allowing unframed bytes on request streams. It limits
> extensibility (as pointed out by Alex) and introduces complexity to
> conventional HTTP/3 server implementations. HTTP desync attacks are
> something that framing protects against, let's not introduce risk for the
> sake of optimization.
>
>
>
> The good news is that DATA frames can span QUIC packets. So if you're ok
> to take the hit once, you can send a very-long DATA frame and just keep
> appending data to it.
>
>
>
> Cheers
>
> Lucas
>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 16 Oct 2020 08:42:52 +0000

It’s not that much the bit overhead but it a) seems just unnecessary, so why not remove it, and b) it’s more the added complexity when the a proxy server that only provides forwarding but no other HTTP servers has to support further HTTP semantics (or even potential future extensions).

Mirja

From: Lucas Pardue <[email-masked]>
Date: Friday, 16. October 2020 at 02:28
To: Mirja Kuehlewind <[email-masked]>
Cc: Mirja Kuehlewind <mirja.kuehlewind=[email-masked]>, Alex Chernyakhovsky <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

A common case for CONNECT will be to tunnel TLS, so if you assume 16Kbyte records the frame overhead of 3 bytes comes out at ~0.02%.

If you have TCP quarks that are smaller than DATA frames it's a different story. But the solution there is to have a large DATA frame and write into it several times. I'd hope that HTTP/3 implementations can offer streaming frame consumption. Buffering is going to cause all sorts of issues.

Extension frames seem like a solid alternative. I know of at least one other proposal for a tighter coupling of DATA-like frame to STREAM.

Cheers
Lucas

Cheers

On Fri, 16 Oct 2020, 00:51 Mirja Kuehlewind, <[email-masked]<mailto:[email-masked]>> wrote:
Damn, missing „not“ below… meant to say that you need the HTTP framing for multiplexing in h2 but you don’t need it for that purpose in h3..

From: Masque <[email-masked]<mailto:[email-masked]>> on behalf of Mirja Kuehlewind <mirja.kuehlewind=[email-masked]<mailto:[email-masked]>>
Date: Friday, 16. October 2020 at 01:44
To: Lucas Pardue <[email-masked]<mailto:[email-masked]>>, Alex Chernyakhovsky <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

HI Lucas,

Mirja

From: Lucas Pardue <[email-masked]<mailto:[email-masked]>>
Date: Thursday, 15. October 2020 at 19:35
To: Alex Chernyakhovsky <[email-masked]<mailto:[email-masked]>>
Cc: Mirja Kuehlewind <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

Hey Mirja,

Cheers
Lucas

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 16 Oct 2020 08:44:42 +0000

Hi David,

I was sending this request to exactly understand if or what the issues are/could be in not having the DATA frame. Can you maybe further explain which issue you see?

Mirja

From: David Schinazi <[email-masked]>
Date: Thursday, 15. October 2020 at 23:48
To: Mike Bishop <[email-masked]>
Cc: Mirja Kuehlewind <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

I agree with Alex, Lucas and Mike here.
Skipping HTTP frames entirely will cause issues.
The best solution IMO here is to define an extension
to HTTP/3 that defines a length-less DATA frame
(call it the DATUM frame perhaps) that extends until
the end of the stream. That would be beneficial for
HTTP GETs as well, not just MASQUE - so I'd
suggest bringing this to the QUIC or HTTP WG.

David

On Thu, Oct 15, 2020 at 11:24 AM Mike Bishop <[email-masked]<mailto:[email-masked]>> wrote:
(For example, see https://mikebishop.github.io/quic-external-data/draft-bishop-quic-external-data.html<https://protect2.fireeye.com/v1/url?k=fb1cf432-a5bc5aff-fb1cb4a9-86073b36ea28-e7aae7271a6c7151&q=1&e=b9526845-8195-474e-90cf-c7f907ea5f65&u=https%3A%2F%2Fmikebishop.github.io%2Fquic-external-data%2Fdraft-bishop-quic-external-data.html>.)

-----Original Message-----
From: Masque <[email-masked]<mailto:[email-masked]>> On Behalf Of Mike Bishop
Sent: Thursday, October 15, 2020 2:19 PM
To: Mirja Kuehlewind <mirja.kuehlewind=[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

We had a similar conversation about the framing of large HTTP responses when the server knows it has no (more) pushes or headers to communicate.  There were proposals for DATA-compatible frames without a length that implicitly last until the end of the stream, as well as references to unframed unidirectional streams.  Ultimately, the WG decided against adopting any of those into the core protocol, but this seems like fertile ground for an HTTP/3 extension.

-----Original Message-----
From: Masque <[email-masked]<mailto:[email-masked]>> On Behalf Of Mirja Kuehlewind
Sent: Thursday, October 15, 2020 12:31 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [Masque] HTTP DATA frames for HTTP CONNECT?

Hi all,

We recently looked into HTTP CONNECT as specified in  draft-ietf-quic-http (section 4.2.) and realized that all payload is required to be encapsulated in HTTP DATA frames after the CONNECT is completed. I can only guess that this is a “left-over” from HTTP/2 as in HTTP/2 this is required to realize multiplexing within the HTTP layer. However, for HTTP/3 multiplexing is realized by QUIC and as such it should be possible to simply forward all payload on a given QUIC stream after the CONNECT without the overhead of any additional HTTP framing. I believe that would actually be inline with how the CONNECT method worked in HTTP/1.1 and could simply save some overhead.

I’m raising this on the mailing list to understand first if there are any other reasons to have the HTTP DATA frame required or if this is maybe an issue that is still worth raising and could be easily addressed at the current state.

We are looking into this for similar use cases as for masque but in cases where only TCP is supported by the target server. For these use cases the proxy could easily support the HTTP CONNECT semantic but any additional HTTP semantics would not only be overhead but also increase complexity. So if it turns out there is no good reason to stuck with the requirement to have payload encapsulated in HTTP DATA frames, it should probably easy to just remove that requirement.

Mirja

--
Masque mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/masque
--
Masque mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/masque

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 16 Oct 2020 12:25:03 +0100

On Fri, 16 Oct 2020, 09:45 Mirja Kuehlewind, <mirja.kuehlewind=
[email-masked]> wrote:

> Hi David,
>
>
>
> I was sending this request to exactly understand if or what the issues
> are/could be in not having the DATA frame. Can you maybe further explain
> which issue you see?
>
I can't speak for David but the issues I forsee come from additional code
paths needed to support the feature. The frame parsing code needs to become
HTTP method-aware, which is not the case in the implementation I own.

With QUIC chair hat on: this is very late to be proposing design changes.
It might even be too late. I'm setting a high bar here for compelling
evidence or support that something needs to be changed.

Cheers
Lucas

masque <https://www.ietf.org/mailman/listinfo/masque>

---

From: Roberto Peon <[email-masked]>
Date: Fri, 16 Oct 2020 19:00:33 +0000

As I’ve considered multipath more and more, I am more convinced that it should be solved between the application and the connection, not between the connection and the transport.
For instance, an HTTP session should be able to use multiple connections (we already have that, kinda, but only for non-overlapping ranges/requests), and so long as we can define and/or discover the relationships between the connections (e.g. when the endpoint is the same but the connections are different), we have solved much of the problem-space.

In other words (and I hope my expression of this is redundant), multi-path really is about defining and mapping the session onto the lower level things.

-=R

From: QUIC <[email-masked]> on behalf of Lucas Pardue <[email-masked]>
Date: Friday, October 16, 2020 at 4:25 AM
To: Mirja Kuehlewind <mirja.kuehlewind=[email-masked]>
Cc: David Schinazi <[email-masked]>, QUIC WG <[email-masked]>, Mike Bishop <[email-masked]>, MASQUE <[email-masked]>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

On Fri, 16 Oct 2020, 09:45 Mirja Kuehlewind, <mirja.kuehlewind=[email-masked]<mailto:[email-masked]>> wrote:
Hi David,

I was sending this request to exactly understand if or what the issues are/could be in not having the DATA frame. Can you maybe further explain which issue you see?
I can't speak for David but the issues I forsee come from additional code paths needed to support the feature. The frame parsing code needs to become HTTP method-aware, which is not the case in the implementation I own.

With QUIC chair hat on: this is very late to be proposing design changes. It might even be too late. I'm setting a high bar here for compelling evidence or support that something needs to be changed.

Cheers
Lucas

masque<https://www.ietf.org/mailman/listinfo/masque>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Fri, 16 Oct 2020 19:09:55 +0000

Hi Lucas,

see inline.

From: Lucas Pardue <[email-masked]>
Date: Friday, 16. October 2020 at 13:25
To: Mirja Kuehlewind <[email-masked]>
Cc: David Schinazi <[email-masked]>, Mike Bishop <[email-masked]>, QUIC WG <[email-masked]>, MASQUE <[email-masked]>
Subject: Re: [Masque] HTTP DATA frames for HTTP CONNECT?

Not sure I understand this correctly. Without the DATA framing you would not need any frame parsing anymore, you only have to remember that a certain stream is converted into a forwarding stream and blinding forward all payload from that stream. The whole point is that then no additional HTTP logic would be need anymore.

I know… that’s why I’m trying to understand the implications instead of just raising an issue on github.

Mirja

Cheers
Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 16 Oct 2020 20:34:53 +0100

Hey,

On Fri, 16 Oct 2020, 20:09 Mirja Kuehlewind, <[email-masked]>
wrote:

> Not sure I understand this correctly. Without the DATA framing you would
> not need any frame parsing anymore, you only have to remember that a
> certain stream is converted into a forwarding stream and blinding forward
> all payload from that stream. The whole point is that then no additional
> HTTP logic would be need anymore.
>

So with quiche, a receiving endpoint calls quiche's HTTP/3 oriented poll()
method. This reads from a transport stream and does HTTP/3 syntax handling
e.g. checking that streams and frames are being used according too all the
rules in the H3 spec. Beyond that quiche doesn't handle anything about the
semantics such as request processing; thats the application's job.

With a modified CONNECT definition, it doesn't change things for other
types of request. I still need to do frame parsing on streams unless an
application tells me otherwise.

On the send side, I'd need the mirror that allows an application to request
sending unframed data.

That causes a bled across my separation of concerns boundary. I need to
account for applications that might misuse this capability to send unframed
data either accidently, or purposefully because they wish to do side step
the quiche's frame checking.

These complications make me nervous. Others might have a different risk
assessment.

Cheers
Lucas

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 19 Oct 2020 10:42:16 +1100

> On 16 Oct 2020, at 10:44 am, Mirja Kuehlewind <mirja.kuehlewind=[email-masked]> wrote:
> 
> HI Lucas,
>  

In the current revision of the spec <https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#CONNECT>, that's followed by:

> Because CONNECT changes the request/response nature of an HTTP connection, specific HTTP versions might have different ways of mapping its semantics into the protocol's wire format.

Note that it *doesn't* say (in either version) that the *connection* is converted into a tunnel; rather, only that one is established. As we now more explicitly point out, the mapping to transport connections depends on the version of the protocol in use; in h3, it isn't the whole connection.

Additional background here: <https://github.com/httpwg/http-core/issues/144>

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 19 Oct 2020 10:03:56 +0000

Hi Mark,

understood. However, given this is version specific, I'm questioning if we should actually convert a whole stream into a tunnel with h3 rather than unnecessary sticking to the same model as used for h2.

Mirja

    Additional background here: <https://protect2.fireeye.com/v1/url?k=21efee0c-7f4f40c1-21efae97-866132fe445e-86d1213e0f8bf20b&q=1&e=ca64d89b-a8c1-4842-81bf-58f120339203&u=https%3A%2F%2Fgithub.com%2Fhttpwg%2Fhttp-core%2Fissues%2F144>

    Cheers,

    --
    Mark Nottingham   https://protect2.fireeye.com/v1/url?k=3f3b8401-619b2acc-3f3bc49a-866132fe445e-607fe695c30ded20&q=1&e=ca64d89b-a8c1-4842-81bf-58f120339203&u=https%3A%2F%2Fwww.mnot.net%2F

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 19 Oct 2020 10:34:47 +0000

Hi Lucas,

see below.

Date: Friday, 16. October 2020 at 21:35

Hey,
On Fri, 16 Oct 2020, 20:09 Mirja Kuehlewind, <[email-masked]<mailto:[email-masked]>> wrote:
Not sure I understand this correctly. Without the DATA framing you would not need any frame parsing anymore, you only have to remember that a certain stream is converted into a forwarding stream and blinding forward all payload from that stream. The whole point is that then no additional HTTP logic would be need anymore.

So with quiche, a receiving endpoint calls quiche's HTTP/3 oriented poll() method. This reads from a transport stream and does HTTP/3 syntax handling e.g. checking that streams and frames are being used according too all the rules in the H3 spec. Beyond that quiche doesn't handle anything about the semantics such as request processing; thats the application's job.

With a modified CONNECT definition, it doesn't change things for other types of request. I still need to do frame parsing on streams unless an application tells me otherwise.

I don’t know the details of your implementation and I didn’t check, so maybe I’m just misunderstand but I would assume that the CONNECT handling itself is done by the HTTP implementation and not the application on top. So your HTTP connection would receive a CONNECT (& convert that stream into a tunnel) and then tell the application that a connect was received. Then when you receive any data on that stream you can forward it directly to the application without even using the frame parser given you have the state information which streams have seen a CONNECT with the HTTP stack. Did you implement CONNECT already?

On the send side, I'd need the mirror that allows an application to request sending unframed data.

What I’m saying is that you have ne a separate interface to generate the CONNECT (again assuming actually generating the CONNECT this is part of your HTTP stack and not the part of the application). This method should than generate a handle that allows you to send unframed data only on that stream (where you are sure that a CONNECT was send and confirmed previously).

That causes a bled across my separation of concerns boundary. I need to account for applications that might misuse this capability to send unframed data either accidently, or purposefully because they wish to do side step the quiche's frame checking.

If you can ensure that your HTTP has initial send a CONNECT (and got a 200 reply) on that stream, I don’t think you need any additional checking.

These complications make me nervous. Others might have a different risk assessment.

Cheers
Lucas

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 19 Oct 2020 12:26:16 +0100

The library doesn't inspect request methods. Doing so for CONNECT would
create a disparity, which is an excellent way to create bugs.

---

From: Ian Swett <[email-masked]>
Date: Tue, 20 Oct 2020 18:05:52 -0400

I'm coming to this conversation late in the game, but this was discussed
fairly extensively on https://github.com/quicwg/base-drafts/issues/1885

And I do believe it's an annoyance, since adding an extra layer of framing
can greatly increase code complexity, harm performance, or both.

However, now that I re-read the HTTP draft, I wonder if it's legal to send
a DATA frame with a huge size and then let the QUIC stream close?  I
couldn't find any text specifically describing what to do in the case where
the DATA frame length does not match the number of bytes read prior to the
QUIC stream being closed, but possibly I missed something.  If this is
allowed, it seems functionally equivalent to the proposal to treat 0 length
DATA frames as terminated when the stream is closed.

Ian

---

From: Ian Swett <[email-masked]>
Date: Tue, 20 Oct 2020 18:43:59 -0400

Alan and Mike answered my question on a different channel, from
https://github.com/quicwg/base-drafts/blob/master/draft-ietf-quic-http.md#frame-layout

Each frame's payload MUST contain exactly the fields identified in its
description. A frame payload that contains additional bytes after the
identified fields or a frame payload that terminates before the end of the
identified fields MUST be treated as a connection error ({{errors}}) of
type H3_FRAME_ERROR.

So we just need to write up an extension draft.

---

From: Lucas Pardue <[email-masked]>
Date: Wed, 21 Oct 2020 00:06:51 +0100

An extension seems entirely reasonable to me.

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 21 Oct 2020 10:31:44 +0900

I agree with others that we do not need to revisit the issue we discussed
in #1885. The conclusion there was that HTTP/3 will always use
application-layer framing.

Regarding how we might fix the issue *in the future*, my preferences goes,
as stated in one of the issue comments, to adding a transport-level
extension that allows "messages" (instead of bytes) to be sent over a QUIC
stream, then use that to send frames in HTTP/3. Doing so reduces the wire
overhead for any frame being sent, reduces the total complexity.

But what we should do *now* is ship QUIC and H3, they do not have to be
changed.

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 21 Oct 2020 03:12:14 -0700

So I am new to this concept of masque, but why do you need to have H3
involved in proxying if the objective is to proxy arbitrary applications as
opposed to acting as as replacement for existing HTTP tunnels?

There is a quite a bit of complexity in H3 that is not necessary when using
QUIC for general purpose proxying.

As to the DATA frame, my intuition tells me that it is probably useful to
retain a concept of tunneling over a single HTTP connection for semantic
reasons, such as an endpoint that is written for HTTP and not for QUIC, but
this need not be related to MASQUE.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 21 October 2020 at 03.32.22, Kazuho Oku ([email-masked]) wrote:

-- 
Kazuho Oku