Subject: Re: WGLC Review of HTTP (-semantics)

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 6 Feb 2021 17:40:01 -0800

> On Jan 31, 2021, at 7:50 PM, Martin Thomson <[email-masked]> wrote:
> 
> I just finished reading this and I have to say that this is very good.  It's a masterful work.  This is a difficult protocol to document, but this document is well-written, clearly organized, and precise.

Hi Martin,

Thanks for the detailed reviews and PRs; they have been very helpful.

> Well, precise to exactly the extent the protocol permits, it's a protocol with lots of warts that have accumulated over its history.  But each of those is uniformly well-treated with clear explanations.
> 
> I raised a few issues and pull requests, but those were mostly minor.  I have just two comments that I think are worth raising at this level.
> 
> 1. Multiple trailer sections
> 
> The document describes trailers in a way that permits multiple field sections to be sent, starting from the point after the end of the header section following a final status code.

To be clear, it doesn't permit them to be sent. It permits them to be understood.

It gives us a framework in which to understand 0, 1, ..., N sections containing
metadata, as opposed to the HTTP/0.9 model of 0 sections, HTTP/1.0 model
of one header section, or the HTTP/1.1 model of one header section and
potentially one trailer section. In short, the description in Semantics allows
for extensions to be defined in the future that can send N sections without
having to re-publish the core HTTP specs.

> This feature has not been implemented and deployed.  Worse, this feature *cannot* be implemented or deployed as there is no version of this protocol, or even an extension to the protocol, that allows this feature to be implemented.  Every current version either does not support trailers (0.9, 1.0) or only supports precisely one trailer section (1.1 with chunked encoding, 2, 3).
> 
> I appreciate the design as a rational, well-constructed thing.  It is very crisply defined.  However, I don't think that it has any place in a document that purports to document an interoperable standard.

It actually has been implemented and deployed, both within experimental/hacked
versions of HTTP and within a deployed METADATA extension frame for HTTP/2.
This is not a new concept. E.g.,

  https://docs.google.com/document/d/1DsPjl6qWxhcF28vzJey0xQJ2O2eRKHv7CBXt3EZyDjc/ <https://docs.google.com/document/d/1DsPjl6qWxhcF28vzJey0xQJ2O2eRKHv7CBXt3EZyDjc/>

We were prevented from defining it within HTTP/2 and HTTP/3
specifically because folks claimed that they could be implemented as
extensions.

However, the definition in Semantics doesn't change deployment status.
All it does is provide for the ability to deploy that feature in the future by
creating a framework in which those semantics can exist. It has absolutely
no effect on existing protocol implementations that have not deployed
multiple trailers (or mid-stream metadata) via HTTP extension points.

This is a chicken and egg problem. What permits extensions to be
deployed in HTTP is the fact that existing recipients are prepared
to ignore them. It is only by defining what can be ignored, before
we deploy, that we can understand both the scope of what can be
safely deployed in the future. Without that, we get stuck in this fear
of anything but the status quo.

IMO, mid-stream metadata is a well understood concept that is common
in other protocols. The reason people haven't deployed it in HTTP is
because we haven't provided a framework in which to do so.

> 2. Upgrade
> 
> Only HTTP/1.1 supports Upgrade in any meaningful sense.  It's possible that HTTP/1.1 is the only version that could use Upgrade in that way.  The way that this mechanism is designed essentially depends on a paradigm that is unique to that version.  I don't think that this is a big deal, but I would prefer to see Sections 7.8, 15.2.2, 15.5.22, and 16.7 moved to the messaging document.  As far as I can tell, these sections are the only ones to mention anything related to Upgrade (aside from listings in the IANA considerations), so my guess is that this would not be especially disruptive a move.  (As these are trailing sections, it probably does not even affect inbound citations to this document, thinking about QUIC.)

I don't think Upgrade has any such limitations. I understand why the
folks implementing h2 were not interested in allowing Upgrade be
signaled on a stream, even when only one stream is in use, but there
is nothing (aside from the spec) preventing Upgrade from functioning
in h2. Likewise, there is nothing 1.1-specific about HTTP/3 clients
indicating that they support some future protocol within QUIC
streams via the Upgrade header field, as opposed to the far more
brittle settings. Websocket++ being the obvious example.

In any case, when we move the semantics defined by HTTP header fields
out of the Semantics document, what we find is that all of the protocol
specs must then make normative references to the 1.1 specification to
forbid the field from being used in those other versions of HTTP.
If, instead, we define the semantics in one place regardless of what
versions of the protocol use them, the other versions only have to
make normative references to Semantics.

Cheers,

....Roy

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 7 Feb 2021 19:55:48 +1300

On 7/02/21 2:40 pm, Roy T. Fielding wrote:
>> On Jan 31, 2021, at 7:50 PM, Martin Thomson wrote:
>>
>> This feature has not been implemented and deployed.  Worse, this 
>> feature *cannot* be implemented or deployed as there is no version of 
>> this protocol, or even an extension to the protocol, that allows this 
>> feature to be implemented.  Every current version either does not 
>> support trailers (0.9, 1.0) or only supports precisely one trailer 
>> section (1.1 with chunked encoding, 2, 3).
>>
>> I appreciate the design as a rational, well-constructed thing.  It is 
>> very crisply defined.  However, I don't think that it has any place in 
>> a document that purports to document an interoperable standard.
> 
> https://docs.google.com/document/d/1DsPjl6qWxhcF28vzJey0xQJ2O2eRKHv7CBXt3EZyDjc/ 
> <https://docs.google.com/document/d/1DsPjl6qWxhcF28vzJey0xQJ2O2eRKHv7CBXt3EZyDjc/>
> 

FTR, Squid core protocol parser has also been refactored over the past 
few years to handle this type of flexible framing. We use it only for 
ICAP at present, but can be extended to future HTTP versions relatively 
easily.

So I support leaving this in.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Sun, 7 Feb 2021 08:45:05 +0100

On Sat, Feb 06, 2021 at 05:40:01PM -0800, Roy T. Fielding wrote:
> > The document describes trailers in a way that permits multiple field
> > sections to be sent, starting from the point after the end of the header
> > section following a final status code.
> 

I'd say that spontaneously I'm not fond of this because I haven't given
it much thought yet, but I do see how that could be useful in the future,
and I agree that semantics must be defined before there's any opportunity
for an extension, so it's better to keep this in I guess.

> > 2. Upgrade
> > 
> > Only HTTP/1.1 supports Upgrade in any meaningful sense.  It's possible that
> > HTTP/1.1 is the only version that could use Upgrade in that way.  The way
> > that this mechanism is designed essentially depends on a paradigm that is
> > unique to that version.  I don't think that this is a big deal, but I would
> > prefer to see Sections 7.8, 15.2.2, 15.5.22, and 16.7 moved to the
> > messaging document.  As far as I can tell, these sections are the only ones
> > to mention anything related to Upgrade (aside from listings in the IANA
> > considerations), so my guess is that this would not be especially
> > disruptive a move.  (As these are trailing sections, it probably does not
> > even affect inbound citations to this document, thinking about QUIC.)
> 
> brittle settings.

Upgrade is a difficult topic because it's actually a declaration of a
drop of semantics. If it was seen as a downgrade rather than an upgrade
it would be clearer (i.e. drop of HTTP semantics).

In haproxy, we're using "multiplexers" to implement the transport part
of the protocol, and we have a version-agnostic HTTP semantic layer
between both ends. We support Upgrade for HTTP/1.1, CONNECT for HTTP/1
and HTTP/2, and map any of them to a "tunnel" in the middle, this
tunnel being associated with an optional protocol (which is how we
can map H2's CONNECT to H1's Upgrade for RFC8441, and conversely). For
me it's pretty clear that Upgrade as it is used is pretty much HTTP/1.1-
specific, there's even the "Connection: Upgrade" requirement. However
the notion of upgraded stream or connection is more generic. I think
that the Upgrade header is to the upgraded protocol what the Host
header is to the authority: just a protocol-specific way to convey it.
But as long as it's possible to reliably negotiate the protocol to
switch to and drop the HTTP semantics, I think various methods are
interoperable and the way we name them or document them will not make
much a difference, since historically at least these two solutions
have existed anyway.

Cheers,
Willy

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 08 Feb 2021 11:53:24 +1100

Leaving upgrade aside, though I think that is equally speculative, because it's more or less what HTTP has always said about upgrading...

> because we haven't provided a framework in which to do so.

This is exactly the sort of discussion we might have if someone were proposing an extension to the protocol.  We can probably agree to try that out.

But this is - otherwise - documenting a protocol that exists.  I don't accept that we need to document this extension to the core semantics in this way.  We are empowered to provide a new document that defines these semantics as an extension to the core semantics.  The same document might define METADATA frames or chunk extensions whatever else is needed to get the thing tested out.

I don't think that the core semantics document is the place to conduct that experiment.  That is all.