Subject: Cache control in trailers?

From: Mark Nottingham <[email-masked]>
Date: Wed, 3 Feb 2021 11:40:41 +1100

Hi all,

One of the fundamental limitations of HTTP that people often bump into is that the status code and headers precede the body,[1] so if something goes wrong while the response is being generated, they can get into awkward situations.

For example, if server-side script has made the response cacheable and encounters an unrecoverable error while generating it, that response will get stored and reused by downstream systems.

Right now, the only ways for it to avoid this is (a) buffer the whole response, (b) don't ever make the response cacheable, in case an error occurs, or (c) purposefully truncate the response (i.e., cause a protocol error), hoping that all downstream caches correctly avoid storing incomplete responses.

None of these seem like great options.

One better way would be to indicate the error in trailers. This has been mused upon before, e.g., allowing the status code to occur in trailers -- but that is likely unworkable, because all of the potential effects of a different status code can't be accounted for when trailers are received.

A more limited approach would be to focus just on the cache's behaviour -- e.g., to allow Cache-Control: no-cache in trailers,[2] updating the semantics of the response to make sure that it's revalidated before it's reused.

What do folks think - would this be useful? Obviously it would need to be implemented in browsers and other caches.

Cheers,

1. Or is that 'content' now?
2. Note that we don't have to reuse CC: no-cache; it could be that a new trailer communicates what's intended more clearly.

--
Mark Nottingham   https://www.mnot.net/

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 03 Feb 2021 01:07:50 +0000

I would prefer some other trailer that indicates that there was a 
problem and that the response should be discarded.

---

From: Ben Schwartz <[email-masked]>
Date: Tue, 2 Feb 2021 20:08:05 -0500

If adding trailers requires the body to be well-formed, then this would
seem to preclude option (c), so this could not be used to indicate an
error.  It would only apply to cases where the body is correct and usable
but less cacheable than expected, which seems like a very narrow use case.

On Tue, Feb 2, 2021 at 7:43 PM Mark Nottingham <[email-masked]> wrote:

> Hi all,
>
> One of the fundamental limitations of HTTP that people often bump into is
> that the status code and headers precede the body,[1] so if something goes
> wrong while the response is being generated, they can get into awkward
> situations.
>
> For example, if server-side script has made the response cacheable and
> encounters an unrecoverable error while generating it, that response will
> get stored and reused by downstream systems.
>
> Right now, the only ways for it to avoid this is (a) buffer the whole
> response, (b) don't ever make the response cacheable, in case an error
> occurs, or (c) purposefully truncate the response (i.e., cause a protocol
> error), hoping that all downstream caches correctly avoid storing
> incomplete responses.
>
> One better way would be to indicate the error in trailers. This has been
> mused upon before, e.g., allowing the status code to occur in trailers --
> but that is likely unworkable, because all of the potential effects of a
> different status code can't be accounted for when trailers are received.
>
> A more limited approach would be to focus just on the cache's behaviour --
> e.g., to allow Cache-Control: no-cache in trailers,[2] updating the
> semantics of the response to make sure that it's revalidated before it's
> reused.
>
> What do folks think - would this be useful? Obviously it would need to be
> implemented in browsers and other caches.
>
> Cheers,
>
> 2. Note that we don't have to reuse CC: no-cache; it could be that a new
> trailer communicates what's intended more clearly.
>
> --

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 03 Feb 2021 12:17:30 +1100

In my experience, these dynamic creation processes use chunked encoding, so you can have a well-formed (from an HTTP sense) body.  The content might be utter garbage though, despite what the headers might have said.

The usual way of dealing with this is buffering the entire response, which is obviously unpleasant.  In h2 or h3, stream resets are probably the best option.  In h1, tearing the connection down also seems preferable.

Mark, would you expect a client to signal its willingness to respect this signal?  I'm thinking that is more important a question to answer than whether to use Cache-Control.

On Wed, Feb 3, 2021, at 12:08, Ben Schwartz wrote:
> error.  It would only apply to cases where the body is correct and 
> usable but less cacheable than expected, which seems like a very narrow 
> use case.
> 
> Attachments:
> * smime.p7s

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 3 Feb 2021 12:37:45 +1100

> On 3 Feb 2021, at 12:17 pm, Martin Thomson <[email-masked]> wrote:
> 

... or the content could be 'sorry, an error happened' -- but with a long cache policy.

> The usual way of dealing with this is buffering the entire response, which is obviously unpleasant.  In h2 or h3, stream resets are probably the best option.  In h1, tearing the connection down als…

I think the use case is 'I want to convey a well-formed body that reflects the error state as best I can, and have the client show it to the user, but I changed my mind on the caching policy due to the aforementioned error.'

Stream resets and tearing the connection down *might* work, but they're not great solutions, because they can also indicate other problems, and some clients might act differently.

> Mark, would you expect a client to signal its willingness to respect this signal?  I'm thinking that is more important a question to answer than whether to use Cache-Control.

I think requiring a signal would make this impractical; no one is going to decide to buffer because they don't get the signal.

I think this could be deployed incrementally; e.g., a CDN could support it so as to limit the damage to other clients. Once enough browsers support it, it would probably be good enough for general use -- which makes browser implementation important.

Since I mentioned CDNs, it occurs that if we adopt CDN-Cache-Control, it'd be useful to have a discussion about whether making *any* cache-control header in a response such a signal is the right approach, or whether having a separate header as a signal is better. I do think that confining the signal to its effects on caches is a better approach than a general signal, however.

If the signal is Cache-Control, it's also worth discussing whether there's a viable use for other directives -- e.g., updating max-age after you send the response body. For content aggregation tools like ESI (and more modern successors, which there seems to be a lot of interest in, thanks to edge computing), I suspect this could be helpful. Again, it's much more useful if it gets into browsers.

Cheers,

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 03 Feb 2021 13:57:26 +1100

On Wed, Feb 3, 2021, at 12:37, Mark Nottingham wrote:
> I think the use case is 'I want to convey a well-formed body that 
> reflects the error state as best I can, and have the client show it to 
> the user, but I changed my mind on the caching policy due to the 
> aforementioned error.'

Why then would this be limited to caching policies?  That suggests you might have a different content-type then.  More seriously though, this suggests that you are reneging on the status code.
 
> Stream resets and tearing the connection down *might* work, but they're 
> not great solutions, because they can also indicate other problems, and 
> some clients might act differently.

I tend to think that this is an easier path to take than a retcon.  You are thinking caching, which is pretty serious, but I think that you are really talking about the status code here and for that I think that resetting streams (or connections) is the best we can reliably do.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 05:25:23 +0100

Hi Mark,

On Wed, Feb 03, 2021 at 11:40:41AM +1100, Mark Nottingham wrote:
> e.g., to allow Cache-Control: no-cache in trailers,[2] updating the semantics
> of the response to make sure that it's revalidated before it's reused.
> 

I think there could be some value in this, but please let's *not* reuse
the same trailer name then. In my experience the vast majority of header
producers have no idea what they do, and they just assemble a collection
of header values+names gathered from Stackoverflow and Wikipedia as a way
to "increase the likelihood this will work". Starting to send the signal
that such an important header field could be placed in either the header
or the trailer part will quickly result in anything important being sent
in both parts "just in case", and I guess none of us wants to have to
deal with that.

But maybe working on the notion of a final status, or maybe an abort
cause could prove useful over the long term. These could be used to
indicate a change of decision regarding caching, deliver a user-friendly
excuse message (e.g. "communication error with origin server"), timing
information (e.g. "processing time 10 ms, transfer time 143 ms"), a
content checksum etc.

Willy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 3 Feb 2021 18:38:50 +1100

> On 3 Feb 2021, at 3:25 pm, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Mark,
> 

I think that's reasonable. 

> information (e.g. "processing time 10 ms, transfer time 143 ms"), a

What I'm trying to avoid is defining something amorphous and unclear about its effects -- such as setting a new status code would do. By limiting its effects to the cache, it's crisp and clear about what should happen. If we make it general, it gets very tricky very quickly.

Of course, that doesn't prevent us from introducing other signals later on, or even adding new behaviours to a signal later.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 08:51:43 +0100

On Wed, Feb 03, 2021 at 06:38:50PM +1100, Mark Nottingham wrote:
> What I'm trying to avoid is defining something amorphous and unclear about
> its effects -- such as setting a new status code would do. By limiting its
> effects to the cache, it's crisp and clear about what should happen. If we
> make it general, it gets very tricky very quickly.

I generally agree with this approach for the same reasons. I tend to think
however that it could be a very good example of how to make intelligent use
of trailers and possibly be extended or replicated in the future for similar
use cases.

I'm also thinking about something, we should not expect caches to follow
trailers for a while, so in order to put some incentive there, we should
have a two-phase response. For example, the cache would pass the request
with "TE: new-cache-header-name", and the server would then emit a
preliminary cache-control with a "no-cache; trailers" or some such value
so that non-compatible caches don't blindly cache the content, but that
compatible caches notice this "trailers" token indicating that the final
verdict will be in the trailers. The compatible cache can then start to
store the response, and decide to commit or destroy it when seeing the
trailer. Others will simply not cache it so a truncated or bogus response
wouldn't be stored anywhere by accident. At least it sounds reasonable to
me like this and I think I'm seeing how this could be implemented without
too much effort, thus encouraging adoption :-)

Willy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 3 Feb 2021 09:09:16 +0100

Am 03.02.2021 um 03:57 schrieb Martin Thomson:
> ...

Right. While I understand why we might want to avoid touching the status
code, this is really what this is about. In particular, an HTTP client
that doesn't cache at all (and thus is not interested in Cache-Control
in the first place) might want to handle this as well.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 08:22:35 +0000

--------
Mark Nottingham writes:

> One of the fundamental limitations of HTTP that people often bump into 
> is that the status code and headers precede the body,[1] so if something 
> goes wrong while the response is being generated, they can get into 
> awkward situations.

...and having an ambiguity between headers and trailers will make for even more awkward situations.

As far as I can tell, allowing trailers to change anything fundamental about the object just shifts the need to buffer the entire object to the next step in the food-chain.

But that can still be a significant improvement, because production and transmission overlaps.

If we want enable that, we should move *all* headers and status to the trailer, so that no confusion is possible.

The way to do that is to define a new 1xx status code, which says the body comes first and should not be touched until trailers tell what it is.

Since there is no credible use-case where the content-length is already known, for HTTP/1.1 chunked encoding can be implied by the 1xx code.

So it could look something like this:

	HTTP/1.1 142 Wait For Trailers<CRNL>
	<CRNL>
	<chunked length><CRNL>
	chunk<CRNL>
	[...]
	<chunked length><CRNL>
	chunk<CRNL>
	0<CRNL>
	HTTP/1.1 200 OK<CRNL>
	Content-Length: 34242<CRNL>
	Cache-Control: blabla<CRNL>
	[...]
	<CRNL>

If we want to make it more robust against misimplementions we move the status line last:

	HTTP/1.1 142 Wait For Trailers<CRNL>
	<CRNL>
	<chunked length><CRNL>
	chunk<CRNL>
	[...]
	<chunked length><CRNL>
	chunk<CRNL>
	0<CRNL>
	Content-Length: 34242<CRNL>
	Cache-Control: blabla<CRNL>
	[...]
	HTTP/1.1 200 OK<CRNL>
	<CRNL>

Finally, we should give senders a way to prevent or at least seriously discourage receivers for peeking until the trailers have arrived: Let the sender XOR the body with a random number, to be disclosed in a trailer-only field.

The final bit is for requestors to indicate that they grok 142, I would be willing to abuse Accept-Encoding for that:

	Accept-Encoding: gzip, brotli, 142

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 03 Feb 2021 19:49:44 +1100

On Wed, Feb 3, 2021, at 19:22, Poul-Henning Kamp wrote:
> If we want enable that, we should move *all* headers and status to the 
> trailer, so that no confusion is possible.

Oddly, I agree, though this comes with a whole lot of problems with message processing at intermediaries and whatnot.

> The final bit is for requestors to indicate that they grok 142, I would 
> be willing to abuse Accept-Encoding for that:
> 

Isn't this sort of thing what TE was originally designed for?

---

From: Julian Reschke <[email-masked]>
Date: Wed, 3 Feb 2021 10:26:48 +0100

> Since there is no credible use-case where the content-length is already known, for HTTP/1.1 chunked encoding can be implied by the 1xx code.

1xx messages can not have a message body.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 09:44:10 +0000

--------
> Oddly, I agree, though this comes with a whole lot of problems with message processing at intermediaries and whatnot.

I'm not sure I see any _major_ problems.

If the client has indicated it can do 142, a non-policy intermediary can just pass stuff along, and that realizes the full production/transmission overlap.

Once the trailers arrive, the intermediary can decide if it wants to cache the object for future requests.

A policy intermediary will need to buffer the entire response before it can do its politiking (unless out-of-band information has given it other options.)

Usually that would be a CDN close to the eventual consumer or a corporate smut-filter, so almost the entire overlap will still be realized.

But yes, there will be implementation details.

> Isn't this sort of thing what TE was originally designed for?

Good point.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 09:48:20 +0000

--------

> 1xx messages can not have a message body.

The message body does not belong to the 142, it belongs to the trailer.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 3 Feb 2021 10:53:51 +0100

> The message body does not belong to the 142, it belongs to the trailer.

Trailers in HTTP/1.1 require chunked encoding, thus a message body. You
can't send a message body absent a final status code message.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 09:59:31 +0000

--------
> can't send a message body absent a final status code message.

This is not "Trailers in HTTP/1.1", so that is not relevant.

This is a new HTTP extension which makes it possible to transmit a response with body and header parts swapped.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 3 Feb 2021 11:07:16 +0100

> This is a new HTTP extension which makes it possible to transmit a response with body and header parts swapped.

I still don't get how you deploy it in HTTP/1.1.

Is your point that you can break compat with HTTP/1.1 messaging
requirements by negotiating this?

Best regards, Julian

---

From: "Martin Thomson" <[email-masked]>
Date: Wed, 03 Feb 2021 21:11:22 +1100

On Wed, Feb 3, 2021, at 20:59, Poul-Henning Kamp wrote:
> This is a new HTTP extension which makes it possible to transmit a 
> response with body and header parts swapped.

What seems more likely to work, without needing to change message framing, is a 3xx status code, indicating that the final response is found later in this message, a chunked body, and a trailer containing a new field that lists the final status code.

That would pass intermediaries, as it follows total message framing. Transposition could be a problem if you really mean to move the control data.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 11:17:03 +0100

On Wed, Feb 03, 2021 at 09:44:10AM +0000, Poul-Henning Kamp wrote:
> If the client has indicated it can do 142, a non-policy intermediary can just
> pass stuff along, and that realizes the full production/transmission overlap.

Please, look at what I proposed before, consisting in letting the origin
server mark the response as non-cacheable first, and announcing that this
might change in the trailers. Non-compatible intermediaries will simply not
cache, compatible ones will be able to decide at the end. We could imagine
supporting more options like short duration by default, long duration once
complete, etc. No need for abusing every corner case and making sure to break
every single intermediary in the middle.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 10:18:48 +0000

--------
Julian Reschke writes:
> >>> The message body does not belong to the 142, it belongs to the trailer=
> .
> >>
> > This is a new HTTP extension which makes it possible to transmit a respo=
> nse with body and header parts swapped.
>

*Only* if a requestor lights whatever "bat-signal" we choose, can the responder send a 142, the chunked body and the headers, in that order.

The "bat-signal" should be designed to not pass through intermediaries which do not understand its significance.  (ie: hop-by-hop header,
mentioned in Connection:)

---

From: Julian Reschke <[email-masked]>
Date: Wed, 3 Feb 2021 11:25:16 +0100

> mentioned in Connection:)

Well.

What you are doing here is an incompatible change of the HTTP/1.1 wire
format (yes, via opt-in). I don't think it's a good idea to deviate from
the wire format and still call the version "1.1" (or "1.x" for that matter).

Furhermore I really do not get why a change in wire format is *needed*.
As Martin said, we can mint a new final status code that indicates that
more status information will be in trailer fields.

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 10:28:52 +0000

--------
Martin Thomson writes:
> What seems more likely to work, without needing to change message framing, is a 3xx status code, indicating that the final response is found later in this message, a 
> chunked body, and a trailer containing a new field that lists the final status code.

I would be fine with 3xx, and that is maybe indeed be a better idea, because it could still allow 'transmission-optimizing' headers, like Marks proposed "response will be approx X MB" to be used.

They crucial part is that "content metadata", content-type, cache-control and all those, live /either/ in the headers /or/ the trailers, but /never/ in both, not even in an "advisory capacity".

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 10:42:17 +0000

--------
Willy Tarreau writes:
> might change in the trailers.

I did, that is actually what gave me idea for my proposal :-)

The problem with trailers has always been that people get hung up over the marginal potential speedups if we allow things to "change later".

But no software in the HTTP ecosystem is prepared for things to "change later".

And therefore trailers still languish, 25 years after they were first proposed.

My suggestion is to finally break the deadlock by nixing the "change things later" aspect once an for all.

Yes, that leaves some slivers of optimization unrealized.

But would finally allow us to lay

	1.  "What if production goes wrong half way?"

and

	2.  "Why cant we (safely) produce and transmit concurrently?".

to bed.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 10:44:07 +0000

--------

> more status information will be in trailer fields.

And as is said, if 3xx works, I'm fine with that.  142 was just my first mockup.

The crucial thing is that if there are trailers there will be no (metadata) headers.

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 3 Feb 2021 11:46:12 +0100

> semantics of the response to make sure that it's revalidated before it's

The problem with putting any kind of information like that in trailers is
that you have to correctly end the body so that you can add any trailers at
all.   This risks telling existing clients/caches that the body is complete
and good.  So this feature does not work well if not implemented.

Currently if a server has a problem whilst generating a response after the
headers have been committed it does have several options for telling the
client/intermediary it is abnormal:

   1. not sending enough bytes if a content-length has been sent
   2. not sending a terminal chunk if chunking transfer encoding is used.
   3. reset stream in HTTP >= 2

To put any error information in a normal trailer cannot be done in 1), and
we invalidate cases 2) and 3)

If you want to not break existing clients/intermediaries then you need a
new feature to terminate a message abnormally, but with metadata saying
why.   I think HTTP/2 already gives this in reset (at least to some
extent), although you can send arbitrary trailers with a reset.

For HTTP/1, you would need to have a way to abnormally terminate the
response AND then provide trailers.

-- 
Greg Wilkins <[email-masked]> CTO http://webtide.com

---

From: Greg Wilkins <[email-masked]>
Date: Wed, 3 Feb 2021 11:56:11 +0100

> response AND then provide trailers.

Which I think the 3xx approach would give, as the trailers could include a
5xx final status code.   Existing clients/intermediaries would then not see
a valid 200 response with some trailers they ignore, but which told them
the response was not valid!

cheers

---

From: Julian Reschke <[email-masked]>
Date: Wed, 3 Feb 2021 11:58:22 +0100

Am 03.02.2021 um 11:46 schrieb Greg Wilkins:
>
>
> On Wed, 3 Feb 2021 at 01:43, Mark Nottingham <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     A more limited approach would be to focus just on the cache's
>     behaviour -- e.g., to allow Cache-Control: no-cache in trailers,[2]
>     updating the semantics of the response to make sure that it's
>     revalidated before it's reused.
>
>
> The problem with putting any kind of information like that in trailers
> is that you have to correctly end the body so that you can add any
> trailers at all.   This risks telling existing clients/caches that the
> body is complete and good.  So this feature does not work well if not
> implemented.
>
> Currently if a server has a problem whilst generating a response after
> the headers have been committed it does have several options for telling
> the client/intermediary it is abnormal:
>
> To put any error information in a normal trailer cannot be done in 1),
> and we invalidate cases 2) and 3)
>

Chunk extensions come to mind (in theory we could send those and then
still abort the chunked response later on).

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 12:24:47 +0100

On Wed, Feb 03, 2021 at 10:18:48AM +0000, Poul-Henning Kamp wrote:
> *Only* if a requestor lights whatever "bat-signal" we choose, can the
> responder send a 142, the chunked body and the headers, in that order.

Pity, no, please don't start to break all the representation. It's already
hard enough to make sure a stack works well for good cases and corner cases,
please let's not start to mix headers and body, or it will be a total mess.

1xx has no body and there are sufficient equipements in the wild relying
on that strong promise so that it does not change. And headers are unsuitable
for stream data for the essential reason that they usually need to be
bufferred and processed as a whole at once.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 12:30:10 +0100

On Wed, Feb 03, 2021 at 10:42:17AM +0000, Poul-Henning Kamp wrote:
> The problem with trailers has always been that people get hung up over the
> marginal potential speedups if we allow things to "change later".

I'm not sure I understand what you mean. What "marginal potential speedups"
are you talking about ?

> But no software in the HTTP ecosystem is prepared for things to "change

Well, gRPC for example is a counter example, it *wants* a trailer for the
status.

I think the ecosystem doesn't exploit trailers that much because trailers
are not much used nor produced, but I think most implementations by now at
least parse them, and that extending them a little bit to support reading
a set of more optimal caching instructions wouldn't be that hard.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 11:41:13 +0000

--------
Willy Tarreau writes:
> I'm not sure I understand what you mean. What "marginal potential speedups" are you talking about ?

For instance having one Cache-Control in the headers and then wanting to modify that with a new one in Trailers ?

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 13:17:23 +0100

> modify that with a new one in Trailers ?

But it's not about a marginal speed up here, it's about a server not being
certain whether it wants its data to be cached or not until they're complete
(think about data being assembled from multiple web service requests), and
preferring to first ask that this response is *not* cached as-is until it
can confirm it was entirely successful with a trailer. I think this totally
makes sense.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 12:22:54 +0000

--------
> certain whether it wants its data to be cached or not until they're complete

And that's fine!

What I'm proposing is making it possible for the server to say "Please grab
this body first, then I'll tell you what it means afterwards."

The marginal speedup we loose because the recipient cannot act on
the body until they see the trailers is regrettable, but we stil
reap the overlap of production and transmission.

The risk of getting two different Cache-Control headers is pretty much precisly why trailers are not being used.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 13:32:29 +0100

> reap the overlap of production and transmission.

It's not just a loss of speed up, but the inability to reject it or
process it. You can't even know how it's encoded, if you need to
decompress it on the fly.

> The risk of getting two different Cache-Control headers is pretty much
> precisly why trailers are not being used.

Note that I purposely asked that we do *not* use the same header name,
otherwise I agree it will be a total mess. If we'd send a
"Cache-Post-Body-Status" at the end that would only be used when
preliminarily advertised in cache-control, that would be safe.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 12:38:36 +0000

--------
> decompress it on the fly.

Yup, and that's too bad.

Today the server has to buffer the body it before it sends the headers,
with my proposal we can at least overlap the transmission and the production.

> Note that I purposely asked that we do *not* use the same header name,

And I'm saying:  Just dont send the preliminary cache-control to begin with:  It's not worth it.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 14:05:38 +0100

> Today the server has to buffer the body it before it sends the headers,

Not if it uses chunked encoding, which was precisely made for this.

> And I'm saying:  Just dont send the preliminary cache-control to begin with:  It's not worth it.

I get your point, but it might make sense to make sure the response
is *not* cached by opportunistic caches on the path if the goal is to
ensure that only totally valid responses are ever cached.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 14:40:27 +0000

--------

> Not if it uses chunked encoding, which was precisely made for this.

Except it doesn't work, because nobody implements trailers - because they dont want to deal with the complexity of merging headers and trailers.

Also, chunked doesn't help you any with the "Oops, production crapped out half way through..." scenario, where you want to make it a 50x instead of 2xx.

> ensure that only totally valid responses are ever cached.

Yes, absolutely.

That's why my initial mockup was structured to make sure such caches wouldn't even consider the response valid, should they by mistake get one.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 16:10:01 +0100

On Wed, Feb 03, 2021 at 02:40:27PM +0000, Poul-Henning Kamp wrote:
> Except it doesn't work, because nobody implements trailers - because they
> dont want to deal with the complexity of merging headers and trailers.

But why do you say "nobody" after I gave an example (gRPC). You don't
necessarily *need* to merge them, just like some headers are forbidden
in trailers, we can very well specify that some fields are only relevant
in trailers. FWIW apparently gRPC does this as I seem to have read
somewhere that a grpc-status trailer is mandatory and that as a corollary,
you may never see an H2 data frame carrying the ES flag.

> Also, chunked doesn't help you any with the "Oops, production crapped out
> half way through..." scenario, where you want to make it a 50x instead of
> 2xx.

It depends what we want to do. Mark mentioned he wanted to make sure
this truncated response is not cached. Just emitting a preliminary
cache-control disabling caching does serve this purpose.

> That's why my initial mockup was structured to make sure such caches wouldn't
> even consider the response valid, should they by mistake get one.

The problem is that even a 100% compliant intermediary will have great
difficulties parsing a body on a 1xx :-/

Willy

---

From: James M Snell <[email-masked]>
Date: Wed, 3 Feb 2021 07:32:59 -0800

One of the likely challenges here is that if a failure happens during
the transmission of the response the server may not even be able to
add a trailer.

Just musing on a slightly different approach... what if we could have
a Cache-Control directive that says, "Only cache if a given trailer is
received (signaling success)".

In the response headers we'd have...

  Cache-Control: cache-on-trailer

Then, only at the end of a successfully transmitted response we'd have
the trailer:

  Cache-Control: cache-commit

The assumption here would be that if the commit trailer is not
received the assumption is that the response should not be cached.

Sorry if this is a repeat or rehash if it's already been discussed, I
haven't had the time to go through the discussion thread.

- James

---

From: Ryan Sleevi <[email-masked]>
Date: Wed, 3 Feb 2021 12:38:22 -0500

> you may never see an H2 data frame carrying the ES flag.

Do you have other examples? To say gRPC holds HTTP “weird” is a bit of an
understatement, in terms of resources and semantics. That is, while it
overlays on the wire protocol, the semantics themselves of gRPC’s
implementation don’t really align nicely with what folks might reasonably
think about HTTP with respect to resources, methods, headers, etc.

This is particularly important when considering that there really isn’t
anywhere near the level of intermediation for gRPC as there is for
“HTTP”/“the Web”. The intermediaries that do exist aren’t agnostic; they
understand the semantics of gRPC and are often gRPC implementations
themselves.

I do have to agree with PHK here: this sort of merging is a state machine
security nightmare, especially when thinking about the interaction with
resources, caching, and the overall semantics of HTTP. It’s exactly this
risk that has discouraged Chrome from exposing trailers, either in our
internal APIs for dealing with resources or in web-developer facing APIs.
Yes, we support it at the wireframe level in our H/2 implementation, but
that’s intentionally not surfaced beyond that. This does mean, for example,
that developers working on Google Chrome are (intentionally) not able to
use gRPC within Chrome: the complexity in implementation and security is
presently not justified.

Admittedly, (DAY-JOB) is primarily dealing in edge cases and weird states
in state machines, so I’m probably over indexed on those concerns, but I
wouldn’t be terribly excited for trailers precisely because it necessitates
careful re-review of every state machine, end to end, to make sure new
issues and surprises aren’t introduced by such semantics. So even if, in
the abstract, it’s good and useful, that sort of complexity may preclude
implementation.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 17:44:09 +0000

--------
Ryan Sleevi writes:

> in state machines, so I'm probably over indexed on those concerns, but I
> wouldn't be terribly excited for trailers precisely because it necessitates
> careful re-review of every state machine, end to end, 

Do you also feel this will be necessary if we add an extension which
allows sending the body before the headers and *only* that ?

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 19:17:37 +0100

On Wed, Feb 03, 2021 at 12:38:22PM -0500, Ryan Sleevi wrote:
> Do you have other examples?

Examples of what, trailers in real use ? I seldom see them in user reports.
The first time I encountered them was about 10 years ago, when haproxy was
not able to correctly forward them and caused trouble with a home-grown
enterprise application. I've seen one application emit signatures of
transfers and another one, maybe two, sending debugging information (timing,
request ID or call trace). I'm seeing here that Fastly makes use of them to
pass timing information:

  https://www.fastly.com/blog/supercharging-server-timing-http-trailers

based on this draft:

  https://www.w3.org/TR/server-timing/

But recently each time I heard about them it was about gRPC.

> To say gRPC holds HTTP "weird" is a bit of an understatement, in terms of
> resources and semantics. That is, while it overlays on the wire protocol,
> the semantics themselves of gRPC's implementation don't really align nicely
> with what folks might reasonably think about HTTP with respect to resources,
> methods, headers, etc.

To be honest, I don't know much of gRPC. I remember looking at the
requirements a while ago when implementing the server-side H2 part in
haproxy, making sure it was compatible (since it was essentially the
main requested use case), and that was all. Right now users appear fine
with it except for some random things like "the client complains about
missing trailer on an error response produced by the proxy" (obviously
since it's H2, we don't have to emit random trailers after an error),
and cancellation which is not easy to deal with.

> This is particularly important when considering that there really isn't
> "HTTP"/"the Web". The intermediaries that do exist aren't agnostic; they

It's inexact, with haproxy we're an HTTP proxy with H1 or H2 on each side,
and gRPC appears to pass correctly over it.

> resources, caching, and the overall semantics of HTTP.

For me, the main ennemy of trailers is the fact that they were considered
as the part of the same namespace as headers, which is what causes this
merging nightmare. But just like some fields are forbidden in trailers, we
could state that they ought to be ignored in headers (in case of merging
by an intermediary), and that's why I really do not want to see both use
the same name. Once you stick to this, there's no merging nightmare nor
security issue anymore: if the trailer is present, you know what to cache.
If it's absent (and being in the header is counted as absent), you don't
cache, period. It's only suboptimal. But suboptimality is what encourages
improvements in products. Breakage encourages fragmentation.

> It's exactly this
> that's intentionally not surfaced beyond that.

Then this sounds like a clean approach to me.

> This does mean, for example,

I agree.

> in state machines, so I'm probably over indexed on those concerns,

We probably have comparable jobs then and I agree that this quickly develops
some immediate reflexes against certain patterns.

> but I
> issues and surprises aren't introduced by such semantics. So even if, in
> the abstract, it's good and useful, that sort of complexity may preclude
> implementation.

What class of issue would you envision with a field which only has semantics
in trailers and which must be ignored in headers ? I mean, say the server
emits this:

    HTTP/1.1 200 OK
    Transfer-encoding: chunked
    Cache-control: no-cache; trailers
    
    b
    0123456789
    0
    Cache-Post-Body-Status: public; max-age=86400

It could be relayed as-is by compliant intermediaries. It could be relayed
like this by those compliant as well but which merge trailers and headers:

    HTTP/1.1 200 OK
    Transfer-encoding: chunked
    Cache-control: no-cache; trailers
    Cache-Post-Body-Status: public; max-age=86400
    
    b
    0123456789
    0

In this case the response is not cached. Transfer-encoding could even be
translated to content-length by the way, the principle remains. The trailer
could also be silently dropped on the path:

    HTTP/1.1 200 OK
    Transfer-encoding: chunked
    Cache-control: no-cache; trailers
    
    b
    0123456789
    0

It wouldn't be cached either.

Then there are those which choke on trailers because they stop after 0 CRLF,
they wouldn't cache either. What I like with this approach is that a degraded
message cannot be restored later in the chain to become correct again. This
wouldn't be the case if using the same field name in both parts.

On the opposite, adding an optional body after 1xx would make cache pollution
attacks trivial through plenty of components. Just upload an image containing
a dummy HTTP response to a server, download it again, and watch intermediaries
stop after the 1xx and parse the body as the final HTTP response, taking it
for the valid contents.

I think the approach is elegant to a certain extent, because it reuses some
protocol elements that have not changed in two decades. That doesn't mean
they're well implemented everywhere, but that their failure modes are more
or less known by now.

Willy

---

From: Ryan Sleevi <[email-masked]>
Date: Wed, 3 Feb 2021 14:00:23 -0500

> Examples of what, trailers in real use ? I seldom see them in user reports.
<snip>

Yes, although this matches my experiences as well.

Granted, we see a lot more poorly written intermediaries (which is the
longer way of spelling antivirus), and so perhaps it's not surprising that
trailers aren't as widely used when transiting multiple
networks/administrative zones.

<snip>

> improvements in products. Breakage encourages fragmentation.

I'm not sure this is entirely correct, in the example you describe below
(and more response there). That is, I think you've described a scenario
where they're still part of the same logical namespace, in practice, even
if they're not meant to be.

<snip>

>From an implementation state machine complexity, this ends up still being
messy. For example, in Chrome at least, we use a multi-process architecture
such that there is a network process, a browser process, and a renderer
process. The renderer process requests a resource from the network process,
and that result is then sent back via IPC through a fixed-size circular
buffer.

Using your example, the header says "don't cache", the trailer says "do
cache". Currently, Chrome's implementation uses the header-defined value to
determine whether it needs to "tee" (in the *nix sense) the response to
both the disk cache and the renderer process, or whether to send it
straight through to the renderer process. In order to ensure backpressure
is properly handled, if the disk IO of the cache activity slows, it
naturally slows the transmission rate to the renderer process. In effect,
we only have a fixed amount of memory in use by a resource at a time.

Using your example, the need to cache after the fact in the trailer
wouldn't be possible, not without buffering the entire response either in
memory or through some temporary file, in order to know what to do with the
response. The same challenge applies in the inverse (where the header says
cache and the trailer says don't cache), which would have us invalidating a
cache entry even though it wasn't necessary or appropriate for the
semantics. That's why I said above that it sounds like they're still in the
same namespace, in as much as it allows trailers to affect the semantics of
headers, and vice-versa, even if their naming isn't identical. This sort of
problem equally exists with the "body first, headers-as-trailers"
suggestion.

These aren't just "correctness" issues, but we view them as security
relevant, because they can allow for things like denial of service (via
resource exhaustion), or interesting attacks in cross-origin timing
scenarios (through probing of any dynamic limits used to mitigate the
exhaustion issue). From a server security standpoint, a number of services
draw security boundaries between "headers" (which should be
trusted/controlled by server admin) and "bodies" (which can be controlled
by hosted code/untrusted parties), and the introduction of "trailers" and
any semantics would have to try to preserve/respect those assumptions. That
is, trailers that redefine the semantics of headers, even if separately
named, would and could be abused, which equally makes trailer bits
unexciting.

I agree it's better for intermediates to use the semantics you described.
That feels similar to the set of mitigations adopted by HSTS/HPKP - namely,
that they only accepted the 'first' occurrence of a header and relied on
the headers not being mergeable, in order to try to preserve the separation
I mentioned above. But it doesn't feel terribly exciting from an
implementation perspective, and feels like it could easily lead to new
classes of security bugs if generally available.

This isn't so much a "hard oppose" to the use case; I think cases like gRPC
show that "novel" uses of the HTTP/2 framing can and do exist, and at a
protocol level, these can enable new and interesting cases. Even the Fastly
example makes general sense for why, and I'm not trying to dismiss that.
However, for the general resource semantics and loading, it feels like
there is the possibility of years of security surprises in store, and so
it's not a particularly exciting or likely thing to implement in a
general-purpose client that tries to be security-opinionated.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 3 Feb 2021 20:50:31 +0100

> if they're not meant to be.

They are by definition from the spec (see 7230#4.1.3 which even explains
how to merge them). What I mean is that by declaring the field's name and
its behavior in both parts we avoid any type of confusion for those who
don't know it and those who receive it molested.

> and that result is then sent back via IPC through a fixed-size circular

OK.

> we only have a fixed amount of memory in use by a resource at a time.

This makes sense.

> response.

But why ? I guess the vast majority of responses received by the browser
are currently cacheable, so how this one would make any difference ? The
condition to decide to tee would just differ, you'd tee if cache-control
says either "public, "private" or "trailers" for example, and never for
other responses like today. And when doing "trailers", you'd finally
decide to either commit the object at the end of the transfer or destroy
it.

Note that I'm not at all trying to infer how this works in details, and
I know we all have our technical limitations for historic reasons or
anthing, but I don't see how an object fetched with a conditional cache
statement would require more storage or disk bandwidth than one fetched
with an explicit or implicit cache statement. You could see it as a final
message saying "finally I'd prefer if you dropped what you've just fetched
from your cache" after a usual response.

> The same challenge applies in the inverse (where the header says
> cache and the trailer says don't cache),

We must not do this, because that would mean that non-compliant
intermediaries will be fooled.

> resource exhaustion),

Here I don't see how, as you don't cache more than what you normally
receive. You just know at the end of the transfer if it's worth keeping
what you just stored in your cache.

> From a server security standpoint, a number of services
> any semantics would have to try to preserve/respect those assumptions.

This remains true. Everything is in the header, only the final verdict
may happen in the trailer. Don't see it as an order for doing something,
rather as an opportunity. The Cache-Control header would say "I consider
it's worth caching this object, but if I encounter any issue, I'll tell
you at the end so that you can freely drop it". The other trailer says
"Object assembly went well enough, according to what I announced in
cache-control, I consider it worth keeping it in your cache if you want".

If you fiddle with cache-control in the middle, that's enough to void the
trailer's effect, so headers clearly are the ones defining everything
related to the semantics here.

> That

This is also why I don't want to use similar header names on the two sides.

> classes of security bugs if generally available.

I hardly see which ones if the header defines the rules and the trailer
only provides an operational status to let the client act as it prefers.

Regards,
Willy

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 03 Feb 2021 21:41:07 +0000

so any policy based on metadata will have to wait until the end to apply 
it.  This is not very workable in many cases.

plenty of people wish to block content by Content-Type, and plenty of 
streams are too big to buffer, and so proxies pass data through as it 
comes.

I think we need metadata up-front, at least the stuff that is immutable 
in order to implement policy based on it.

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 03 Feb 2021 21:44:47 +0000

+1

seems like a much more robust strategy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 21:48:40 +0000

--------
> it.  This is not very workable in many cases.

Well, if it doesn't work for them, they shouldn't ask for this transmission
mode then.

Since it will need to be, by definition, hop-by-hop, that's a nobrainer.

---

From: Ben Schwartz <[email-masked]>
Date: Wed, 3 Feb 2021 16:49:53 -0500

Restarting on a different tack.

> (c) purposefully truncate the response (i.e., cause a protocol error),
> hoping that all downstream caches correctly avoid storing incomplete
> responses.
>

Have you encountered caches that don't handle this correctly?

This may seem inelegant, but if it works in practice, I think we can codify
it into a respectable solution (e.g. forcing a particular kind of protocol
error).

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 03 Feb 2021 21:54:22 +0000

------ Original Message ------
From: "Poul-Henning Kamp" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Julian Reschke" <[email-masked]>; "[email-masked]" 
<[email-masked]>
Sent: 4/02/2021 10:48:40 am
Subject: Re: Cache control in trailers?

>Since it will need to be, by definition, hop-by-hop, that's a nobrainer.

why not design a solution that allows both requirements to work?  
Otherwise any intermediary that may want to enforce policy based on 
metadata will have to always down-grade the requests which makes the 
whole exercise less valuable.

---

From: Ian Swett <[email-masked]>
Date: Wed, 3 Feb 2021 16:57:30 -0500

In regards to the topic, I can see the usefulness, but all the options are
fairly unappealing to me, even without thinking about the impact of relying
more on trailers(which by itself is unappealing).

> Since it will need to be, by definition, hop-by-hop, that's a nobrainer.

In the past, I've been told headers are never really hop-by-hop, so I'm a
bit surprised to see you say this.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Wed, 03 Feb 2021 21:57:43 +0000

--------

> why not design a solution that allows both requirements to work?

Isn't that *precisely* what trailers have always been designed to do ?

> whole exercise less valuable.

Only for them.

The rest of us can still enjoy the speedup, which can be significant
in many API contexts.

---

From: Ryan Sleevi <[email-masked]>
Date: Wed, 3 Feb 2021 17:26:21 -0500

> from your cache" after a usual response.

The "commit the object at the end or destroy it" is precisely the issue,
however. It's a new edge in the state machine, in which an object in the
cache could be not-yet-committed, which then makes a number of assumptions
potentially problematic. I agree that there are many ways to potentially
tackle this, but I highlight it as an example of why it's not very
exciting, and thus unlikely to be implemented (just as trailers are not
supported in any of our layers beyond basic H/2 message framing, at which
point, they get dropped).

For example, consider a situation where there are multiple requests for the
same resource. Request 1 comes in, opens up the resource, begins reading
the resource while streaming the response into the file and to the caller.
Request 2 for that resource comes in, sees that it's available in the cache
(even though not fully completed), begins reading that response and
streaming to the other caller. In the semantics described, the need for an
explicit Commit stage prohibits this, in as much as it's not clear that
Request 2 can reuse that resource, because Request 1 might change its mind
at the end of the stream.

This is akin to the problem others have raised, with respect to policy
controls applied early-on/up-front.

Equally, consider a situation where there is a pre-existing resource in the
cache, and during opportunistic revalidation, a newer version is available.
If that Request decides at the end to not cache, we've now doomed an
otherwise usable resource, and need to commit to removing it from the cache.

> what you just stored in your cache.

Sorry, the resource exhaustion remark was with respect to internally
buffering the resource to determine whether or not to tee. While the
analogy isn't perfect, the situation is you can't tee > /dev/null and then
expect to copy that data back out to a file. And if you tee to a file, and
then have to move it to /dev/null at the end, you've equally consumed
resources in the intermediate steps (such as a set-aside pre-commit file).
Important policy related checks (such as total storage for an origin, and
invalidating entries to ensure sufficient space based on content-length)
now run at the end, after the resources have been committed, rather than
before.

Again, I don't doubt that there can and are possibilities of using this in
interesting ways. But the complexity trade-off is not particularly
compelling or exciting, both in terms of practical implementation, but then
also reasoning about protocol state machines, which is why we haven't
implemented/exposed trailers beyond process-frame-and-throwaway. I know for
certain it's made a number of folks sad, including the fine folks at
Fastly, because it does limit some cases.

And this is, of course, ignoring that for HTTP/1.1 we more or less consider
this doomed on the client because of how many broken intermediaries they
have to deal with :) H/2+ using encrypted connections at least have bought
us a little more flexibility here, and even though there are still bad
implementations of those, they're at least newer implementations, so the
mistakes aren't yet as firmly ossified or unnoticed :)

---

From: David Benjamin <[email-masked]>
Date: Wed, 3 Feb 2021 17:58:11 -0500

In addition to the state machine issues Ryan points out, I don't think the
original premise in focusing on just the cache's behavior holds.

Suppose a server has encountered an unrecoverable error generating a
resource. The cache must know for future requests, but the current request
also must know. Truncated and complete bodies are different. Truncated
downloads may be communicated to the user as failed. Truncated script and
style resources do not execute because doing so has security consequences.

Resetting the stream sends *exactly* the signal you need. Doing anything
else in the HTTP frontend is a bug, likely with security consequences.
Likewise, an HTTP cache failing to notice the reset is a bug, likely with
security consequences. If some HTTP cache is broken here, it'll need a code
change for Cache-Control trailers anyway. That code change is better spent
on fixing this bug.

On Wed, Feb 3, 2021 at 5:29 PM Ryan Sleevi <[email-masked]> wrote:

> For example, consider a situation where there are multiple requests for
> the same resource. Request 1 comes in, opens up the resource, begins
> reading the resource while streaming the response into the file and to the
> caller. Request 2 for that resource comes in, sees that it's available in
> the cache (even though not fully completed), begins reading that response
> and streaming to the other caller. In the semantics described, the need for
> an explicit Commit stage prohibits this, in as much as it's not clear that
> Equally, consider a situation where there is a pre-existing resource in
> the cache, and during opportunistic revalidation, a newer version is
> available. If that Request decides at the end to not cache, we've now
> doomed an otherwise usable resource, and need to commit to removing it from
> the cache.
>
>
> And this is, of course, ignoring that for HTTP/1.1 we more or less
> consider this doomed on the client because of how many broken
> intermediaries they have to deal with :) H/2+ using encrypted connections
> at least have bought us a little more flexibility here, and even though
> there are still bad implementations of those, they're at least newer
> implementations, so the mistakes aren't yet as firmly ossified or unnoticed
> :)
>

---

From: James M Snell <[email-masked]>
Date: Wed, 3 Feb 2021 15:29:03 -0800

On Wed, Feb 3, 2021 at 2:29 PM Ryan Sleevi <[email-masked]> wrote:
[snip]
>
> The "commit the object at the end or destroy it" is precisely the issue, however. It's a new edge in the state machine, in which an object in the cache could be not-yet-committed, which then makes a number of assumptions potentially problematic. I agree that there are many ways to potentially tackle this, but I highlight it as an example of why it's not very exciting, and thus unlikely to be implemented (just as trailers are not supported in any of our layers beyond basic H/2 message framing, at which point, they get dropped).
>
> For example, consider a situation where there are multiple requests for the same resource. Request 1 comes in, opens up the resource, begins reading the resource while streaming the response into the file and to the caller. Request 2 for that resource comes in, sees that it's available in the cache (even though not fully completed), begins reading that response and streaming to the other caller. In the semantics described, the need for an explicit Commit stage prohibits this, in as much as it's not clear that Request 2 can reuse that resource, because Request 1 might change its mind at the end of the stream.
>

Yes, but there are ways of handling that. If Response to Request 1
indicates a clear signal at the start that it's possible that the
response could be truncated and that it shouldn't be cached unless it
was successfully entirely received, then there are options for Request
2, with the only-if-cached request directive being the obvious blunt
force approach. But response 2 could also be served with either a
status code or a response cache control directive that clearly
indicates that it is pulling from an incomplete cached representation
that could end up failing or truncated, and it could include its own
trailer to indicate its own status accordingly.

> This is akin to the problem others have raised, with respect to policy controls applied early-on/up-front.
>
> Equally, consider a situation where there is a pre-existing resource in the cache, and during opportunistic revalidation, a newer version is available. If that Request decides at the end to not cache, we've now doomed an otherwise usable resource, and need to commit to removing it from the cache.
>

That needn't be the case. The existing resource in the cache can
remain until the new representation is committed, and can even be
served while the new resource is being received and the commit is
pending.

Either way you're right that this non-trivially adds to the state
machine which definitely brings along a number of non-trivial
complexities.

- James

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Feb 2021 11:01:39 +1100

Hey James,

> On 4 Feb 2021, at 2:32 am, James M Snell <[email-masked]> wrote:
> 

That's explicitly out of scope here -- HTTP already defines how to handle truncated / badly delimited responses.[1] This is really just 'I changed my mind about caching policy because something happened while I was generating the response.' Although, see below.

> received the assumption is that the response should not be cached.

That's definitely interesting. In my mind, the important thing to consider here is whether we think this mechanism is going to be used by content that's currently marked as cacheable or uncacheable; it may well be the latter, in which case this approach makes a lot of sense.

If we specify it really carefully, it might also be possible to define its interaction with other cache directives in a way that allows the origin to still control the behaviour of non-implementing caches.

Also, an extension taking this approach probably should define whether or not the commit is valid if the response is truncated; I'd assume it's not. It'd also need to talk about the relationship to collapsed requests,[2] as Ryan alludes to.

I might start working on a draft; would you like to co-author?

Cheers,

1. https://httpwg.org/http-core/draft-ietf-httpbis-cache-latest.html#incomplete.responses
2. https://httpwg.org/http-core/draft-ietf-httpbis-cache-latest.html#constructing.responses.from.caches

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 4 Feb 2021 11:04:13 +1100

Hi David,

> On 4 Feb 2021, at 9:58 am, David Benjamin <[email-masked]> wrote:
> 
> In addition to the state machine issues Ryan points out, I don't think the original premise in focusing on just the cache's behavior holds.
> 
> Suppose a server has encountered an unrecoverable error generating a resource. The cache must know for future requests, but the current request also must know. Truncated and complete bodies are different. Truncated downloads may be communicated to the user as failed. Truncated script and style resources do not execute because doing so has security consequences.
> 
> Resetting the stream sends exactly the signal you need. Doing anything else in the HTTP frontend is a bug, likely with security consequences. Likewise, an HTTP cache failing to notice the reset is a bug, likely with security consequences. If some HTTP cache is broken here, it'll need a code change for Cache-Control trailers anyway. That code change is better spent on fixing this bug.

That's explicitly not the case I'm trying to address here -- what I'm interested in is the case where there is a _recoverable_ error, but it isn't desirable to reuse the resulting response on the same terms as was optimistically thought when the headers were sent. Usually, this will be in generating HTML.

If an unrecoverable error is encountered, resetting the stream is indeed the right approach.

Cheers,

---

From: James M Snell <[email-masked]>
Date: Wed, 3 Feb 2021 16:11:01 -0800

I'd be happy to.

On Wed, Feb 3, 2021 at 4:01 PM Mark Nottingham <[email-masked]> wrote:
[snip]

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 4 Feb 2021 07:43:52 +0100

> I'd be happy to.

Please at least check the rest of this thread then, as there were similar
proposals using distinct names to avoid the issues caused by getting them
merged.

Willy

---

From: Stefan Eissing <[email-masked]>
Date: Thu, 4 Feb 2021 10:30:40 +0100

I so far like the proposal from Willy the most. Especially as it has defined semantics in presence of unaware, merging intermediates.

As to the "real life" use of trailers: in apache httpd I only ever saw issues raised with gRPC.

FTR: adding optional bodies to 1xx responses would break all sort of things. Not an option.

Cheers,
Stefan

> Am 03.02.2021 um 20:50 schrieb Willy Tarreau <[email-masked]>:
>

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 4 Feb 2021 10:47:13 +0100

On Thu, 4 Feb 2021 at 10:33, Stefan Eissing <[email-masked]>
wrote:

> I so far like the proposal from Willy the most. Especially as it has
> defined semantics in presence of unaware, merging intermediates.
>

It's really hard to work out exactly what proposals are on the table.  I
think everybody is not that far away, but arguing vigorously over little
differences.
There are several points I can see are:

   - General consensus that trailers are under utilized, but that there are
   various use cases for them and that defining a new mechanism may help with
   uptake.
   - There is a range of views on the scope: should there be a proposal
   just to make cache control possible in trailers or should it be a more
   general solution that includes status and other metadata.
   - Whatever the mechanism is, it should only be used if acceptable to all
   hops.  Not yet seen a concrete proposal for this.
   - Once the mechanism is used, there needs to be an indication at the
   start of the response that more info will be in the trailers. This can be
   done with a status (eg 3xx) or with 200 responses with new headers and/or
   header values.   I've not yet seen a proposal for using headers that
   addresses the problem of clients that don't know the new headers mistaking
   the 200 response as valid/complete.
   - There is a question of should the data only be in the trailers, or can
   it be sent in the headers and then changed in the trailers.
   - Once we get to trailers, will they use new headers or re-use already
   defined ones.

Then separately to how the mechanism indicates that there is more data in
the trailers, there is the need to well define the caching state machine
that will handle late and/or changed cache-control and/or status.

But while that list is moderately long, I don't really see any position
that far apart.     Perhaps a few more concrete drafts that address all
those concerns?

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 4 Feb 2021 11:04:52 +0100

On Thu, Feb 04, 2021 at 10:47:13AM +0100, Greg Wilkins wrote:
>    I've not yet seen a proposal for using headers that

For me, pre-announcing cache-control in the headers was targetting this
requirement: if you indicate the response is not cacheable until the
final verdict, this is no more a problem.

At least I'm seeing something positive in all this discussion, it emphasizes
the importance of distinguishing the term "field" from the term "header" :-)

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Feb 2021 10:53:35 +0000

--------

>    it be sent in the headers and then changed in the trailers.

The current definition of trailers, the one nobody has wanted to
touch with a ten feet pole for 25 years, has metadata in both headers
and trailers, and that is the primary, almost the sole reason, people
cite for not supporting trailers.

We can cling to that, for what seems at best marginal or even
speculative benefits, and trailers will continue to not be used,
or we can get rid of that, and they may stand a chance.

The downside is that no semantic processing can happen downstream,
until both the body and trailers have arrived, but that is still
significantly faster than having the server buffer until the response.
is complete.

>    -Whatever the mechanism is, it should only be used if acceptable to all

No matter what kind of "magic marker" we use to enable an extension,
there will be buggy-proxies out there which let the marker through,
yet do not understand what they just consented to handle.

Willy's proposal to strike a middle ground is sympathetic:  Formulate
responses in such a way that buggy implementations will handle them
safely, for instance by a boiler-plate "neutering" Cache-Control
in the headers.  Unfortunately that just postpones the trouble until
the buggy-proxy attempts the next transaction on the connection and
barf on the orphan trailers.

I can only imagine two mechanisms that can /eliminate/ buggy-proxies mangling things:

1.  Break the on-wire format enough that buggy-proxies will reject
    the response as broken.

2.  Scramble the body with a nonce only available in the trailers,
    so that clients will never see good data if the trailers did not survive.

There seems to be little appetite for number one, so maybe we should
talk about #2 ?

Here's a strawman:

	A new 3xx response code which means: "Everything is in the trailers."

	The allowed header fields:

		Transfer-Encoding: chunked
		Content-Length
		Connection

	(Optional:  Allow other headers but only if they are listed in the Connection header)

	The sender XOR scrambles the body with a N*64bit randomly chosen nonce.

	The nonce is disclosed in a "Trailer-Nonce" field (as RFC8941 Byte Sequence).

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 4 Feb 2021 14:11:46 +0100

> final verdict, this is no more a problem.

If the mechanism is only intended to deal with cache-control, then that is
probably sufficient.
However, I think the use-case of discovering after committing a response
that the content is no longer going to be cacheable yet can be completed in
a way that is still a valid response for the one request is vanishingly
small.

Alternatively, if deferred cache-control is only an exemplar of a more
general mechanism to signal that metadata fields are
expected in the trailer, then I don't think it is sufficient.  Something
like the 3xx proposal feels safest to avoid accidental interpretation of a
200 response.

I know Mark's proposed scope was just cache-control, but it's probably
worthwhile at least considering if a more general solution is possible.

cheers

---

From: Greg Wilkins <[email-masked]>
Date: Thu, 4 Feb 2021 15:28:45 +0100

>                 Transfer-Encoding: chunked

Why everything in the trailers?  Isn't it sufficient to say that
authoritative fields are in trailers and that any in the header should be
considered just hints.

Isn't entirely dependent on the fields themselves? Some will make sense to
only be in a header (eg Content-Length, Transfer-Encoing), some might only
be practical in a trailer (checksums), others can be in either place but
would be nonsensical to have them in both (eg Content-Type can't change mid
way through the content), others can be in both places and it may make some
sense for the value to change (eg Cache-Control, :Status, Set-Cookie )

Fields like Date, Retry-After, Age, Expires, Last-Modified might well
benefit from being set in the header and then updated in a trailer if the
transmission took a long time.  A strong ETAg might be able to be generated
on the fly an added to the trailer

All the Accept- fields don't make much sense in a trailer as they relate to
what the server will accept on other requests which may be concurrent with
the transfer.

I can't see why any Content- field should change or not be known during the
header generation.  I can see content becoming invalid, but that is a
change of status and not a new Content- field

So I'm not seeing a good reason that a mechanism should only apply to
Cache-Control, because there are at least a few other related fields that
areq equally applicable to change as Cache-Control (Date, Retry-After, Age,
Expires, Last-Modified).     But on the flip side, I'm not seeing many
fields other than those that are related to caching in some way that are
applicable.

So a thorough enumeration of all the fields is really needed to justify why
add a mechanism for just one field or for a subset or for all.

        The sender XOR scrambles the body with a N*64bit randomly chosen
> nonce.

I'm not seeing the benefit of this other than if you really want to melt
the polar ice caps with wasted CPU cycles.  If we can't trust endpoints to
interpret the mechanism correctly, then we may as well invent a new
protocol.

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 4 Feb 2021 16:03:15 +0100

> worthwhile at least considering if a more general solution is possible.

In fact I think that acting on the cache duration could solve many issues.
This would consist in only extending the caching duration. If an object was
rendered with some errors or incomplete information but was not aborted
during generation, it's probably still acceptable to be delivered again
for the time during which the service is expected to recover. So one could
imagine pre-setting the max-age to, say, 5 seconds in the header, and extend
it to one hour in the trailer if the result is satisfying. This might also
help avoiding the issue of cacheable vs non-cacheable etc, as it would be
much clearer that the decision doesn't change, only the validity duration.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Feb 2021 16:10:32 +0000

--------
Greg Wilkins writes:
> --0000000000001c746805ba8386a3

> authoritative fields are in trailers and that any in the header should be

Because I want to keep it simple.

> on the fly an added to the trailer

That is what we have today, and nobody wants to touch that with a ten feet pole.

> I'm not seeing the benefit of this [...]

The benefit is that the sender can trust that the content will not be
interpreted until it has been "released", and that we can trust buggy
proxies to make such a hash of it (pun intended) that it will be
found and fixed.

---

From: David Benjamin <[email-masked]>
Date: Thu, 4 Feb 2021 12:17:59 -0500

On Wed, Feb 3, 2021 at 7:04 PM Mark Nottingham <[email-masked]> wrote:

> > In addition to the state machine issues Ryan points out, I don't think
> the original premise in focusing on just the cache's behavior holds.
> >
> > Resetting the stream sends exactly the signal you need. Doing anything
> That's explicitly not the case I'm trying to address here -- what I'm
> interested in is the case where there is a _recoverable_ error, but it
> isn't desirable to reuse the resulting response on the same terms as was
> optimistically thought when the headers were sent. Usually, this will be in
> generating HTML.
>

Ah, I see. The original email mentioned an unrecoverable error, so I
interpreted this to mean the script is unable to properly service the
request.

Are you saying that the script, despite the error, is still able to
generate a passable, but suboptimal, response to the request? Could you
give an example? Is the server simply truncating the HTML and doing
something cleverer? I would think that, if you've gotten far enough to
start streaming output, your ability to change gears is somewhat limited.

(For HTML and browsers, a stream reset won't unload the document or
anything. I suspect a stream reset would still work fine there. We handle
truncated bodies differently in different contexts.)

David

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 04 Feb 2021 23:30:26 +0000

------ Original Message ------

>proxies to make such a hash of it (pun intended) that it will be

at the cost of the ability to block on any metadata without having to 
spool the entire resource (and delay it).

this is a non-starter as far as I'm concerned.

Some fields belong only in the headers and should never be seen in 
trailers.  I would rather see clarity around this.  E.g. any Content- 
field belongs in the header only.

Isn't there a warning field that can be used for more general cases?

Adrien

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Thu, 04 Feb 2021 23:52:39 +0000

--------
Adrien de Croy writes:

> at the cost of the ability to block on any metadata without having to=20

Uhm, we must be talking waaaaay past each other here...

The scenario where trailes offer low hanging fruit is:

	client:			server:
	-------------------------------------------------------
	request -->

				start production of dynamic response body
				since we cannot know what it will be, we
				have to spool it locally first.
				chew
				chew
				chew	
				chew	
				AhA!  Now we are certain what metadata properties
				the response should have
				produce headers
				send headers
				send body
	receive headers
	chugga
	chugga
	chugga
	chugga
	chugga
	finish receiving body

Today nobody supports trailers, so the server is forced to spool
the body locally, so the transmission can only start once the
entire, possibly very large body has been produced.

With my proposal:

	client:			server:
	-------------------------------------------------------
	request -->

				Hmm, we need trailers
				send new 3xx code, saying metadata comes last
	receive 3xx
				start production of dynamic response body
				send stuff as soon as it is produced
				chew
	receive body		chew
				chew	
	receive more body	chew	
				send final chunk
				AhA!  Now we know what metadata properties the response should have
	receive end of body
				produce headers
				send headers
	receive headers
	act on headers & body

In short, the spooling is moved from the server to the client,
which means the transmission of it overlaps the production,
and the server gets to determine the metadata as production
progresses.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 5 Feb 2021 10:50:46 +1100

> On 5 Feb 2021, at 4:17 am, David Benjamin <[email-masked]> wrote:
> 
> Ah, I see. The original email mentioned an unrecoverable error, so I interpreted this to mean the script is unable to properly service the request.

Aha - my apologies for the imprecision.

> Are you saying that the script, despite the error, is still able to generate a passable, but suboptimal, response to the request? Could you give an example? Is the server simply truncating the HTML and doing something cleverer? I would think that, if you've gotten far enough to start streaming output, your ability to change gears is somewhat limited.

Absolutely. A common architecture when you have to scale is to use a thin "front-end" layer to assemble responses from back-end HTTP APIs, with generous lashings of caching. This is the function of ESI[1], it was the approach we used in many Yahoo! properties back in the day, my understanding is that much of Amazon is built this way, and emerging "edge computing" architectures[2][3][4] can be used in this fashion as well. There's also a fair amount of interest in using this approach client-side in various ways.[5]

In doing so, you necessarily have to handle failure in one or more of those service calls. ESI, for example, allows you to use a backup URL or text (depending on whose extensions you're using) to paper over the problem.[6] That makes the response complete, and it serves that client as best as it can, but you wouldn't want to cache the result for reuse.

To date, this problem hasn't been a dealbreaker because most use of this approach has been for personalised or otherwise dynamic content -- the inputs are cached, but the output isn't cacheable (or when it is, it's private). However, I'm recently hearing of more uses where output caching is desired, because people find this kind of architecture beneficial for more than that.

> (For HTML and browsers, a stream reset won't unload the document or anything. I suspect a stream reset would still work fine there. We handle truncated bodies differently in different contexts.)

Understood. That's one of the reasons I don't want to use it as a signal; it isn't explicit.

Cheers,

1. https://www.w3.org/TR/esi-lang/
2. https://www.fastly.com/products/edge-compute/serverless/
3. https://workers.cloudflare.com
4. https://docs.aws.amazon.com/lambda/latest/dg/welcome.html
5. https://symfony.com/doc/current/templating/hinclude.html
6. A lot of effort is often put into making sure that the engine can stream as much as it possibly can while still reserving the ability to dynamically handle problems like this. It's Hard[tm].

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 05 Feb 2021 01:33:07 +0000

------ Original Message ------
From: "Poul-Henning Kamp" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Greg Wilkins" <[email-masked]>; "Stefan Eissing" 
<[email-masked]>; "Willy Tarreau" <[email-masked]>; "Ryan 
Sleevi" <[email-masked]>; "Martin Thomson" <[email-masked]>; 
"[email-masked] Group" <[email-masked]>
Sent: 5/02/2021 12:52:39 pm
Subject: Re: Cache control in trailers?

>The scenario where trailes offer low hanging fruit is:

not in my experience. All the scripts I've seen returning content 
explicitly set Content-Type first before they start spewing content, and 
it's streamed out, the servers don't have to spool at all.

What sort of metadata is it envisioned would require waiting until the 
end of transmission to set and be based on the entire content?  And have 
to be sniffed by the server?  I can't think of any.  We've had 
discussions in the past about clients sniffing response message bodies 
and it was decided it was a bad thing for security reasons, but I think 
everyone still does it or the internet breaks.

Adrien

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Feb 2021 07:39:35 +0000

--------
Adrien de Croy writes:

> not in my experience.

Good for you!

But it is the number #1 thing on peoples wishlist whenever they ask about
trailers in Varnish Cache.

> end of transmission to set and be based on the entire content?

Cache-Control, Set-Cookies, and the multitude of private headers people
use for CDN and walled-garden applications.

> We've had

Which is *precisely* why I propose we give the server the option
to XOR scramble the body until the metadata is ready.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 5 Feb 2021 08:46:02 +0100

> to XOR scramble the body until the metadata is ready.

No please Poul-Henning, really please, no more addition of such horrors
that we had to deal with in WebSocket. It serves no purpose and prevents
anyone along the chain from efficiently processing the contents, including
but not limited to, H1<->H2 translation, compression, inspection, etc.
Please do not add ugliness on top of ugly things, you usually have much
better and cleaner proposals, let's get back to deployed standards and
stick to what works where relevant :-/

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Feb 2021 07:46:48 +0000

--------
Mark Nottingham writes:

> In doing so, you necessarily have to handle failure in one or more of 
> those service calls. ESI, for example, allows you to use a backup URL or 
> text (depending on whose extensions you're using) to paper over the 
> problem.[6] That makes the response complete, and it serves that client 
> as best as it can, but you wouldn't want to cache the result for reuse.

Not to mention assembling all the tracking set-cookies from all the
included items.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Feb 2021 07:51:47 +0000

--------

> but not limited to, H1<->H2 translation, compression, inspection, etc.

First, note that if your downstream understand this new extension, nothing
prevents you from forwarding in a streaming fashion and leaving the XOR'ing
to the next sucker in the chain.

Second: Yes, that is *precisely* why I want to offer servers that
option:  So they have a sure-fire way to prevent sneak-peeks which
they have reason to belive will get things wrong.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 5 Feb 2021 09:16:19 +0100

Poul,

If there is a proposal to make trailers more palatable/usable by some 3xx
status code or some other signal in the header, then there is no need to
include some XOR mechanism. If servers want that kind of behaviour, then
they can come up with a new content type and send the nonce in the trailer.

The conversation here is really talking past each other.  The
original proposal is to support cache-control fields in trailers.  I don't
think anyone has said that is a really bad idea and most a
general supportive (although I'm a bit dubious on the use-case , but
recent posts have clarified that).  The counter is not that we shouldn't do
that, but that that trailers are generally not used and that there are some
proposals to make them more usable generally and not just for
cache-control.   So I see lots of actual agreement, but lots of posts like
there is disagreement???

So Mark... your original proposal was just about the Cache-Control field.
 Do you still think that it is only that field that needs support in
trailers?  What about Set-Cookie, Last-Modified, ETag, Retry-After, Age,
Expires, etc.   aren't these fields also affected by the use-case you have
outlined?   Do you still think a single field proposal is sufficient or is
a more general solution needed?

cheers

---

From: Julian Reschke <[email-masked]>
Date: Fri, 5 Feb 2021 09:38:12 +0100

Am 05.02.2021 um 09:16 schrieb Greg Wilkins:
> ...
> So Mark... your original proposal was just about the Cache-Control
> field.   Do you still think that it is only that field that needs
> support in trailers?  What about Set-Cookie, Last-Modified,
> ETag, Retry-After, Age, Expires, etc.   aren't these fields also
> affected by the use-case you have outlined?   Do you still think a
> single field proposal is sufficient or is a more general solution needed?
> ...

Note: "Etag" is already marked as acceptable in trailers
(<https://github.com/httpwg/http-core/issues/262>).

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Feb 2021 08:42:37 +0000

--------
Greg Wilkins writes:

> include some XOR mechanism.

The XOR mechanism is primarily there to protect against proxies which do not understand the new 3xx code.

After some rollout period, I expect it will be safe to reduce the 'N' in 'N*64 bits' to zero.

---

From: Greg Wilkins <[email-masked]>
Date: Fri, 5 Feb 2021 10:00:31 +0100

> (<https://github.com/httpwg/http-core/issues/262>).

Sure, but one of the suggestions to come out of this thread is that it
might be valuable to put something in the header to say that the final
value will be in the trailers.   This could be for an individual header (eg
Willy suggested Cache-Control: no-cache; trailers)  or a more general
mechanism like the 3xx proposal.

So in the case of Etag, would it be worthwhile for cache implementers to
know that the content they are currently receiving will/may have an etag in
the trailers?    If so, that suggests a more general mechanism for
hinting at the contents of the trailers is needed.

It is hard to focus on actual solutions whilst the scope of the issue is
not resolved.

cheers

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 5 Feb 2021 10:09:34 +0100

> they have reason to belive will get things wrong.

The problem is that we all know that breakage is never black or white,
it's always gray in that non-compliant intermediaries get stuck in the
middle with something that desynchronizes them, or worst, get attacked
by some body placed at a location where they were expecting some headers.

Willy

---

From: Julian Reschke <[email-masked]>
Date: Fri, 5 Feb 2021 10:45:20 +0100

Am 05.02.2021 um 10:00 schrieb Greg Wilkins:
>
>
> On Fri, 5 Feb 2021 at 09:41, Julian Reschke <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>     (<https://github.com/httpwg/http-core/issues/262
>     <https://github.com/httpwg/http-core/issues/262>>).
>
>
> value will be in the trailers.   This could be for an individual header
> (eg Willy suggested Cache-Control: no-cache; trailers)  or a more
> general mechanism like the 3xx proposal.
> ...

For fields that are *only* available as trailer, we have the Trailer
response header field...:

   Trailer: ETag

Now do we beed a generic solution for saying "I have this field as
header field, but I want the trailer field to override it"?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Fri, 05 Feb 2021 09:49:41 +0000

--------

> hinting at the contents of the trailers is needed.

There are three fundamental problems with current trailers;

1. You dont know which header-fields are only provisional.

2. Combining the header and trailer field values is fraught with
   peril, both for sender, middle-boxes and receivers.

3. Buggy-proxies will mangle things in transit.

The first problem can be solved with a new header which lists which
fields which might appear in the trailers:

	Provisional:  Cache-control, etag, Last-modfied

The second problem can be solved with a clear-cut rule which says that:

A) Header values for fields marked provisonal are discarded when
   the same field appears in the trailers[1].

B) Trailer fields not marked provisional in the headers are discarded.

Something like that could make me reconsider implementing trailers in Varnish.

But I still think it's much easier, more robust and almost as efficient
to simply transmit the all the semantic metadata fields after the body.

There is no solution for the third problem, except to make sure the
buggy-proxies break bad enough that somebody fixes them.

Poul-Henning

PS: Yes, I know IETF "does not do APIs" but we should at least
    think about how our solution can be used from a CGI API.

[1] There's a detail here:  We need to ensure that a trailer
    fields without content survive, so the sender can say "Sorry,
    didn't want an Etag anyway" etc.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 5 Feb 2021 18:37:32 +0100

> to simply transmit the all the semantic metadata fields after the body.

This is called Trailers :-)

> buggy-proxies break bad enough that somebody fixes them.

I'm not that much worried about them nowadays. Plenty of them will
simply drop the trailers before forwarding. That's fine and permitted.
The ugliest ones will stop parsing after 0 CRLF, and will break when
finding that remaining string in the buffer, either when the connection
is left idle, or when they expect to parse the response to the second
request sent over that connection, thus breaking keep-alive. In any
case, these are not the ones we'll often find where decoding SSL is
required, nor inside the network where reliability is a business
metric so I consider that they're mostly off-path nowadays.

>     think about how our solution can be used from a CGI API.

Why ? CGIs do not even support chunked encoding in the first place,
let alone sending final meta-data past the body. The common failure
case for CGI is to exit() and let the server close the communication.

>     didn't want an Etag anyway" etc.

Sure and actually this would tend to encourage not to merge headers
and trailers. I don't know how many intermediaries do merge them
nowadays, maybe we could check among implementations here and
complete the list started below ?

   - haproxy: drop chunked extensions, pass trailers as-is, no merge

Willy

---

From: Wesley Oliver <[email-masked]>
Date: Fri, 5 Feb 2021 20:13:05 +0200

Hi,

Well if there an error then going to have to be a signal to indicate a
change, ideally if an error occur I would preferable want to push a full
new response, and headers to avoid a redirect, the action in my mind should
invalidate the whole previous response headers and body. Should be
basically deleted. Then new response body would need to include a new url
for the pushed response, to replace the current page.
I think of it like an implicit embedded redirect. So in my mind there is
first a Control signal quick, then receipt of a new full response http
response that is pushed and mabye could then follow by a redirect push, if
possible not sure that possible. Where if control signals failed, then
local client, would need decided, which parts are still valid for caching.

In which case revalidation of some chucks already receive that could have
been cached need to revalidate. But I need catch up here, but in my mind I
want partial chucks, to be able revalidate using hash or best new
equivalent.
Don't want to fully redownload a file of a few gigs.
For each chuck received, were just the streaming chucks need revalidate
there hashes.. I haven't done deep dive here.

In essence, cache being setup, to must immediatly revalidate, but can mabye
display stale content, in the mean time, like error page that previously
transmitted, then browser saying attempting or trying again, that be like
at browser page level management.

Woundering for fetch, benift from somthing like that, but then fetch would
need to be able have another call back, to signal it to refetch content as
been updated. This allow mabye display old unavailable content while
fetching the new. So for like this to work need another call back to signal
it.

Mabye http header could be used to indicated support for return stale error
page, while revalidation.

Just my too cents. But not having to buffer response is going to be great
improvement for memory pressure on server and overhead and time till first
packet recieved.

My two cents but have my previous proposal of optermistics response
previously.

Kind Regards,

Wesley Oliver

---

From: Wesley Oliver <[email-masked]>
Date: Fri, 5 Feb 2021 20:29:00 +0200

Hi,

The other thing I tought of is that an generic error page could be
spesified in the first response headers, so if fail to push and dynamic
more spesific error page. A generic static one could be fetchee for
display, mabye js could pass error codes or response pushed to enhance it
with clientside fecth.

But any error that so fatal that error page couldn't be pushed, then
reverted to the error page to display that spesified in the response
headers.

I think like the idear of first eesponse header for an error page to show
if fatal.

Guess in this case, may want fatal page to be able fetch old url, which
errors, so interigate the url, mabye some headers or title improve its
messages. So like browser client would pass url of fatal page to that as
param to the error page.

In this case one would want to be able to fetch old failed response, to
display correct failure details, so want fetch to be able to fetch the old
failed url from the static fatal page, that redirect to.
So mabye a header for fetch, to choose what to serve from cache while
revalidation and then another status, saying response been revalidated.

That way a static page with fetch, provide a nice error page show details
by interogating previous failed fetch and wait for revalidation, in which
it could redirect and recover immediately. So might see fail page like for
a second then show second successfull attempt, that went off during the
error page display, that caused the revalidation and on success a redirect,
happing locally, based in cache.

Just think is one able to return cache item I'd, such that prevent 3rd
revalidation and fetch, if no content expire was sent, as redirect then
won't be a local cache hit.

I like idea spesify error page in the response header, as if fails
critical, can push dynamic error response page.

But definitely need cache to require forced revalidation.

Kind Regards,

Wesley Oliver

On Fri, 05 Feb 2021, 20:13 Wesley Oliver, <[email-masked]> wrote:

> In essence, cache being setup, to must immediatly revalidate, but can
> mabye display stale content, in the mean time, like error page that
> previously transmitted, then browser saying attempting or trying again,
> that be like at browser page level management.
>
> Mabye http header could be used to indicated support for return stale
> error page, while revalidation.
>
>> A more limited approach would be to focus just on the cache's behaviour
>> -- e.g., to allow Cache-Control: no-cache in trailers,[2] updating the

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 5 Feb 2021 11:01:18 -0800

If we are going to continue this conversation, I insist that participants
actually read the current definition of trailers under WGLC in 

  https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#trailer.fields

and

  https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#field.trailer

and then study existing use beyond what you have implemented yourself.

It takes only five minutes of searching to dig up many articles from people
using trailers in practice within APIs, CDNs, backend systems, etc.  E.g.,

   https://engineering.pivotal.io/post/http-trailers/
   https://www.fastly.com/blog/supercharging-server-timing-http-trailers
   https://webtide.com/http-trailers-in-jetty/
   https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md
   https://grpc.github.io/grpc/cpp/md_doc__p_r_o_t_o_c_o_l-_w_e_b.html
   https://golang.org/pkg/net/http/#example_ResponseWriter_trailers
   https://github.com/envoyproxy/envoy/issues/2394

HTTP is not limited by what browsers currently use! We should all know that
by now without me having to repeat it ad nauseum. Browsers are important
but make up less than 1% of the total HTTP implementations. Nevertheless,
we have deliberately fixed several aspects of trailers in the revision, specifically
so that they can be better implemented by intermediaries and browsers, and
won't override header field values by default.

This is not to say that the current description is perfect. It is somewhat
limited by past definition and usage of trailers and the various ways they
might be sent for HTTP/1, 2, and 3. We have worked hard to define what
has been implemented widely now and avoid prohibiting what has only
been implemented as experiments, such as mid-stream trailers.

Any statements suggesting that trailers are "not in current use" or that
"nobody implements trailers" are obviously false. They are being used.
They would be used a lot more if we don't arbitrarily break them without
understanding the existing implementations.

To answer the original question, there is nothing stopping a server from
sending

    Cache-Control: must-revalidate
    Trailer: cache-control
... body...
    Cache-Control: public

or

    Cache-Control: max-age=3200
    Trailer: end-status
... body...
    End-Status: 500

or

    Cache-Control: max-age=3200
    Trailer: end-cache-control
... body...
    End-Cache-Control: oops

and simply ask cache implementations to do the right thing once you define
what that right thing might be. These are already supported by the core spec,
but they would need to be defined as extensions to be interoperable.

Personally, I think end-status is the easiest and most reusable solution, for
any number of features that might need to know if something broke. However,
Willy is right that saying must-revalidate up front and then softening that
at the end would be the safer choice where completion is more important
than default performance. I suggest that choice needs to be resource-specific.

Please, if there is something specifically wrong with the current definitions
of Trailers in the specs currently in WGLC, and its not just a personal opinion
about their usage, suggest a better definition that doesn't break other
people's software, or suggest some additional sentences that might clarify
the work so that the next generation of apps can use this feature without
prior shared agreement.

....Roy

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 05 Feb 2021 21:12:48 +0000

------ Original Message ------
From: "Roy T. Fielding" <[email-masked]>

>than default performance. I suggest that choice needs to be resource-specific.
The "Something broke" could also apply to a scanning intermediary.  E.g. 
some message body was found to contain a virus.

If an intermediary could signal that the final status may be different, 
and rely on clients to obey that, then it could safely stream unscanned 
data to the client, and indicate the result at the end, knowing the 
client will discard the message body.

This would avoid all manner of hacks which don't work very well but are 
required to stop client agents from e.g. timing out due to lack of data 
(as intermediary spools and scans).

It might be necessary if defining a new field which indicates a final 
result, to indicate to authors that it might be a good idea not to 
render a message body until the final status is received (to avoid 
susceptibility to exploits).

Adrien

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 6 Feb 2021 13:34:20 +1100

Hey Greg,

My .02 -

> On 5 Feb 2021, at 7:16 pm, Greg Wilkins <[email-masked]> wrote:
> 
> So Mark... your original proposal was just about the Cache-Control field.   Do you still think that it is only that field that needs support in trailers?

I never said it was the *only* field that needs support in trailers. However, I believe it's most realistic to focus on a defined use case that has limited impact on implementations, rather than trying to boil the ocean of moving buffering between different parties in the protocol. 

>  What about Set-Cookie, Last-Modified, ETag, Retry-After, Age, Expires, etc.   aren't these fields also affected by the use-case you have outlined?   Do you still think a single field proposal is sufficient or is a more general solution needed?

I think the property that PHK pointed out -- giving more flexibility for tracking -- is a great argument against doing anything to make Set-Cookie more useful.

As was already pointed out, ETag can be carried in a trailer today; it just remains for implementations to support it. I suppose we could update Core so that Last-Modified can be too, but I don't think the use case is as compelling.

Retry-After isn't terribly useful for 99%+ of the web, and isn't supported in browsers AFAIK, so it's not very interesting.

Updating Age doesn't make sense; you need it to be as reliable as possible, and adding doubt about that is just going to make things worse.

Expires - *shrug* I guess. I'd rather kill it in favour of CC: max-age -- again, we should think about what behaviours we're encouraging.

As discussed, the elephant in the room that you don't mention is the status code -- but the problem is that a variety of behaviours are triggered by status codes, some specified, some not. In contrast, Cache-Control is targeted just at caches, and so you can reason about changes to it much more easily.

Cheers,

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 6 Feb 2021 09:09:05 +0100

On Fri, Feb 05, 2021 at 09:12:48PM +0000, Adrien de Croy wrote:
> If an intermediary could signal that the final status may be different, and
> rely on clients to obey that, then it could safely stream unscanned data to
> the client, and indicate the result at the end, knowing the client will
> discard the message body.

Or even better, indicate a suspected infection so that the client can
apply extra measures including quarantine, second inspection, ask user
for consent etc, which are not possible today.

This is a very good idea which illustrates very well how passing metadata
after a body can be beneficial.

Willy

---

From: Wesley Oliver <[email-masked]>
Date: Sat, 6 Feb 2021 10:52:35 +0200

Hi,

Be keen to get through previous list specs, just load shedding for next few
hours, so I'll have wait patiently.

For virus, I would imagion body would be truncated, signal fail, as no
wants a virus to be saved. May be good to implement a different status
error code here or class of codes, for programs that snoop the traffic, so
one can easily differentiate them. Mabye that they like old Norton did
inject js, inject an identifier, which means for error page or anyone to
better debug what's gone wrong and where.
Recommend ways handling the contentz based on that status code.

Waiting for the final status, means that pages would be blocking the client
wouldn't be able to process the page and start down loading content,
depending on the definition. Because anyone choose not obey starting to
process the body. I feel that depending on use cases may like to indicate
different behaviours.

Allow body to be processed incremental, just not rendered, one option,
another progressive loaded pages, one want to be rendered and on error, two
ways to handle it, one continue display current page in progressive non
complete form and like popuo page fail, OK to the error page, or mabye
browse basically, show split frame or popup error, so error page loads and
displays as an overlay, may want top-left top-right bottom-left
bottom-right.
The other split, like resizable frame set window, which could be closed.
Like a retry button or auto retry be attempted.

Gather the behaviour in which body should be handled needs have different
http response header modes, for like the above behaviours. Which can
evolved over time, as people think new ways to do things.

More important part would be ensure caching, drops the body, and ensure it
knows that content needs to be revalidated. So whether communicate for
cache if body content is kept, so it can display what every it had while
revalidstinh if wanted. I am thinking
Way in which cache gets handled, need be communicated in error, set cache
status to must-revalidate, or mabye adjusted shorter failed cache time.

Mabye on there being signal of failure the behaviour of what is to happen
to the cache, should be left programmable by a response header, which
decide mabye under crazy load what every that must continue fail for
5minutes, instead like the normal 1hour cache or mabye a few weeks. I think
allowing http response header, to spesified cache content expiry behaviour
override, may be alot more flexible in future as ways in which we use and
improve things and instead of pining it down to must revalidate, think
having a default behaviour of must revalidate, if now http response header
for onerror or httpstatus code map cache behaviours.
Feel a default behaviour and ability to still override it dynamically,
which come with time new a alternative use cases.

Just my perspective on things.

Kind Regards,

Wesley Oliver

---

From: Greg Wilkins <[email-masked]>
Date: Sat, 6 Feb 2021 10:31:42 +0100

Sorry, I'm definitely guilty of getting distracted by the conversation and
forgetting aspects of the trailer mechanism...

Going back to Mark's original post.   He had two proposals: "indicate the
error in trailers"; "to allow Cache-Control: no-cache in trailers".

For the first proposal, I'll repeat my initial feedback: it is a bad idea
to handle errors during generation by correctly terminating a 200 response
in order to put an error status into the trailers.
Clients/intermediaries that don't support the specific error/status field
would see this as a valid response.  Specifically I think it is a bad idea
to do:
    HTTP/1/1 200 OK
    Trailer: end-status
    ... body...
    End-Status: 500

I can only see this as being workable in a 200 response if there is a
really reliable way for the client and all intermediaries to indicate their
support of End-Status specifically and not just support for Trailers
generally.   It could be argued that TE: trailers should indicate support
for all standard fields, but I can't see how it can be assumed to indicate
support for newly defined fields.

For the second proposal, as Roy indicates, it is already possible. So what
exactly is being proposed? Clarifications to the Cache-Control field
definition or a new Cache-Update field?   Again, either way, it does appear
that being able to indicate support for a specific field in the trailers
rather than the trailer mechanism in general would be valuable.

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 6 Feb 2021 10:42:08 +0100

On Sat, Feb 06, 2021 at 10:31:42AM +0100, Greg Wilkins wrote:
> generally.

Agreed. In general I'm not against the use of a status, but we currently
don't have statuses that mean "suboptimal response, please don't rely too
much on it". Conversely, cache-control semantics allow to send a short
"max-age" and a "must-revalidate", to warn the client about possibly poor
content quality while letting the server recover from the bad condition.

This is why I'd prefer a trailer field completing cache-control, but I'd
be fine with just a status if we define one meaning just this.

Willy

---

From: Wesley Oliver <[email-masked]>
Date: Sat, 6 Feb 2021 16:48:55 +0200

Hi,

I typically agree with rather moving to max-age. I recently had some great
fun with authentication, where mobile phones local time could be out
and so can the server, extra. that basically I typically came up with the
idea, when generating expire for like jwt, that one wants to encode the
local client device time and server time in there
and work to the one,which is gong to make things work the best, when
evaluating expire, wanting to refresh a jwt before it expires, using local
device time, as may not be insync with the server.
Then again also just just invalidate all signature on deployment, forcing
all jwt to become invalid and fail authentication.

I would be for both a status code and for the ability to override the
Cache-Control header in trailing headers, or something similar.
Obviously one would need the request to specify its support for these
features, either protocol version or optional request support headers,
So that on clients that don't support the behaviour, the old approach can
still be taking of buffering everything.

The semantics of page erroring, one client immediately tries again, in
which the default behaviour of must-revalidate,
 in subsequent request on may want to change that to reduce the load caused
by the internal server error, however, this would
require that one could count the client requests, which becomes complex, if
this is not like a statless jwt put in request forward headers, which don't
exist yet,
so one would need to use a cookie, not idea, as that for bowser really.
Upon reaching a certain count, would could put out a different expire time
of 5 minutes, however, right now the logic for system behind load balance,
could become quite complex to implement.

If one is able to easily monitor URL errors and dynamically change the
cache-control header, one would be able to reduce load continuously being
created by this error,
However, it may be simpler that for certain classes of HTTP status codes,
on set maybe different Cache-Control overides, but starting with since one
for one is great, see how we go.
as setting it to 5 minutes, would reduce massive amounts of load, to
generate the content on the server, however, fail at some point when
sending it to the client with (I hate autocorrect now aws=was)
AWS your are basically paying for this, so maybe one adds in a circuit
breaker here to any upstream, with the 5 minutes that would allow it to
break the load on the upstreams.
but one has to wrap to manage list of all forward requests.

it could be the case, where typically I don't have a lot of load, from this
functionality, so its working fine, however, if were to fail the serverside
overhead and waisted cpu cycle and
partial bandwidth would be terrible. In this case one could possible
recommend, that if a server failed with trailing headers, that it should
revert for the next URL unique request to buffering if possible,
to reduce the bandwidth consumption to the client, by first ensuring it
could generate a response with no errors, there after it could resume
normal streaming mode or proxy could convert the request,
if response headers allowed for it.
But allowing one to override the default Cache-Control would be great to
prevent this from happening to a degree.

One thing new feature that would be possible is if the downstream are allow
to change the response to streaming response with trailing headers, one
would need to indicate
in a response header if that are allowed to do this, as may just be a
blanket setting turn on for the outside facing world, which the result in a
behaviour that is not desirable
for the consumer of the response for rendering extra.

Maybe one may want a backoff period instead, which can be overridden by
forced refresh request, the backoff would just reduce load attract on the
system, by my hitting try again,
successively and abusively, give overloading the servers, with failed
requests, however, stream trailing headers, would mean I would be paying a
bandwidth penalty.

Kind Regards,

Wesley Oliver

<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avg.com
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

-- 
----
GitHub:https://github.com/wesleyolis
LinkedIn:https://www.linkedin.com/in/wesley-walter-anton-oliver-85466613b/
Blog/Website:https://sites.google.com/site/wiprogamming/Home
Skype: wezley_oliver
MSN messenger: [email-masked]

---

From: Wesley Oliver <[email-masked]>
Date: Sat, 6 Feb 2021 17:09:28 +0200

Hi,

The thing is that on error, that need to signal a communications protocol
error, signify the data position of the body that has been transmitted, so
it can figure out
where the trailing headers will be, versus the trailing headers being
actually being transmitted in the chuck encoding. In m mind one requires an
external
communication mechanism, to break the expected data to be received, in
which trailing headers, would be misinterpreted as the body of the response.

Kind Regards,

Wesley Oliver

On Sat, Feb 6, 2021 at 4:48 PM Wesley Oliver <[email-masked]> wrote:

>  in subsequent request on may want to change that to reduce the load
> caused by the internal server error, however, this would
> require that one could count the client requests, which becomes complex,
> if this is not like a statless jwt put in request forward headers, which
> don't exist yet,
> it could be the case, where typically I don't have a lot of load, from
> this functionality, so its working fine, however, if were to fail
> the serverside overhead and waisted cpu cycle and
> One thing new feature that would be possible is if the downstream are
> allow to change the response to streaming response with trailing headers,
> one would need to indicate
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free.
> www.avg.com
> <#m_-7276197523490670123_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 7 Feb 2021 21:59:35 +1300

On 6/02/21 3:34 pm, Mark Nottingham wrote:
> Hey Greg,
> 
> My .02 -
> 
>> On 5 Feb 2021, at 7:16 pm, Greg Wilkins wrote:
>>

With my web-dev hat on it is quite common for CMS controlled sites to 
have a huge mix of modification times on scripts and content sources. It 
is a huge pain to calculate a reliable L-M before output in this 
environment. It would much cleaner to (be able to) accumulate a 
modification time across the scripts and dump a Trailer header with the 
true L-M value.

> Updating Age doesn't make sense; you need it to be as reliable as possible, and adding doubt about that is just going to make things worse.

Ack. No need for Age at all on freshly generated content use-cases.

> Expires - *shrug* I guess. I'd rather kill it in favour of CC: max-age -- again, we should think about what behaviours we're encouraging.

Very much agreed on deprecating Expect. Though the value limit on 
max-age is smaller than possible Expect values. So there is a use-case 
for extremely long duration archiving to use Expect.

Amos

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 08 Feb 2021 08:16:27 +0000

--------

> modification time across the scripts and dump a Trailer header with the 

This is pretty much the first thing people who start using ESI asks for,
the second being set-cookie.

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 08 Feb 2021 08:51:51 +0000

--------
Roy T. Fielding writes:

> of Trailers in the specs currently in WGLC, [...]

There is, take for instance 6.5.1:

	Many fields cannot be processed outside the header section
	because their evaluation is necessary prior to receiving
	the content, such as those that describe message framing,
	routing, authentication, request modifiers, response controls,
	or content format.

The unstated assumption is that the recipient cannot be required to
spool up the body and wait for the trailers before doing anything
with it.

Of the listed examples, only "framing" cannot wait[1], the rest of
the examples provided are simply wrong:  None of that information
is in any way required to receive and temporarily spool the body[2].

I dont know when or why this assumption was made silent canon in
the HTTP protocol, but I will argue that it is:

A) Bad Architecture.

B) A really big roadblock for trailers.

C) Preventing us from reaping obvious and significant speed benefits.

D) Not documented anywhere.

If, after discussion, consensus is that there are good reasons for
keeping this assumption, we should document that, if there are not,
we should get rid of it.

Poul-Henning

[1] And one could reasonably ask what is that even doing in the
semantics in the first place ?  Yes, *I* know, you dont need to
explain it, but if you came to this fresh, you would wonder.

[2] That is not to say that the headers cannot provide useful
hints such as "expect around 3 megabytes" or "you'll need a
sha256 digest later", but they are just hints.

---

From: Wesley Oliver <[email-masked]>
Date: Mon, 8 Feb 2021 11:37:00 +0200

Hi,

But that is why send those headers upfront  if they not sent it would a be
a communication protocol error for HTTP as they had be sent upfront and
then along send
possible headers in this case and some other that would logically be
allowed to be sent as trailing headers and some which have overriding
behaviour.

External communications control protocol upon which HTTP runs, would need
to support a signal that there was and error and that at certain byte offset
the assume override or trailing headers in the transmission could be found.
Otherwise, the response would need to be padded, to meet the mini
communicated blocks transmissions size,
leading to a lot of more waisted bandwidth.As you can't predict and error.

Kind Regards,

Wesley Oliver

---

From: Wesley Oliver <[email-masked]>
Date: Mon, 8 Feb 2021 12:05:40 +0200

Hi,

One other thought is that if there was an error then the last 10 bytes of
the stream will signal it,
Which would be a signed payload, as not confuse with normal body content,
the signing could be sent in the headers,
which basically say from this position back 300bytes for trailing headers
or that it is all good.
This would be simpler and upon encapsulating control protocol signally the
end of transmission, the last 10 bytes of the response
would always need to be checked as to ascertain if HTTP was success or
failure. That way no modifications would be
required to the encapsulating control protocol.
The only thing, similar to file upload, I do worry about is an error
occurring perfectly on the founder of the response body, but if
always has to be present for successful response and failure response it
shouldn't be a problem. However, without an HTTP protocol
flag support specifying this mode, it can't be implicit, because if
implicitly, the backwards compatibility with implicit support,
would suffer from the chance that the last part of the body, by chance
matches the signature ad last 10bytes indicate success or failure.

This mode of transmission for the protocol definitely will be required to
avoid this, and expectation of support to be communicated upfront in the
first headers or protocol version indicate support for
there always being a signed 10bytes of success or failure indicator.

Kind Regards,

Wesley Oliver

On Mon, Feb 8, 2021 at 11:37 AM Wesley Oliver <[email-masked]> wrote:

> the assume override or trailing headers in the transmission could be
> found. Otherwise, the response would need to be padded, to meet the mini

---

From: Stefan Eissing <[email-masked]>
Date: Mon, 8 Feb 2021 11:12:56 +0100

> Am 06.02.2021 um 10:42 schrieb Willy Tarreau <[email-masked]>:
> 

tr-{xxx}: yyy      # replace header 'xxx' with value 'yyy', when this field appears in trailers

is that what we are discussing?

- Stefan

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 8 Feb 2021 11:32:28 +0100

> spool up the body and wait for the trailers before doing anything

... and watch the whole Web collapse because suddenly everyone
needs to buffer every byte in flight before even knowing where to
forward it if at all. This is the e-mail architecture that you're
describing. Far less reliable and fast than the web. Every single
message has to be spooled to disk and some get delivered very late
when not lost. Imagine an HTTP response passing through 10 layers
of intermediaries, you'd divide the effective bandwidth on this
request by 10 because of the store-and-forward approach!

Conversely, announcing upfront who to deal with a message allows to
add controls and to stream it. This is so much important that Expect
was invented to save users from having to upload a whole attachment
for 2 minutes in their webmail before seeing a login pop-up and
figuring they'd have to do it again.

For sure there *are* information that would be useful at the end
(hence the creation of trailer) and possibly even in the middle,
(e.g. timestamping or quality information for streamed contents
maybe). But let's not suddenly declare that everything would be
better at the end because this is utterly wrong.

And it's very fortunate that Authorization and Cookie are sent prior
to any upload, otherwise a number of servers would be forced to store
a lot of crap before figuring if the sender was allowed to upload it
in the first place. This could even have legal implications by the
way.

Regards,
Willy

---

From: Wesley Oliver <[email-masked]>
Date: Mon, 8 Feb 2021 12:37:02 +0200

Hi,

So basically something along the lines of the following will need to be
sent to prevent same issues as file upload chance of happening.
if you were using quick, say when doing a special stream buffer flush,
those protocols upon which HTTP can't provide this end frame of transmitted
bytes up until such a point will possibly have the chance of suffering same
issues as file upload, or some part of the trailing body
gets recognised as final 10 bytes.
RST frame with final byte offset
Data frame with FIN bit set

Kind Regards,

Wesley Oliver

On Mon, Feb 8, 2021 at 12:05 PM Wesley Oliver <[email-masked]> wrote:

>> But that is why send those headers upfront  if they not sent it would a
>> be a communication protocol error for HTTP as they had be sent upfront and
>> then along send
>> LinkedIn:
>> https://www.linkedin.com/in/wesley-walter-anton-oliver-85466613b/

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 08 Feb 2021 11:11:39 +0000

--------
Willy Tarreau writes:
> forward it if at all.
> [rant]
> But let's not suddenly declare that everything would be

Nice strawman, but nobody has advocated anything like that.

Now, if you have any considered opinions on offering *an extension*
which allows all *semantic metadata* fields be put at the end, I'd
love to hear them.

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 8 Feb 2021 13:41:27 +0100

> Nice strawman, but nobody has advocated anything like that.

This is still the way I read it: "buffer body before processing headers"
will almost always mean "store and forward".

> which allows all *semantic metadata* fields be put at the end, I'd

No I don't because I think it is not a good idea. Supporting *some*
may make sense but definitely not all, and not in all cases, as there
are always useful information there that are better retrieved before
the body. For example, if you pass Content-Type too late, you may
lose the opportunity to perform compression on the fly.

Maybe we need to implement something between TE and Trailers so that
the client could suggest the server to place certain fields in the
trailer part. This would obviously be a per-connection header field.
But as to why a client would ask for these beyond the rare examples
already suggested seems unclear to me.

Willy

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 08 Feb 2021 13:53:46 +0000

--------

> will almost always mean "store and forward".

First, again: In any situation where the authoritative headers are known
at header tranmit time, one would be stupid not to send them as headers.

Second: Only if your job *is* to forward, you have heard of browsers, right ?  ;-)

Third: The *only* use-case we are talking about is when authoritative
headers are not available until after the entire body has been produced.

In that case it will be faster if the body can be transmitted as it
is produced, followed by the headers, than to spool up the body
on the server, and send it after the headers.

> For example, if you pass Content-Type too late, you may

I have no problem with Content-Type being in the headers or in the trailers.

My problem is when it is in both and they disagree.

Or when it takes complex semantic editing to join two wildly differing
Cache-Control headers.

> trailer part.

Another bogus straw-man, as you yourself note: The need for this
arises on the server side, the client as no idea if any particular
object will be subject to such issues or not.

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 8 Feb 2021 16:01:03 +0100

> the client could suggest the server to place certain fields in the

I think this is the crux of the matter. Servers can already put pretty
arbitrary fields into trailers if the client says
  TE: trailers

But that is too coarse a switch, because the server cannot know which
particular fields the client is able to handle in the trailers.  It may be
expecting only a check-sum or similar in the trailer, so if the server
starts sending Cache-Control and/or End-Status fields, then it has no way
to know if the client will actually interpret them.

Perhaps we need to send requests something like

  Accept-Trailer: Cache-Control
  TE: trailers
  Connection: TE, Accept-Trailer

If an intermediary is happy to buffer the entire content before acting,
then it can send requests like:

  Accept-Trailer: *
  TE: trailers
  Connection: TE, Accept-Trailer

Servers that send trailer fields that are not listed in an Accept-Trailer
should only consider them advisory and cannot assume that the client will
actually interpret them... which is kind of the state today anyway.

cheers

---

From: Wesley Oliver <[email-masked]>
Date: Tue, 9 Feb 2021 09:02:41 +0200

Hi,

Basically, I would agree with you here.

The issues about the encapsulation protocol, requiring to be able to
communicate like a buffer end of section transmission end by indicating the
bytes sent,
such that the recipient can then inspect the last bit of information
received and jump back to the offset of the trailing headers if
encapsulating protocol is not able to
reliably signal such an event, or can only signal the end of a section
possibly, where there can be false negatives, without the final offset.
Then basically the 8 bytes of end data signalling offset and success that
must always be there in this mode would need to support an escaping scheme,
such that the last select of received data can be scanned, for the ending
of transmission, because you won't know the length of data upfront and even
if you do on error,
it won't have transmitted all that data, so you would have to send padding
and wasted bandwidth.

Obviously, this has an impact for speed, so ideally on wants some form of
the control signal to be able to trigger the scanning opposed to scanning
everything.
If one is able to rely on a specific buffer size of transmission always
being full, then only need to check on the server and the client that the
last buffer that is not expected to be full,
before transmission and after receiving, which give you the chances to
escape content, before transmission, however, you don't want to double
buffer the transmission, you preferable
like to directly update the buffer written to for transmission. The same
would happen on the client-side upon receiving a non-full buffer.
If a buffer where to perfectly end on the buffer size, then is bit of a
problem, so upon final dynamic generation or content generation, one would
need to write and explicit
check-in, that if the buffer size match that, need to escape and
transmitted another buffer of content, to trigger the client-side checking.
Relying on partially full last buffer size check would reduce the need for
the encapsulating protocol, signal.

Ability to have buffer size that naturally allows alignment would reduce
scanning overhead, however, using triggering on buffer not be full, would
eliminate the need for
escaping, because just made this constraint condition existing, which when
violated, no need to check for last trailing 8 bytes giving the offset
backwards.

The other thing that would be nice is that one can override the HTTP status
and cache status, however, still follow up with an additional
redirect, location header, so only error there is automatically redirected
to and error page as well.

I think that a new name for AcceptTrailers CacheControl would be needed and
changed to something like
NonBufferResponse, which would be equivalent to CacheControl | Status |
Location, as a location now has a temporary redirect
implied status, when used with CacheControl.

Kind Regards,

Wesley Oliver

---

From: Wesley Oliver <[email-masked]>
Date: Tue, 9 Feb 2021 18:43:13 +0200

Hi,

The one obvious thing missing is you are relaying of the buffer size being
smaller, then
basically, there will be padding to cause a next smaller buffer at a
certain point.
In this case, one is going to need the content-length header to be
transmitted as well in the trailing-headers
because one will need to be able to strip away the padding that used to
create the end buffer trigger.
There is another additional header that is required.

Kind Regards,

Wesley OLiver

On Tue, Feb 9, 2021 at 9:02 AM Wesley Oliver <[email-masked]> wrote:

> The other thing that would be nice is that one can override the HTTP
> status and cache status, however, still follow up with an additional
> I think that a new name for AcceptTrailers CacheControl would be needed
> and changed to something like
>> starts sending Cache-Control and/or End-Status fields, then it has no
>> way to know if the client will actually interpret them.
>>

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 09 Feb 2021 22:59:24 +0000

There are special problems with scanning for viruses.  Truncating is not 
reliably safe.  This is because truncated content may still contain 
enough data to be dangerous, and browsers don't delete downloads on 
truncation (none of them did last time I checked anyway).  So if you 
want your virus to go through a scanning intermediary, just pad it out 
first so the payload after truncation is still virulent.

Browsers probably should fix their treatment of truncated content.  E.g. 
close on connection during chunked download without 0\r\n\r\n should 
raise some kind of flag?  Same with connection close with C-L prior to 
advertised number of bytes arriving.  The pathological case is the 
HTTP/1.0 mode with no C-L and there's no way to indicate an abort.

Timeouts are the other big problem.  Client agents tend to time out if 
they send a request, and don't get a response in a timeframe they deem 
reasonable.  Scanning intermediaries have to pass something back through 
to prevent clients just retrying.  Spooling and scanning can take a LONG 
time, e.g. large archives, which need unpacking and scanning of 
individual contained files.  This was the impetus behind my I-D for 
using 1xx codes to indicate scanning progress.

There hasn't been really any support on this working group for properly 
supporting applications like content inspection.  HTTP really doesn't 
lend itself to this.  It could of course.  Some kind of directive in 
trailers could be an interesting approach, but as you say would require 
the client to defer processing until the final directive.

I'd suggest using another 2xx code to indicate that processing should be 
deferred.  And support for this would need to be indicated by the 
client.

Adrien

------ Original Message ------
From: "Wesley Oliver" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Roy T. Fielding" <[email-masked]>; "HTTP Working Group" 
<[email-masked]>
Sent: 6/02/2021 9:52:35 pm
Subject: Re: Cache control in trailers?

>Hi,
>
>Be keen to get through previous list specs, just load shedding for next 
>few hours, so I'll have wait patiently.
>
>error code here or class of codes, for programs that snoop the traffic, 
>so one can easily differentiate them. Mabye that they like old Norton 
>did inject js, inject an identifier, which means for error page or 
>anyone to better debug what's gone wrong and where.
>Waiting for the final status, means that pages would be blocking the 
>client wouldn't be able to process the page and start down loading 
>content, depending on the definition. Because anyone choose not obey 
>starting to process the body. I feel that depending on use cases may 
>like to indicate different behaviours.
>
>another progressive loaded pages, one want to be rendered and on error, 
>two ways to handle it, one continue display current page in progressive 
>non complete form and like popuo page fail, OK to the error page, or 
>mabye browse basically, show split frame or popup error, so error page 
>loads and displays as an overlay, may want top-left top-right 
>bottom-left bottom-right.
>Gather the behaviour in which body should be handled needs have 
>different http response header modes, for like the above behaviours. 
>Which can evolved over time, as people think new ways to do things.
>
>More important part would be ensure caching, drops the body, and ensure 
>it knows that content needs to be revalidated. So whether communicate 
>for cache if body content is kept, so it can display what every it had 
>while revalidstinh if wanted. I am thinking
>Way in which cache gets handled, need be communicated in error, set 
>cache status to must-revalidate, or mabye adjusted shorter failed cache 
>time.
>
>Mabye on there being signal of failure the behaviour of what is to 
>happen to the cache, should be left programmable by a response header, 
>which decide mabye under crazy load what every that must continue fail 
>for 5minutes, instead like the normal 1hour cache or mabye a few weeks. 
>I think allowing http response header, to spesified cache content 
>expiry behaviour override, may be alot more flexible in future as ways 
>in which we use and improve things and instead of pining it down to 
>must revalidate, think having a default behaviour of must revalidate, 
>if now http response header for onerror or httpstatus code map cache 
>behaviours.

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 10 Feb 2021 08:30:42 +0100 (CET)

On Tue, 9 Feb 2021, Adrien de Croy wrote:

> Browsers probably should fix their treatment of truncated content.

That's *exactly* what I thought when I grabbed that work item in the Firefox 
code back in 2014. I went into that task with confidence based on the 
knowledge that curl is not that lax on these areas. Yeah, I'm that naive! =)

Several months later, after having tried lots of things (that broke the web 
experience for a lot of peeps), I mostly gave up [1].

> close on connection during chunked download without 0\r\n\r\n should raise 
> some kind of flag.

Back then at least, there were widely used web components that generated such 
contents with enough users that we got a fair amount of bug reports when we 
tried it. More specifically they often left out the final zero length chunk.

> Same with connection close with C-L prior to advertised number of bytes 
> arriving.

It's been a while since I did this of course, but the wrong C-L case was still 
then fairly common for gzip-compressed content. At least common enough that we 
deemed we couldn't impose this behavior strictly on users.

I don't know if others have revisited these things since I left Mozilla in 
late 2018.

[1] = 
https://daniel.haxx.se/blog/2014/10/26/stricter-http-1-1-framing-good-bye/

-- 

  / daniel.haxx.se

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 10 Feb 2021 10:01:57 +0100

Hi Daniel,

On Wed, Feb 10, 2021 at 08:30:42AM +0100, Daniel Stenberg wrote:
> > raise some kind of flag.
> Back then at least, there were widely used web components that generated
> such contents with enough users that we got a fair amount of bug reports
> when we tried it. More specifically they often left out the final zero
> length chunk.

As a user, I hate it when an interrupted download is suddenly lost. You
may see an image start to display and suddenly disappear, or a download
breaking at 90% due to your ADSL being flaky and having to try again,
etc. For me each downloaded byte is valuable. Of course I would appreciate
it if my browser asked me "this transfer was interrupted, it may be a bug
on the site, a network condition error or an anti-malware detecting
dangerous content, what do you want to do, try again, try to continue,
abort, keep it truncated ?" and that would be done.

One of the problems with the state of the web today *is* that we try to
focus too much on user experience and avoid as much as possible to mention
possible breakage. If sites started to read everywhere about them causing
popups on browsers on each and every listing of articles on sale because
a CRLF is missing after a chunk, you can be sure that for most important
sites it would already be fixed. As long as we silently fix, users are
happy, and is we break without explanation, users rightfully blame the
browser.

Just my two cents,
Willy

---

From: Daniel Stenberg <[email-masked]>
Date: Wed, 10 Feb 2021 10:24:06 +0100 (CET)

On Wed, 10 Feb 2021, Willy Tarreau wrote:

> As a user, I hate it when an interrupted download is suddenly lost. You may 
> see an image start to display and suddenly disappear, or a download breaking 
> at 90% due to your ADSL being flaky and having to try again, etc. For me 
> each downloaded byte is valuable.

A cut off download can in most cases get retried and even resumed. Finding out 
that it was truncated does not need to equal disappearing. That's a UI 
problem, not a HTTP problem.

A cut off javascript download could probably become a security problem if it 
was cut off at exactly a very unfortunate point in the script. But again, not 
a HTTP problem really.

> Of course I would appreciate it if my browser asked me "this transfer was 
> interrupted, it may be a bug on the site, a network condition error or an 
> anti-malware detecting dangerous content, what do you want to do, try again, 
> try to continue, abort, keep it truncated ?" and that would be done.

If that happened in N% of your sessions and that other browser wouldn't annoy 
the user as much, users might find that as reason to just go to the other 
instead...

> possible breakage.

I'm with you a 100% on this, but that doesn't actually help much.

I claim the reason "we" are this sensitive is because how the browser economy 
works. Users equals money to browsers so if you scare off your users to make 
them go use another browser that doesn't show the scary warning or is more 
okay with "broken" sites, the browser vendor lose money. They rather keep the 
users and instead compromise on protocol strictness (or at least hide some of 
the issues).

-- 

  / daniel.haxx.se

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 10 Feb 2021 10:34:25 +0100

On Wed, Feb 10, 2021 at 10:24:06AM +0100, Daniel Stenberg wrote:
> > Of course I would appreciate it if my browser asked me "this transfer
> > was interrupted, it may be a bug on the site, a network condition error
> > or an anti-malware detecting dangerous content, what do you want to do,
> > try again, try to continue, abort, keep it truncated ?" and that would
> > be done.
> 
> If that happened in N% of your sessions and that other browser wouldn't
> annoy the user as much, users might find that as reason to just go to the
> other instead...

Sure and this has been used as an argument against user notifications
for quite a while, but it's more of a matter of having the few remaining
players agree on a strategy here. After all, they all managed to make it
really painful to access sites behind expired certificates. After all, we
now all suffer from the annoyingly stpuid GDPR banners on every site which
announces that it will set a cookie to remember our refusal of cookies, is
way more annoying than an occasional browser pop-up, and we're all forced
to live with that.

> I claim the reason "we" are this sensitive is because how the browser
> economy works. Users equals money to browsers so if you scare off your users
> to make them go use another browser that doesn't show the scary warning or
> is more okay with "broken" sites, the browser vendor lose money. They rather
> keep the users and instead compromise on protocol strictness (or at least
> hide some of the issues).

Yes, I can understand this point, which is also where standards definition
can help get all of them on the same boat for very specific stuff, without
the fear of losing users.

Cheers,
Willy

---

From: David Benjamin <[email-masked]>
Date: Wed, 10 Feb 2021 12:15:31 -0500

(This thread has gotten extremely long. I've only read the last few
messages.)

On browser handling of truncation, I believe Chrome already enforces
truncation checks in HTTP/1.1. If we don't get the full Content-Length
bytes or if the final zero-length chunk is missing, the network stack will
report an error. The one hack we need is some buggy servers will send the
unencoded length rather than the encoded one. So if we see a truncation but
Content-Length exactly matches the decoded length, we'll clear the error
before reporting it to the caller. (But I believe that happens above all
the connection reuse logic, so protocol-level framing still cares.)

What the calling code does with the error depends on the codepath. Scripts
and CSS only execute atomically (security requirement). HTML is rendered
streaming and, as far as I know, doesn't really care whether there was an
error at the end. Not sure about images. Downloads show an error in the UI.

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 10 Feb 2021 20:58:44 +0000

This makes me wonder whether it would be useful to have a header that a 
scanning intermediary could use to indicate that an abort on the 
connection should be treated as fatal.

So the thing that has the information can impart it to the thing that 
needs it.

------ Original Message ------
From: "Daniel Stenberg" <[email-masked]>
To: "Willy Tarreau" <[email-masked]>
Cc: "Adrien de Croy" <[email-masked]>; "Wesley Oliver" 
<[email-masked]>; "HTTP Working Group" <[email-masked]>
Sent: 10/02/2021 10:24:06 pm
Subject: Re: Cache control in trailers?

>>As a user, I hate it when an interrupted download is suddenly lost. You may see an image start to display and suddenly disappear, or a download breaking at 90% due to your ADSL being flaky and having to try again, etc. For me each downloaded byte is valuable.
>
>A cut off download can in most cases get retried and even resumed. Finding out that it was truncated does not need to equal disappearing. That's a UI problem, not a HTTP problem.
>
>A cut off javascript download could probably become a security problem if it was cut off at exactly a very unfortunate point in the script. But again, not a HTTP problem really.
>
>>Of course I would appreciate it if my browser asked me "this transfer was interrupted, it may be a bug on the site, a network condition error or an anti-malware detecting dangerous content, what do you want to do, try again, try to continue, abort, keep it truncated ?" and that would be done.
>
>If that happened in N% of your sessions and that other browser wouldn't annoy the user as much, users might find that as reason to just go to the other instead...
>
>I claim the reason "we" are this sensitive is because how the browser economy works. Users equals money to browsers so if you scare off your users to make them go use another browser that doesn't show the scary warning or is more okay with "broken" sites, the browser vendor lose money. They rather keep the users and instead compromise on protocol strictness (or at least hide some of the issues).
>
>--
>  / daniel.haxx.se

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 10 Feb 2021 22:37:23 +0100

On Wed, Feb 10, 2021 at 08:58:44PM +0000, Adrien de Croy wrote:
> scanning intermediary could use to indicate that an abort on the connection
> should be treated as fatal.
> 
> So the thing that has the information can impart it to the thing that needs
> it.

This sounds like an excellent idea. Maybe something for the Proxy-Status
header field ?

Willy