Subject: Feedback from first read of QUIC/HTTP draft 08

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 8 Dec 2017 00:45:30 -0800

This was originally #1001, but moved here on upon request.

Reading through the QUIC/HTTP-draft-08 a few things appeared unclear. This
is just a summary based on my first pass over the document, so take it for
what it is. I have only superficial knowledge of HTTP/2 in advance.

   1. A bit too much discussion of HTTP/2 instead of leaving that for
   section 8. It does not help understanding QUIC/HTTP from scratch. And
   perhaps a bit too much assumption about HTTP/2 knowledge while implementers
   might have skipped that step in their evolution.
   2. No explanation of how methods are communicated. Header fields
   reference HTTP 1.1 draft, but from skimming HTTP/2 doc, it is clear that
   these (as suspected) are transfered via special headers known as pseudo
   headers).
   3. Lots of frame type flags, none used, and only a few types. Why not
   either remove flags and have more types, or have a type bit encoding for
   flags presence (this may be due to early state of draft, just mentioning).
   4. Why are settings 16 bits instead of varints, and why is the space so
   large. A 256 or even 2K space would be easy to have in a table, 64K
   requires a map or switching logic. A settings space still requires that
   both ends understands a signficant portion of settings to make any real
   sense. Who decides to add more settings. Is this to have space for the next
   century, or just until the next QUIC version?
   5. There is an End Header Block something, but it is unclear what it is.
   Is it a flag to be defined, or some pseudo header?
   6. The GOWAY discussion could probably be shortened and made it bit
   clearer - it takes a few readings to understand the point of multiple GOWAY
   frames and why the ID can decrease (not sure I do fully understand).
   7. A basic example of a short HTTP 1.1 request and how it looks in the
   protocol would be helpful.
   8. The HPACK, to be replaced, doc is hard to parse, and it doesn't sound
   like it is as complicated as it reads. Hopefully a future QUIC equivalent
   would be a bit easier to digest.
   9. The entire push hierarchy sounds very complex - I wonder what the
   takeaway from HTTP/2 is - is it really worthwhile, or is no-one using it
   yet, but in 10 years it will be huge?

---

From: Dmitri Tikhonov <[email-masked]>
Date: Fri, 8 Dec 2017 08:01:23 -0500

On Fri, Dec 08, 2017 at 12:45:30AM -0800, Mikkel Fahnøe Jørgensen wrote:
>    8. The HPACK, to be replaced, doc is hard to parse

There are three HPACK replacement proposals (neither of which is
referenced by [draft-ietf-quic-http-08]): QPACK, QCRAM, and QMIN.
Which one are you referring to?

>    and it doesn't sound like it is as complicated as it reads.
>    Hopefully a future QUIC equivalent would be a bit easier to
>    digest.

If you offered some specific examples, I think it would be appreciated:
we want to make the documents comprehensible on the first read.

  - Dmitri.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 8 Dec 2017 09:40:05 -0500

I’m just referring to the fact that I’m aware the HPACK is not the long
term solution, but this was the document I was referring to. I have not
been looking into Q* yet.

My intuition is that HPACK could largely be described by the following
text, although details may be a bit off as I haven’t digged into all the
details:

HPACK represents HTTP headers in a compact form. Every valid name string
and value string can be stored as a HPACK field. A valid name or value
string conforms to HTTP 1.1 strings without space, ‘:’ and CRLF delimited.
A name may also be a pseudo-name starting with ‘:’ for example ‘:method’
used in HTTP/2.

The fields are conceptually ordered similar to how the text representation
is ordered in HTTP 1.1 such that a similar header can be recreated after
decoding. The result of decoding a HPACK stream is exactly such a list, and
updated internal state which makes the next exchange smaller in size. This
internal state is kept in the dynamic name table discussed below.

Some header names are used often and have a dedicated encoding in a static
table. Some names are less general but may still be reused across requests,
such names can be stored in a dynamic table for reuse. And finally some
names should not compressed for security reasons or because it is not
deemed efficient. The sender (encoder) decides which form to use, but can
only choose the static form if the name is in the static table. Names are
always case insensitive.

The header values are always just explicit strings that are not compressed
(or are they? - did not get that far in the text). We refer to such a
string as ValueString.

For security reasons and implementation simplicity, the format is simple
and not extensible. Instead HPACK as a whole may be replaced by another
compression if needed. HPACK depends on the sender (encoder) sending all
operations in a specific order such that the decoder can reverse the
encoding unambiguously.

In the following the three forms of name encoding are discussed further,
followed by how the entire list is encoded.

The static table is list of known names that is a assigned an index between
1 and K, where K is the length of the static table. Each name is
represented as an integer index into this table. We call this value
StaticName.

The following may not be exactly correct, but this is how I imagine it work
from a quick skim.

The dynamic table has a fixed size which is negotiated. The size may
change, but always in a way that both sender and receiver is aware of it
and only in a way the receiver permits. If the sender grows the table
beyond the granted limit, this is a fatal error. The current size of the
dynamic table is N. The table may not use all entries. If a name is stored
in the table it is represented by the integer DynamicName which is in the
range (1 + K … N + K). The size N may be zero in which case no names are
encoded as a dynamic name. Otherwise the table is filled up such that the
first name is given the value N + K, the next name the value N + K - 1
until the table fills up. After that, the process starts over with the next
value again being N + K and so forth. The decoder has enough information to
maintain the table as it changes. The encoded issues commands to add names
to the table but does not explicitly evict names. The encoder can grow or
shrink the table with a single command. In either case the new size takes
effect once the table wraps the next time around. We refer to the dynamic
table size N as the integer DynamicTableSize.

When an encoder adds a name to the dynamic table, it does not send the name
explicitly but via a huffman compression scheme based on static huffman
codes. The net effect is a command with a bit string containing the
compressed name to be added to the dynamic table. No index information is
needed because the decoder is in sync. The encoder, by definition, does not
need to add names to static table so there is no corresponding operation.
We refer to the huffman encoded name as DynamicNameString.

Finally the name can be stored explicitly without compression. There is no
update command for this. We refer to an explicit string as ExplicitName.

The encoding is a stream of commands which we list below, followed by a
wire format representation.

PushDynamicHeader(DynamicNameSring)

ResizeDynamicTable(DynamicTableSize)

AddImplicitHeader(StaticName | DynamicName, ValueString)

AddExplicitHeader(ExplicitName, ValueString)

Done()

The encoder sends a list of commands in a very specific order. An
implementation is free to choose how it orders these commands, except
Done() comes last and AddImplicitHeader must only be called when there is a
valid entry in the table being addressed.

The decoder processes these commands in order and aborts the operation in
case of any violation. The result of the decoding process is an order list
of name/value pairs with exactly one field for each AddImplicit/Explicit
commands. In addition the decoder now potentially has a populated dynamic
table that can be reused by the next request which must be handled strictly
after the previous in order to maintain sync. If the decoder wants to
reduce or increase its dynamic table it can send this information to the
decoder via command X (TBD). This is an async operation. The decoder cannot
require the sender to go below a previous sent limit, but it can allow it
to grow larger and advise it to shrink. The initial dynamic table size is
an out of band parameter similar to the huffman table and the static table.

An example is the following HTTP1.1 header where the status line with the
GET method is replaced by HTTP/2 pseudo names:

… example to follow …

The dynamic table then looks like: ...

The Huffman encoding is the list of symbols in appendix A along with the
huffman algorithm,

The static table is the list names in appendix B, or another set of names
if agreed out of band.

The wire format of ValueString is …

The wire format of  DynamicName is …

The wire format of ExplicitString is ….

The wire format of command X header is …

Appendix...

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 8 Dec 2017 15:02:54 +0000

Hi Mikkel,

There was a recent thread on httpbis mailing list where we touched on the idea of HTTP header compression independent of version mapping. HPACK is quite a generic thing really - there's not much that is specific to HTTP/2.

Your proposed text seems related to that thread, where do you see that belonginging? Do you think that RFC 7838 does not already explain it in an appropriate form?

Lucas

________________________________________
From: QUIC [[email-masked]] on behalf of Mikkel Fahnøe Jørgensen [[email-masked]]
Sent: 08 December 2017 14:40
To: Dmitri Tikhonov
Cc: IETF QUIC WG
Subject: Re: Feedback from first read of QUIC/HTTP draft 08

I’m just referring to the fact that I’m aware the HPACK is not the long term solution, but this was the document I was referring to. I have not been looking into Q* yet.

My intuition is that HPACK could largely be described by the following text, although details may be a bit off as I haven’t digged into all the details:

HPACK represents HTTP headers in a compact form. Every valid name string and value string can be stored as a HPACK field. A valid name or value string conforms to HTTP 1.1 strings without space, ‘:’ and CRLF delimited. A name may also be a pseudo-name starting with ‘:’ for example ‘:method’ used in HTTP/2.

The fields are conceptually ordered similar to how the text representation is ordered in HTTP 1.1 such that a similar header can be recreated after decoding. The result of decoding a HPACK stream is exactly such a list, and updated internal state which makes the next exchange smaller in size. This internal state is kept in the dynamic name table discussed below.

Some header names are used often and have a dedicated encoding in a static table. Some names are less general but may still be reused across requests, such names can be stored in a dynamic table for reuse. And finally some names should not compressed for security reasons or because it is not deemed efficient. The sender (encoder) decides which form to use, but can only choose the static form if the name is in the static table. Names are always case insensitive.

The header values are always just explicit strings that are not compressed (or are they? - did not get that far in the text). We refer to such a string as ValueString.

For security reasons and implementation simplicity, the format is simple and not extensible. Instead HPACK as a whole may be replaced by another compression if needed. HPACK depends on the sender (encoder) sending all operations in a specific order such that the decoder can reverse the encoding unambiguously.

In the following the three forms of name encoding are discussed further, followed by how the entire list is encoded.

The static table is list of known names that is a assigned an index between 1 and K, where K is the length of the static table. Each name is represented as an integer index into this table. We call this value StaticName.

The following may not be exactly correct, but this is how I imagine it work from a quick skim.

The dynamic table has a fixed size which is negotiated. The size may change, but always in a way that both sender and receiver is aware of it and only in a way the receiver permits. If the sender grows the table beyond the granted limit, this is a fatal error. The current size of the dynamic table is N. The table may not use all entries. If a name is stored in the table it is represented by the integer DynamicName which is in the range (1 + K … N + K). The size N may be zero in which case no names are encoded as a dynamic name. Otherwise the table is filled up such that the first name is given the value N + K, the next name the value N + K - 1 until the table fills up. After that, the process starts over with the next value again being N + K and so forth. The decoder has enough information to maintain the table as it changes. The encoded issues commands to add names to the table but does not explicitly evict names. The encoder can grow or shrink the table with a single command. In either case the new size takes effect once the table wraps the next time around. We refer to the dynamic table size N as the integer DynamicTableSize.

When an encoder adds a name to the dynamic table, it does not send the name explicitly but via a huffman compression scheme based on static huffman codes. The net effect is a command with a bit string containing the compressed name to be added to the dynamic table. No index information is needed because the decoder is in sync. The encoder, by definition, does not need to add names to static table so there is no corresponding operation. We refer to the huffman encoded name as DynamicNameString.

Finally the name can be stored explicitly without compression. There is no update command for this. We refer to an explicit string as ExplicitName.

The encoding is a stream of commands which we list below, followed by a wire format representation.

Done()

The encoder sends a list of commands in a very specific order. An implementation is free to choose how it orders these commands, except Done() comes last and AddImplicitHeader must only be called when there is a valid entry in the table being addressed.

The decoder processes these commands in order and aborts the operation in case of any violation. The result of the decoding process is an order list of name/value pairs with exactly one field for each AddImplicit/Explicit commands. In addition the decoder now potentially has a populated dynamic table that can be reused by the next request which must be handled strictly after the previous in order to maintain sync. If the decoder wants to reduce or increase its dynamic table it can send this information to the decoder via command X (TBD). This is an async operation. The decoder cannot require the sender to go below a previous sent limit, but it can allow it to grow larger and advise it to shrink. The initial dynamic table size is an out of band parameter similar to the huffman table and the static table.

An example is the following HTTP1.1 header where the status line with the GET method is replaced by HTTP/2 pseudo names:

… example to follow …

The Huffman encoding is the list of symbols in appendix A along with the huffman algorithm,

The static table is the list names in appendix B, or another set of names if agreed out of band.

Appendix...

-----------------------------
http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and
may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in
error, please delete it from your system.
Do not use, copy or disclose the
information in any way nor act in reliance on it and notify the sender
immediately.
Please note that the BBC monitors e-mails
sent or received.
Further communication will signify your consent to
this.
-----------------------------

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Fri, 8 Dec 2017 10:39:21 -0500

On 8 December 2017 at 16.02.56, Lucas Pardue ([email-masked]) wrote:

Hi Mikkel,

There was a recent thread on httpbis mailing list where we touched on the
idea of HTTP header compression independent of version mapping. HPACK is
quite a generic thing really - there's not much that is specific to HTTP/2.

Your proposed text seems related to that thread, where do you see that
belonginging? Do you think that RFC 7838 does not already explain it in an
appropriate form?

Hi Lucas,

My original text was really on the first impressions of QUIC/HTTP and not
on HPACK as such and hence the HTTP/2 reference. HPACK was just a single
bullet noting that my initial impression of the HPACK text (admittedly
quite late) was a fair bit more confusing than I would have expected. When
asked to provide more details on why I found HPACK a bit complicated, I
opted to suggest a simpler text as an example based on my overall
understanding - just a quick write down. However, I cannot say if the HPACK
has a good reason to be as it is without reading the document very
carefully and understanding all the details. So my input should be taken
from a naive first readers perspective. Those with more experience can
better judge if the text can be used a guideline to simplify future packing
algorithms, but as I stated in the beginning, take it for what it is.

On the actual packing algorithms, I could have an opinion, but right now
that is fairly low on my list of things to focus on. I’m sure someone has
thought long and hard over async complications. I can say though, is that
overall ZSTD looks like a promising general purpose compression library
that does support pre-registered dictionaries and now has a more liberal
license than it used to have.

As to QUIC/HTTP itself, I was a bit put off a bit about HTTP/2 prose. It is
not necessarily bad or unwarranted, but from my perspective I didn’t feel
the need to dig into HTTP/2 unless there was a very specific reason such
as: this is the list of permitted pseudoheaders and defined in HTTP/2
section x.

And finally this wasn’t meant to put down the efforts placed on either
document. But I consider the initial uninformed impressions valuable
because once you understand a problem, you can no longer see where others
struggle and this is why I shared this.

---

From: Lucas Pardue <[email-masked]>
Date: Fri, 8 Dec 2017 16:20:06 +0000

Hi Mikkel,

I didn't mean to sound adversarial, rather in that httpbis thread I had raised a question about extracting common semantics out of the HTTP/2 doc; your text seems quite aligned to that idea which I had independently. To give a different example, HTTP server push could, IMO, benefit from be abstracted away from the HTTP/2 specifics in which it is defined. On that topic, I raised a few issues with push in HTTP/QUIC where I think some editorial changes could help improve its readability. There are other changes (like uni streams) that needed to be shaken out before tackling improvements, so somebody issues remain open. (I'll get around to doing a PR soon).

Fresh eyes are great for the very reason you describe. I'm sure many people wouldn't go to the effort of writing up issues or suggesting improvents, so thanks for doing so. I think I agreed with most of what you wrote up.

Regards
Lucas

Ps in my previous email I meant to say RFC 7541 (HPACK). 

________________________________________
From: Mikkel Fahnøe Jørgensen [[email-masked]]
Sent: 08 December 2017 15:39
To: Lucas Pardue; Dmitri Tikhonov
Cc: IETF QUIC WG
Subject: RE: Feedback from first read of QUIC/HTTP draft 08

On 8 December 2017 at 16.02.56, Lucas Pardue ([email-masked]<mailto:[email-masked]>) wrote:
Hi Mikkel,

Hi Lucas,

My original text was really on the first impressions of QUIC/HTTP and not on HPACK as such and hence the HTTP/2 reference. HPACK was just a single bullet noting that my initial impression of the HPACK text (admittedly quite late) was a fair bit more confusing than I would have expected. When asked to provide more details on why I found HPACK a bit complicated, I opted to suggest a simpler text as an example based on my overall understanding - just a quick write down. However, I cannot say if the HPACK has a good reason to be as it is without reading the document very carefully and understanding all the details. So my input should be taken from a naive first readers perspective. Those with more experience can better judge if the text can be used a guideline to simplify future packing algorithms, but as I stated in the beginning, take it for what it is.

On the actual packing algorithms, I could have an opinion, but right now that is fairly low on my list of things to focus on. I’m sure someone has thought long and hard over async complications. I can say though, is that overall ZSTD looks like a promising general purpose compression library that does support pre-registered dictionaries and now has a more liberal license than it used to have.

As to QUIC/HTTP itself, I was a bit put off a bit about HTTP/2 prose. It is not necessarily bad or unwarranted, but from my perspective I didn’t feel the need to dig into HTTP/2 unless there was a very specific reason such as: this is the list of permitted pseudoheaders and defined in HTTP/2 section x.

And finally this wasn’t meant to put down the efforts placed on either document. But I consider the initial uninformed impressions valuable because once you understand a problem, you can no longer see where others struggle and this is why I shared this.