Subject: Add extension work to Interop matrix

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 7 Jan 2020 09:31:15 -0500

Hello,

There are now a few extensions that have been proposed and implemented.
We should add them to the next Interop matrix.  I suggest using the
following codes:

    g   Datagram extension
    r   HTTP priorities extension
    q   Loss bits extension ('l' looks too much like '1')
    1   One-way delay extension

'g' and 'r' can be added to the third row, while 'q' and '1' can be
added to the second row (dropping the "use uppercase letters only"
requirement).

  - Dmitri.

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 7 Jan 2020 15:16:48 +0000

> added to the second row (dropping the "use uppercase letters only"

Personally I disagree with adding extension-related letters to the matrix.
Some of these are in call for adoption, while others are far off that. I'd
rather keep the main matrix for features of the core documents.

Its great that people want to test these extensions out, but where its
difficult to see why these extensions should be included an not others.
We're at 19 "codes" right now, and not many implementations satisfy them
all. I'd like to find a solution that is more sustainable in the longer
term that can support extensions along with QUIC v2 or whatever comes next.
I don't know what that looks like but I'd take some inspiration from things
like the QUIC Network Simulator interop runner and Web Platform Tests (
https://wpt.fyi/results/?label=experimental&label=master&aligned).

Cheers
Lucas

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 7 Jan 2020 10:56:27 -0500

> >     1   One-way delay extension
--- 8< --- SNIP --- 8< ---
> difficult to see why these extensions should be included an not others.

I did not state that we should exclude any extensions.  I believe
the list above is exhaustive.  If not, I am all for adding to it.

> term that can support extensions along with QUIC v2 or whatever comes next.

The good thing about the current form of the Interop matrix is that
it is easy to start a new sheet and that older matrices become useless
rapidly.  The English alphabet has 26 letters, so we are using
19 / (26 * 2 + 10) = 31% of all easily readable code available to us.
Going up to 23 codes (37%) is not a big deal.  I don't expect us to
run out.  Even if we do, we can always do something new with the
next version of the matrix.

(I considered proposing E1, E2, or Eq, Eg, but one-letter codes are
simpler and fit with the current model.)

> https://wpt.fyi/results/?label=experimental&label=master&aligned).

My proposal is just to capture extension work in a simple, low entry
cost, manner.  I am not willing to commit to create a framework such
as to which you refer above.

  - Dmitri.

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 7 Jan 2020 17:46:13 +0000

> the list above is exhaustive.  If not, I am all for adding to it.

A salient view of QUIC-related I-Ds is at
https://datatracker.ietf.org/wg/quic/documents/ which is 19 documents
strong, albeit some do not relate to specific implementation work e.g. use
cases. No doubt there are others not accounted for on that page.

The recent email threads related to the topic "Re-chartering for extension
work" [1] show there are a lot of people that want to discuss a range of
topics.

> as to which you refer above.

I think the simple approach will scale poorly given the active discussion
about QUIC extensibility. There are three aspects to consider: what to test
and how to test it, where to capture results, and how to represent results.
Today we achieve that with "Implementation drafts" and the interop matrix.
Your proposal addresses the second and third aspects. To address the first
one, we could brake it into a set of questions:

1) Should QUIC adopted extensions be in scope for the next interop?
  * Datagram and Version Negotiation are in calls for adoption ending on
January 15.
2) Should QUIC non-adopted extensions be in scope for the next interop?
  * If so, which ones?
3) Who is willing to help define or review interoperability tests for the
above?
4) Who is interested in testing during the interop event?
5) Is it important to record the results?
  * If so, where?

The answers to these questions then help to decide if testing is
incorporated into an implementation draft and the results included in the
Interop matrix.

Personally, I have interest in interop testing of Datagram. I don't think
it needs to be included in the implementation draft nor the interop matrix
but capturing the results in some other location might be interesting.

Cheers
Lucas

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 7 Jan 2020 18:20:21 +0000

An interop is not an IETF event, so its “scope and agenda” is whatever the people who show up want to work on.   If more than one implementation is interested in implementing an extension, it seems simple to designate a letter and have them go at it.  We’ll not run out of letters (60+ case-sensitive alphanumerics).

  *   Igor

From: Lucas Pardue <[email-masked]>
Sent: Tuesday, January 7, 2020 12:46 PM

On Tue, Jan 7, 2020 at 3:56 PM Dmitri Tikhonov <[email-masked]<mailto:[email-masked]>> wrote:
I did not state that we should exclude any extensions.  I believe
the list above is exhaustive.  If not, I am all for adding to it.

A salient view of QUIC-related I-Ds is at https://datatracker.ietf.org/wg/quic/documents/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_wg_quic_documents_&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=NH5Oty6zwbwiXZ3F7JSL5CuvXbThJpB1Kivfb-biWb8&s=bk8-1eXj1pTq4sN0iOEyGbMwwr4ExjZENuCX2ZBWrQQ&e=> which is 19 documents strong, albeit some do not relate to specific implementation work e.g. use cases. No doubt there are others not accounted for on that page.

The recent email threads related to the topic "Re-chartering for extension work" [1] show there are a lot of people that want to discuss a range of topics.

> https://wpt.fyi/results/?label=experimental&label=master&aligned<https://urldefense.proofpoint.com/v2/url?u=https-3A__wpt.fyi_results_-3Flabel-3Dexperimental-26label-3Dmaster-26aligned&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=NH5Oty6zwbwiXZ3F7JSL5CuvXbThJpB1Kivfb-biWb8&s=kMvFr_9jKMBPTRiarfRxr44oC2kMeGYR3exUj1T8VVA&e=>).

I think the simple approach will scale poorly given the active discussion about QUIC extensibility. There are three aspects to consider: what to test and how to test it, where to capture results, and how to represent results. Today we achieve that with "Implementation drafts" and the interop matrix. Your proposal addresses the second and third aspects. To address the first one, we could brake it into a set of questions:

1) Should QUIC adopted extensions be in scope for the next interop?
  * Datagram and Version Negotiation are in calls for adoption ending on January 15.
2) Should QUIC non-adopted extensions be in scope for the next interop?
  * If so, which ones?
3) Who is willing to help define or review interoperability tests for the above?
4) Who is interested in testing during the interop event?
5) Is it important to record the results?
  * If so, where?

The answers to these questions then help to decide if testing is incorporated into an implementation draft and the results included in the Interop matrix.

Personally, I have interest in interop testing of Datagram. I don't think it needs to be included in the implementation draft nor the interop matrix but capturing the results in some other location might be interesting.

Cheers
Lucas

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 7 Jan 2020 13:49:13 -0500

> about QUIC extensibility.

I get the feeling we are talking past each other.  Do you not
acknowledge the evanescent nature of the Interop matrix?

You want to impose artificial strictures on the way the Interop
results are recorded because it may scale poorly, but this argument
is invalid:  To "scale," we can simply choose to do record things
differently for the next Interop.

  - Dmitri.

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 7 Jan 2020 18:51:48 +0000

On Tue, Jan 7, 2020 at 6:20 PM Lubashev, Igor <[email-masked]> wrote:

> An interop is not an IETF event, so its “scope and agenda” is whatever the
> people who show up want to work on.   If more than one implementation is
> interested in implementing an extension, it seems simple to designate a
> letter and have them go at it.  We’ll not run out of letters (60+
> case-sensitive alphanumerics).
>

That's a fair point. I wasn't suggesting it was the IETF's role to make
such decisions but the ones involved in populating and consuming the matrix
should probably have some framework for discussion how to use it. To date
things have grown a bit organically, with safety rails of a constrained
charter scope and many implementations focusing on the same super-set of
core protocol features.

I'm not so concerned about the availability of codes/letters but the
managability of the the interop matrix as a shared resource. Why is it
important to capture the results of such extension interop in this one
resource? Are all extensions equal? To me it would be just as useful to
record such results in standalone documents, its slightly more work for
those testing extensions but reduces the cognitive overload of those
consuming the presented information for core protocol features.

Cheers
Lucas

---

From: Christian Huitema <[email-masked]>
Date: Tue, 7 Jan 2020 09:06:49 -1000

> added to the second row (dropping the "use uppercase letters only"

Looking at https://interop.seemann.io/, it seems that the worst interop
problem so far is the H09/H3 split. Of the 7 implementations listed
there, we see 2 that do not support H3 and at least one that does not
support H09.

Most of the failures in the "measurement" matrix are related: clients
and servers are programmed to request or serve only H09, and fail if the
peer does not support it. But some propose a list of supported ALPN and
the ALPN negotiation succeeds.

It seems that ALPN negotiation is going to be a practical requirement
going forward. Not so much for negotiating H09 versus H3, we can expect
H09 to fade away at some point. But we will have to negotiate h3-24 vs
h3-25 and similar transitions for a good bit of time, and then we will
probably move to h4-00, h4-01, etc. So maybe we should start testing that.

-- Christian Huitema

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 7 Jan 2020 19:18:09 +0000

On Tue, Jan 7, 2020 at 6:49 PM Dmitri Tikhonov <[email-masked]>
wrote:

> acknowledge the evanescent nature of the Interop matrix

> differently for the next Interop.

The standardisation process is iterative but to date we've followed a
fairly rudimentary pattern where each incarnation of the matrix relates to
a specific implementation draft, which in turn relates to some set of
versioned drafts. The proposal changes the pattern to now consider
extensions. Across the sheets we have little evidence of burning or
substituted letters, so its not clear how people would read into such a
occurrence.

The scaling problem is not invalid. Parsing the interop matrix, even in its
current form, is not user-friendly, especially for those not actively
engaged in the process.

We all share the matrix. Adding letters might help you but it doesn't help
me.

Lucas

---

From: Dmitri Tikhonov <[email-masked]>
Date: Tue, 7 Jan 2020 14:45:44 -0500

> probably move to h4-00, h4-01, etc. So maybe we should start testing that.

I would be for it.  Is the testing as simple as

    1. Client sends (h3-X, h3-Y), server responds with h3-X; and
    2. Client sends (h3-Y, h3-X), server responds with h3-Y?

Of course, this assumes that the server support both h3-X and h3-Y.
For example, supporting drafts 23 and 24 at the same time was easy;
24 and 25 might not be as easy.

  - Dmitri.

---

From: Andrei Popov <[email-masked]>
Date: Tue, 7 Jan 2020 19:49:52 +0000

>     2. Client sends (h3-Y, h3-X), server responds with h3-Y?

Not necessarily. The server will likely choose an ALPN ID based on the server's application protocol preferences, rather than the client's.

Cheers,

Andrei

-----Original Message-----
From: QUIC <[email-masked]> On Behalf Of Dmitri Tikhonov
Sent: Tuesday, January 7, 2020 11:46 AM
To: Christian Huitema <[email-masked]>
Cc: IETF QUIC WG <[email-masked]>
Subject: [EXTERNAL] Re: ALPN negotiation (was Re: Add extension work to Interop matrix)

  - Dmitri.

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 7 Jan 2020 19:51:54 +0000

Probably a good idea.

IIUC I think you could solve the multi- version problem by creating two (or
more) aliases for each version of the same protocol. This means testing of
ALPN ordering but implementations don't need to support protocol level
changes.

The upshot is that this could also be used for Alt-Svc testing.

Lucas

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 7 Jan 2020 22:42:19 +0000

Lucas, I think you are worried that letters were used to identify required features of QUIC V1, and adding optional “extension” letters to the mix would cause confusion.  Is that correct?

A single matrix with letters proved to be an effective way to summarize and report on the interop, so it is pretty clear why people interested in implementing extensions want to record their progress in this way.  Would it make more sense to you if there was a clear separation of QUIC V1 feature interop from Extension interop results?  Like two matrices: “QUIC V1” and “Extensions”?

  *   Igor

On Tue, Jan 7, 2020 at 6:49 PM Dmitri Tikhonov <[email-masked]<mailto:[email-masked]>> wrote:

The standardisation process is iterative but to date we've followed a fairly rudimentary pattern where each incarnation of the matrix relates to a specific implementation draft, which in turn relates to some set of versioned drafts. The proposal changes the pattern to now consider extensions. Across the sheets we have little evidence of burning or substituted letters, so its not clear how people would read into such a occurrence.

The scaling problem is not invalid. Parsing the interop matrix, even in its current form, is not user-friendly, especially for those not actively engaged in the process.

We all share the matrix. Adding letters might help you but it doesn't help me.

Lucas

  - Dmitri.

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 7 Jan 2020 23:27:46 +0000

On Tue, 7 Jan 2020, 22:43 Lubashev, Igor, <[email-masked]> wrote:

> Lucas, I think you are worried that letters were used to identify required
> features of QUIC V1, and adding optional “extension” letters to the mix
> would cause confusion.  Is that correct?
>

Yes my main concern is avoiding overloading things.

>  A single matrix with letters proved to be an effective way to summarize
> and report on the interop, so it is pretty clear why people interested in
> implementing extensions want to record their progress in this way.  Would
> it make more sense to you if there was a clear separation of QUIC V1
> feature interop from Extension interop results?  Like two matrices: “QUIC
> V1” and “Extensions”?
>
I think this is a good articulation of the benefit. Adding non WG
extensions to the matrix makes reporting on interop during the QUIC
sessions clunky.

IMO separating out V1 from other stuff is useful. Whether that's a single
matrix for all extensions or a collection of them is something that can be
worked through.

I am supportive of extension work and now seems like a good time to distill
what worked well and create some patterns to help and empower people,
rather than have them reinvent or feel restricted.

Extensions vary in their complexity, so it's hard to predict what letters
would be useful to articulate each features and interop tests. Just
considering DATAGRAM, this immediately gets complicated with the nuance of
HTTP/3 datagram. I can think of several useful tests in this domain. We
made some interop headway in Singapore, it would be nice to have a record
of that.

Cheers
Lucas

---

From: Christian Huitema <[email-masked]>
Date: Tue, 7 Jan 2020 21:20:10 -1000

On 1/7/2020 9:51 AM, Lucas Pardue wrote:
> Probably a good idea. 
>
> IIUC I think you could solve the multi- version problem by creating
> two (or more) aliases for each version of the same protocol. This
> means testing of ALPN ordering but implementations don't need to
> support protocol level changes.
>

The aliasing part is interesting, but it requires some WG conventions,
e.g. stating that "hq-24" and "a24-hq" are equivalent. We would have to
agree on the convention as part of interop testing. Also, if the list
proposed by the client is "a24-hq, hq-24", how do you define success?
Negotiating the first value in the list? Negotiating any of the values
in the list?

Related thought: is there an ALPN range reserved for greasing?

-- Christian Huitema

---

From: <[email-masked]>
Date: Wed, 8 Jan 2020 12:19:35 +0000

Hi

+1, I am supporting this since we initiated the work on the spinbit.

Regards
Emile

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of Dmitri Tikhonov
Sent: mardi 7 janvier 2020 15:31
To: IETF QUIC WG
Subject: Add extension work to Interop matrix

Hello,

    g   Datagram extension

  - Dmitri.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Lars Eggert <[email-masked]>
Date: Wed, 8 Jan 2020 14:27:39 +0200

Hi,

On 2020-1-7, at 16:31, Dmitri Tikhonov <[email-masked]> wrote:
> We should add them to the next Interop matrix.

maybe we should add a fourth row for extensions?

I'm with Lucas on that I'd like to keep the sheet focused on protocol features (and now possibly extensions) that have seen WG adoption. The rough idea is that unless we believe that the majority of implementations are going to implement something, it's probably not worthwhile adding it to the sheet.

Lars

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 9 Jan 2020 08:54:20 -0500

That's true.

Then, what would the test be?

  - Dmitri.

---

From: Lucas Pardue <[email-masked]>
Date: Thu, 9 Jan 2020 14:59:28 +0000

While writing up a response I was reminded of David Benjamin's TLS
Extensibility draft, draft-ietf-tls-grease-04 [1], which probably covers
some of the discussion in this thread and defines ALPN grease values.

If we want to do something QUIC specific we may want to incorporate some of
that. Interop tests would probably require some idea of the client knowing
what to expect the server to select, without that, how success is measured
is tricky (as mentioned above).

I'm starting to wonder if such a test becomes a little contrived though.
ALPN for HTTP/1.1 and HTTP/2 seems to be working fine on the Internet.
There might be merit in expanding the scope a little to think about testing
connection establishment including negotiation extension. The example case
is DATAGRAM, where "h3-NN" is agreed in ALPN but there is a mismatch of the
QUIC connection parameters for  max_datagram_frame_size.

[1] https://tools.ietf.org/html/draft-ietf-tls-grease-04

---

From: Andrei Popov <[email-masked]>
Date: Thu, 9 Jan 2020 18:36:56 +0000

The test would need to consider the server's ALPN preferences, e.g.:
Client: (h3-X, h3-Y), server: (h3-Y, h3-X) => server chooses h3-Y.
Client: (h3-X, h3-Y), server: (h3-X, h3-Y) => server chooses h3-X.
Client: (h3-Y, h3-X), server: (h3-Y, h3-X) => server chooses h3-Y.
Client: (h3-Y, h3-X), server: (h3-X, h3-Y) => server chooses h3-X.

There are a lot of other possible test cases.

Cheers,

Andrei

-----Original Message-----
From: Dmitri Tikhonov <[email-masked]> 
Sent: Thursday, January 9, 2020 5:54 AM
To: Andrei Popov <[email-masked]>
Cc: Christian Huitema <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: [EXTERNAL] Re: ALPN negotiation (was Re: Add extension work to Interop matrix)

That's true.

  - Dmitri.

On Tue, Jan 07, 2020 at 07:49:52PM +0000, Andrei Popov wrote:
> > It seems that ALPN negotiation is going to be a practical 
> > requirement going forward. Not so much for negotiating H09 versus 
> > H3, we can expect
> > h3-25 and similar transitions for a good bit of time, and then we 
> > will probably move to h4-00, h4-01, etc. So maybe we should start testing that.
>