Subject: Re: [tsvwg] L4S drafts: Next Steps

From: Bob Briscoe <[email-masked]>
Date: Wed, 10 Mar 2021 12:53:14 +0000

David,

As you've just pointed this old thread out, I'll continue on this 
thread. See [BB] inline...

On 09/12/2020 22:28, Black, David wrote:
>
> Bob,
>
> Reminder of the goal for [1] (Rework the “Prague Requirements” into 
> the following two entities ...):
>
>      >> Success of this exercise will be judged by the WG reaching 
> “rough consensus” that multiple transport protocol implementations
>      >> currently meet and/or are reasonably expected to be able to 
> meet the normative requirements (a).
>
> If the L4S folks believe that this has already been done, then it’s 
> time to start bringing forward the multiple transport protocols that 
> demonstrate success of that exercise.
>

[BB] This is understood and fine. This is where our focus is.

I assume readers are aware that the IETF's requirement for 2 independent 
implementations is for progressions from Proposed Standard to full 
Internet Standard, and few RFCs have ever taken that step (only 118 RFCs 
are STDs or parts of STDs) (e.g. RFC3168 is not).
https://www.rfc-editor.org/search/rfc_search_detail.php?sortkey=Number&sorting=DESC&page=All&pubstatus%5B%5D=Standards%20Track&std_trk=Internet%20Standard

I accept that we're messing with core protocols here, so the bar should 
be higher. But, 2 independent full implementations for an EXP would be 
way beyond any interpretation of IETF process.

> >Nonetheless, if there are specific items in S.4 that someone believes 
> should or shouldn't be there, or should be worded differently, let's 
> have those discussions now on the list - I'm all ears
>
> Speaking only for myself as an individual beyond this point, these two 
> normative requirements struck me as ones that might be better stated 
> as design and implementation guidelines:
>
>     o  A scalable congestion control MUST eliminate RTT bias as much as
>        possible in the range between the minimum likely RTT and typical
>        RTTs expected in the intended deployment scenario (see
>        Appendix A.1.5  <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-12#appendix-A.1.5>  for rationale).
>     o  A scalable congestion control SHOULD remain responsive to
>        congestion when typical RTTs over the public Internet are
>        significantly smaller because they are no longer inflated by
>        queuing delay (seeAppendix A.1.6  <https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-12#appendix-A.1.6>  for rationale).
>
> The use of “as much as possible” in the first bullet makes it hard to 
> figure out whether a specific congestion control meets that mandatory 
> requirement for participation in the L4S experiment.  The second 
> bullet appears to involve a prediction of the future as currently written.
>

[BB] I'll refrain from getting into text-specifics in this thread, 'cos 
I'm trying to be absolutely sure I understand what the chairs want. 
We've felt subjected to a "No, bring me a different rock" process so 
far, so we want to be absolutely clear what sort of rock you want.

This part of your original email (which was as a chair) asked us to 
/restructure/ the drafts. I said they already have the structure you 
want. No reply to that point. Now what you talk about above (albeit as 
an individual now) are wording changes not restructuring. We've been 
doing wording changes with the survey of implementers. Fine with that.

But I'd like confirmation (as a chair) that restructuring is not needed.

==============================================================================================
FYI, here's the structure of the transport requirements (I already 
explained this in the previous email in this thread):

ecn-l4s-id
* Sec.4. (Normative) Transport Layer Behaviour
     All the factors that involve potential harm to others (containing 
MUSTs and now some SHOULDs)
     With a pointer from each one to further guidance in Appx A.1
* Sec 5 (Normative) Network Node Behaviour
...
* Appx A.1 (Informative) Requirements for Scalable Transport Protocols
     Guidance on features that involve potential harm to others
* Appx A.2 (Informative) Scalable Transport Protocol Optimizations
    Guidance on features that would benefit the flow itself

draft-briscoe-iccrg-prague-congestion-control (just posted)
* Detailed guidance, specification, implementation notes, etc.

Bob

> Thanks, --David
>
> *From:* Bob Briscoe <[email-masked]>
> *Sent:* Tuesday, December 8, 2020 6:01 PM
> *To:* Black, David; [email-masked]
> *Subject:* Re: [tsvwg] L4S drafts: Next Steps
>
> [EXTERNAL EMAIL]
>
> David and chairs,
>
> On 04/12/2020 15:54, Black, David wrote:
>
>     The WG chairs have consulted among ourselves and with our AD
>     (Martin Duke).
>
>     Our guidance to the WG and the L4S draft authors is that at least
>     the following two things need to be done before WGLC will become
>     possible on the L4S drafts:
>
>     [1] Rework the “Prague Requirements” into the following two
>     entities, which will overlap:
>
>     a.Transport-protocol-agnostic normative requirements for all
>     congestion-controlled traffic that uses the L4S low latency queue.
>
>     ·The DualQ AQM design relies upon traffic adhering to these
>     requirements.
>
>     b.Design and implementation guidelines for new congestion controls
>     that meet the normative requirements (a).
>
>     ·Ultimately, the choice of whether a particular aspect is a
>     guideline (b) or requirement (a) is a WG “rough consensus” decision.
>
>
> [BB] I believe draft-ietf-tsvwg-ecn-l4s-id is already structured for 
> this division between normative requirements and design and 
> implementation guidelines.
> * [1]a) is Section 4 "Prerequisite Transport Layer Behaviour". Nearly 
> every paragraph also refers off to subsections of Appendix A.1 for 
> extra non-normative info, headed "Requirements for Scalable Transport 
> Protocols". [Actually, it might be useful to swap these two headings]
> * [1]b) is Appendix A.2: "Scalable Transport Protocol Optimizations"
>
> The rule that determined which aspect went in which was:
> [1]a) if it's about an aspect of one transport's behaviour that has 
> potential to impact/harm others
> [1]b) if it's solely about how a transport can improve it's own 
> performance.
> They have been divided like that from the very first ad hoc meeting in 
> Prague that formulated them (in 2015)
>
> Code to address all the requirements and most of the optimizations has 
> been implemented. Over the last year, 3 or 4 of the later requirements 
> in a) have become SHOULDs not MUSTs. Personally I would have left most 
> as MUSTs, but I've tried to reflect what the WG wanted. Those demoted 
> to SHOULD have generally been considered a lower risk of harm (either 
> low likelihood of occuring or minor harm if they do) - taking into 
> account the complexity of implementing them.
>
> We could certainly flag the point where the transition from MUSTs to 
> SHOULDs occurs within section 4. But I think we should still group all 
> the items with potential for harm together in the same section. 
> Because assessment of risk will change as the Internet landscape 
> changes (possibly over the duration of the experiment).
>
> Now I've explained, I hope the chairs will all agree that "rework the 
> Prague Requirements into two entities" (normative requirements and 
> design and implementation guidelines) is already done and dusted.
>
> Nonetheless, if there are specific items in S.4 that someone believes 
> should or shouldn't be there, or should be worded differently, let's 
> have those discussions now on the list - I'm all ears.
>
>
>
> Bob
>
>
>          Success of this exercise will be judged by the WG reaching
>     “rough consensus” that multiple transport protocol implementations
>          currently meet and/or are reasonably expected to be able to
>     meet the normative requirements (a).
>
>     [2] Build the safety case for L4S experimental Internet-wide
>     deployment in the L4S Ops draft.
>
>     a.This safety case does not rely on endpoints running TCP Prague,
>     though it can assume endpoints are meeting the reworked Prague
>     requirements
>
>     b.Ops draft will provide guidance on how to detect L4S problems on
>     their RFC 3168 network, and how to mitigate them.
>
>     c.Ops draft must consider these scenarios:
>
>     · An unsophisticated user purchases an L4S endpoint and runs it on
>     a service provider's single-queue RFC 3168 network.
>
>     · L4S traffic from another domain enters an RFC 3168 single-queue
>     network (e.g. a peer-to-peer application)
>
>          Success of this exercise will be judged by the WG reaching
>     “rough consensus” that deployment of the L4S experiment
>          is unlikely to cause significant damage to the Internet, and
>     that any damage that results is expected to be tolerable.
>
>     Thanks, --David [as TSVWG WG co-chair]
>
>     ----------------------------------------------------------------
>
>     David L. Black, Senior Distinguished Engineer
>
>     Dell Technologies, Infrastructure Systems Group
>
>     176 South St., Hopkinton, MA  01748
>
>     +1 (774) 350-9323 <tel:+17743509323> Mobile: +1 (978) 394-7754
>     <tel:+19783947754>
>
>     [email-masked] <mailto:[email-masked]>
>
>     ----------------------------------------------------------------
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Black, David" <[email-masked]>
Date: Wed, 10 Mar 2021 18:01:22 +0000

Bob,

Reproducing text from the slide used in today's meeting:

------------------------
L4S Drafts: What Needs To Be Done

1. Transport-protocol-independent requirements to use L4S service
- Goal: Interoperability - multiple transport protocols able to meet requirements
* Specifications and/or implementations - WG decides what is sufficient
- Location: draft-ietf-tsvwg-ecn-l4s-id

2. Safety of Internet-wide L4S experiment
- Goal: L4S experiment unlikely to cause significant damage to the Internet, and
* Any damage that results is expected to be tolerable
- Location: draft-white-tsvwg-l4s-ops (candidate for WG adoption)
* WG Last Call on l4s-ops draft not required to meet safety goal

* Criteria: WG "rough consensus" that each goal has been reached
- WG chairs will run initial consensus calls before next IETF meeting week
------------------------

This discussion is about the first goal: Transport-protocol-independent requirements to use L4S service

> I accept that we're messing with core protocols here, so the bar should be higher.
> But, 2 independent full implementations for an EXP would be way beyond any interpretation of IETF process.

Two independent full implementations of protocols that meet the requirements ought to suffice, but may not be necessary.
As stated on the slide (and quoted above): " Specifications and/or implementations - WG decides what is sufficient."

> This part of your original email (which was as a chair) asked us to /restructure/ the drafts. I said they already have the structure you want.
> No reply to that point. Now what you talk about above (albeit as an individual now) are wording changes not restructuring.
> We've been doing wording changes with the survey of implementers. Fine with that.
>

Confirmed.

Thanks, --David (as a TSVWG chair)

From: Bob Briscoe <[email-masked]>
Sent: Wednesday, March 10, 2021 7:53 AM
To: Black, David; [email-masked]
Subject: Re: [tsvwg] L4S drafts: Next Steps

[EXTERNAL EMAIL]
David,

As you've just pointed this old thread out, I'll continue on this thread. See [BB] inline...
On 09/12/2020 22:28, Black, David wrote:
Bob,

Reminder of the goal for [1] (Rework the "Prague Requirements" into the following two entities ...):
     >> Success of this exercise will be judged by the WG reaching "rough consensus" that multiple transport protocol implementations
     >> currently meet and/or are reasonably expected to be able to meet the normative requirements (a).
If the L4S folks believe that this has already been done, then it's time to start bringing forward the multiple transport protocols that demonstrate success of that exercise.

I assume readers are aware that the IETF's requirement for 2 independent implementations is for progressions from Proposed Standard to full Internet Standard, and few RFCs have ever taken that step (only 118 RFCs are STDs or parts of STDs) (e.g. RFC3168 is not).
https://www.rfc-editor.org/search/rfc_search_detail.php?sortkey=Number&sorting=DESC&page=All&pubstatus%5B%5D=Standards%20Track&std_trk=Internet%20Standard

I accept that we're messing with core protocols here, so the bar should be higher. But, 2 independent full implementations for an EXP would be way beyond any interpretation of IETF process.

>Nonetheless, if there are specific items in S.4 that someone believes should or shouldn't be there, or should be worded differently, let's have those discussions now on the list - I'm all ears

Speaking only for myself as an individual beyond this point, these two normative requirements struck me as ones that might be better stated as design and implementation guidelines:

      Appendix A.1.5<https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-12#appendix-A.1.5> for rationale).

      queuing delay (see Appendix A.1.6<https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-12#appendix-A.1.6> for rationale).

The use of "as much as possible" in the first bullet makes it hard to figure out whether a specific congestion control meets that mandatory requirement for participation in the L4S experiment.  The second bullet appears to involve a prediction of the future as currently written.

[BB] I'll refrain from getting into text-specifics in this thread, 'cos I'm trying to be absolutely sure I understand what the chairs want. We've felt subjected to a "No, bring me a different rock" process so far, so we want to be absolutely clear what sort of rock you want.

This part of your original email (which was as a chair) asked us to /restructure/ the drafts. I said they already have the structure you want. No reply to that point. Now what you talk about above (albeit as an individual now) are wording changes not restructuring. We've been doing wording changes with the survey of implementers. Fine with that.

==============================================================================================
FYI, here's the structure of the transport requirements (I already explained this in the previous email in this thread):

ecn-l4s-id
    All the factors that involve potential harm to others (containing MUSTs and now some SHOULDs)

Bob

Thanks, --David

From: Bob Briscoe <[email-masked]><mailto:[email-masked]>
Sent: Tuesday, December 8, 2020 6:01 PM
To: Black, David; [email-masked]<mailto:[email-masked]>
Subject: Re: [tsvwg] L4S drafts: Next Steps

[EXTERNAL EMAIL]
David and chairs,
On 04/12/2020 15:54, Black, David wrote:
The WG chairs have consulted among ourselves and with our AD (Martin Duke).
Our guidance to the WG and the L4S draft authors is that at least the following two things need to be done before WGLC will become possible on the L4S drafts:
[1] Rework the "Prague Requirements" into the following two entities, which will overlap:
1.      Transport-protocol-agnostic normative requirements for all congestion-controlled traffic that uses the L4S low latency queue.
1.      The DualQ AQM design relies upon traffic adhering to these requirements.
2.      Design and implementation guidelines for new congestion controls that meet the normative requirements (a).
1.      Ultimately, the choice of whether a particular aspect is a guideline (b) or requirement (a) is a WG "rough consensus" decision.

[BB] I believe draft-ietf-tsvwg-ecn-l4s-id is already structured for this division between normative requirements and design and implementation guidelines.
* [1]a) is Section 4 "Prerequisite Transport Layer Behaviour". Nearly every paragraph also refers off to subsections of Appendix A.1 for extra non-normative info, headed "Requirements for Scalable Transport Protocols". [Actually, it might be useful to swap these two headings]
* [1]b) is Appendix A.2: "Scalable Transport Protocol Optimizations"

The rule that determined which aspect went in which was:
[1]a) if it's about an aspect of one transport's behaviour that has potential to impact/harm others
[1]b) if it's solely about how a transport can improve it's own performance.
They have been divided like that from the very first ad hoc meeting in Prague that formulated them (in 2015)

Code to address all the requirements and most of the optimizations has been implemented. Over the last year, 3 or 4 of the later requirements in a) have become SHOULDs not MUSTs. Personally I would have left most as MUSTs, but I've tried to reflect what the WG wanted. Those demoted to SHOULD have generally been considered a lower risk of harm (either low likelihood of occuring or minor harm if they do) - taking into account the complexity of implementing them.

We could certainly flag the point where the transition from MUSTs to SHOULDs occurs within section 4. But I think we should still group all the items with potential for harm together in the same section. Because assessment of risk will change as the Internet landscape changes (possibly over the duration of the experiment).

Now I've explained, I hope the chairs will all agree that "rework the Prague Requirements into two entities" (normative requirements and design and implementation guidelines) is already done and dusted.

Nonetheless, if there are specific items in S.4 that someone believes should or shouldn't be there, or should be worded differently, let's have those discussions now on the list - I'm all ears.

Bob

     Success of this exercise will be judged by the WG reaching "rough consensus" that multiple transport protocol implementations
     currently meet and/or are reasonably expected to be able to meet the normative requirements (a).
[2] Build the safety case for L4S experimental Internet-wide deployment in the L4S Ops draft.
1.      This safety case does not rely on endpoints running TCP Prague, though it can assume endpoints are meeting the reworked Prague requirements
2.      Ops draft will provide guidance on how to detect L4S problems on their RFC 3168 network, and how to mitigate them.
3.      Ops draft must consider these scenarios:
1.       An unsophisticated user purchases an L4S endpoint and runs it on a service provider's single-queue RFC 3168 network.
2.       L4S traffic from another domain enters an RFC 3168 single-queue network (e.g. a peer-to-peer application)
     Success of this exercise will be judged by the WG reaching "rough consensus" that deployment of the L4S experiment
     is unlikely to cause significant damage to the Internet, and that any damage that results is expected to be tolerable.
Thanks, --David [as TSVWG WG co-chair]
----------------------------------------------------------------
David L. Black, Senior Distinguished Engineer
Dell Technologies, Infrastructure Systems Group
176 South St., Hopkinton, MA  01748
+1 (774) 350-9323<tel:+17743509323>           Mobile: +1 (978) 394-7754<tel:+19783947754>
[email-masked]<mailto:[email-masked]>
----------------------------------------------------------------

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 11 Mar 2021 18:42:06 +0000

David,

On 10/03/2021 18:01, Black, David wrote:
> _L4S Drafts: What Needs To Be Done_
>
> __
>
> – Goal: Interoperability – multiple transport protocols able to meet 
> requirements
>
> • Specifications and/or implementations – WG decides what is sufficient
>
> – Location: draft-ietf-tsvwg-ecn-l4s-id
>
> – Goal: L4S experiment unlikely to cause significant damage to the 
> Internet, and
>
> • Any damage that results is expected to be tolerable
>
> – Location: draft-white-tsvwg-l4s-ops (candidate for WG adoption)
>
> • WG Last Call on l4s-ops draft not required to meet safety goal
>
> • Criteria: WG “rough consensus” that each goal has been reached
>
> – WG chairs will run initial consensus calls before next IETF meeting week
>
> ------------------------
>
> This discussion is about the first goal: 
> Transport-protocol-independent requirements to use L4S service
>
> > But, 2 independent full implementations for an EXP would be way 
> beyond any interpretation of IETF process.
>
> Two independent full implementations of protocols that meet the 
> requirements ought to suffice, but may not be necessary.
>
> As stated on the slide (and quoted above): " Specifications and/or 
> implementations – WG decides what is sufficient."
>

[BB] I think the number 'two' comes from wanting to be sure that 
developers other than the author-team can understand the spec. 
sufficiently to implement.

Pls confirm that could also be satisfied by /each/ requirement having 
been implemented by /at least one other/ team than the authors (or 
reasonably expected to implement).

The reason this is important is that there is a heavy performance 
evaluation burden attached to each requirement, because they are nearly 
all about impact on the queuing delay of other hosts (or impact on 
throughput of others in the case of the Classic ECN AQM one).

For a second CC developer to invest the time and effort in the 
performance testing necessary for every requirement, they would have to 
be very confident that the IETF was going to follow through with 
codepoint assignment. A great job has been done in undermining that 
confidence in the last couple of years.

To be clear, there are many implementations of the network part, and a 
load of network operators ready to deploy the network part. There's no 
problem there (other than all those projects have gone cold waiting for 
the IETF). The focus here is CC development, I assume.

> /restructure/ the drafts. I said they already have the structure you want.
>
> > No reply to that point. Now what you talk about above (albeit as an 
> individual now) are wording changes not restructuring.
>

Good. Thank you.

Bob
> *Sent:* Wednesday, March 10, 2021 7:53 AM
>     Reminder of the goal for [1] (Rework the “Prague Requirements”
>     into the following two entities ...):
>
>          >> currently meet and/or are reasonably expected to be able
>     to meet the normative requirements (a).
>
>     If the L4S folks believe that this has already been done, then
>     it’s time to start bringing forward the multiple transport
>     protocols that demonstrate success of that exercise.
>
>
> I assume readers are aware that the IETF's requirement for 2 
> independent implementations is for progressions from Proposed Standard 
> to full Internet Standard, and few RFCs have ever taken that step 
> (only 118 RFCs are STDs or parts of STDs) (e.g. RFC3168 is not).
> <https://www.rfc-editor.org/search/rfc_search_detail.php?sortkey=Number&sorting=DESC&page=All&pubstatus%5B%5D=Standards%20Track&std_trk=Internet%20Standard>
>
> I accept that we're messing with core protocols here, so the bar 
> should be higher. But, 2 independent full implementations for an EXP 
> would be way beyond any interpretation of IETF process.
>
>
>
>     >Nonetheless, if there are specific items in S.4 that someone
>     believes should or shouldn't be there, or should be worded
>     differently, let's have those discussions now on the list - I'm
>     all ears
>
>     Speaking only for myself as an individual beyond this point, these
>     two normative requirements struck me as ones that might be better
>     stated as design and implementation guidelines:
>
>       
>
>     The use of “as much as possible” in the first bullet makes it hard
>     to figure out whether a specific congestion control meets that
>     mandatory requirement for participation in the L4S experiment. 
>     The second bullet appears to involve a prediction of the future as
>     currently written.
>
>
> [BB] I'll refrain from getting into text-specifics in this thread, 
> 'cos I'm trying to be absolutely sure I understand what the chairs 
> want. We've felt subjected to a "No, bring me a different rock" 
> process so far, so we want to be absolutely clear what sort of rock 
> you want.
>
>     <mailto:[email-masked]>
>     *To:* Black, David; [email-masked] <mailto:[email-masked]>
>         Our guidance to the WG and the L4S draft authors is that at
>         least the following two things need to be done before WGLC
>         will become possible on the L4S drafts:
>
>         1.Transport-protocol-agnostic normative requirements for all
>         1.The DualQ AQM design relies upon traffic adhering to these
>         requirements.
>
>         2.Design and implementation guidelines for new congestion
>         controls that meet the normative requirements (a).
>
>         1.Ultimately, the choice of whether a particular aspect is a
>     [BB] I believe draft-ietf-tsvwg-ecn-l4s-id is already structured
>     for this division between normative requirements and design and
>     * [1]a) is Section 4 "Prerequisite Transport Layer Behaviour".
>     Nearly every paragraph also refers off to subsections of Appendix
>     A.1 for extra non-normative info, headed "Requirements for
>     Scalable Transport Protocols". [Actually, it might be useful to
>     swap these two headings]
>     They have been divided like that from the very first ad hoc
>     meeting in Prague that formulated them (in 2015)
>
>     Code to address all the requirements and most of the optimizations
>     has been implemented. Over the last year, 3 or 4 of the later
>     requirements in a) have become SHOULDs not MUSTs. Personally I
>     would have left most as MUSTs, but I've tried to reflect what the
>     WG wanted. Those demoted to SHOULD have generally been considered
>     a lower risk of harm (either low likelihood of occuring or minor
>     harm if they do) - taking into account the complexity of
>     implementing them.
>
>     We could certainly flag the point where the transition from MUSTs
>     to SHOULDs occurs within section 4. But I think we should still
>     group all the items with potential for harm together in the same
>     section. Because assessment of risk will change as the Internet
>     landscape changes (possibly over the duration of the experiment).
>
>     Now I've explained, I hope the chairs will all agree that "rework
>     the Prague Requirements into two entities" (normative requirements
>     and design and implementation guidelines) is already done and dusted.
>
>     Nonetheless, if there are specific items in S.4 that someone
>     believes should or shouldn't be there, or should be worded
>     differently, let's have those discussions now on the list - I'm
>     all ears.
>
>
>
>     Bob
>
>
>
>              Success of this exercise will be judged by the WG
>         reaching “rough consensus” that multiple transport protocol
>         implementations
>         1.This safety case does not rely on endpoints running TCP
>         Prague, though it can assume endpoints are meeting the
>         reworked Prague requirements
>
>         2.Ops draft will provide guidance on how to detect L4S
>         problems on their RFC 3168 network, and how to mitigate them.
>
>         3.Ops draft must consider these scenarios:
>
>         1. An unsophisticated user purchases an L4S endpoint and runs
>         it on a service provider's single-queue RFC 3168 network.
>
>         2. L4S traffic from another domain enters an RFC 3168
>         single-queue network (e.g. a peer-to-peer application)
>
>              Success of this exercise will be judged by the WG
>         reaching “rough consensus” that deployment of the L4S experiment
>              is unlikely to cause significant damage to the Internet,
>         and that any damage that results is expected to be tolerable.
>

---

From: "Black, David" <[email-masked]>
Date: Thu, 11 Mar 2021 20:08:22 +0000

> [BB] I think the number 'two' comes from wanting to be sure that developers other than the author-team can understand the spec. sufficiently to implement.
>
> Pls confirm that could also be satisfied by /each/ requirement having been implemented by /at least one other/ team than the authors (or reasonably expected to implement).

No, definitely not confirmed.

Use of the ECT(1) codepoint entails meeting all the requirements.  If only TCP Prague will be able to meet all the requirements, then that set of requirements for ECT(1) codepoint usage is not transport-protocol-independent.

Thanks, --David

From: Bob Briscoe <[email-masked]>
Sent: Thursday, March 11, 2021 1:42 PM
To: Black, David; [email-masked]
Subject: Re: [tsvwg] L4S drafts: Next Steps

[EXTERNAL EMAIL]
David,

------------------------

[BB] I think the number 'two' comes from wanting to be sure that developers other than the author-team can understand the spec. sufficiently to implement.

Pls confirm that could also be satisfied by /each/ requirement having been implemented by /at least one other/ team than the authors (or reasonably expected to implement).

The reason this is important is that there is a heavy performance evaluation burden attached to each requirement, because they are nearly all about impact on the queuing delay of other hosts (or impact on throughput of others in the case of the Classic ECN AQM one).

For a second CC developer to invest the time and effort in the performance testing necessary for every requirement, they would have to be very confident that the IETF was going to follow through with codepoint assignment. A great job has been done in undermining that confidence in the last couple of years.

To be clear, there are many implementations of the network part, and a load of network operators ready to deploy the network part. There's no problem there (other than all those projects have gone cold waiting for the IETF). The focus here is CC development, I assume.

Confirmed.

Good. Thank you.

Bob

[EXTERNAL EMAIL]
David,

ecn-l4s-id

Bob

Thanks, --David

[EXTERNAL EMAIL]
David and chairs,

Bob

--

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 11 Mar 2021 21:18:03 +0000

David,

Where does the concern about transport protocol independence come from? 
No-one expects that to be a problem for L4S implementers.

Similarly RFC3168 became a PS when ECN was solely supported (and solely 
defined) for TCP over IP. No-one believed that it would not be possible 
to implement over other transports. Of course there could be practical 
difficulties, e.g. the API for setting ECN codepoints, different 
deployment difficulties, due to middlebox treatments, etc. But I don't 
think you are talking about those sort of difficulties, because you are 
talking about the requirements in the draft (that the authors set up as 
our goals for the L4S service right from the start).

Actually, there are already implementations of scalable CC over TCP 
(two), RMCAT/RTP/UDP, a proprietary protocol over UDP, there was a port 
to QUIC/UDP produced during a hackathon a couple of years ago, I don't 
know of others, but that doesn't mean there aren't people out there 
working on them. This doesn't say anything about how 'full' they are, 
but it does say that no-one's reported any problems with the 
requirements that are due to choice of transport protocol.

This is what I mean by "No, bring me a different rock" - new arbitrary 
requirements popping up, that are not good tests of whether the 
experiment will succeed, but which would send everyone spinning hamster 
wheels if we allowed them to pass unchallenged.

Bob

On 11/03/2021 20:08, Black, David wrote:
> >>                        – Goal: Interoperability – multiple 
> transport protocols able to meet requirements
>
> > Pls confirm that could also be satisfied by /each/ requirement 
> having been implemented by /at least one other/ team than the authors 
> (or reasonably expected to implement).
>
> Use of the ECT(1) codepoint entails meeting all the requirements.  If 
> only TCP Prague will be able to meet all the requirements, then that 
> set of requirements for ECT(1) codepoint usage is not 
> transport-protocol-independent.
>
> Thanks, --David
>
> *Sent:* Thursday, March 11, 2021 1:42 PM
>     – Goal: Interoperability – multiple transport protocols able to
>     meet requirements
>
>     should be higher.
> evaluation burden attached to each requirement, because they are 
> nearly all about impact on the queuing delay of other hosts (or impact 
> on throughput of others in the case of the Classic ECN AQM one).
>
> performance testing necessary for every requirement, they would have 
> to be very confident that the IETF was going to follow through with 
> problem there (other than all those projects have gone cold waiting 
> for the IETF). The focus here is CC development, I assume.
>
>
>     > This part of your original email (which was as a chair) asked us
>     to /restructure/ the drafts. I said they already have the
>     structure you want.
>
>     > No reply to that point. Now what you talk about above (albeit as
>     an individual now) are wording changes not restructuring.
>
>              >> Success of this exercise will be judged by the WG
>              >> currently meet and/or are reasonably expected to be
>         able to meet the normative requirements (a).
>
>     independent implementations is for progressions from Proposed
>     Standard to full Internet Standard, and few RFCs have ever taken
>     that step (only 118 RFCs are STDs or parts of STDs) (e.g. RFC3168
>     is not).
>     should be higher. But, 2 independent full implementations for an
>     EXP would be way beyond any interpretation of IETF process.
>
>
>
>
>         differently, let's have those discussions now on the list -
>         I'm all ears
>
>         Speaking only for myself as an individual beyond this point,
>         these two normative requirements struck me as ones that might
>         be better stated as design and implementation guidelines:
>
>           
>
>         The use of “as much as possible” in the first bullet makes it
>         hard to figure out whether a specific congestion control meets
>         that mandatory requirement for participation in the L4S
>         experiment.  The second bullet appears to involve a prediction
>         of the future as currently written.
>
>
>     process so far, so we want to be absolutely clear what sort of
>     rock you want.
>
>     This part of your original email (which was as a chair) asked us
>     to /restructure/ the drafts. I said they already have the
>     structure you want. No reply to that point. Now what you talk
>     about above (albeit as an individual now) are wording changes not
>     restructuring. We've been doing wording changes with the survey of
>     implementers. Fine with that.
>
>             Our guidance to the WG and the L4S draft authors is that
>             at least the following two things need to be done before
>             WGLC will become possible on the L4S drafts:
>
>             [1] Rework the “Prague Requirements” into the following
>             two entities, which will overlap:
>
>             1.Transport-protocol-agnostic normative requirements for
>             all congestion-controlled traffic that uses the L4S low
>             latency queue.
>
>             1.The DualQ AQM design relies upon traffic adhering to
>             these requirements.
>
>             1.Ultimately, the choice of whether a particular aspect is
>             a guideline (b) or requirement (a) is a WG “rough
>             consensus” decision.
>
>
>         [BB] I believe draft-ietf-tsvwg-ecn-l4s-id is already
>         structured for this division between normative requirements
>         and design and implementation guidelines.
>         Nearly every paragraph also refers off to subsections of
>         Appendix A.1 for extra non-normative info, headed
>         "Requirements for Scalable Transport Protocols". [Actually, it
>         might be useful to swap these two headings]
>         Code to address all the requirements and most of the
>         optimizations has been implemented. Over the last year, 3 or 4
>         of the later requirements in a) have become SHOULDs not MUSTs.
>         Personally I would have left most as MUSTs, but I've tried to
>         reflect what the WG wanted. Those demoted to SHOULD have
>         generally been considered a lower risk of harm (either low
>         likelihood of occuring or minor harm if they do) - taking into
>         We could certainly flag the point where the transition from
>         MUSTs to SHOULDs occurs within section 4. But I think we
>         should still group all the items with potential for harm
>         together in the same section. Because assessment of risk will
>         change as the Internet landscape changes (possibly over the
>         duration of the experiment).
>
>         Now I've explained, I hope the chairs will all agree that
>         "rework the Prague Requirements into two entities" (normative
>         requirements and design and implementation guidelines) is
>         already done and dusted.
>
>         differently, let's have those discussions now on the list -
>         I'm all ears.
>
>
>
>         Bob
>
>
>
>
>             reaching “rough consensus” that multiple transport
>             protocol implementations
>                  currently meet and/or are reasonably expected to be
>             able to meet the normative requirements (a).
>
>             [2] Build the safety case for L4S experimental
>             Internet-wide deployment in the L4S Ops draft.
>
>             1. An unsophisticated user purchases an L4S endpoint and
>             runs it on a service provider's single-queue RFC 3168 network.
>
>                  is unlikely to cause significant damage to the
>             Internet, and that any damage that results is expected to
>             be tolerable.
>
>             +1 (774) 350-9323 <tel:+17743509323> Mobile: +1 (978)
>             394-7754 <tel:+19783947754>
>

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 11 Mar 2021 23:35:18 +0200

> On 11 Mar, 2021, at 11:18 pm, Bob Briscoe <[email-masked]> wrote:
> 
> Where does the concern about transport protocol independence come from? No-one expects that to be a problem for L4S implementers. 

If it's not expected to be a problem, then there shouldn't be any difficulty meeting the requirement, should there?

I think the concern arises because there is not yet even *one* complete implementation that meets *all* of the necessary requirements (including, for example, good behaviour when encountering an RFC-3168 AQM).  This points to the likely difficulty of a second implementer's task in replicating the feat.

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 11 Mar 2021 22:07:48 +0000

Jonathan,

The concern is requiring unnecessary work to prove things that are 
"bring me a different rock" exercises.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 12 Mar 2021 00:55:47 +0200

> On 12 Mar, 2021, at 12:07 am, Bob Briscoe <[email-masked]> wrote:
> 
> The concern is requiring unnecessary work to prove things that are "bring me a different rock" exercises.

That does appear to be *your* concern at the moment.  I am merely trying to interpret the concerns of other people for your benefit.  Neither you nor I are the sole arbiters of what valid concerns may or may not exist.  I would however encourage you to consider what I have recently said about "externalised risk"; I can find a link to the YouTube recording of my talk the other day, if that would be helpful.

What I believe David Black is referring to - and I'm sure he'll correct me if necessary - is the (hopefully uncontroversial) notion that a specification is useless if it cannot be implemented, and indeed not very useful if distinct implementations are not automatically interoperable with each other.  This is part of the IETF credo of "rough consensus and running code".

As he explained, it is not necessary for two independent implementations to actually exist for an Experimental or Proposed Standard RFC, so long as there's a reasonable expectation that a second complete implementation is achievable by an ordinarily skilled practitioner of the art (to borrow some patent law jargon).  The existence of *one* complete implementation would, however, appear to be a prerequisite for such a determination.  If such an implementation is open-source and well documented, it's reasonable to expect a second implementer to refer to it in minor cases of doubt - but the major facets of requirement must also be well described in the specification.

I have entirely separate concerns about the specification itself, which I will keep to other threads.

 - Jonathan Morton

---

From: "Black, David" <[email-masked]>
Date: Fri, 12 Mar 2021 00:20:52 +0000

> What I believe David Black is referring to - and I'm sure he'll correct me if necessary - is the (hopefully uncontroversial) notion that a specification
> is useless if it cannot be implemented, and indeed not very useful if distinct implementations are not automatically interoperable with each other.

That's the core idea.  In more detail, the transport requirements for use of the L4S service (section 4 of draft-ietf-tsvwg-ecn-l4s-id-14, particularly section 4.3 on congestion response) define a functional interface between the network and transport protocols, an interface that needs to interoperate with a variety of transport protocols in order to realize the L4S goal of being a widely usable service.  Looking back over the past few years, in 20/20 hindsight, some of these requirements were initially written too strictly to the point of being infeasible.

Turning to the current situation, the claim (in email below) that "No-one expects that [meeting these requirements] to be a problem for L4S implementers" does not align with the two "Needs experimental data" slides in Koen's meeting presentation on these requirements (see slides 6 and 7 of: https://datatracker.ietf.org/meeting/110/materials/slides-110-tsvwg-sessb-73-prague-requirements-00).  I have no problem with evaluation of some of these requirements being goals of the L4S experiment - if that is done, then mandating that the requirements involved be met in order to participate in the L4S experiment appears to set the bar for experimental use of ECT(1) too high.  To make this concrete, one of the "Needs experimental data" items (first item on slide 7) is:

		o MUST implement monitoring to detect non_L4S ECN AQM...
		   - Is detection itself required?
		   - Robust detection scheme needs real deployment experience.
		   - Combination with delay based control could minimize potential issues
		   - Develop during experiment as needed.

To purse such a course of action, "MUST implement monitoring to detect non_L4S ECN AQM..." cannot survive as a normative requirement (MUST) for use of the ECT(1) codepoint in the L4S experiment, although it would be fine to encourage that monitoring to be implemented.  If this sort of approach of converting problematic requirements to implementation guidance and/or goals to be evaluated by the experiment is followed for all of the "Needs experimental data" items, then the resulting WG consensus call on feasibility of the remaining requirements (effectively the rest of the contents of Koen's slides, including removal of the objectionable documentation requirements) should be quick and easy.

The l4s-ops draft would need to align with any resulting changes.

Thanks, --David

-----Original Message-----
From: Jonathan Morton <[email-masked]> 
Sent: Thursday, March 11, 2021 5:56 PM
To: Bob Briscoe
Cc: Black, David; [email-masked]
Subject: Re: [tsvwg] L4S drafts: Next Steps

[EXTERNAL EMAIL] 

 - Jonathan Morton

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 12 Mar 2021 01:48:16 +0000

David,

> That's the core idea.  In more detail, the transport requirements for use of the L4S service (section 4 of draft-ietf-tsvwg-ecn-l4s-id-14, particularly section 4.3 on congestion response) define a…

I am not questioning that there is work to be done to implement those 
requirements (by us - the authors). I am questioning the notion that at 
least one developer (other than the authors) has to do all the same work 
as well, irrespective of how many other developers do many parts of the 
work.

The functional interfaces are a tiny part of the job, i.e. the ECT(1) 
codepoint and e2e transport feedback.

However, 90% of the requirements are about performance impact on other 
hosts (mostly latency but also throughput). I.e. all the safety 
requirements in 4.3. These are /not/ functional interfaces.

Functional interface requirements are relatively easy to set up tests 
for. In contrast performance safety testing requires huge resources to 
test satisfactorily. And the results are easy to rubbish as meaningless 
if not done over the real Internet (Jonathan can give you the playbook 
on that). But no-one can do this over the real Internet without the 
codepoint. So this type of testing requires even more resources to set 
up an artificial but realistic environment.

I agree this stuff has to be done. It just makes a hugely hard task even 
harder if you arbitrarily set the bar so that at least one of the 
developers has to do it /all/ *as well as the authors*. When it would be 
no less meaningful if testing of each requirement was covered by at 
least one of the developers (as well as the authors).

>
> Turning to the current situation, the claim (in email below) that "No-one expects that [meeting these requirements] to be a problem for L4S implementers" does not align with the two "Needs experimental data" slides in Koen's meeting presentation on these requirements (see slides 6 and 7 of: https://datatracker.ietf.org/meeting/110/materials/slides-110-tsvwg-sessb-73-prague-requirements-00).

[BB] You're now quoting me out of context (presumably due to Jonthan's 
snippers). My words were:
>
>> Where does the concern about transport protocol independence come 
>> from? No-one expects that to be a problem for L4S implementers.

I was arguing against your specific focus on *transport protocol 
independence*. That suddenly adds another dimension to the test matrix.

Say, for instance that the second implementer likely to do the first 
full set of testing is Google with BBRv2. They have done person-years of 
work but, with this sudden call to bring you a different rock, all their 
work suddenly won't count, 'cos their implementation is over TCP, which 
is the same transport protocol as TCP Prague. Yes, Google could probably 
do the work over QUIC instead". But only if you had asked them 2 years 
ago. Suddenly inventing this new requirement throws a spanner into 
everyone's plans. The projects are set up. The people involved have 
expertise in the particular protocols, then suddenly there's a click of 
the fingers for a different rock.

Which transport protocol is used is not even important to the 
requirements in the draft. it's just an arbitrarily different rock.

> I have no problem with evaluation of some of these requirements being goals of the L4S experiment - if that is done, then mandating that the requirements involved be met in order to participate in the L4S experiment appears to set the bar for experimental use of ECT(1) too high.  To make this concrete, one of the "Needs experimental data" items (first item on slide 7) is:
>

[BB] This is fine, and good, and consistent with what we've been led to 
understand. Thank you.
It's the sudden demands for different rocks I am picking up on: sudden 
demands about how many transport protocols, sudden constraints on which 
implementers have to do how much, etc.

Bob

---

From: "Black, David" <[email-masked]>
Date: Fri, 12 Mar 2021 02:43:16 +0000

[...skipping to the end ...]

>> To purse such a course of action, "MUST implement monitoring to detect non_L4S ECN AQM..." cannot survive as
>> a normative requirement (MUST) for use of the ECT(1) codepoint in the L4S experiment, although it would be fine
>> to encourage that monitoring to be implemented.  If this sort of approach of converting problematic requirements
>> to implementation guidance and/or goals to be evaluated by the experiment is followed for all of the "Needs experimental data"
>> items, then the resulting WG consensus call on feasibility of the remaining requirements (effectively the rest of the contents
>> of Koen's slides, including removal of the objectionable documentation requirements) should be quick and easy.
>>
> [BB] This is fine, and good, and consistent with what we've been led to understand. Thank you.

If this is what the L4S team intends to do, then we're aligned, and I see no need to respond to most of the discussion skipped above.

I'll attempt clarify one more thing:

> Say, for instance that the second implementer likely to do the first full set of testing is Google with BBRv2.
> They have done person-years of work but, with this sudden call to bring you a different rock, all their work
> suddenly won't count, 'cos their implementation is over TCP, which is the same transport protocol as TCP Prague.

For the purposes under discussion, TCP Prague and TCP BBRv2 look like different transport protocols to me.

Thanks, --David

-----Original Message-----
From: Bob Briscoe <[email-masked]> 
Sent: Thursday, March 11, 2021 8:48 PM
To: Black, David
Cc: [email-masked]; Jonathan Morton
Subject: Re: [tsvwg] L4S drafts: Next Steps

[EXTERNAL EMAIL] 

David,

On 12/03/2021 00:20, Black, David wrote:
>> What I believe David Black is referring to - and I'm sure he'll 
>> correct me if necessary - is the (hopefully uncontroversial) notion that a specification is useless if it cannot be implemented, and indeed not very useful if distinct implementations are not automatically interoperable with each other.

I am not questioning that there is work to be done to implement those requirements (by us - the authors). I am questioning the notion that at least one developer (other than the authors) has to do all the same work as well, irrespective of how many other developers do many parts of the work.

The functional interfaces are a tiny part of the job, i.e. the ECT(1) codepoint and e2e transport feedback.

However, 90% of the requirements are about performance impact on other hosts (mostly latency but also throughput). I.e. all the safety requirements in 4.3. These are /not/ functional interfaces.

Functional interface requirements are relatively easy to set up tests for. In contrast performance safety testing requires huge resources to test satisfactorily. And the results are easy to rubbish as meaningless if not done over the real Internet (Jonathan can give you the playbook on that). But no-one can do this over the real Internet without the codepoint. So this type of testing requires even more resources to set up an artificial but realistic environment.

I agree this stuff has to be done. It just makes a hugely hard task even harder if you arbitrarily set the bar so that at least one of the developers has to do it /all/ *as well as the authors*. When it would be no less meaningful if testing of each requirement was covered by at least one of the developers (as well as the authors).

>> from? No-one expects that to be a problem for L4S implementers.

I was arguing against your specific focus on *transport protocol independence*. That suddenly adds another dimension to the test matrix.

Say, for instance that the second implementer likely to do the first full set of testing is Google with BBRv2. They have done person-years of work but, with this sudden call to bring you a different rock, all their work suddenly won't count, 'cos their implementation is over TCP, which is the same transport protocol as TCP Prague. Yes, Google could probably do the work over QUIC instead". But only if you had asked them 2 years ago. Suddenly inventing this new requirement throws a spanner into everyone's plans. The projects are set up. The people involved have expertise in the particular protocols, then suddenly there's a click of the fingers for a different rock.

Which transport protocol is used is not even important to the requirements in the draft. it's just an arbitrarily different rock.

Bob

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 12 Mar 2021 08:24:27 +0000

I also agree on that...

> For the purposes under discussion, TCP Prague and TCP BBRv2 look like different transport protocols to me.

+1.  I was looking for different ways of instantiating the method - not 
necessarilly in different protocols, and this was helpful to me.

Gorry

---

From: Greg White <[email-masked]>
Date: Fri, 12 Mar 2021 17:56:00 +0000

Ok, so is this a reasonable summary of specifically what needs to happen to satisfy item 1 in the chairs' "What needs to be done"* to start WGLC on the L4S drafts?

The multiplicity of existing experimental L4S implementations is considered sufficient. The one remaining MUST statement (monitoring to detect non_L4S ECN) in ECN-L4S-ID that was not universally agreed to by the L4S implementers** needs to be rectified (the 2 SHOULDs are fine as is).  The wordsmithing of one MUST statement (react to packet loss in a friendly way) will need to be finished.  Once the authors believe this is done, the chairs will run a WG consensus call to gauge whether there is evidence that the requirements are implementable. If there is rough consensus, this goal has been achieved.

* https://datatracker.ietf.org/meeting/110/materials/slides-110-tsvwg-sessb-71-chairs-slides-00
** https://datatracker.ietf.org/meeting/110/materials/slides-110-tsvwg-sessb-73-prague-requirements-00 

I would appreciate a comment from the chairs that this either describes ROCK #1 sufficiently, or if it doesn't, please describe the rock that you are looking for.

Thanks,
Greg

On 3/12/21, 1:24 AM, "tsvwg on behalf of Gorry Fairhurst" <[email-masked] on behalf of [email-masked]> wrote:

    I also agree on that...

    Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 12 Mar 2021 18:03:13 +0000

> I would appreciate a comment from the chairs that this either describes ROCK #1 sufficiently, or if it doesn't, please describe the rock that you are looking for.
If you ask "the Chairs" plural - we do have three heads plus an AD (that 
makes perhaps 4 heads), the group of us need to talk - and we will.

 From this Chair: be assured I'll look at the full list of 
SHOULD/MUST/MAY and will give any comments I can.

Gorry

---

From: Sebastian Moeller <[email-masked]>
Date: Fri, 19 Mar 2021 23:21:31 +0100

Hi Bob,

i believe nobody is asking for rocks (nor shrubberies) here, this is really about a convincing demonstration that the whole L4S design, consisting of network and protocol bits actually behaves like a coherent design and not two disjoint sub-projects that will never really harmonize? ATM what I see is you off-loading problems in the network bits by claiming the protocol will need to fix this up (RTT-bias comes to mind), and currently not even a single protocol convincingly demonstrates that the requirements a) can be met, and that b) doing so results in an L4S system that performs as promised without causing a safety issue for the existing internet.

	As it stands not even TCP Prague meaningfully fulfills the L4S transport requirements*, so we have zero proof that these requirements are actually feasible (I am simply going to ignore the questionnaire you send around, as neither party claimed, let alone demonstrated that their protocol actually meets all the requirements; these are essentially mere pre-implementation judgements whether people see conceptual stumbling blocks a priori, but as we all know the devil is in the details, so...)

Best Regards
	Sebastian

*) Partly because these requirements are unclear and impossible to fulfill, the RTT-bias thing comes to mind again, where you require the impossible from transport protocols (as that essentially requires faster than light travel) to fix up grave misdesigns in L4S' network bits core, the AQM which makes RTT-bias considerably worse as compared to the status quo (as seen in Pete's data))

> On Mar 11, 2021, at 23:07, Bob Briscoe <[email-masked]> wrote:
> 
> Jonathan,
>

---

From: "Livingood, Jason" <[email-masked]>
Date: Sat, 20 Mar 2021 14:33:01 +0000

> i believe nobody is asking for rocks (nor shrubberies) here, this is really about a convincing demonstration that the whole L4S design, consisting of network and protocol bits actually behaves like a coherent design and not two disjoint sub-projects that will never really harmonize?

Maybe I am being too simplistic with my operator mindset, but I'd prefer that any long-term decision be data-driven. And to collect significant data you need to conduct an experiment. Which in turn seems to potentially depend on an experimental RFC. So if we want the convincing demonstration sought & real data to drive a long-term decision then how is an experimental RFC not the logical next step?

Jason

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 20 Mar 2021 16:48:09 +0200

> On 20 Mar, 2021, at 4:33 pm, Livingood, Jason <Jason_Livingood=[email-masked]> wrote:
> 

Field studies would require the published RFC.  Lab studies can already be performed (and to some extent have been).  The lab studies have shown, far from a promising system that might be ready for a field trial, one that shows worrying failure modes that primarily affect innocent bystanders.  These effects, moreover, were fairly predictable by theory (in that I did, in fact, predict them before the lab study results were available).

What we're asking for is a system that at least behaves reasonably under lab conditions, chosen to represent realistic challenge scenarios likely to occur in the real world.  Only then can we have any confidence that L4S will not cause problems anywhere and everywhere that it is deployed.

 - Jonathan Morton

---

From: "Livingood, Jason" <[email-masked]>
Date: Sat, 20 Mar 2021 15:08:02 +0000

> The lab studies have shown

IMO the challenge with lab studies is that there are a lot of variables that are artificial and/or the test environment or test traffic does not fully reflect reality (production). 10 or 15 years ago I would have been very focused on extensive QA testing and an elaborate lab test environment. These days its more how to do controlled testing directly in the production network, with a controlled/minimized blast radius in case of problems, easy/quick rollback, rapid code/config iteration.

> far from a promising system that might be ready for a field trial, one that shows worrying failure modes that primarily affect innocent bystanders.

Only one way to find out - test it in a real production experiment with appropriate risk controls. The status quo seems to suggest years more debate without (IMO) sufficient data. An alternative is a controlled production experiment, appropriately risk-managed, between willing participants. I see no downside to allowing that to occur via experimental RFC. We're setting the bar at the proposed standard or standard level, which I think is incorrect. I would say instead enable the experiment to accelerate the learning & improvement & drive data-driven decisions.

Jason

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 20 Mar 2021 15:13:58 +0000

> Only one way to find out - test it in a real production experiment with appropriate risk controls. The status quo seems to suggest years more debate without (IMO) sufficient data. An alternative is…

Jason,

Thanks for writing this so clearly!

This sounds like what I was expecting. The labs tests have been useful, 
but the next proposed steps are the deployment experiment, which as far 
as I can see what has driven the development of these drafts in the 
working group.

Gorry

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 20 Mar 2021 17:43:36 +0200

> On 20 Mar, 2021, at 5:08 pm, Livingood, Jason <[email-masked]> wrote:
> 

Ah yes, the fail fast, fail often strategy.  And if your self-driving car happens to run into the side of a couple of 18-wheelers in the process, oh well, we'll do better next year.  Or next decade.  Look behind you, a three-headed monkey!

I know that the lab is an artificial environment in which not all possible conditions can be realistically predicted and simulated.  But if your system can't even cope reliably with conditions that the lab *can* easily simulate, what chance does it have in the much less predictable conditions in the real world?  And how do you debug it when it does fail, without being able to replicate some approximation of those conditions in the lab?

> Only one way to find out - test it in a real production experiment with appropriate risk controls. The status quo seems to suggest years more debate without (IMO) sufficient data. An alternative is…

I would have somewhat more confidence in this approach if that was what the l4s-ops draft was written to require.  My suggestions to that effect do not appear, at this moment, to have been taken seriously by its author.  To eliminate bystander risk, as you suggest, requires that the experimental traffic be explicitly confined to the participating network(s) at a minimum - but that is not in the draft.

 - Jonathan Morton

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 20 Mar 2021 17:12:06 +0100

Hi Jason,

The answer to your question, 'why is it not time for declaring L4S experimental' at the current time is quite simple.
Team L4S has not even scratched the surface on safety and functionality tests that can be performed over the internet or at least participating ASs. Really all requests for data demonstrating functionality over long RTT paths have just been ignored. And basically all tests by other parties demonstrated L4S failing to reach its promises, all the while making imperfect situations considerably worse. 

Just look at Pete's data at https://github.com/heistp/l4s-tests, these are "simple" lab tests that demonstrate that:

L4S's network bits: the dual queue coupled AQM fails to equitable share between the LL and non-LL queue at the same RTT (see second pair of bars under:
687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f32306d735f32306d732e737667 <https://camo.githubusercontent.com/a83b3e8c3aa1824e4ff38b5d0d9136079f431c2e917bb347c15b8b17d64b6a32/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f32306d735f32306d732e737667>
and
687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f38306d735f38306d732e737667 <https://camo.githubusercontent.com/572896b1f713045d3e146dd32d3431a441d1b6a442e0f3a101d39660094bf93d/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f38306d735f38306d732e737667>
and
687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667 <https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667>

ostensibly its main reason to exists!

L4S protocol bits: Watch TCP Praque not survive the contact with a FIFO bottlenech as is the norm over arbitrary internet path at the fifth (and in the last plot also the sixth pair of bars:

687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f32306d735f32306d732e737667 <https://camo.githubusercontent.com/a83b3e8c3aa1824e4ff38b5d0d9136079f431c2e917bb347c15b8b17d64b6a32/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f32306d735f32306d732e737667>
and
687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f38306d735f38306d732e737667 <https://camo.githubusercontent.com/572896b1f713045d3e146dd32d3431a441d1b6a442e0f3a101d39660094bf93d/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f38306d735f38306d732e737667>
and
687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667 <https://camo.githubusercontent.com/0ca81a2fabe48e8fce0f98f8b8347c79d27340684fe0791a3ee6685cf4cdb02e/687474703a2f2f7363652e646e736d67722e6e65742f726573756c74732f6c34732d323032302d31312d3131543132303030302d66696e616c2f73312d6368617274732f727474666169725f63635f71646973635f31306d735f3136306d732e737667>

Why are we even discussing with a straight face that L4S would be ready for prime-time?

So far team L4S has NOT refuted Pete's results in any meaningful way (that is by presenting data showing that the issues are remedied).

	So let me ask bluntly, have you actually looked at the data yourself, or are you relaying on team L4S' subjective responses to the data? Or what makes you believe that L4S is ready for an internet-wide roll-out?

	Anyway talking procedurally, I am opposed to validate such quick* and dirty engineering practices as demonstrated by L4S by granting RFC status (which alas, is not under my control, as I fully realize). The precedence this is going to set is: no matter how incomplete an internet draft or how much at odds with observable data, persistence alone will get you RFC status. IMHO the idea behind a standardization process should not be to simply accept proposals as if 'set in stone', but rather to aim at improving the drafts and technologies behind those drafts, especially in the context of an experiment. 
	Speaking of L4S as an 'experiment', so far there are no clear hypothesis that an L4S experiment is going to explore, nor any objective criteria how to evaluate such an experiment. IMHO the whole 'experiment' is going to be can L4S be successfully rolled-out, without a proper definition of what success is going to be. 

	And with L4S I see a lot of room for improvements, bur mostly I see a lot of work for L4S to even reach its stated promises. The secret to success is under promise, over deliver, L4S appears to aim for the opposite approach.

Best Regards
Sebastian

*) Odd that after a decade iL4S still feels somewhat rushed, but that might be subjective.

P.S.: I am reminded of SpaceX/Tesla here, we as a WG seem to endorse a SpaceX, fail often, fail early, but also evolve quickly approach, but not in the context of the empty skies about Texas, but rather as if Tesla would employ the same haphazard approach to develop the autonomous driving facilities in customer cars in existing traffic... But  IMHO that is what releasing the incomplete L4S design and implementation on the internet. UNLESS our goal is to build a fast priority track for short RTT TCP Prague flows over everything else, in that case current L4S might be a contender. Sorry for the lack of sugar coat here, but the whole episode leaves me somewhat jaded. 

On 20 March 2021 15:33:01 CET, "Livingood, Jason" <[email-masked]> wrote:
i believe nobody is asking for rocks (nor shrubberies) here, this is really about a convincing demonstration that the whole L4S design, consisting of network and protocol bits actually behaves like a coherent design and not two disjoint sub-projects that will never really harmonize?

Jason

---

From: Sebastian Moeller <[email-masked]>
Date: Sat, 20 Mar 2021 17:21:57 +0100

Hi Jason,

> On Mar 20, 2021, at 16:08, Livingood, Jason <Jason_Livingood=[email-masked]> wrote:
> 
> IMO the challenge with lab studies is that there are a lot of variables that are artificial and/or the test environment or test traffic does not fully reflect reality (production).

	[SM] Nobody hindered team L4S to actually vary those parameters.

> 10 or 15 years ago I would have been very focused on extensive QA testing and an elaborate lab test environment. These days its more how to do controlled testing directly in the production network, with a controlled/minimized blast radius in case of problems, easy/quick rollback, rapid code/config iteration.

	[SM] Controlled testing over real networks is actually possible already, without the IETF's blessing, as long as one makes sure the potential fall-out is restricted to voluntarily participating nodes. To claim there is nothing between tests in the lab and "release the kraken" onto the whole internet, does not strike my as aterribly nuanced nor realistic position?

> Only one way to find out - test it in a real production experiment with appropriate risk controls.

	[SM] Like a guarding DSCP that makes sure the fall-out is restricted mainly to willingly cooperaring parties? Such common-sense safeguard have been summarily rejected by team L4S... In other words there will be no risk controls, let alone appropriate ones. I come to this conclusion as extrapolation of L4S lab test trajectory. Why do you assume the approach towards safety and functionality engineering is going to change?

> The status quo seems to suggest years more debate without (IMO) sufficient data.

	[SM] So all I need t do to get an incomplete draft to RFC status is to drag my feet? Surely that can not be your serious position?

> An alternative is a controlled production experiment, appropriately risk-managed, between willing participants.

	[SM] +1, but that is not what is under discussion here. We are talking about "release the kraken" based on theoretical considerations, that L4S will work and there will be no incentive to exploit its known failure modes. Engineering by "hope and prayers"...

> I see no downside to allowing that to occur via experimental RFC. We're setting the bar at the proposed standard or standard level, which I think is incorrect. I would say instead enable the experiment to accelerate the learning & improvement & drive data-driven decisions.

	[SM] Again, for a "a controlled production experiment" no buy in of the IETF is required, it never has. So why pretend otherwise?

Sebastian

P.S.: I missed the last meeting, so if team L4S actually presented data that answers all my questions, I am willing to eat my hat, so did they?

---

From: Pete Heist <[email-masked]>
Date: Mon, 22 Mar 2021 09:03:43 +0100

Hi Jason, one comment below...

On Sat, 2021-03-20 at 15:08 +0000, Livingood, Jason wrote:
> IMO the challenge with lab studies is that there are a lot of variables
> that are artificial and/or the test environment or test traffic does
> not fully reflect reality (production). 10 or 15 years ago I would have
> been very focused on extensive QA testing and an elaborate lab test
> environment. These days its more how to do controlled testing directly
> in the production network, with a controlled/minimized blast radius in
> case of problems, easy/quick rollback, rapid code/config iteration.
> 
> > far from a promising system that might be ready for a field trial,
> > one that shows worrying failure modes that primarily affect innocent
> > bystanders.
> >  What we're asking for is a system that at least behaves reasonably
> > under lab conditions, chosen to represent realistic challenge
> > scenarios likely to occur in the real world.  Only then can we have
> > any confidence that L4S will not cause problems anywhere and
> > everywhere that it is deployed.
> 

Along these lines, there is nothing stopping anyone from using DSCP (as
an appropriate risk control within participating ASs), to perform
experiments to reproduce issues that can be demonstrated in the lab.
Here are a few: https://github.com/heistp/l4s-tests/

It would be nice if someone took the time to set up some production
gear with fq_codel built into it, like routers and/or WiFi access
points, and test various kinds of real-world traffic, through tunnels
and not. One could combine traffic to L4S servers within the
experimental network, along with conventional traffic to anywhere on
the Internet. That seems like a useful step before starting an
Internet-wide experiment. Perhaps even better would be to fix what
safety issues are known so that they can't occur in the first place.

Regards,
Pete

>  The status quo seems to suggest years more debate without (IMO)
> sufficient data. An alternative is a controlled production experiment,
> appropriately risk-managed, between willing participants. I see no
> downside to allowing that to occur via experimental RFC. We're setting
> the bar at the proposed standard or standard level, which I think is
> incorrect. I would say instead enable the experiment to accelerate the
> learning & improvement & drive data-driven decisions.
> 
> Jason
>

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 22 Mar 2021 09:54:19 +0100

Hi Pete, hi List,

speaking of operational issues one could/should employ to make a potential L4S "experiment" safer. I think adding a section about how to guard an L4S experiment with an additional DSCP seems like a good addition to the ops draft. Yes, I understand that the main L4S drafts do not want to go there, but I see no reason why team L4S' reluctance to add this to the main internet drafts should stop the informational ops draft to describe such an approach well enough to give practitioners a pointer how to implement such a guarded deployment quickly.
	This is in addition to the section about what implementers are supposed to do at the end of the experiment for which I proposed/volunteered text already (but have heard exactly nothing back).

Best Regards
	Sebastian

> On Mar 22, 2021, at 09:03, Pete Heist <[email-masked]> wrote:
>

---

From: Sebastian Moeller <[email-masked]>
Date: Mon, 22 Mar 2021 11:31:12 +0100

Hi Jason,

given your deep involvement with an ISP that seems interested in deploying L4S, have you actually performed any tests in your network that could help assess L4S fitness for purpose? That is have you done any tests looking at how safe and how functional L4S is actually in a real network? While I do not hope for posting of internal data, it would be great if you could confirm whether your assessment of L4S fitness is based on data outside of data presented by the L4S proponents or critics in this thread and/or IETF meeting presentations?
I	t is well possible that I simply have not seen the data yet, that convincingly demonstrated L4S robustly and reliably delivering on its promises, but that is something I am happy to remedy, if such data exists.

Best Regards
	Sebastian

P.S.: Not a new question and not a question restricted to Jason. Anybody with independent data showing L4S achieving its promises (or failing to do so) please at least let the list know that you have seen such data.

> On Mar 20, 2021, at 15:33, Livingood, Jason <[email-masked]> wrote:
>