Subject: [ippm] Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-method-06: (with DISCUSS)

From: Magnus Westerlund via Datatracker <[email-masked]>
Date: Thu, 25 Feb 2021 06:18:43 -0800

Magnus Westerlund has entered the following ballot position for
draft-ietf-ippm-capacity-metric-method-06: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-ippm-capacity-metric-method/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

A) Section 8. Method of Measurement

I think the metrics are fine, what makes me quite worried here is the
measurement method. My concerns with it are the following.

1. The application of this measurement method is not clearly scoped. Therefore
I will assume that across the Internet measurements are possible. However in
that context I think the definition and protection against severe congestion
has significant short comings. The main reason is that the during a
configurable time period (default 1 s) the sender will attempt to send at a
specified rate by a table independently on what happens during that second.

2. The algorithm for adjusting rate is table driven but give no guidance on how
to construct the table and limitations on value changes in the table. In
addition the algorithm discusses larger steps in the table without any
reflection of what theses steps sides may represent in offered load.

3. Third the algorithms reaction to any sequence number gaps is dependent on
delay and how it is related to unspecified delay thresholds. Also no text
discussion how these thresholds should be configured for safe operation.

B) Section 8. Method of Measurement

There are no specification of the measurement protocol here that provides
sequence numbers, and the feedback channel as well as the control channel. Is
this intended to use TWAMP?

>From my perspective this document defines the metrics on standards track level.
However, the method for actually running the measurements are not specified on
a standards track level. No one can build implementation. And if the section is
intended to provide requirements on a protocol that performs these measurements
I think several aspects are missing. There appear several ways forward here to
resolve this; one is to split out the method of measurement and define it
separately to standard tracks level using a particular protocol, another is to
write it purely as requirements on a measurement protocols.

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Fri, 26 Feb 2021 06:05:13 +0000

Hi Magnus, Thanks for your review.

Please see replies to your review and comments from Len and me, consolidated and marked [acm] below,

Al

> -----Original Message-----
> From: Magnus Westerlund via Datatracker [mailto:[email-masked]]
> Sent: Thursday, February 25, 2021 9:19 AM
> To: The IESG <[email-masked]>
> Cc: [email-masked]; [email-masked];
> [email-masked]; Ian Swett <[email-masked]>; [email-masked];
> [email-masked]
> Subject: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-metric-
> method-06: (with DISCUSS)
> 
...
> I will assume that across the Internet measurements are possible. 
[acm]

Like all the ad hoc Internet Speed measurements in the world today, our primary interest is access measurement.  I added "Applicability" to the Scope and Goals section 2, as follows:

NEW paragraph
The primary application of this metric and method of measurement described here is the same as in Section 2 of RFC7479, where:

o The access portion of the network is the focus of this problem statement. The user typically subscribes to a service with bidirectional access partly described by rates in bits per second.

-=-=-=-=-=-=-=-

[acm] Obviously we won't be able to stop application beyond access, just like we can't stop people from grabbing one of any number of utilities and anywhere and sending at any rate they can achieve, but we have made an applicability statement. 

> However in
> configurable time period (default 1 s) the sender will attempt to send at
> a specified rate by a table independently on what happens during that
> second.
[acm] 
Not quite, 1 second is the default measurement interval for Capacity, but sender rate adjustments occur much faster (and we add a default at 50ms). This is a an important point (and one that Ben also noticed, regarding variable F in section 8.1). So, I have added FT as a parameter in section 4:

o FT, the feedback time interval between status feedback messages communicating measurement results, sent from the receiver to control the sender. The results are evaluated to determine how to adjust the current offered load rate at the sender (default 50ms)

-=-=-=-=-=-=-=-
Note that variable F in section 8.1 is redundant with parameter F in Section 4, 
the number of flows (in-06). So we changed the section 8.1 variable F to FT 
in the working text.

> reflection of what theses steps sides may represent in offered load.
[acm] 
We can add (Len suggested the following text addition):
OLD
8.1. Load Rate Adjustment Algorithm

A table SHALL be pre-built defining all the offered load rates that
will be supported (R1 through Rn, in ascending order, corresponding
to indexed rows in the table). Each rate is defined as datagrams of...

NEW
8.1. Load Rate Adjustment Algorithm

A table SHALL be pre-built defining all the offered load rates that
will be supported (R1 through Rn, in ascending order, corresponding
to indexed rows in the table). It is RECOMMENDED that rates begin with
0.5 Mbps at index zero, use 1 Mbps at index one, and then continue in
1 Mbps increments to 1 Gbps. Above 1 Gbps, and up to 10 Gbps, it is
RECOMMENDED that 100 Mbps increments be used. Above 10 Gbps,
increments of 1 Gbps are RECOMMENDED. Each rate is defined as...

-=-=-=-=-=-=-=-

> discussion how these thresholds should be configured for safe operation.
[acm] 
We can add some details in the paragraph below:
OLD
If the feedback indicates that sequence number anomalies were detected OR 
the delay range was above the upper threshold, the offered load rate is decreased. 
Also, if congestion is now ...
NEW
If the feedback indicates that sequence number anomalies were detected OR 
the delay range was above the upper threshold, the offered load rate is decreased. 
The RECOMMENDED values are 0 for sequence number gaps and 30-90 ms for lower 
and upper delay thresholds. Also, if congestion is now ...

-=-=-=-=-=-=-=-

Please also note many requirements for safe operation in Section 10, 
Security Considerations.

> sequence numbers, and the feedback channel as well as the control channel.
[acm] 
That is correct. The Scope does not include protocol development.

> Is this intended to use TWAMP?
[acm] 
Maybe, but a lot of extensions would be involved.

> 
> From my perspective this document defines the metrics on standards track
> level. However, the method for actually running the measurements are not
> specified on a standards track level. 
[acm]
In IPPM work, the methods of measurement are described more broadly than 
the metrics, as actions and operations the Src and Dst hosts perform to 
send and receive, and calculate the results. 

IPPM Methods of Measurement have not included protocol requirements in 
the past, in any of our Standards Track Metrics RFCs.  In fact, we developed 
a measurement-specific criteria for moving our RFCs along the standards track 
that has nothing to do with protocols or interoperability. 
See BCP 176 aka RFC 6576:  https://tools.ietf.org/html/rfc6576
IP Performance Metrics (IPPM) Standard Advancement Testing 

> No one can build implementation. 
[acm] 
I'm sorry, but that is not correct.  Please see Section 8.4.

> And if the section is
> separately to standard tracks level using a particular protocol, another
> is to write it purely as requirements on a measurement protocols.
[acm] 
As stated above, connecting a method with a single protocol is not IPPM's way.

This seems to be the type of document you are looking for, completely separate 
from the IPPM Metric the Method RFCs:

RFC 7497 https://tools.ietf.org/html/rfc7497
    Rate Measurement Test Protocol Problem Statement and Requirements

Abstract

   This memo presents a problem statement for access rate measurement
   for test protocols to measure IP Performance Metrics (IPPM).  Key
   rate measurement test protocol aspects include the ability to control
   packet characteristics on the tested path, such as asymmetric rate
   and asymmetric packet size.

RFC7497 doesn't anticipate the degree of rate control with feedback 
that we have described in the Capacity draft, but it has some useful aspects 
(access scope). It also provides evidence that the IPPM WG separates 
protocol requirements and protocol specification work (such as TWAMP and STAMP) 
from the RFCs on Metrics and Methods of Measurement.

This memo replaces ad hoc, inaccurate TCP-based measurements of "Internet Speed"
with a standardized metric and method. It allows the protocol development to begin; 
that's exactly what we said in the scope!

I have asked for agenda time at the IPPM session to briefly present and get 
feedback on the protocol used in the running code, the next step.

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 26 Feb 2021 10:37:43 +0000

Hi Al,

Please see inline. 

On Fri, 2021-02-26 at 06:05 +0000, MORTON, ALFRED C (AL) wrote:
> Like all the ad hoc Internet Speed measurements in the world today, our
> primary interest is access measurement.  I added "Applicability" to the Scope
> and Goals section 2, as follows:
> 
> NEW paragraph
> The primary application of this metric and method of measurement described
> here is the same as in Section 2 of RFC7479, where:
> 
> o The access portion of the network is the focus of this problem statement.
> The user typically subscribes to a service with bidirectional access partly
> described by rates in bits per second.
> 

Good, I guess that this was the case, but it wasn't stated explicitly. 

The reason I am worried is that this metric and also the method of measurement
as noted creates impact on the network. What makes me extra worried is that the
congestion avoidance property is currently not well documented and the rather
big parameterization space that is left without guidance and will impact the
properties. 

> -=-=-=-=-=-=-=-
> 
> [acm] Obviously we won't be able to stop application beyond access, just like
> we can't stop people from grabbing one of any number of utilities and anywhere
> and sending at any rate they can achieve, but we have made an applicability
> statement. 
> 
> > However in
> Not quite, 1 second is the default measurement interval for Capacity, but
> sender rate adjustments occur much faster (and we add a default at 50ms). This
> is a an important point (and one that Ben also noticed, regarding variable F
> in section 8.1). So, I have added FT as a parameter in section 4:
> 
> o FT, the feedback time interval between status feedback messages
> communicating measurement results, sent from the receiver to control the
> sender. The results are evaluated to determine how to adjust the current
> offered load rate at the sender (default 50ms)
> 
> -=-=-=-=-=-=-=-

Okay, that makes things clearer. With all the equal intervals in the metrics I
had missinterpreted that also the transmission would be uniform during the
measurement intervals. 

However, when rereading Section 8.1 I do have to wonder if the non-cumaltive
feedback actually creates two issues. First, it appears to loose information for
reordering that crosses the time when the FT timer fires, due to reset. In
addition if the feedback is not reliable it looses the information for that
interval. And making feedback reliabel could cause worse HOL issues for reacting
to later feedback that are recived prior to the lost one. 

> increments of 1 Gbps are RECOMMENDED. Each rate is defined as...

Is this what you actually used in your test implementation? At my first glance
this recommendation looks to suffer from rather severe step effects and also
make the respond to losses behave strange around the transitions. Wouldn't some
type of logarithmic progression be more appropriate here for initial probing?

If I have 1 GBPS line rate, there is a 1000 steps in the table to this value.
Even if I increase with the suggested 10 steps until first congestion seen, it
will take 100 steps, and with 50 ms feedback interval that is 5 seconds before
it is in the right ball park. And I get one random loss at 10 mbps, then its 990
steps, In such a situation the whole measurement period (10 s) would be over
before one has reached actual capacity. 

To me it appears that the probing (slow start) equivalent do need logarithmic
increase to reach likely capacity quickly. Then how big the adjustment is
actually dependent on what extra delay one consider the target for the test.
Having a step size of 1 GBPS if probing a 2.5 GBPS path would likely make it
very hard to keep the delay in the intended interval when it would fluxtuate
between 500 mbps to much traffic and then 500 mbps to little. Sure with
sufficiently short FT it will likely work in this algorithm. However, I wonder
about regulation stability here for differnet RTT, FTs and buffer depth
fluxtuations. 

From my perspective I think this is a indication that the load rate adjustment
algorithm is not ready to be a standards track specification. 

I would recommend that you actually take out the control algorithm and write a
high level functional description of what needs to happen when measuring this
capacity. 

If I understand this correctly the requirements on the measurement are the
following.

- Need to seek the available capacity so that several measurement period are
likely to be done at capacity
- Must not create persistent congestion as the capacity measurement should be
based on traffic capacity that doesn't cause more standing queue that X, where X
is some additiona delay in ms compared to minimal one way delay. And X is
actually something that is configurable for a measurement campagin as capacity
for a given one way delay and delay variation can be highly relevant to know. 

What else is needed?

Are synchronized clocks needed or just relative delay changes necessary? 

Ok, but the delay values as I noted before highly dependent of what my goal with
the capacity metric is. If I want to figure out the capacity for like say XR or
cloud gaming applications that maybe have much lower OWD variances and absolute
values so maybe my values are 10-25 ms. 

How much explaration have you done of the control stability over a range of
parameters? Do you have any material about that? 

> I'm sorry, but that is not correct.  Please see Section 8.4.

Sorry, that was poorly formulated. I mean that you can't give this specification
to a guy on a island without external communication and have them implemented it
and it will work with someone else implementation. You have clearly implemented
a solutiont that works for some set of parameters. And I am asking how much of
the reasonable parameter space you have tested. 

Based on this discussion I don't think I can build an implementation that
fulfills the measurement goals, becasue I have questions about them. And I
suspect it would take substantial amount of experimentation to get it to work
correctly over a broader range of input parameters. 

> As stated above, connecting a method with a single protocol is not IPPM's way.

That is fine. However, I find the attempt to specify a specific load regulator
in the method of measurement to take this specification beyond a general method
of measurment. The high level requirement appear to be that to correctly find
the capacity, and that requires that one load to the point where buffers are
filled sufficiently to introduce extra delay or where AQM starts dropping or
marking some of the load. Thus, I am questioning if the described algorithm will
adqueately solve that issue over a wider range of parameters. 

So if you have more information to show at least which range it has been proven
to do its work and with what input parameters? I hope you understand that I
expect this load control algorithm to get simularly scrutinized to congestion
control algorithms that we standardize in IETF. 

I would very much prefer to take out the load algorithm and place it in a
seperate document where it can have a tighter scope description and more
discussion about about that it does its job. 

I hope this clarifies what my concerns are with this document in its current
form. 

Cheers

Magnus

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 26 Feb 2021 10:47:25 +0000

(Resending as I got an error that it wasn't delivered due to signature)

Hi Al,

Please see inline. 

What else is needed?

Cheers

Magnus

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 27 Feb 2021 18:54:07 +0000

Hi Magnus,
Thanks for your further clarifications. We have more work ahead,
so I am deleting the topics where we appear to agree on changes 
in the first exchange (Applicability statement).

Consolidated replies and good ideas (we think) from the co-authors 
are in-line below.
thanks,
Al

> From: Magnus Westerlund [mailto:[email-masked]]
> Sent: Friday, February 26, 2021 5:47 AM
> To: [email-masked]; MORTON, ALFRED C (AL) <[email-masked]>
> Cc: [email-masked]; [email-masked]; draft-ietf-ippm-capacity-
> [email-masked]; [email-masked]; [email-masked]
> Subject: Re: Magnus Westerlund's Discuss on draft-ietf-ippm-capacity-
> metric-method-06: (with DISCUSS)
> 
> > Please see replies to your review and comments from Len and me,
> consolidated
> > and marked [acm] below,
> >
> > Al
> >
> > > 1. The application of this measurement method is not clearly scoped.
<snip> 
[acm] we agreed on text adding "access" applicability to the scope section. 

> > Note that variable F in section 8.1 is redundant with parameter F in
> Section
> > 4,
> reordering that crosses the time when the FT timer fires, due to reset.
[acm]
I don't understand how the sequence error counting "loses information" when 
reordered packets cross a measurement feedback boundary. I'm not sure what 
aspect of measurement you are "resetting", but I assume it is 
    "The accumulated statistics are then
     reset by the receiver for the next feedback interval."

Suppose I have two measurement intervals and I receive:

||  1  2  3  5  6 || 4  7  8  9 ...||

where || is the measurement feedback boundary.

Packet 4 arrives late enough from its original position to span the boundary.
The 3->5 sequence is one sequence error, and the 4->7 sequence is another error.
This example produces two sequence errors in different feedback intervals, 
but that's a typical measurement boundary problem. We can't get rid of 
measurement boundaries, and they affect many measurements.

Note that a reordered packet contributes to IP-Layer Capacity,
by definition.  

Perhaps you had some other scenario in mind?

> In addition if the feedback is not reliable it looses the information for that
> interval. 
[acm] 
That's right, and:
1. the sending rate does not increase or decrease without feedback
2. the feedback is traveling the reverse path that the test do not congest with 
   test traffic
3. the running code has watchdog time-outs that *terminate the connection* if
   either the sender or receiver go quiet 

In essence, the test method is not reliable byte stream transfer like TCP.
We can shut-down the test traffic very quickly if something is wrong and 
useful measurement is in question.

> And making feedback reliabel could cause worse HOL issues for
> reacting to later feedback that are recived prior to the lost one.
> 
[acm] 
So the alternative to un-reliable feedback can be worse? 
Good thing it's not planned.

> Is this what you actually used in your test implementation? 
[acm] 
Yes, except that the current table stops at 10Gbps. We haven't had the 
opportunity to test >10Gbps.

> At my first glance
[acm] 
Len and I considered various algorithms for the search.  

Logarithmic increase typically means more rate overshoot than Linear increases.
Unfortunately, a large rate overshoot means that the queues will fill and need
a longer time to bleed-off, meaning that rate reductions will take you far from
the "right neighborhood" again.

Our experience is that we avoid large overshoot with fast or slow linear increases.
It means we've taken some care to keep the network running. We haven't broken 
any test path yet, and we bail-out quickly and completely if something goes wrong.

> it is in the right ball park.
[acm] 
Your math is correct.

Remember that the only assumption we made when building the table of sending rates
is that the maximum *somewhere between 500kbps and 10Gbps*. Our lab tests used 
unknown rates between 50Mbps and 10Gbps, as the "ground truth" that we asked UDP 
and TCP-based methods to measure correctly. Measurements on production networks 
encountered many different technologies. Some subscriber rates were 5 to 10 Mbps
on upstream.  

> And I get one random loss at 10 mbps, then its 990
[acm] 
I'm sorry, that's not quite correct, assuming delay range meets the criteria
below, which would be consistent with "one random loss".

The text says:
  If the feedback indicates that sequence number anomalies were detected OR 
  the delay range was above the upper threshold, the offered load rate is 
  decreased.  (by one step)

But when the next feedback message arrives with no loss, and the "congestion"
state has not been declared, the relevant text is:

  If the feedback indicates that no sequence number anomalies were detected AND 
  the delay range was below the lower threshold, the offered load rate is increased. 
  If congestion has not been confirmed up to this point, the offered load rate is 
  increased by more than one rate (e.g., Rx+10).

and we return to the high speed increases, because:

  Lastly, the method for inferring congestion is that there were sequence 
  number anomalies AND/OR the delay range was above the upper threshold for 
  *two* consecutive feedback intervals.

So, there is a single step back for the single random loss, but then immediately
back to Rx+10 increases.

> actually dependent on what extra delay one consider the target for the test.
[acm] 
Our delay variation values are low, but need to accommodate the relatively high 
delay variation of some access technologies. We learned this during our testing
on production networks.

> about regulation stability here for differnet RTT, FTs and buffer depth
[acm] 
I'm sorry, that's not quite correct, the text we proposed to add says:

   It is RECOMMENDED that rates begin with 0.5 Mbps at index zero, 
   use 1 Mbps at index one, and then continue in 1 Mbps increments to 1 Gbps. 
   Above 1 Gbps, and up to 10 Gbps, it is RECOMMENDED that 100 Mbps increments be used. 
and                                                        ^^^^^^^^
   Above 10 Gbps, increments of 1 Gbps are RECOMMENDED.

Your example falls in the 1Gbps to 10Gbps range, where table increments are 
100Mbps. 

> algorithm is not ready to be a standards track specification.
[acm] 
Given several corrections above, the authors ask that you reconsider your position.
Please read on.

> high level functional description of what needs to happen when measuring this
[acm]
We worked for several weeks in December to make the current high-level description
an accurate one. The IESG review has resulted added some useful details that 
I have shared along the way.

> for a given one way delay and delay variation can be highly relevant to
[acm] 
These are not identical to our requirements.  For example:
 - Both the metric and method consider a number of measurement intervals, 
and the Maximum IP-Layer Capacity is determined from one (or more) of the intervals.

> Are synchronized clocks needed or just relative delay changes necessary?
[acm] 
Just delay variation, and the safest is RT delay variation.

> values so maybe my values are 10-25 ms.
[acm] 
We intend to measure the limit of the access technology, with a set of parameters
that work well for all technologies we have tested so far.  

Notice that I didn't type the word "application" above. Or "user experience".

Sure, there is sensitivity to the parameters chosen, and we supplied our 
well-tested defaults to maximize results comparability and technology coverage
(with no twiddling).

> 
> How much explaration have you done of the control stability over a range
> of parameters? Do you have any material about that?
[acm] 
Yes. There are several parameter ranges we examined.

If we set the delay thresholds high enough, we see the RTT grow as the queues
fill to max and tail-drop finally restricts the rate. We can measure the 
extent of buffer bloat this way (if it is present). It's Not our goal.

We have used lower thresholds of delay variation, which work fine on the 
PON 1Gbps access services.

In the collaborative testing of the Open Broadband Open Source project,
one participant contributed tests with a 5G system that exhibited systematic
low-level loss and reordering in his lab. For this unusual case, Len added
the features to set a loss threshold above zero, and to tolerate reordered
and duplicate packets with no penalty in rate adjustment.

We have tried a range of test durations (I=20, 30 for example). 

We have tried different steep-ness of ramp-up slope. Rate += 10 steps 
works well, even when measuring rates separated by 3 orders of magnitude.

But for the co-authors, it was more important that the load adjustment search
produce the correct Maximum IP-Layer Capacity for each of the lab conditions 
we created (including challenging conditions with competing traffic, long delay
etc.), and the many access technologies we tested in production use 
(where again we encountered similar challenging conditions).

> > See BCP 176 aka RFC 6576:
https://urldefense.com/v3/__https://tools.ietf.org/html/rfc6576__;!!BhdT!3 RMQMZOAFloQ1FvtC_LIrUsDqBGxQeJZ1F4GFt8TpbQ60w841wT54sLLoKS9-Yc$
> to a guy on a island without external communication and have them
> implemented it and it will work with someone else implementation. 
[acm] 
But then you are asking for protocol-level interoperability, Magnus.
That is not our scope, or the scope of any IPPM Metric and Method RFCs.
The procedures of BCP 176 tell us when independent implementations produce 
equivalent results, which is IETF's definition of "works with" for metrics
and methods.

> You have clearly
> implemented a solutiont that works for some set of parameters. And I am asking how
> much of the reasonable parameter space you have tested.
[acm] 
Right. I answered this question qualitatively above, but the co-authors
claim that an equally important question is the breadth of access technologies
we have tested.

The tests conducted over 2+ years used the following production access types:

1. Fixed: DOCSIS 3.0 cable modem with "triple-play" capability and embedded WiFi and
Wired GigE switch (two manufacturers).
2. Mobile: LTE cellular phone with a Cat 12 modem (600 Mbps Downlink, 50 Mbps uplink).
3. Fixed: passive optical network (PON) "F", 1 Gbps service.
4. Fixed: PON "T", 1000 Mbps Service.
5. Fixed: VDSL, service, at various rates <100 Mbps.
6. Fixed: ADSL, 1.5 Mbps.
7. Mobile: LTE enabled router with ETH LAN to client host 
8. Fixed: DOCSIS 3.1 cable modem with "triple-play" capability and embedded WiFi and
Wired GigE switch (two other manufacturers).

> suspect it would take substantial amount of experimentation to get it to
> work correctly over a broader range of input parameters.
[acm] 
Now we refer you to the references in the memo, particularly Appendix X 
of Y.1540:

   [Y.1540]   Y.1540, I. R., "Internet protocol data communication
              service - IP packet transfer and availability performance
              parameters", December 2019,
              <https://www.itu.int/rec/T-REC-Y.1540-201912-I/en>.

   [Y.Sup60]  Morton, A., Rapporteur, "Recommendation Y.Sup60, Interpreting
              ITU-T Y.1540 maximum IP-layer capacity measurements", June
              2020, <https://www.itu.int/rec/T-REC-Y.Sup60/en>.

and Liaisons, were many of the experimental results are summarized:

   [LS-SG12-A]
              12, I. S., "LS - Harmonization of IP Capacity and Latency
              Parameters: Revision of Draft Rec. Y.1540 on IP packet
              transfer performance parameters and New Annex A with Lab
              Evaluation Plan", May 2019,
              <https://datatracker.ietf.org/liaison/1632/>.

   [LS-SG12-B]
              12, I. S., "LS on harmonization of IP Capacity and Latency
              Parameters: Consent of Draft Rec. Y.1540 on IP packet
              transfer performance parameters and New Annex A with Lab &
              Field Evaluation Plans", March 2019,
              <https://datatracker.ietf.org/liaison/1645/>.

Also, see our slides from the Hackathons at IETF 105 and 106, and the 
IPPM WG sessions slides beginning with IETF-105, July 2019.
You might also look into the discussions on the mailing list.
Some other results are available to those with ITU-TIES accounts.

The load adjustment algorithm itself was improved after experimentation,
adding the fast ramp-up with rate += 10 when feedback indicates no
impairments. The original/current algorithms appear in Y.1540 Annexes A 
and B, respectively. 

> to do its work and with what input parameters? 
[acm] 
Yes. See ~10 references  and replies above.

> I hope you understand that I
[acm] 
Yes, although it is a surprising at this point, we certainly understand your 
current position.

However, Rüdiger made a relevant point in our discussions (why our algorithm's 
role is different from Transport Area congestion control algorithms, 
and need not be subjected to the same scrutiny):

    This is a measurement method designed for infrequent and sensible maximum 
    capacity assessment, instantiated only in an OAM or diagnostic tool. 

    It is not a blueprint for a congestion control algorithm (CCA) in a bulk 
    transfer protocol that runs by default and is globally deployed by 
    commodity stacks.

We don't want to re-create any TCP CCA: they weren't designed for accurate 
measurement of maximum rate (as the referenced measurements show). 

It appears that the most recent (2018) standardized and widely used 
CCA is Cubic (RFC 8312 https://tools.ietf.org/html/rfc8312 ).  

The great TCP CCA Census (2019)
https://datatracker.ietf.org/meeting/109/materials/slides-109-iccrg-the-great-internet-tcp-congestion-control-census-00

finds that BBR versions account for greater popularity on Alexa-250 sites
(25.2%) than CUBIC, and more than 40% of downstream traffic on the Internet
(slide 14). I found some references to BBR in ICCRG drafts, but no RFC.
I would guess that BBR has already provided CCA for more traffic than the 
test traffic complying to this memo ever will.

Our overall method works similar to BBR: Received rate per RTT is the 
feedback to the sender. 

We added Applicability to the access portion, not the global Internet 
where standardized transport protocol CCAs must operate.

We are not specifying a transport CCA that must support many applications.
Measurement is the *only* application (for an IP-layer metric).

Early impressions have been formed on several erroneous assumptions 
regarding algorithm stability (operation above 1Gbps) and 
suitability for purpose (one random loss case).

Ad-hoc methods resulting in TCP-based under-estimates of Internet Speed 
are the problem we attack here! Implementation of harmonized industry 
standards are the solution.

We believe in rough consensus and running code.

We also ask that you understand our position, that tests with many different 
access technologies in production, and careful comparison of ad-hoc methods
claiming to make the similar measurements in the lab and the field are equally, 
if not more important than even more parameter investigations at this point. 
I have personally been running lab tests since September 2018 with various tools. 
Len released his first version of the code in Feb 2019,
and we immediately focused on tests with his utility instead of UDP packet 
blasters like iPerf, and Trex with my own Binary Search with Loss verification
algorithm that we use in device benchmarking (cross-over with BMWG).

> I hope this clarifies what my concerns are with this document in its
> current form.
[acm] 
Yes, and we have rather exhaustively argued to go ahead here, especially 
since a much less-frequently-used testing-only algorithm is a different situation 
than specifying a CCA for global TCP deployment: transport area's usual role.

We hope you can now appreciate the years of study, experimentation and 
running code that you apparently first encountered last Thursday, 
and will look into some more of the supporting background material.

Thanks again.