Subject: [alto] Lars Eggert's Discuss on draft-ietf-alto-performance-metrics-19: (with DISCUSS and COMMENT)

From: Lars Eggert via Datatracker <[email-masked]>
Date: Mon, 29 Nov 2021 05:10:05 -0800

Lars Eggert has entered the following ballot position for
draft-ietf-alto-performance-metrics-19: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/blog/handling-iesg-ballot-positions/
for more information about how to handle DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

This document needs to become much more formal about how it defines the
metrics it wishes to use with ALTO. This could either be done either by
identifying and normatively referencing existing metrics the IETF has defined,
or by defining them here. When normatively referencing existing IETF metrics, it
would need to explain why their use with ALTO makes sense.

At the moment, the document informatively points to a somewhat arbitrary
collection of prior IETF metrics (most of which are from IPPM, residual
bandwidth from IS-IS TE, but then reservable bandwidth from OSPF TE?). But it
only refers to them as "examples", without actually defining how exactly they
are to be used with ALTO, or - if not those - which actual metrics are supposed
to be used.

Defining a mechanism for exposing metric information to clients isn't really
useful unless the content of that information is much more clearly specified.

Section 4.1.3. , paragraph 2, discuss:
>    Intended Semantics: To give the throughput of a TCP congestion-
>    control conforming flow from the specified source to the specified
>    destination; see [RFC3649, Section 5.1 of RFC8312] on how TCP
>    throughput is estimated.  The spatial aggregation level is specified
>    in the query context (e.g., PID to PID, or endpoint to endpoint).

A TCP bandwidth estimate can only be meaningfully be derived for bulk TCP
transfers under a set of pretty strict and simplistic assumptions, making this
metric a meaningless at best and misleading at worst, given that the source of
this information doesn't know what workload, congestion controller and network
conditions the user of this information will use or see.

Also, RFC3649 is an Experimental RFC (from 2003!) and RFC8312 is an
Informational RFC. Since this document normatively refers to them, it needs to
cite them, and this will cause DOWNREFs for PS document. I would argue that
at least RFC3649 is certainly not an appropriate DOWNREF.

Why define this metric at all? The material you point to is the usual
model-based throughput calculation based on RTT and loss rates; a client that
intended to predict TCP performance could simply query ALTO for this and perform
their own computation, which will likely be more accurate, since the client will
hopefully know which congestion controller they will use for the given workload,
and what the characteristics of that workload are.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Section 1. , paragraph 6, comment:
>    The purpose of this document is to ensure proper usage of the
>    performance metrics defined in Table 1; it does not claim novelty of
>    the metrics.  The "Origin Example" column of Table 1 gives an example
>    RFC that has defined each metric.

I don't understand what the purpose of the "origin example" column is. Most of
these point to IPPM metrics, which have a pretty clear and narrowly-defined area
of applicability. Since ALTO isn't performing IPPM-style network testing, it's
not clear why IPPM metrics are referenced here?

Section 2.2. , paragraph 23, comment:
>    If a cost metric string does not have the optional statical operator
>    string, the statistical operator SHOULD be interpreted as the default
>    statical operator in the definition of the base metric.  If the

What is a "statical" operator; I am not familiar with the term and it doesn't
seem to appear in other RFCs? (Also occurs elsewhere in this document.)

Section 3.1.4. , paragraph 4, comment:
>    link statistics.  Another example of a source to estimate the delay
>    is the IPPM framework [RFC2330].  It is RECOMMENDED that the

IPPM defines measurement metrics. How would they be a source for estimates?

Section 3.3. , paragraph 1, comment:
> 3.3.  Cost Metric: Delay Variation (delay-variation)

Is this supposed to apply to the one-way or bidirectional delay? Also, delay
variation is not independent from path utilization (c.f. bufferbloat), so why is
it being reported independently?

Section 3.5. , paragraph 1, comment:
> 3.5.  Cost Metric: Loss Rate (lossrate)

What is this metric supposed to capture? Loss is generally not independent from
network utilization (apart from random corruption loss). So it should be zero
for unloaded networks, and depends on utilization otherwise. Also, is this
unidirectional or bidirectional loss (wording below is unclear)?

Using lowercase "not" together with an uppercase RFC2119 keyword is not
acceptable usage. Found: "MUST not"

The document has 6 authors, which exceeds the recommended author limit. I
assume the sponsoring AD has agreed that this is appropriate?

No reference entries found for: [RFC3649] and [RFC8312].

Found terminology that should be reviewed for inclusivity; see
https://www.rfc-editor.org/part2/#inclusive_language for background and more
guidance:

 * Term "man"; alternatives might be "individual", "people", "person".

-------------------------------------------------------------------------------
All comments below are about very minor potential issues that you may choose to
address in some way - or ignore - as you see fit. Some were flagged by
automated tools (via https://github.com/larseggert/ietf-reviewtool), so there
will likely be some false positives. There is no need to let me know what you
did with these suggestions.

"Abstract", paragraph 2, nit:
-    types of cost metric.  Since the ALTO base protocol (RFC 7285)
+    types of cost metrics.  Since the ALTO base protocol (RFC 7285)
+                        +

Section 1. , paragraph 2, nit:
> ] on registering ALTO cost metrics. Hence it specifies the identifier, the in
>                                     ^^^^^
A comma may be missing after the conjunctive/linking adverb "Hence".

Section 2.2. , paragraph 2, nit:
> of the observations. median: the mid point (i.e., p50) of the observations.
>                                  ^^^^^^^^^
This word is normally spelled with a hyphen.

"IPPM ", paragraph 2, nit:
>  Also, delay variation is not independent from path utilization (c.f. buffer
>                               ^^^^^^^^^^^^^^^^
The usual collocation for "independent" is "of", not "from". Did you mean
"independent of"?

Section 3.3.3. , paragraph 7, nit:
> apture? Loss is generally not independent from network utilization (apart fr
>                               ^^^^^^^^^^^^^^^^
The usual collocation for "independent" is "of", not "from". Did you mean
"independent of"?

Section 3.4.3. , paragraph 6, nit:
> imation" method. See Section 3.1.4 on on related discussions such as summing
>                                    ^^^^^
Possible typo: you repeated a word.

Section 3.5.4. , paragraph 3, nit:
>  [RFC8312]), it helps to specify as much details as possible on the the cong
>                                     ^^^^
Use "many" with countable plural nouns like "details".

Section 3.5.4. , paragraph 3, nit:
> ify as much details as possible on the the congestion control algorithm used
>                                    ^^^^^^^
Two determiners in a row. Choose either "the" or "the".

These URLs in the document can probably be converted to HTTPS:
 * http://www.iana.org/assignments/alto-protocol/alto-protocol.xhtml#cost-metrics

---

From: Martin Duke <[email-masked]>
Date: Mon, 29 Nov 2021 12:01:57 -0800

1. One problem here is that 3649 and 8312 aren't in the references at all,
so that should be fixed if we retain the text

2. IMO collection of bandwidth information is out of scope of this
document; in general collection of network information is out of scope for
ALTO. I viewed the references as informative examples rather than a
prescription of how it's collected.

3. I agree, in retrospect, that 3649 and 8312 are bad suggestions, as their
result is very time-dependent and noisy. From the rest of section 4, this
metric is clearly in the context of bandwidth reservations and min(raw link
capacity) is probably a better indicator. However, the rest of these
sections don't really talk about metric definitions, so it might be best
simply to delete the parenthetical.

So my suggestion would be to take Lars's request to define what the authors
mean a little precisely (raw capacity?), but not to the granularity that
IPPM defines metrics, and get rid of the references entirely.

Is that satisfactory to you, Lars?

Martin

On Mon, Nov 29, 2021 at 5:10 AM Lars Eggert via Datatracker <
[email-masked]> wrote:

>  *
> http://www.iana.org/assignments/alto-protocol/alto-protocol.xhtml#cost-metrics
>
>
>
>

---

From: "Y. Richard Yang" <[email-masked]>
Date: Tue, 30 Nov 2021 20:49:35 -0500

Hi Lars,

Thanks for the review! Please see below.

On Mon, Nov 29, 2021 at 8:10 AM Lars Eggert via Datatracker <
[email-masked]> wrote:

> bandwidth from IS-IS TE, but then reservable bandwidth from OSPF TE?).

To give some background, the WG derived the list of metrics from RFC 8571
(BGP - Link State (BGP-LS) Advertisement of IGP Traffic Engineering
Performance
Metric Extensions), focusing on network->application. The list added Hop
Count
(exists in original ALTO RFC 7285), Round-trip (to avoid two queries, and
many apps
use RTT), and TCP Throughput, and removed Unidirectional Available
Bandwidth
and Unidirectional Utilized Bandwidth, to reduce the number of bandwidth
metrics.

> But it
> only refers to them as "examples",

I searched the word "example" and do not see where the document says that
they
are examples. It says that "Since different applications may use different
cost metrics,
the ALTO base protocol introduces an ALTO Cost Metric Registry (Section
14.2 of
[RFC7285]), as a systematic mechanism to allow different metrics to be
specified. For
example, a delay-sensitive application may want to use latency-related
metrics, and
a bandwidth-sensitive application may want to use bandwidth-related
metrics."

Does this paragraph give an impression that the metrics are only examples?
If so,
do you suggest removing the "For example" phrase to reduce the impression?

The document does have the sentence " The "Origin Example" column of Table
1 gives an example RFC that has defined each metric." Here the word
"example"
word means one existing work.

> without actually defining how exactly they

The document has "... the ALTO base protocol introduces an ALTO Cost Metric
Registry
(Section 14.2 of [RFC7285]), as a systematic mechanism to allow different
metrics
 to be specified. " and "When an ALTO server supports a cost metric defined
in this document,
it should announce this metric in its information resource directory (IRD)
as defined in
 Section 9.2 of [RFC7285]." Does this provide enough on how exactly they
should be used?
The function of this document is to satisfy the registry and the use will
be in the base protocol
(RFC7285). If there is a specific suggestion, it will be good to have.

> I agree with this statement that information should be specified as
clearly as
possible, but at the same time, we need abstraction to reduce the
complexity.
One guiding principle in the design is that ALTO information provides
reasonable guidance, not mathematical precision.

> transfers

Yes. It is intended for bulk transfer.

> under a set of pretty strict and simplistic assumptions, making this
> metric a meaningless at best and misleading at worst,

I will say that TCP throughput formula in general has turned out to be
quite useful.

> given that the source of

Network (the source) is in a pretty good position to estimate the potential
TCP
throughput. In a high multiplexing setting (small fish in a big pond),
network can
have access to estimated loss rate, RTT, and typical packet size to compute
the TCP throughput formula. In a low multiplexing setting (big fish in a
small pond),
network can know the set of flows and estimate the bandwidth share. See the
citation
of the Prophet work in the document and the G2 work in SIGMETRICS'21 and
SIGCOMM'21. The congestion controller info is part of the metric (the link
points
to standard TCP/Reno). I made some minor edits to clarify.

> at least RFC3649 is certainly not an appropriate DOWNREF.
Good suggestion! I added the reference to 3649 from the second paragraph of
Sec. 5.1 of RFC8312 (you are a co-author). It reads "The average
window sizes of Standard TCP and HSTCP are from [RFC3649].  The
average window size of CUBIC is calculated using Eq. 6 and the CUBIC
TCP-friendly region for three different values of C." Our plan, which is
already suggested
by Martin but it is my fault to not update yet, is to remove 3649 and use
RFC8312bis.
Make sense?

> and what the characteristics of that workload are.

The throughput formula is for a very limited setting, i.e., the small fish
setting. What
we found useful is the low multiplexing setting, where the loss rate is the
output,
not the input, of the convergence process. It has good use cases. Please see
the Prophet paper and one most recent example is the use cases, such as
accelerating time-bound constrained flows, in Sec. 3 of
https://www.reservoir.com/wp-content/uploads/2021/08/G2_QTBS_TR_2021.pdf
The paper uses max-min fairness but the Internet uses other fairness.

The metrics that this document use are defined in multiple IETF documents
before.
The intention of the sentence is to give early work credit.

> Apology for the typo. statical operator -> statistical operator. They are
fixed in
an internal version but we did not upload.

> IPPM defines measurement metrics. How would they be a source for estimates?
The intention was to refer to the measurement methodology in 6.2 of RFC
2330, but
I can see the potential confusion now. How about we change the wording to
"Another example of a source to estimate the delay is through active
measurements,
for example, considering the IETF IPPM framework [RFC2330]."

> Is this supposed to apply to the one-way or bidirectional delay?

This is the current specification: "
3.3.3.  Intended Semantics and Use

   Intended Semantics: To specify spatial and temporal aggregated delay
   variation (also called delay jitter)) with respect to the minimum
   delay observed on the stream over the one-way delay from the
   specified source and destination.  The spatial aggregation level is
   specified in the query context (e.g., PID to PID, or endpoint to
   endpoint)."

So it is one way.

Also, delay
> it being reported independently?

Not sure I understand the suggestion. We see reports of jitter
(e.g., https://cpr.att.com/pdf/se/0001-0003.pdf) reported independently (in
the sense
as a single metric, without specifying as conditional
values/probabilities).

> unidirectional or bidirectional loss (wording below is unclear)?

It is meaningful in high multiplexing settings. There can also be an
load-independent
(I can see that you may see interference can be load as well) loss rate
when there are
wireless links.

It is intended to be unidirectional: "3.5.3.  Intended Semantics and Use

   Intended Semantics: To specify spatial and temporal aggregated packet
   loss rate from the specified source and the specified destination.
   The spatial aggregation level is specified in the query context
   (e.g., PID to PID, or endpoint to endpoint)."

How about the following change:
" To specify spatial and temporal aggregated packet
   loss rate from the specified source and the specified destination."
=>
 To specify spatial and temporal aggregated packet
   loss rate, in one way, from the specified source and the specified
destination."

> acceptable usage. Found: "MUST not"
Got it. We have fixed the case:
"The total length of the cost metric string MUST not exceed 32"
=>
"The total length of the cost metric string MUST NOT exceed 32"

> No reference entries found for: [RFC3649] and [RFC8312].
Thanks for pointing it out. It was missing after an update and pointed out
by
Martin. It is fixed in the next version which we will upload soon.

> Hah. You mean change
"man-in-the-middle (MITM) attacks"
=>
"person-in-the-middle attacks".

I looked and indeed see PITM (
https://en.wikipedia.org/wiki/Man-in-the-middle_attack).

Interesting and fixed. Thanks!

The edits below are great and fixed. Thanks again!

Richard

> _______________________________________________
> alto mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/alto
>

-- 
-- 
 =====================================
| Y. Richard Yang <[email-masked]>   |
| Professor of Computer Science       |
| http://www.cs.yale.edu/~yry/        |
 =====================================

---

From: "Y. Richard Yang" <[email-masked]>
Date: Tue, 30 Nov 2021 21:27:40 -0500

Hi Lars,

Just posted an updated version, and a diff from the previous version is
available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-alto-performance-metrics-20

Thanks!
Richard

On Tue, Nov 30, 2021 at 8:49 PM Y. Richard Yang <[email-masked]> wrote:

> The document has "... the ALTO base protocol introduces an ALTO Cost
> Metric Registry
>  to be specified. " and "When an ALTO server supports a cost metric
> defined in this document,
> Network (the source) is in a pretty good position to estimate the
> potential TCP
> network can know the set of flows and estimate the bandwidth share. See
> the citation
> we found useful is the low multiplexing setting, where the loss rate is
> the output,
> (e.g., https://cpr.att.com/pdf/se/0001-0003.pdf) reported independently
> (in the sense