Subject: [aqm] Review comments on draft-kuhn-aqm-eval-guidelines-02

From: [email-masked]
Date: Fri, 1 May 2015 08:12:34 +0100

This is a review of AQM Evaluation Guidelines 
(draft-kuhn-aqm-eval-guidelines-02)

I think this document has a useful set of tests.

I think the document has value and an updated version of this document
should be published. There are likely to be more variants of AQM schemes
that emerge and this provides a valuable framework to help compare them
and to provide better insight into their properties.

I sent a set of minor corrections and formatting issues to the document
editors as a separate email - I’ll not list them here, since these are now
planned for the next revision. I’ll focus on three topics:

(1) The document is a standards-track specification of tests. While the
tests seemed generally good, I found the original recommendations
inconsistent.

Section 1: I think this section is informative. I propose moving the one
new requirement (recommending documenting drop-tail comparison is placed
in the relevant section. The language needs to be adjusted to avoid RFC
2119 language until this is defined.

Section 2: There is a RFC2119 keyword at the start of this section. I
think to may be good to consider how this is expressed: “These metrics are
RECOMMENDED to assess the performance of an AQM  scheme.” This could be
better perhaps to say: “This section provides normative requirements for
metrics that can be used to assess the performance of an AQM scheme.”

- I suggest rephrasing one sentence to:“ It is therefore not necessary to
measure all of the following metrics, and guidance is provided for each
metric.” This is to avoid the possibility that the reader thinks all
recommendations are simply optional!

- Section 2.3: Packet loss synchronization - Does not make a
recommendation, is this SHOULD, MUST or MAY - saying it is important is
not sufficient. I suspect it is a “MUST”.

- I think sections 3.1 and 3.2 are informative. Section 4.4 - likewise. In
general, I think the RFC2119 usage needs to be tightened to clearly
conclude what is to be tested in each case, whether the test is optional,
etc. Specifically, I think the language for the requirement should be to
“evaluate” each of the test cases, not just to implement a scenario.

- Section 12: This combines “SHOULD” with “sufficiently detailed” - is it
ok to say SHOULD provide a detailed … or something that does leaves the 
judgment of “sufficient” part to the evaluater, ensuring some detail is
provided.

- It may be useful to add a summary table. I do like the idea of a summary
table at the end, but to be clear if this is done, please use the EXACT
RFC2119 keywords from the sections, to prevent people detecting
inconsistencies here.

(2) Scenarios and test cases - I think we need to be clear (above) what is
required.

I’m also suggest to introduce a multi-AQM scenario where we identify that
there is a need to consider the implications of more than one congested
bottleneck along a path that has an active AQM.  This case came out of
Gen-ART review for the AQM Recommendations, and I think we should identify
this as a test case: e.g. "“Transports operating under the control of AQM
experience the effect of multiple control loops that react over different
timescales. It is therefore important that proposed AQM schemes are seen
to be stable when they are deployed at multiple points of potential
congestion along an Internet path. The pattern of congestion signals (loss
or ECN-marking) arising from AQM methods also need to not adversely
interact with the dynamics of the transport protocols that they control.”

(3) I have sent text (offlist) that attempts to correct some deviations in
language usage from the AQM recommendations ID, and suggested to cite this
where possible, rather than say things similar. These details proved to be
minor, I suggested some additional points and rewording for others.

I think though a small amount of additional work is still needed to agree
which test evaluations and discussions are a MUST and which are a MAY and
there seems to be some contradictions, which willneed to be checked in the
next revision.
Best wishes,

Gorry

---

From: [email-masked]
Date: Fri, 1 May 2015 08:14:46 +0100

I did review the latest WG draft, (sorry for the wrong name in the title.

Gorry

> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

---

From: Nicolas Kuhn <[email-masked]>
Date: Thu, 21 May 2015 10:04:49 +0200

Hi Gorry, all, 

We have posted a -03 version that features Gorry’s comments and Gorry’s work 
on making this document consistent with the recommendations document. 

The major modifications in the -03 version are:
- one new multi-aqm scenario;
- a conclusive table that clearly states the requirements for each scenario.

We are currently working on Jim’s review and we will be back as soon as possible.

Please see some comments on Gorry’s review inline. 

> On 01 May 2015, at 09:12, [email-masked] wrote:
> 

The name is wrong, but the reviewed document is: AQM Characterization Guidelines (draft-ietf-aqm-eval-guidelines-02)

> editors as a separate email - Ill not list them here, since these are now
> planned for the next revision. Ill focus on three topics:
> 

We acknowledge that there might be some inconsistencies and ambiguity between MUSTs and SHOULDs, etc;
but thanks to Gorry’s help, we have made sure that the usage of these words is more accurate.
If there is still some inconsistency, we believe the conclusive table would help make the document more
consistent.
We also encourage people on ML to point out any potential inconsistency that may still exist. Our aim is to 
have zero inconsistency and complete clarity in the paper’s text. 

> in the relevant section.

This section is indeed informative. We propose to move the 
“
   One key objective behind formulating the guidelines is to help
   ascertain whether a specific AQM is not only better than drop-tail
   but also safe to deploy.  Testers therefore need to provide a
   reference document for their proposal discussing performance and
   deployment compared to those of drop-tail.
"

To the Methodology sections. 

> The language needs to be adjusted to avoid RFC

The paragraphs of the first section has been moved around to avoid this situation.

> think to may be good to consider how this is expressed: These metrics are
> RECOMMENDED to assess the performance of an AQM  scheme. This could be
> better perhaps to say: This section provides normative requirements for
> metrics that can be used to assess the performance of an AQM scheme.
> 

Thanks.

> - I suggest rephrasing one sentence to: It is therefore not necessary to
> metric. This is to avoid the possibility that the reader thinks all

Thanks.

> not sufficient. I suspect it is a MUST.
> 

We would say it is a “SHOULD” as this section only provides guidance on each metric - and it is not necessary to measure all of them. 

> etc.

We hope that the conclusive table would help to clearly stipulate the test scenarios using RFC2119 terminology.

Even if a test may be considered, the description of the test may include “must do this / that”, to clearly say what must be done
if the test is selected. We hope that we have made that clearer in the -03 version. 

> Specifically, I think the language for the requirement should be to
> evaluate each of the test cases, not just to implement a scenario.
> 

Agree.

> - Section 12: This combines SHOULD with sufficiently detailed - is it
> ok to say SHOULD provide a detailed 
> or something that does leaves the 
> judgment of sufficient part to the evaluater, ensuring some detail is
> provided.
> 

Agree - the text now says : 

“   A description of each test setup SHOULD be detailed to allow this"

> RFC2119 keywords from the sections, to prevent people detecting

The conclusion now features a conclusive table summing up the requirement on each
test. 

> Im also suggest to introduce a multi-AQM scenario where we identify that
> this as a test case: e.g. "Transports operating under the control of AQM
> interact with the dynamics of the transport protocols that they control.
> 

We have added a multi-AQM scenario with a specific parking-lot topology. 
For the sake of simplicity, the AQM scheme should be the same in each router, but
alternative configurations could be considered. 

> minor, I suggested some additional points and rewording for others.

Thanks.

> there seems to be some contradictions, which willneed to be checked in the

Kind regards, 

The authors.