Subject: Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

From: Wesley Eddy <[email-masked]>
Date: Tue, 1 Dec 2015 14:19:52 -0500

Thanks for making this update.

I don't think Roland and Polina who had made last call comments had yet 
had the time to check that the changes in the last revision met what 
they were expecting, so I'd like to give them (and anyone else who has 
comments) a couple of weeks to check this revision out, and assuming 
there are no major issues or objections by around 12/15, will plan to 
end the working group last call, and send the document to the area 
director for publication.

On 11/27/2015 5:50 AM, Kuhn Nicolas wrote:
> Dear all,
>
> This updated version integrates:
> - modification on the buffer sizes, following some discussion points raised by Michael Scharf on the tcpm mailing list [1];
> - some nits raised by Greg Skinner.
>
> Kind regards,
>
> Nicolas KUHN
>
> [1] https://www.ietf.org/mail-archive/web/tcpm/current/msg09894.html
>
> -----Message d'origine-----
> De : aqm [mailto:[email-masked]] De la part de [email-masked]
> Envoyé : vendredi 27 novembre 2015 11:44
> À : [email-masked]
> Cc : [email-masked]
> Objet : [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt
>
>
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>   This draft is a work item of the Active Queue Management and Packet Scheduling Working Group of the IETF.
>
>          Title           : AQM Characterization Guidelines
>          Authors         : Nicolas Kuhn
>                            Preethi Natarajan
>                            Naeem Khademi
>                            David Ros
> 	Filename        : draft-ietf-aqm-eval-guidelines-09.txt
> 	Pages           : 35
> 	Date            : 2015-11-27
>
> Abstract:
>     Unmanaged large buffers in today's networks have given rise to a slew
>     of performance issues.  These performance issues can be addressed by
>     some form of Active Queue Management (AQM) mechanism, optionally in
>     combination with a packet scheduling scheme such as fair queuing.
>     The IETF Active Queue Management and Packet Scheduling working group
>     was formed to standardize AQM schemes that are robust, easily
>     implementable, and successfully deployable in today's networks.  This
>     document describes various criteria for performing precautionary
>     characterizations of AQM proposals.  This document also helps in
>     ascertaining whether any given AQM proposal should be taken up for
>     standardization by the AQM WG.
>
>
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/
>
> There's also a htmlized version available at:
> https://tools.ietf.org/html/draft-ietf-aqm-eval-guidelines-09
>
> A diff from the previous version is available at:
> https://www.ietf.org/rfcdiff?url2=draft-ietf-aqm-eval-guidelines-09
>
>
> Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org.
>
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>
>

---

From: Polina Goltsman <[email-masked]>
Date: Mon, 7 Dec 2015 13:32:18 +0100

Dear all,

Apologies for the long delay.

I have review the latest version of the draft. In my opinion the 
document still needs a detailed review about the representation (the 
text): the structure of the document would make more sense if the 
information in section 2, 3, and 4 would appear in different order, the 
document still could use a more consistent terminology, and the 
structure of section 5-12 could be more consistent. I am happy to 
provide the detailed review, but since it is a lot of work, I have 
several concerns about the content of the document, which I would prefer 
to clear first.

The document proposes the evaluation methodology for AQMs which consists 
of the following topology and scenarios:

    0. the experiments should be performed on the topology, described in
    Figure 1 in Section 3, where the bottleneck between routers L and R
    is of *unspecified* capacity, which SHOULD include*both symmetric
    and asymmetric*, and the buffer MAY be set to a BDP. If I understand
    correctly, it is further RECOMMENDED to use a range of input
    parameters for the evaluated AQM (this may only be required if
    several AQMs are being compared)

    1. (section 5) several experiments with different TCP congestion
    control schemes (+UDP) which should be performed on *unspecified*
    bottleneck capacity and *unspecified* RTT, which should or should
    not be the same as the one with which the AQM is configured.

    2. (section 6) RTT fairness, where two groups of *unspecified*
    number of flows share a bottleneck of *unspecified* capacity, where
    the first group sees RTT of 100ms and the second is in range [5ms;560ms]

    3. (section 7) burst absorption, is a group of four scenarios, which
    include web-traffic, bursty video frames, which should be generated
    in an unspecified manner (*left to the tester to decide*), CBR
    traffic of *unspecified* rate, and TCP flow, all of which is
    evaluated using a set of metrics that MAY be generated. I might be
    missing something, but I don't understand how any of these metrics
    can be used to  characterize burst absorption.

    4. (section 8) stability, including (a) varying the number of flows
    on a bottleneck of *unspecified* capacity with *unspecified* RTT,
    according to the provided formulas, (b) varying network capacity
    between 10Mbps an 100Mbps for one TCP flow (if I understand it
    correctly)

    5. (section 9) which includes (a) traffic mix of a combination
    applications, including TCP, web-traffic, bi-direction VoIP using
    *unspecified congestion* control (I am not sure, but I think there
    are options), CBR of *unspecified* rate, and adaptive video
    streaming also with *unspecified* congestion control, with one
    combination required and other left to the tester to decide; and (b)
    bi-directional traffic, which should evaluate the effect on dropping
    DNS/TCP Syn packets* using a specified number of bulk TCP flows
    using the the throughput-delay tradeoff graph; (in both scenarios
    capacities and RTT are *unspecified* as well)

* I assume that DNS/SYN packets are mentioned in this section by mistake

Do you think that a person without a substantial background knowledge on 
the evaluation of AQM schemes can perform this evaluation (resolve all 
unspecified conditions) in a reasonable amount of time?

In the abstract, the document says that it describes characterization 
guidelines for an AQM proposal, to decide whether it should be adopted 
by the AQM WG. The WG currently has two AQMs (dropping/marking policy) 
in last call. Did someone evaluate these AQMs according to the specified 
guidelines?

Moreover, it seems to me that the WG is about to conclude. What exactly 
is the purpose of standardizing this document then ?

Regards,
Polina

On 12/01/2015 08:19 PM, Wesley Eddy wrote:
> I don't think Roland and Polina who had made last call comments had 
> yet had the time to check that the changes in the last revision met 
> what they were expecting, so I'd like to give them (and anyone else 
> who has comments) a couple of weeks to check this revision out, and 
> assuming there are no major issues or objections by around 12/15, will 
> plan to end the working group last call, and send the document to the 
> area director for publication.
>
>
>
>> - modification on the buffer sizes, following some discussion points 
>> raised by Michael Scharf on the tcpm mailing list [1];
>> Please note that it may take a couple of minutes from the time of 
>> submission until the htmlized version and diff are available at 
>> tools.ietf.org.
>>

---

From: Kuhn Nicolas <[email-masked]>
Date: Fri, 11 Dec 2015 14:20:39 +0000

Dear Polina, all,

Thank you for your email and I understand your concerns. I will provide more detail answer below for each of your points, but I would like first to clarify a general issue.

The scope of this document is to present how to assess the performance of AQMs - without focusing on a specific context (3G networks, data-centers, satellite networks, home routers, ISP core network) where Bufferbloat may (or may not) occur (yet). This is the main reason why this draft does not provide specific details on the bottleneck capacities, numbers of flows, or traffic generation. Every strict number or specific flow characteristic limits the scope of this document that is not only focusing on web traffic latency reduction in home routers.

Please see inline for specific answer to your concerns.

Cheers,

Nicolas

De : aqm [mailto:[email-masked]] De la part de Polina Goltsman
Envoyé : lundi 7 décembre 2015 13:32
À : Wesley Eddy; [email-masked]
Cc : Bless, Roland (TM)
Objet : Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

Dear all,

I have review the latest version of the draft. In my opinion the document still needs a detailed review about the representation (the text): the structure of the document would make more sense if the information in section 2, 3, and 4 would appear in different order, the document still could use a more consistent terminology, and the structure of section 5-12 could be more consistent. I am happy to provide the detailed review, but since it is a lot of work, I have several concerns about the content of the document, which I would prefer to clear first.

NK : we changed the ToC of the draft several times and I think the first sections (2, 3 and 4) are quite independent. I do not see a specific need for changing their order.
Regarding sections 5-12, we could still add some "Motivation" and "Recommended tests" for the section 5 for increasing the consistency. The sections that do not need performance evaluations have another organization (Motivation then Recommended discussion), but that seems consistent to me. We had several checks within the list for the terminology (such as the one done by Gorry), but if you spot anything, feel free to send your comments.

The document proposes the evaluation methodology for AQMs which consists of the following topology and scenarios:

0. the experiments should be performed on the topology, described in Figure 1 in Section 3, where the bottleneck between routers L and R is of unspecified capacity, which SHOULD include both symmetric and asymmetric, and the buffer MAY be set to a BDP. If I understand correctly, it is further RECOMMENDED to use a range of input parameters for the evaluated AQM (this may only be required if several AQMs are being compared)

NK : The Figure 1 in section 3 MAY be used. The dumbbell topology is useful to evaluate the performance of congestion control (such as mentioned in the common TCP evaluation suite of the IRTF).

NK: Such as for the different parameters (buffer sizing, bottleneck capacity, etc.), we understand that having unspecified parameters may be annoying, but I do not see how we can define such parameters for satellite networks, data-centers, home gateways, core Internet providers' network, etc. As one example, the capacity is (not only) related to the mechanisms from lower layers, such as link layer reliability schemes, channel access mechanisms, etc. which highly varies from one context to the other. Also, the buffer sizing depends on the BDP, the amount of traffic, the presence of PEP (such as in satellite gateways), thus I am afraid we can hardly define a buffer sizing that suits every use case. The amount of input parameters for the evaluated AQM that is to be looked at is the amount of parameters that are exhibited by the AQM developers (i.e. to what extent the AQM is a black box or not). I hardly see how we can have guidelines that specify number and parameterization that work for any use case scenario and deployment context.

1. (section 5) several experiments with different TCP congestion control schemes (+UDP) which should be performed on unspecified bottleneck capacity and unspecified RTT, which should or should not be the same as the one with which the AQM is configured.

NK: see previous comment.

2. (section 6) RTT fairness, where two groups of unspecified number of flows share a bottleneck of unspecified capacity, where the first group sees RTT of 100ms and the second is in range [5ms;560ms]

NK: see previous comment.

3. (section 7) burst absorption, is a group of four scenarios, which include web-traffic, bursty video frames, which should be generated in an unspecified manner (left to the tester to decide), CBR traffic of unspecified rate, and TCP flow, all of which is evaluated using a set of metrics that MAY be generated. I might be missing something, but I don't understand how any of these metrics can be used to  characterize burst absorption.

NK : see previous comment.

4. (section 8) stability, including (a) varying the number of flows on a bottleneck of unspecified capacity with unspecified RTT, according to the provided formulas, (b) varying network capacity between 10Mbps an 100Mbps for one TCP flow (if I understand it correctly)

NK: see previous comment.

5. (section 9) which includes (a) traffic mix of a combination applications, including TCP, web-traffic, bi-direction VoIP using unspecified congestion control (I am not sure, but I think there are options), CBR of unspecified rate, and adaptive video streaming also with unspecified congestion control, with one combination required and other left to the tester to decide; and (b) bi-directional traffic, which should evaluate the effect on dropping DNS/TCP Syn packets* using a specified number of bulk TCP flows using the the throughput-delay tradeoff graph; (in both scenarios capacities and RTT are unspecified as well)
* I assume that DNS/SYN packets are mentioned in this section by mistake
NK: see previous comment for the rates. The comment on DNS and TCP Syn packets have been introduced after Jim's review [https://mailarchive.ietf.org/arch/msg/aqm/pS9WlUlK5gExF8Ohu_syUXrNiUk ].
Do you think that a person without a substantial background knowledge on the evaluation of AQM schemes can perform this evaluation (resolve all unspecified conditions) in a reasonable amount of time?
NK: The unspecified parameters do not require specific knowledge on AQMs - the unspecified parameters are (1) the RTT or the capacities, which are related to the network underneath; (2) and the characteristics of the traffic generation, which is related to the traffic that is run between clients and servers. Thus, I think that a person without a substantial background knowledge on the evaluation of AQM schemes can perform this evaluation in a reasonable amount of time.
In the abstract, the document says that it describes characterization guidelines for an AQM proposal, to decide whether it should be adopted by the AQM WG. The WG currently has two AQMs (dropping/marking policy) in last call. Did someone evaluate these AQMs according to the specified guidelines?
Moreover, it seems to me that the WG is about to conclude. What exactly is the purpose of standardizing this document then ?

NK: I am not sure whether the WG is about to conclude and do not have such information. The purpose in standardizing this document as INFORMATIVE would be to provide guidelines for evaluating AQM for the specific issue of the bufferbloat. You may believe that the WG is about to conclude, this draft may be useful for other initiative in marking/dropping in routers.
Kind regards,
Nicolas

Regards,
Polina

I don't think Roland and Polina who had made last call comments had yet had the time to check that the changes in the last revision met what they were expecting, so I'd like to give them (and anyone else who has comments) a couple of weeks to check this revision out, and assuming there are no major issues or objections by around 12/15, will plan to end the working group last call, and send the document to the area director for publication.

Dear all,

Kind regards,

Nicolas KUHN

-----Message d'origine-----
De : aqm [mailto:[email-masked]] De la part de [email-masked]<mailto:[email-masked]>
Envoyé : vendredi 27 novembre 2015 11:44
À : [email-masked]<mailto:[email-masked]>
Cc : [email-masked]<mailto:[email-masked]>
Objet : [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

Abstract:

_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

---

From: Polina Goltsman <[email-masked]>
Date: Fri, 11 Dec 2015 17:59:55 +0100

Dear Nicolas,

Thank you very much for the explanation, I have however two questions 
regarding your comments:

1) why is the explanation about why are parameters unspecified not 
present in the document?

2) in this case, I assume that the remark about "asymmetric link 
scenario evaluates an AQM mechanism in a more realistic setup;" in 
section 3.1 and all cases where the parameters for capacity and 
round-trip time ARE present (see items 2 and 4 below), should be removed ?

I guess now I have to write a longer review...  :(

Regards,
Polina

On 12/11/2015 03:20 PM, Kuhn Nicolas wrote:
>
> Dear Polina, all,
>
> Thank you for your email and I understand your concerns. I will 
> provide more detail answer below for each of your points, but I would 
> like first to clarify a general issue.
>
> The scope of this document is to present how to assess the performance 
> of AQMs – without focusing on a specific context (3G networks, 
> data-centers, satellite networks, home routers, ISP core network) 
> where Bufferbloat may (or may not) occur (yet). This is the main 
> reason why this draft does not provide specific details on the 
> bottleneck capacities, numbers of flows, or traffic generation. Every 
> strict number or specific flow characteristic limits the scope of this 
> document that is not only focusing on web traffic latency reduction in 
> home routers.
>
> *De :*aqm [mailto:[email-masked]] *De la part de* Polina Goltsman
> *Envoyé :* lundi 7 décembre 2015 13:32
> *À :* Wesley Eddy; [email-masked]
> *Cc :* Bless, Roland (TM)
> *Objet :* Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt
>
> Dear all,
>
> information in section 2, 3, and 4 would appear in different order, 
> the document still could use a more consistent terminology, and the 
> several concerns about the content of the document, which I would 
> prefer to clear first.
>
> NK : we changed the ToC of the draft several times and I think the 
> first sections (2, 3 and 4) are quite independent. I do not see a 
> specific need for changing their order.
>
> Regarding sections 5-12, we could still add some “Motivation” and 
> “Recommended tests” for the section 5 for increasing the consistency. 
> The sections that do not need performance evaluations have another 
> organization (Motivation then Recommended discussion), but that seems 
> consistent to me. We had several checks within the list for the 
> terminology (such as the one done by Gorry), but if you spot anything, 
> feel free to send your comments.
>
>
> The document proposes the evaluation methodology for AQMs which 
> consists of the following topology and scenarios:
>
> Figure 1 in Section 3, where the bottleneck between routers L and R is 
> of *unspecified* capacity, which SHOULD include*both symmetric and 
> asymmetric*, and the buffer MAY be set to a BDP. If I understand 
> parameters for the evaluated AQM (this may only be required if several 
> AQMs are being compared)
>
> NK : The Figure 1 in section 3 MAY be used. The dumbbell topology is 
> useful to evaluate the performance of congestion control (such as 
> mentioned in the common TCP evaluation suite of the IRTF).
>
> NK: Such as for the different parameters (buffer sizing, bottleneck 
> capacity, etc.), we understand that having unspecified parameters may 
> be annoying, but I do not see how we can define such parameters for 
> satellite networks, data-centers, home gateways, core Internet 
> providers’ network, etc. As one example, the capacity is (not only) 
> related to the mechanisms from lower layers, such as link layer 
> reliability schemes, channel access mechanisms, etc. which highly 
> varies from one context to the other. Also, the buffer sizing depends 
> on the BDP, the amount of traffic, the presence of PEP (such as in 
> satellite gateways), thus I am afraid we can hardly define a buffer 
> sizing that suits every use case. The amount of input parameters for 
> the evaluated AQM that is to be looked at is the amount of parameters 
> that are exhibited by the AQM developers (i.e. to what extent the AQM 
> is a black box or not). I hardly see how we can have guidelines that 
> specify number and parameterization that work for any use case 
> scenario and deployment context.
>
>
> bottleneck capacity and *unspecified* RTT, which should or should not 
> be the same as the one with which the AQM is configured.
>
> 2. (section 6) RTT fairness, where two groups of *unspecified* number 
> of flows share a bottleneck of *unspecified* capacity, where the first 
> group sees RTT of 100ms and the second is in range [5ms;560ms]
>
> include web-traffic, bursty video frames, which should be generated in 
> an unspecified manner (*left to the tester to decide*), CBR traffic of 
> *unspecified* rate, and TCP flow, all of which is evaluated using a 
> set of metrics that MAY be generated. I might be missing something, 
> but I don't understand how any of these metrics can be used to  
> characterize burst absorption.
>
> 4. (section 8) stability, including (a) varying the number of flows on 
> a bottleneck of *unspecified* capacity with *unspecified* RTT, 
> between 10Mbps an 100Mbps for one TCP flow (if I understand it correctly)
>
> *unspecified congestion* control (I am not sure, but I think there are 
> options), CBR of *unspecified* rate, and adaptive video streaming also 
> with *unspecified* congestion control, with one combination required 
> and other left to the tester to decide; and (b) bi-directional 
> traffic, which should evaluate the effect on dropping DNS/TCP Syn 
> packets* using a specified number of bulk TCP flows using the the 
> throughput-delay tradeoff graph; (in both scenarios capacities and RTT 
> are *unspecified* as well)
>
> NK: see previous comment for the rates. The comment on DNS and TCP Syn 
> packets have been introduced after Jim’s review 
> [https://mailarchive.ietf.org/arch/msg/aqm/pS9WlUlK5gExF8Ohu_syUXrNiUk ].
>
> Do you think that a person without a substantial background knowledge 
> on the evaluation of AQM schemes can perform this evaluation (resolve 
> all unspecified conditions) in a reasonable amount of time?
>
> NK: The unspecified parameters do not require specific knowledge on 
> AQMs – the unspecified parameters are (1) the RTT or the capacities, 
> which are related to the network underneath; (2) and the 
> characteristics of the traffic generation, which is related to the 
> traffic that is run between clients and servers. Thus, I think that a 
> person without a substantial background knowledge on the evaluation of 
> AQM schemes can perform this evaluation in a reasonable amount of time.
>
> in last call. Did someone evaluate these AQMs according to the 
> specified guidelines?
>
> Moreover, it seems to me that the WG is about to conclude. What 
> exactly is the purpose of standardizing this document then ?
>
> NK: I am not sure whether the WG is about to conclude and do not have 
> such information. The purpose in standardizing this document as 
> INFORMATIVE would be to provide guidelines for evaluating AQM for the 
> specific issue of the bufferbloat. You may believe that the WG is 
> about to conclude, this draft may be useful for other initiative in 
> marking/dropping in routers.
>
> Kind regards,
>
> Nicolas
>
>
> Regards,
> Polina
>
>     I don't think Roland and Polina who had made last call comments
>     had yet had the time to check that the changes in the last
>     revision met what they were expecting, so I'd like to give them
>     (and anyone else who has comments) a couple of weeks to check this
>     revision out, and assuming there are no major issues or objections
>     by around 12/15, will plan to end the working group last call, and
>     send the document to the area director for publication.
>
>
>
>     - modification on the buffer sizes, following some discussion
>     points raised by Michael Scharf on the tcpm mailing list [1];
>     De : aqm [mailto:[email-masked]] De la part de
>     [email-masked] <mailto:[email-masked]>
>     À : [email-masked] <mailto:[email-masked]>
>     Cc : [email-masked] <mailto:[email-masked]>
>       This draft is a work item of the Active Queue Management and
>     Packet Scheduling Working Group of the IETF.
>
>     [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Kuhn Nicolas <[email-masked]>
Date: Mon, 14 Dec 2015 09:09:03 +0000

Dear Polina, all,

1)      The explanation was mostly on why parameters are not specified in the document because the words unspecified was in a bold font in your email ; this made me think that this was your concern about the document.

2)      I think it is not an issue to have some parameters specified in the document.

You do not "have to" write a longer review. Do it  only if you think it is necessary and helpful in having a better document :).

As Gorry said at IETF93, "this doc will never be perfect"[1].

Kind regards,

Nicolas

[1] https://www.ietf.org/proceedings/93/minutes/minutes-93-aqm

De : Polina Goltsman [mailto:[email-masked]]
Envoyé : vendredi 11 décembre 2015 18:00
À : Kuhn Nicolas; Wesley Eddy; [email-masked]
Cc : Bless, Roland (TM)
Objet : Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

Dear Nicolas,

Thank you very much for the explanation, I have however two questions regarding your comments:

1) why is the explanation about why are parameters unspecified not present in the document?

2) in this case, I assume that the remark about "asymmetric link scenario evaluates an AQM mechanism in a more realistic setup;" in section 3.1 and all cases where the parameters for capacity and round-trip time ARE present (see items 2 and 4 below), should be removed ?

Regards,
Polina

Cheers,

Nicolas

De : aqm [mailto:[email-masked]] De la part de Polina Goltsman
Envoyé : lundi 7 décembre 2015 13:32
À : Wesley Eddy; [email-masked]<mailto:[email-masked]>
Cc : Bless, Roland (TM)
Objet : Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

Dear all,

Regards,
Polina

Dear all,

Kind regards,

Nicolas KUHN

-----Message d'origine-----
De : aqm [mailto:[email-masked]] De la part de [email-masked]<mailto:[email-masked]>
Envoyé : vendredi 27 novembre 2015 11:44
À : [email-masked]<mailto:[email-masked]>
Cc : [email-masked]<mailto:[email-masked]>
Objet : [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

Abstract: