Subject: [bmwg] first WGLC on draft-ietf-bmwg-traffic

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 23 Sep 2014 08:55:44 -0400

BMWG (and AQM):

A WG Last Call period for the Internet-Draft on
Traffic Management Benchmarking:

   http://tools.ietf.org/html/draft-ietf-bmwg-traffic-management

will be open from 23 September 2014 through October 21 2014.

These drafts are beginning the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html

which says:
...
The primary change is a requirement that approximately four WG members
complete a review template for the draft (or set of drafts),
upon entry to the Last Call Process. The WG reviewers should be
outside the body of active contributors (editors + authors).
The goal is to produce a quality review with valuable feedback
on the draft(s), and not to set a minimum number of people willing
to complete the review template. Directed feedback,
regardless of quantity, is a better condition by which BMWG can
more clearly assess a draft's readiness to advance.
(the template is appended to the archived message)

Please read and express your opinion on whether or not these
Internet-Drafts should be forwarded to the Area Directors for
publication as Informational RFCs.  Send your comments
to this list or [email-masked] and [email-masked]

Al
bmwg co-chair

---

From: [email-masked]
Date: Thu, 25 Sep 2014 08:33:23 +0100

I have a few comments from a quick pass through this document. I think it
is useful to define such tests, and I suspect those performing the tests
may understand the wider context of testing a specific mechanism, but I
have a concern that the tests would produce strange results if deployed
with sophisticated traffic management, and this is not really discussed. I
think this needs to be addressed, and if necessary the cope of the tests
clearly stated.

I am worried that while the tests may help characterise a single
shaper/policer, there is a need to extend the document at least to comment
on more complex interactions that can occur.

QoS Frameworks define policies (e.g. DS) that can map DSCPs (or MF
classified traffic) to treatments, possibly aggregating behaviours for
different traffic aggregates. Treatments are not independent - traffic
with one marking can and does traffic with a different marking (well its
intended to). Testing multiple DSCPs in DS at the same time is more
complex, it should not be assumed that each of these can be characterised
separately.

AQM seeks to manage buffering, and this reaction is likely to be triggered
by test sequences. When triggered, the response (loss, delay) can impact
the flow’s treatment long after the traffic burst that triggered AQM in
the first place.

Many AQM methods are also used in combination with scheduling, again this
can be and often is non-trivial and depends on the number of flows and
sometimes the hashing of the flows to queues, etc within the scheduling
framework.

Other specific comments follow.

Gorry

---

Section 1:
Introduction needs to define terms when first used.
- would be good to define VLAN and DSCP

/sent onto to the next network/sent to the next network/
- is it also possible to say “forwarded” rather than “sent”?

Active Queue Management (AQM):
- would be good to cite:
https://datatracker.ietf.org/doc/draft-ietf-aqm-recommendation/

Please define MEF

/of it's egress ports/of its egress ports/

/The Network Delay Emulator (NDE) is a requirement for the TCP
   stateful tests/
- probably true of any control-loop based congestion control (i.e. TCP is
an example, SCTP, DCCP and other congestion controls layered on UDP would
also need this (including some uses of RTP).

Please define BDPs

Section 3:

/Another goal is to devise methods that utilize flows with
   congestion-aware transport (TCP) /
- I think TCP is an example here, and needs an e.g., to avoid it seeming
like this is the only test case. Note also this comment applies later in
the same para.

When it comes to an active technique, the concept of Burst Size Achieved
(BSA)  and Lost packets (LP) is less clear, since won’t this also depend
on the recent history of the flow traffic patterns and could be
significantly influenced by scheduling mechanisms between flows (i.e.
whether the flow is hashed to a separate queue, whether the flow is
classified as “new” or “old” and the total number of flows being
serviced). Performance could be quite different for a single flow test.

With Active management, Packet Delay (PD) also becomes a function of traffic.

Section 4.2 - I think the document should mention the need to test for
other non TCP congestion-controlled traffic, although I could see that for
simplicity the tests are defined only for TCP.

5.1.1 Burst Hunt with Stateless Traffic
- A cautionary note here could be helpful, that advises that if AQM is
used, the bursts must be sufficiently separated in time that they do not
cumulatively trigger a response - since AQM algorithms can have history,
treatment of a subsequent burst can be influenced by an earlier burst.

5.2.1. and 6.4:
- could note that multiple flows may also be queued and scheduled
separately within the device.

Informative References
-missing

> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>

---

From: ramki Krishnan <[email-masked]>
Date: Thu, 25 Sep 2014 10:02:51 -0700

Hi Gorry,

Many thanks for your detailed comments.

The goal of this draft is to describe the commonly used traffic management techniques in switches and routers. AQM (RED, WRED etc.) is an advanced traffic management technique which we plan on  addressing in a separate draft. Tail drop is the commonly used technique in switches and routers for addressing congestion control. Please see references to Cisco and Brocade configuration guides - (http://www.cisco.com/c/en/us/td/docs/routers/10000/10008/configuration/guides/qos/qoscf/10qqueue.html#wp1041728; http://www.brocade.com/downloads/documents/product_manuals/B_NetIron/NetIronUnified_05400a_ConfigGuide.pdf).

BTW, David asked us at the last IETF meeting about AQM and the above response was well received by the working group.

We will address all your non AQM comments in the next version of the draft.

Thanks,
Ramki

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Thursday, September 25, 2014 12:33 AM
To: [email-masked]; MORTON, ALFRED C (AL)
Cc: [email-masked]
Subject: Re: [bmwg] [aqm] first WGLC on draft-ietf-bmwg-traffic

I have a few comments from a quick pass through this document. I think it is useful to define such tests, and I suspect those performing the tests may understand the wider context of testing a specific mechanism, but I have a concern that the tests would produce strange results if deployed with sophisticated traffic management, and this is not really discussed. I think this needs to be addressed, and if necessary the cope of the tests clearly stated.

I am worried that while the tests may help characterise a single shaper/policer, there is a need to extend the document at least to comment on more complex interactions that can occur.

QoS Frameworks define policies (e.g. DS) that can map DSCPs (or MF classified traffic) to treatments, possibly aggregating behaviours for different traffic aggregates. Treatments are not independent - traffic with one marking can and does traffic with a different marking (well its intended to). Testing multiple DSCPs in DS at the same time is more complex, it should not be assumed that each of these can be characterised separately.

AQM seeks to manage buffering, and this reaction is likely to be triggered by test sequences. When triggered, the response (loss, delay) can impact the flow's treatment long after the traffic burst that triggered AQM in the first place.

Many AQM methods are also used in combination with scheduling, again this can be and often is non-trivial and depends on the number of flows and sometimes the hashing of the flows to queues, etc within the scheduling framework.

Gorry

---

Section 1:

/sent onto to the next network/sent to the next network/
- is it also possible to say "forwarded" rather than "sent"?

Please define MEF

/The Network Delay Emulator (NDE) is a requirement for the TCP
   stateful tests/
- probably true of any control-loop based congestion control (i.e. TCP is an example, SCTP, DCCP and other congestion controls layered on UDP would also need this (including some uses of RTP).

Please define BDPs

Section 3:

/Another goal is to devise methods that utilize flows with
   congestion-aware transport (TCP) /
- I think TCP is an example here, and needs an e.g., to avoid it seeming like this is the only test case. Note also this comment applies later in the same para.

When it comes to an active technique, the concept of Burst Size Achieved
(BSA)  and Lost packets (LP) is less clear, since won't this also depend on the recent history of the flow traffic patterns and could be significantly influenced by scheduling mechanisms between flows (i.e.
whether the flow is hashed to a separate queue, whether the flow is classified as "new" or "old" and the total number of flows being serviced). Performance could be quite different for a single flow test.

Section 4.2 - I think the document should mention the need to test for other non TCP congestion-controlled traffic, although I could see that for simplicity the tests are defined only for TCP.

5.1.1 Burst Hunt with Stateless Traffic
- A cautionary note here could be helpful, that advises that if AQM is used, the bursts must be sufficiently separated in time that they do not cumulatively trigger a response - since AQM algorithms can have history, treatment of a subsequent burst can be influenced by an earlier burst.

5.2.1. and 6.4:
- could note that multiple flows may also be queued and scheduled separately within the device.

Informative References
-missing

> BMWG (and AQM):
>
> A WG Last Call period for the Internet-Draft on Traffic Management 
> Benchmarking:
>
> complete a review template for the draft (or set of drafts), upon 
> entry to the Last Call Process. The WG reviewers should be outside the 
> body of active contributors (editors + authors).
> The goal is to produce a quality review with valuable feedback on the 
> draft(s), and not to set a minimum number of people willing to 
> complete the review template. Directed feedback, regardless of 
> quantity, is a better condition by which BMWG can more clearly assess 
> a draft's readiness to advance.
> publication as Informational RFCs.  Send your comments to this list or 
> [email-masked] and [email-masked]
>
> Al
> bmwg co-chair
>

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: [email-masked]
Date: Fri, 26 Sep 2014 09:20:42 +0100

I think that it is a good plan to expect a separate draft on more advanced
techniques.

However,  AQM has been deployed in places (and likely more so), so what I
was hoping was that this particular draft could scope itself carefully to
state the constraints of the tests. I don't think it is wise simply to
ignore the AQM-related issues, although this scoped could well be
addressed in the start and end of the document.

I agree the details of AQM tests and QoS-based tests are likely to be more
complicated and best declared out of scope.

Gorry

> The goal of this draft is to describe the commonly used traffic management
> techniques in switches and routers. AQM (RED, WRED etc.) is an advanced
> traffic management technique which we plan on  addressing in a separate
> draft. Tail drop is the commonly used technique in switches and routers
> for addressing congestion control. Please see references to Cisco and
> Brocade configuration guides -
> (http://www.cisco.com/c/en/us/td/docs/routers/10000/10008/configuration/guides/qos/qoscf/10qqueue.html#wp1041728;
> http://www.brocade.com/downloads/documents/product_manuals/B_NetIron/NetIronUnified_05400a_ConfigGuide.pdf).
>
> BTW, David asked us at the last IETF meeting about AQM and the above
> response was well received by the working group.
>
> the flow's treatment long after the traffic burst that triggered AQM in
> the first place.
>
> (BSA)  and Lost packets (LP) is less clear, since won't this also depend
> classified as "new" or "old" and the total number of flows being

---

From: Barry Constantine <[email-masked]>
Date: Fri, 26 Sep 2014 11:50:12 +0000

Hi Gorry,

We will add the scoping language that you requested and also incorporate your core document comments.

Thank you for taking the time to review this work.

Barry Constantine

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Friday, September 26, 2014 4:21 AM
To: ramki Krishnan
Cc: [email-masked]; MORTON, ALFRED C (AL); [email-masked]; [email-masked]
Subject: Re: [bmwg] [aqm] first WGLC on draft-ietf-bmwg-traffic

I think that it is a good plan to expect a separate draft on more advanced techniques.

However,  AQM has been deployed in places (and likely more so), so what I was hoping was that this particular draft could scope itself carefully to state the constraints of the tests. I don't think it is wise simply to ignore the AQM-related issues, although this scoped could well be addressed in the start and end of the document.

I agree the details of AQM tests and QoS-based tests are likely to be more complicated and best declared out of scope.

Gorry

> The goal of this draft is to describe the commonly used traffic 
> management techniques in switches and routers. AQM (RED, WRED etc.) is 
> an advanced traffic management technique which we plan on  addressing 
> in a separate draft. Tail drop is the commonly used technique in 
> switches and routers for addressing congestion control. Please see 
> references to Cisco and Brocade configuration guides - 
> (http://www.cisco.com/c/en/us/td/docs/routers/10000/10008/configuratio
> n/guides/qos/qoscf/10qqueue.html#wp1041728;
> I have a few comments from a quick pass through this document. I think 
> it is useful to define such tests, and I suspect those performing the 
> tests may understand the wider context of testing a specific 
> mechanism, but I have a concern that the tests would produce strange 
> results if deployed with sophisticated traffic management, and this is 
> not really discussed. I think this needs to be addressed, and if 
> necessary the cope of the tests clearly stated.
>
> shaper/policer, there is a need to extend the document at least to 
> comment on more complex interactions that can occur.
>
> with one marking can and does traffic with a different marking (well 
> its intended to). Testing multiple DSCPs in DS at the same time is 
> more complex, it should not be assumed that each of these can be 
> characterised separately.
>
> AQM seeks to manage buffering, and this reaction is likely to be 
> triggered by test sequences. When triggered, the response (loss, 
> delay) can impact the flow's treatment long after the traffic burst 
> that triggered AQM in the first place.
>
> Many AQM methods are also used in combination with scheduling, again 
> this can be and often is non-trivial and depends on the number of 
> flows and sometimes the hashing of the flows to queues, etc within the 
> scheduling framework.
>
> - probably true of any control-loop based congestion control (i.e. TCP 
> is an example, SCTP, DCCP and other congestion controls layered on UDP 
> would also need this (including some uses of RTP).
>
> Please define BDPs
>
> Section 3:
>
> - I think TCP is an example here, and needs an e.g., to avoid it 
> seeming like this is the only test case. Note also this comment 
> applies later in the same para.
>
> (BSA)  and Lost packets (LP) is less clear, since won't this also 
> depend on the recent history of the flow traffic patterns and could be 
> other non TCP congestion-controlled traffic, although I could see that 
> for simplicity the tests are defined only for TCP.
>
> used, the bursts must be sufficiently separated in time that they do 
> not cumulatively trigger a response - since AQM algorithms can have 
> history, treatment of a subsequent burst can be influenced by an earlier burst.
>
> 5.2.1. and 6.4:
>> The primary change is a requirement that approximately four WG 
>> members complete a review template for the draft (or set of drafts), 
>> publication as Informational RFCs.  Send your comments to this list 
>> or [email-masked] and [email-masked]
>>
>> Al
>> bmwg co-chair
>>

---

From: Dave Taht <[email-masked]>
Date: Fri, 26 Sep 2014 06:46:48 -0700

> BTW, David asked us at the last IETF meeting about AQM and the above response was well received by the working group.

I was delighted to see this new wg attempting to tackle more complex
scenarios for characterizing switch and router behavior, up to, and
*almost* including fq and aqm techniques. A lot of the test ideas are
still relevant at that level, and I'd hoped some of it would feed
forward into the aqm evaluation guidelines.

And while I do hope to one day live in a world where fq and aqm
technologies are deployed by default,  my principal hope from the bmwg
is that the latency gets measured and reported everywhere appropriate
and weighted more heavily than loss.

I also note that the lightweight techniques netperf-wrapper uses, and
the common data result format, could be extended to cover many of the
tests discussed there so far, and it would be a boon to be able to
gather and share detailed metrics rather than single point ones.
netperf-wrapper has now been tested successfully up to 40GigE, and
some folk are now using it to analyze the effects of shaving
nanoseconds off of host generated traffic at the lowest layer
interaction between the tx ring and qdisc...

https://patchwork.ozlabs.org/patch/393034/

So I wouldn't mind a mention of netperf, d-itg, owamp, in the draft.
netperf is used, rather than iperf, in the linux kernel networking
community, because A) it's trusted at high rates, and B) the author
responds to email.

-- 
Dave Täht

https://www.bufferbloat.net/projects/make-wifi-fast

---

From: Barry Constantine <[email-masked]>
Date: Fri, 26 Sep 2014 22:23:10 +0000

Hi David,

Sounds good, we'll include references to netperf-wrapper in the next version.

We will also review the common data result format you mentioned and see how this might fit into the draft.

Thanks again for your interest, ideas, and support.

Thank you,
Barry Constantine

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of Dave Taht
Sent: Friday, September 26, 2014 9:47 AM
To: ramki Krishnan
Cc: [email-masked]; MORTON, ALFRED C (AL); [email-masked]; [email-masked]
Subject: Re: [bmwg] [aqm] first WGLC on draft-ietf-bmwg-traffic

> BTW, David asked us at the last IETF meeting about AQM and the above response was well received by the working group.

I was delighted to see this new wg attempting to tackle more complex scenarios for characterizing switch and router behavior, up to, and
*almost* including fq and aqm techniques. A lot of the test ideas are still relevant at that level, and I'd hoped some of it would feed forward into the aqm evaluation guidelines.

And while I do hope to one day live in a world where fq and aqm technologies are deployed by default,  my principal hope from the bmwg is that the latency gets measured and reported everywhere appropriate and weighted more heavily than loss.

I also note that the lightweight techniques netperf-wrapper uses, and the common data result format, could be extended to cover many of the tests discussed there so far, and it would be a boon to be able to gather and share detailed metrics rather than single point ones.
netperf-wrapper has now been tested successfully up to 40GigE, and some folk are now using it to analyze the effects of shaving nanoseconds off of host generated traffic at the lowest layer interaction between the tx ring and qdisc...

So I wouldn't mind a mention of netperf, d-itg, owamp, in the draft.
netperf is used, rather than iperf, in the linux kernel networking community, because A) it's trusted at high rates, and B) the author responds to email.

--
Dave Täht

---

From: Toke Høiland-Jørgensen <[email-masked]>
Date: Sat, 27 Sep 2014 10:19:12 +0200

> see how this might fit into the draft.

Hi

Author of netperf-wrapper chiming in: If you have any questions let me
know. I'd also be happy to work with you to specify the data format
and/or make necessary adjustments in the interest of having a common
format to store and exchange test results.

-Toke