Subject: Re: [bmwg] draft-constantine-bmwg-traffic-management-00

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Sun, 3 Mar 2013 12:21:42 -0500

I'll add a few comments on this draft, below.
as a participant, with one exception indicated below,
Al

=============================

1. Intro

Comment: there is a tendency to over-use the term "QoS" in our
industry to mean many different things, and the Intro falls
into that trap. There is "network QoS" which is the end-to-end
performance that the service/path achieves, and then there are
"QoS mechanisms" and traffic management tools are part of that
aspect. In short, just saying QoS is ambiguous unless the context
is clear, and it's always better to be more specific in a standard.
So, substitute a specific term for "QoS" wherever possible.

Also, since were are only testing a single device, not an entire
service, I suggest to avoid the term SLA and refer to "limits"
instead.

I suggest, in the first sentence:
OLD
   Traffic management (i.e. policing, shaping, etc.) is an increasingly
   important component in today's networks.
NEW

   Traffic management (i.e. policing, shaping, etc.) is an increasingly
   important component when engineering network Quality of Service (QoS) today.

and then

OLD
   The benchmarking framework can also be used as a test procedure to
   assist in the tuning of Quality of Service (QoS) parameters before
   field deployment.
NEW
   The benchmarking framework can also be used as a test procedure to
   assist in the tuning of traffic management parameters before
   field deployment.

later, in 1.1:
OLD
   In general, a device with traffic management capabilities performs
   the following QoS functions:
NEW
   In general, a device with traffic management capabilities performs
   the following functions:

OLD
. Traffic classification: identifies traffic according to various QoS
NEW
. Traffic classification: identifies traffic according to various configuration

OLD
  . Traffic policing: rate limits traffic that enters a router
     according to the traffic classification.  If the traffic exceeds
     the contracted Service Level Agreement (SLA), the traffic is either
     dropped or remarked and sent onto to the next network node
NEW
  . Traffic policing: limits the rate of traffic that enters a router
     according to the traffic classification.  If the traffic exceeds
     the contracted limits, the traffic is either
     dropped or remarked and sent onto to the next network node.

OLD
  . Traffic shaping: is a traffic control measure of actively buffering
     and metering the output rate of traffic in an attempt to adapt
     bursty traffic to the SLA.
NEW
  . Traffic shaping: a traffic control measure of actively buffering
     and metering the output rate of traffic in an attempt to adapt
     bursty traffic to the configured limits.

OLD
  .  Traffic Scheduling: provides QoS within the network device by
     storing packets in various types of queues and applies a
     dispatching algorithm to assign the forwarding sequence of packets.
NEW
  .  Traffic Scheduling: provides traffic classification within the network device by
     directing packets to various types of queues and applies a
     dispatching algorithm to assign the forwarding sequence of packets.

OLD
  . Congestion Management: monitors the status of internal queues and
     actively drops packets, which causes the sending hosts to back-off
     and in turn can alleviate queue congestion.
NEW
  . Active Queue Management (AQM): monitors the status of internal queues and
     actively drops packets, which causes the sending hosts to back-off
     and in turn can alleviate queue congestion.
(since you refer to AQM later)

Figure 1 refers to "QoS mechanisms", or you could remove "QoS" everywhere.

Section 2 Conventions:

I agree with Gilles,
s/Exceeded/Excess/
in EBS and EIR, these are the more conventional terms.

I don't think the lab set-up belongs in the section on Conventions,
it should probably have its own section.

Also:

    +--------------+     +-------+     +----------+    +-----------+
    | Transmitting |     |       |     |          |    | Receiving |
    | Test Host    |     |       |     |          |    | Test Host |
    |              |-----| DUT   |---->| Network  |--->|           |
    |              |     |       |     | Delay    |    |           |
    |              |     |       |     | Emulator |    |           |
    |              |<----|       |<----|          |<---|           |
    |              |     |       |     |          |    |           |
    +--------------+     +-------+     +----------+    +-----------+

I think that, for a more thorough device characterization, we should
be thinking in terms of testing multiple ports, interface cards, etc.
simultaneously on the DUT. Traffic management feature performance could be
affected by total offered load, for the same reasons that you are concerned
with NDE performance under load, when you say:

   Also note that the Network Delay Emulator (NDE) should be passive in
   nature such as a fiber spool.  This is recommended to eliminate the
   potential effects that an active delay element (i.e. test impairment
   generator) may have on the test flows. In the case that a fiber spool
   is not practical due to the desired latency, an active NDE must be
   independently verified to be capable of adding the configured delay
   without loss.  This requirement will vary from test to test on
   desired traffic speed and should be calibrated before any test
   requiring delay, which can add a significant additional amount of
   testing to each step.

You could probably have some baseline testing that removes the DUT from the
setup above, in order to characterize an active/software NDE.

Also, later in section 3, you say:

   Within this framework, the metrics are defined for each traffic
   management test but do not include pass / fail criterion, which is
   not within the charter of BMWG.  This framework does not attempt to
   rate the performance of one manufacturer's network equipment versus
   another, but only to provide benchmarks to conduct repeatable,
   comparative testing.

<chair_hat>

I discussed this with Barry last year, the focus in BMWG is on black-box
comparison of benchmarks across different vendor's equipment. I think there
may be ways to augment what you proposed here to include such tests, for example
to compare the total capacity of each DUT in terms of the number of CBR traffic
flows it can police while maintaining the specified/configured limits.
Inherently, that testing includes the "functional" test that you are seeking
now (If I configure 64kbps CBR, does the DUT enforce/achieve that limit).

I suspect this sort of benchmarking is interesting for the same reasons
you are concerned with an active NDE keeping up with the load.

</chair_hat>

Section 4. Traffic Benchmarking Metrics

   The metrics to be measured during the benchmarks are divided into two
   (2) sections: packet layer metrics used for the stateless traffic
   testing and metrics used for the stateful traffic testing

I noted that the "stateless" traffic requires inclusion of sequence numbers,
but this wasn't flagged later in the Test Equipment capabilities section.

Section 4.1
   - Burst Size Achieved (BSA): for the traffic policing and network
   queue tests, the tester will be configured to send bursts to test
   either the Committed Burst Size (CBS) or Exceeded Burst Size (EBS) of

>>> need references for CBS and EBS

   a policer or the queue / buffer size configured in the DUT.  The
   Burst Size Achieved metric is a measure of the actual burst size
   received at the egress port of the DUT with no lost frames.  As an
   example, the
>>> insert "configured"
               CBS of a DUT is 64KB and after the burst test, only a 63
   KB can be achieved without frame loss.  Then 63KB is the BSA.

   - Lost Frames (LF): For all traffic management tests, the tester will
   transmit the test frames into the DUT ingress port and the number of
   frames received at the egress port will be measured.  The difference
   between frames transmitted into the ingress port and received at the
   egress port is the number of lost frames as measured at the egress
   port.  These frames must have unique identifiers such that only the
   test frames are measured.

How long will you wait for missing Frames to arrive?  important aspect of the metric.

   - Out of Sequence Frames (OOS): in additions to LF metric, the test
   frames must be monitored for sequence and the out-of-sequence (OOS)
   frames will be counted per RFC-???? or is this ITU??.

That's where the req. for seq numbers comes from. It's RFC 4737.

Regarding FD and FDV:
   - Frame Delay (FD): the Frame Delay metric is the difference between
   the timestamp of the received egress port frames and the frames
   transmitted into the ingress port and specified in ITU-1564.

   - Frame Delay Variation (FDV): the Frame Delay Variation metric is
   the variation between the timestamp of the received egress port
   frames and specified in ITU-1564.
It's fine to use ITU-T Recs for reference, but isn't there a BMWG definition
that will suit the need? RFC 2285 may be helpful in some areas...

In 4.2. Metrics for Stateful Traffic Tests
How do you define RTT?  Time from Send Segment to ACK for that Segment?
what about delayed ACKs?
What is "Baseline RTT"?

In 5.2
    The TCP implementation used by the test host must be specified in the
    test results (i.e. OS version, i.e. LINUX OS kernel using TCP New Reno,
    TCP options supported, etc).
There may be other, more repeatable alternatives to using a full TCP
implementation of a particular type, at least it's something to discuss
in-person.

In
6.1. Policing Tests

   The intent of the policing tests is to verify the policer performance
   parameters of CIR-CBS and EIR-EBS. The tests will verify that the
   device can handle the CIR rate with CBS and the EIR rate with EBS and
   will use back-back frame testing concepts from RFC 2544 (but adapted
   to burst size algorithms and terminology).  Also MEF-14,19,37 provide
   some basis for specific components of this test.

I think Scott Bradner mentioned difficulties with the 2544 back-to-back
tests during the preliminary presentation in November (repeatability).

>>>>>>> that may be enough to deal with in the first draft

---

From: Barry Constantine <[email-masked]>
Date: Mon, 4 Mar 2013 17:01:24 +0000

Al and Gilles,

Thank you for the detailed comments, they are all really good and will be incorporated in the next round.

For Al, the Section 3 comment.  It would be good to discuss the functional test aspects of a benchmark test versus a pure benchmark test; I want to make sure that this is 100% clear and we can discuss in Orlando.

Thanks again,
Barry

From: [email-masked] [mailto:[email-masked]] On Behalf Of MORTON JR., ALFRED (AL)
Sent: Sunday, March 03, 2013 12:22 PM
To: Gilles Forget; [email-masked]
Subject: Re: [bmwg] draft-constantine-bmwg-traffic-management-00

1. Intro

and then

OLD

later, in 1.1:
OLD

OLD

OLD

OLD

OLD

OLD

Section 2 Conventions:

I agree with Gilles,
s/Exceeded/Excess/

Also:

<chair_hat>

</chair_hat>

Section 4.1

Regarding FD and FDV:

In 5.2

In
6.1. Policing Tests

---

From: Shishio Tsuchiya <[email-masked]>
Date: Wed, 13 Mar 2013 14:25:07 -0400

BMWG
I also support this draft.
As I commented on the meeting,Tranparents LAN service is very popular in Japan.
Service provider are just providing Ethernet interface to customer,but service speed are variable.
1)another media used in the line:

|CPE|--ethernet--|LAN TA|----DSL,T1,ATM----|LAN TA|---|Switch|

Interface and line speed differed,and LAN TA does not enough buffer to protect user traffic,so CPE are using shaping and queueing.

2)sub-rate ethernet:
|CPE|---ethernet---|Switch|
media: 10Mbps service speed:1Mbps-10Mbps(step by 1Mbps)
     :100Mbps service speed:20Mbps-100Mbps(step by 10Mbps)
     :  1Gbps service speed:200Mbps-1Gbps(step by 100Mbps)
Service provider's switch is executing policing according to service speed.
Therefore CPE shaping and queueing to protect user traffic.

Enterprise customer are testing shaping accuracy,queing effectivity and CPE cpu utilizations.
But they are testing it by own method,because the standard method is not defined yet.
Service provider doing policing but not shows detail information of parameters.
I don't know whether another countries also have same situation as Japan.

But if the draft could define test method and parameters such as acceptable value,I think it would be more useful for network operators.

Regards,
-Shishio

(2013/02/19 20:53), Gilles Forget wrote:
> Hello BMWG,
> 
> I support this work as a network specialist who has many times tested traffic management capabilities in both lab and field situations.
> 
> I think this -00 revision is a good start and here are my first comments:
> 
> 1. Throughout the document, the words “network device” is referenced and other times “network node” is used.
>      Comment: In my opinion, "network device" seems more representative and should be consistently used.
> 
> 2. Section 1.1. “Traffic policing: rate limits traffic that enters a router a…”
>      Comment: It could also be a switch or other network appliances, so I suggest replacing the word "router" by “network device".
> 
> 3. In section 2, EIR and EBS should use the word “Excess” instead of “Exceeded” in their definitions.
> 
> 4. Section 5.1  “Since iperf is a software based tool, there will be performance limitations at higher link speeds".
>      Comment: I would like to suggest replacing by “Since iperf is software based, there will be performance limitations at higher speeds (e.g. 1G, 10G, etc…)”
> 
> 5. Section 5.5, “This interactive traffic is not uni-directional in nature but is chatty.
>      Comment: I would suggest, “These traffic are bi-directional and can be chatty at times.”
> 
> 6. Section 6.1.  “Policing tests will only use stateless traffic since a policer only operates at Layer 2 .  Stateful TCP test traffic would not yield any benefit to test a policer.”
>      Comment:  I’m not sure about this, I think a policer can operate at the packet layer also. And I can assure you that TCP traffic is affected by a policer.
> 
> 7. In section 6.1, Tc and Te are defined, in my opinion this should be done in section 2 instead.
> 
> 8. In sections 6.2.1, 6.2.2, and 6.3, QL, Ti, BB, BSA, NDE,  SR, SBS are defined, in my opinion this should be done in section 2 instead.
> 
> 9. Section 6.3.2 “By cumulative TCP window, this equates to:”
>      Comment: I would like to suggest, “The cumulative TCP Window Sizes* (RWND at the receiving end & CWND at the transmitting end) equates to:”
> 
> 10. Section 6.3.2 “* TCP window size is used per RFC 6349 and is the minimum of the TCP WIN and the Send Socket Buffer (SSB )”
>        Comment: I would like to suggest. “* as described in section 3 of RFC6349, the SSB MUST be large enough to fill the BDP.”
> 
> 11. Section 6.4.  “AQM techniques vary, but the basic principal is to discard traffic before the queue overflows (FIFO).  This discard in effect sends congestion notification  warning to protocols such as TCP,”
>        Comment: This is not true when RFC3168 is not implemented. In fact it just causes packet discards forcing TCP to adjust its TCP CWND.
> 
> Regards,
> 
> Gilles Forget, CCNP.
> T : 450-473-5718
> C : 514-895-8212
> @ : [email-masked] <mailto:[email-masked]>
> 
> Politique de Confidentialite des Courriels <http://www.bell.ca/enterprise/EntGnl_EmailConfidentialityWarning.page?language=fr> | Email Confidentiality Warning <http://www.bell.ca/enterprise/EntGnl_EmailConfidentialityWarning.page?language=en>
> 
> 
> 
> 
> 
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg
>