Subject: Re: [bmwg] Second WGLC on draft-ietf-bmwg-traffic-management

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 5 Jan 2015 14:10:10 -0500

BMWG -FYI this WGLC ends tomorrow.

Hi Barry and Ramki,

These are my comments on
                      Traffic Management Benchmarking
		      draft-ietf-bmwg-traffic-management-01.txt

Al
(as participant)

Note: as Document Shepherd, I see quite a few nits generated
by the nits-checker:
https://tools.ietf.org/idnits?url=https://tools.ietf.org/id/draft-ietf-bmwg-traffic-management-01.txt

High-level:
   7. Security Considerations

   8. IANA Considerations

   9. Conclusions

Security and IANA sections are Mandatory, and cannot be blank.

I saw a few "TBD" in section 6, questions for Ramki. . .

The new procedures add considerable needed details, but also
raise a few questions.

When considering repeatability over multiple test runs, there could
be a statistical measure of deviation for the key benchmarks so that
the audience can easily assess the degree of consistency observed. 

Detailed comments on a partial version of the draft are attached.

________________________________________
From: bmwg [[email-masked]] On Behalf Of MORTON, ALFRED C (AL)
Sent: Wednesday, December 10, 2014 2:30 PM
To: [email-masked]; [email-masked]
Subject: [bmwg] Second WGLC on draft-ietf-bmwg-traffic-management

BMWG (and AQM):

A WG Last Call period for the Internet-Draft on
Traffic Management Benchmarking:

   http://tools.ietf.org/html/draft-ietf-bmwg-traffic-management

will be open from 10 December 2014 through 6 January, 2015.

The first WGLC (on -00) closed October 21 2014, with substantial
comments and the authors believe they are now addressed.

This draft is continuing the BMWG Last Call Process. See
http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html

Please read and express your opinion on whether or not this
Internet-Draft should be forwarded to the Area Directors for
publication as an Informational RFC.  Send your comments
to this list or [email-masked] and [email-masked]

Al
bmwg co-chair

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: Barry Constantine <[email-masked]>
Date: Mon, 12 Jan 2015 16:29:30 +0000

Hi Al,

Thanks again for the review and we'll make the changes in the next couple of weeks.

We never ran it through full nit check for submission, but I can see it's time to do that and will get that taken case of as well.

Thank you,
Barry Constantine

JDSU Network and Service Enablement
Principal Member Technical Staff
301-325-7069

-----Original Message-----
From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C (AL)
Sent: Monday, January 05, 2015 2:10 PM
To: [email-masked]; [email-masked]
Subject: Re: [bmwg] Second WGLC on draft-ietf-bmwg-traffic-management

Hi Barry and Ramki,

These are my comments on

Al
(as participant)

Note: as Document Shepherd, I see quite a few nits generated by the nits-checker:
https://tools.ietf.org/idnits?url=https://tools.ietf.org/id/draft-ietf-bmwg-traffic-management-01.txt

High-level:

   8. IANA Considerations

   9. Conclusions

The new procedures add considerable needed details, but also raise a few questions.

When considering repeatability over multiple test runs, there could be a statistical measure of deviation for the key benchmarks so that the audience can easily assess the degree of consistency observed. 

BMWG (and AQM):

A WG Last Call period for the Internet-Draft on Traffic Management Benchmarking:

The first WGLC (on -00) closed October 21 2014, with substantial comments and the authors believe they are now addressed.

This draft is continuing the BMWG Last Call Process. See http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html

Please read and express your opinion on whether or not this Internet-Draft should be forwarded to the Area Directors for publication as an Informational RFC.  Send your comments to this list or [email-masked] and [email-masked]

Al
bmwg co-chair

---

From: Barry Constantine <[email-masked]>
Date: Thu, 15 Jan 2015 12:51:44 +0000

Hi Al,

Concerning the several comments in the area of Measurement units, I am unsure what is intended for some of these.  One example below:

   - Shaper Rate (SR): the Shaper Rate is only applicable to the  traffic shaping tests.  The SR represents the average egress output rate (bps) over the test interval.

ACM] determine and add: 
Measurement Units for reported values.

In this example, the SR is bps, so sorry I must be missing something obvious concerning your request.

Thanks,
Barry

From: bmwg [[email-masked]] on behalf of MORTON, ALFRED C (AL) [[email-masked]]

Hi Barry and Ramki,

These are my comments on

Al
(as participant)

High-level:

   8. IANA Considerations

   9. Conclusions

BMWG (and AQM):

Al
bmwg co-chair

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 15 Jan 2015 07:58:29 -0500

you're right, apparently I pasted this comment one too many times...

> From: Barry Constantine [mailto:[email-masked]]
> Sent: Thursday, January 15, 2015 7:52 AM
> To: MORTON, ALFRED C (AL); [email-masked]
> Subject: RE: Second WGLC on draft-ietf-bmwg-traffic-management
> 
> Hi Al,
> 
> Concerning the several comments in the area of Measurement units, I am
> unsure what is intended for some of these.  One example below:
> 
>    - Shaper Rate (SR): the Shaper Rate is only applicable to the  traffic
> shaping tests.  The SR represents the average egress output rate (bps)
> over the test interval.
> 
> ACM] determine and add:

---

From: Barry Constantine <[email-masked]>
Date: Sat, 17 Jan 2015 14:02:42 +0000

Hi Al,

Looking at another comment in the area of repeatability:

[ACM]
> When considering repeatability over multiple test runs, there could be 
> a statistical measure of deviation for the key benchmarks so that the 
> audience can easily assess the degree of consistency observed.

The test is divided into pure packet, stateless generation with tradition metrics such as loss, delay jitter and stateful TCP traffic with RFC 6349 metrics.  As we both are aware, TCP variation will be greater than traditional packet based.

I'm struggling a little how to phrase the measure of variation even for the traditional (Scott Bradnor please chime in).

Let me throw out, 1% deviation for stateless and 3% for stateful tests as guidance for acceptable repeatability?

Thank you,
Barry Constantine

-----Original Message-----
From: MORTON, ALFRED C (AL) [mailto:[email-masked]] 
Sent: Thursday, January 15, 2015 7:58 AM
To: Barry Constantine; [email-masked]
Subject: RE: Second WGLC on draft-ietf-bmwg-traffic-management

>    - Shaper Rate (SR): the Shaper Rate is only applicable to the  
> traffic shaping tests.  The SR represents the average egress output 
> rate (bps) over the test interval.
> 
> ACM] determine and add:
> https://tools.ietf.org/idnits?url=https://tools.ietf.org/id/draft-ietf
> -
> bmwg-traffic-management-01.txt
> 
> 
> High-level:
> When considering repeatability over multiple test runs, there could be 
> a statistical measure of deviation for the key benchmarks so that the 
> audience can easily assess the degree of consistency observed.
> 
> publication as an Informational RFC.  Send your comments to this list 
> or [email-masked] and [email-masked]
> 
> Al
> bmwg co-chair
> 
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 17 Jan 2015 09:57:16 -0500

I don't think we need a threshold for acceptability,
just a metric of variation across trials. You are trying
to give some guidance on when the number of trials is
sufficient, or confidence that the trials have characterized
the results variation sufficiently. If you reach a point where
variation is confidently characterized as a significant portion
of the measured values, the next step may be to revise the methods
to achieve better consistency...  variation may be an indicator of
difficulty repeating the results on another day, in another lab, etc.

Al

> Sent: Saturday, January 17, 2015 9:03 AM
> The test is divided into pure packet, stateless generation with tradition
> metrics such as loss, delay jitter and stateful TCP traffic with RFC 6349
> metrics.  As we both are aware, TCP variation will be greater than
> traditional packet based.
> 
> I'm struggling a little how to phrase the measure of variation even for
> the traditional (Scott Bradnor please chime in).
>