Subject: Re: [bmwg] WGLC: draft-ietf-bmwg-mpls-forwarding-meth-00

From: "Chip Popoviciu (cpopovic)" <[email-masked]>
Date: Sun, 12 Oct 2008 18:36:37 -0400

Dear Al and BMWG,

This is my last call review for "MPLS Forwarding Benchmarking
Methodology" (draft-ietf-bmwg-mpls-forwarding-meth-00.txt).

I fully support this work. Please let me know if you need additional
information.

Best regards,
Chip

------------------------------------------------------------------------
-------------------------------------------------

I-D Title(s): MPLS Forwarding Benchmarking Methodology
Filename(s): draft-ietf-bmwg-mpls-forwarding-meth-00.txt
Reviewer Name: Ciprian Popoviciu
Date: 10/12/08

Review Summary:

I believe the recommendations of this document are useful in the
industry and the work does a very good job in both methodology and
coverage.

Overall:

   * Does/Do the draft(s) provide clear identification of the
     scope of work? E.g., is the class of device, system, or
     service being characterized clearly articulated.

Yes.

   * If a terminology memo, are the measurement areas clearly
     defined or otherwise cited?  Is the working set of
     supporting terminology sufficient and correct?  To your
     knowledge, are the areas of the memo that may conflict
     with other bodies of work? Are there any measurements or
     terminology that are superfluous?  Are any missing?

   * If a methodology memo, does the methodology AND its
     corresponding terminology adequately define a benchmarking
     solution for its application area? Do the methodologies present
     sufficient detail for the experimental control of the benchmarks?

This document properly and completely describes the relevant methodology
needed for basic benchmarking of MPLS forwarding.

   * If neither a terminology or methodology, does the offered
     memo offer complementary information important to the use
     or application of the related benchmarking solution?

   * Do you feel there are undocumented limitations or caveats to
     the benchmarking solution being proposed?  If so, please
     describe.

No.

   * Does the memo attempt to define acceptance criteria for
     any of the benchmark areas?

No, this document does not define acceptance criteria.

Technical Content:  (Accuracy, Completeness of coverage)

   Are definitions accurate? Is the terminology offered relevant?

Yes.

   To your knowledge, are there technical areas that are erroneous?
   Are there questionable technical areas that need to be re-examined
   or otherwise scrutinized.

No.

   Does the solution adequately address IPv6?

Not applicable to IPv6.

   Do you feel the memo(s) being offered are technically mature enough
   for advancement to informational RFC?

Yes.

Clarity and Utility:

  If you had a need, would you utilize the benchmarking solutions
  advocated by this and its related memos?  If not, why?

Yes.

Conformance to BMWG principles: (see BMWG charter)
http://www.ietf.cnri.reston.va.us/html.charters/bmwg-charter.html

  Do you have confidence that the benchmarks, as explicitly
  defined, will yield consistent results if repeated on the
  same device (DUT/SUT), multiple times for a given test condition.
  If not, cite benchmark(s) and issue(s).

Yes.

  Do you have confidence that the benchmarks, if executed for a
  given test condition, utilizing the documented methodology
  on multiple test infrastructure (e.g., test equipment), would
  yield correct and consistent results on the same DUT/SUT?
  (Said differently, are the benchmark's methodology written
  with enough exacting detail, that benchmark implementation
  differences do not yield a difference in the measured quantities?)
  If not, cite benchmark(s) and issue(s).

Yes.

  Do you feel that the benchmarks form a basis of comparison between
  implementations of quantity being characterized? (I.e., are the
  benchmarks suitable for comparing solutions from different vendors.)

Yes.

  If not, cite benchmarks and issues.

  For those benchmarks cited above, do you feel that the benchmarks,
  as specified, have universal applicability for the given
  behavior being characterized?  (i.e., benchmarks might not form
  a basis for cross-vendor comparison, can be used universally
  in a different role.)

Yes.

Editorial Comments:
(includes any deficiencies noted w.r.t. I-D Nits, spelling, & grammar)

None.
_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Rajiv Papneja" <[email-masked]>
Date: Mon, 13 Oct 2008 18:42:24 -0400

Hello Al,

Here is my review in response to the BMWG initial WGLC for MPLS Forwarding
Benchmarking Methodology co-authored by Aamer Akhter and Rajiv Asati. 

I-D Title(s): MPLS Forwarding Benchmarking Methodology
Filename(s): draft-ietf-bmwg-mpls-forwarding-meth-00.txt
Reviewer Name: Rajiv Papneja
Date: October 13, 2008

Review Summary:

Let me start by saying that since inception of this effort, I have been
supporter of this draft as it provides guidelines for test engineers for
measuring the MPLS Forwarding performance within the bounds of RFC 2544.  

Overall:

Yes, I believe so. It does define the scope of work, and also relates it as
an extension of the RFC 2544. 

Test methodology section suggests a common topology to be used. Recommends
test considerations including the initial conditions before starting the
test. Also defines the control plane requirements before forwarding tests
could be executed, for example support of IGP, and label distribution
mechanisms. It is very important for the consistency of the test results
that the initial conditions are correctly met. The methodology provides
recommended values frame sizes, trial run time, and TTL handling.  

I do not believe so.

There is no acceptance criterion. However, this benchmark presents a
methodology for comparing performance of various MPLS enabled forwarding
devices.

Yes, and are elaborated where necessary.

I do not think so.

NA

I believe so.

Clarity and Utility:

Yes, we have already used these in some of our tests

As noted earlier, the initial conditions and requirements are laid out
clearly in the document. I believe that the results obtained using the
benchmarks defined would yield consistent results.

Yes

Yes

Yes

---

From: Al Morton <[email-masked]>
Date: Wed, 15 Oct 2008 14:28:03 -0400

Reminder!

A little more than a week is left on this WGLC.

Have you read a draft today? bmwg needs you!
Al
bmwg chair

At 10:13 AM 9/24/2008, Al Morton wrote:
>BMWG:
>
>A WG Last Call period for the Internet-Draft on
>
>"MPLS Forwarding Benchmarking Methodology"
>draft-ietf-bmwg-mpls-forwarding-meth-00.txt
>
>will be open from 24 September through October 24, 2008.
>
>This is the "first" WGLC on the draft.  It was developed as
>draft-akhter-bmwg-mpls-meth-04.txt and adopted as a WG draft.
>
>As with all initial WGLCs, volunteers are needed to complete
>the Last Call Review Template, available here:
>http://home.comcast.net/~acmacm/BMWG/LastCallTemplate.txt
>
>Use of the template and bmwg's last call process can be found here:
>http://www.ietf.org/mail-archive/web/bmwg/current/msg00846.html
>
>Please weigh-in on whether or not you feel that this Internet-Draft
>should be given to the Area Directors and IESG for consideration in
>publishing the Draft as an Informational RFC.  Send your comments
>to this list or [email-masked].
>
>A URL for this Internet-Draft is:
>http://tools.ietf.org/html/draft-ietf-bmwg-mpls-forwarding-meth-00
>(which replaced http://tools.ietf.org/id/draft-akhter-bmwg-mpls-meth-04.txt )
>
>Al
>bmwg chair
>

---

From: "Silvija Andrijic Dry (sdry)" <[email-masked]>
Date: Thu, 16 Oct 2008 11:47:09 -0400

Al and BMWG,

Below is my WGLC review of draft-ietf-bmwg-mpls-forwarding-meth-00. 
In general I'm in support of progressing this draft.

Thx,
Silvija

Last Call Review Template

I-D Title(s): MPLS Forwarding Benchmarking Methodology
Filename(s): draft-ietf-bmwg-mpls-forwarding-meth-00.txt
Reviewer Name: Silvija Dry
Date: 10/16/08

Please organize your comments in the following categories below.

Review Summary:

Overall:

Yes.

N/A

The methodology is defined with adequate level of detail to be useful
benchmarking solution for MPLS products.  

N/A

No.

No.

Yes.

No.

N/A

Yes.

Clarity and Utility:

Yes.

Yes.

Yes.

Yes.

Yes.

Editorial Comments:

None.

---

From: Al Morton <[email-masked]>
Date: Thu, 16 Oct 2008 11:59:41 -0400

>will be open from 24 September through October 24, 2008.

I have three comments as WG chair:

First, this is a very clearly written draft, and I'd like to
thank the authors for their persistent and skillful efforts
to pare this draft down to the minimum.

I'd like to see the "standard security paragraphs" incorporated
in the text.  This way, it's clear to anyone who picks-up the
RFC that the scope is limited to lab characterization, etc.
There are people who recently referenced RFC 2544 for live network
OAM testing in their standards documents. Not Good.
Clear Scope = Good.

The draft seems to use Frame and Packet interchangeably, and that's
bound to confuse some readers.  Pick the right one.

My remaining comments are below, as a participant.
Al

Section 2, Scope.
There are a few good sentences in the abstract that should be
also appear in the text of the scope. This also solves the problem of
not mentioning RFC 2544 until page 5.

I suggest:
2. Document Scope

    The purpose of this draft is to describe a methodology specific to
    the benchmarking of MPLS forwarding devices. The scope of this
    benchmarking will be limited to various types of packet-forwarding
    and delay measurements in a laboratory setting.
    It builds upon the tenets set forth in RFC2544 [RFC2544], RFC1242
    [RFC1242] and other IETF Benchmarking
    Methodology Working Group (BMWG) efforts.

    MPLS [RFC3031] is a foundation enabling technology for other more
    ...

Section 4, Test Methodology.
The example calculates the number of ports needed using the throughput,
but how do we know the throughput before establishing the test setup?
Maybe there is an estimated throughput available, and it seems that there
are a couple of considerations to mention.

I suggest:
The exact throughput is a measured quantity obtained through testing.
Throughput may vary depending on the number of ports used,
and other factors. The number of ports used (p) MUST be reported.

Section 4.1.2, editorials:
...(includes when the label bindings change). The most commonly used
    protocols are Label Distribution Protocol (LDP) [RFC5036], RSVP-TE ...
            ^^^^^
...This memo discourages the use of static label to establish the MPLS
         ^^^^

Section 4.1.3, Frame sizes
Although most active test people know what an IMIX is,
we don't have that term in bmwg lingo.

I suggest (in place of the last 2 paragraphs):
...In addition to the individual frame size trials, results MAY ALSO
    be collected with multiple simultaneous frame sizes (sometimes
    called an IMIX). There is no standard for mixtures of frame sizes,
    and the results are subject to wide interpretation. See section 18
    of RFC 2544.

    When running trials using multiple simultaneous frame sizes, the DUT
    configuration MUST remain the same.

Section 4.1.4, TTL and Hop Count
I suggest to add:

    If TTL/Hop Count Decrement is a configurable option, the setting
    MUST be reported.

Section 4.1.5, Trial Duration
I suggest to reword the last paragraph, for clarity:

    The longer trial time used for dynamic routing protocols
    is to verify that the DUT is able to maintain a stable
    control plane when the data-forwarding plane is under stress.

Section 4.1.5.1. Traffic Verification
I suggest to reword the last sentence of the 1st paragraph:

       ...In addition, the
    presence or absence of MPLS header on the packet MUST be verified,
    as well as checksum, frame sequencing and correct MPLS TTL
    decrement action.

Also, is the last paragraph in quotes needed? possibly an editing issue:

    "In all cases the sent traffic MUST be accounted for, whether it was
    received on the wrong port, correct port or not received at all. In
    addition, the MPLS header...."
??

Section 4.1.7. Abbreviations Used
Looks like a typo:
OLD
    RN := Remote Network (can also be thought of as a network that is
    reachable via) Mp.
NEW?
    RN := Remote Network (can also be thought of as a network that is
    reachable via Mp).
                 ^^^^

Section 5. Reporting Format
1st paragraph, suggest:
s/recommended/RECOMMENDED/
Table header, suggest:
s/Unit/Units/

Section 6. MPLS Forwarding Benchmarking tests
1st para, suggest:
s/than one MPLS headers/than one MPLS header/
s/those of described in RFC2544/those described in RFC2544/

2nd para, suggest:
s/follow the below 'Test Setup' and 'Test Procedure.'/follow the 
'Test Setup' and 'Test Procedure' below./

3rd para says:
        ... In such case, the tool traffic should use BpRN1 and
      BpAN as the IP destinations in a weighted round robin fashion. The
      weighting ratio between  BpRN1 and BpAN is a constant test
      parameter. A suggested ratio is 1:100 with BpAN:BpRN1. The traffic
      streams offered MUST conform to section 16 of RFC 2544.

but the weighting seems to conflict with the referenced section of 2544,
specifically the sentence:  http://tools.ietf.org/html/rfc2544
     ...The specified tests are
    run using the same data rate being offered to each of the input
    ports.

 >>> Need some text to reconcile this difference, and possibly discuss
 >>> this point on the list.

Section 6, un-numbered section on Test Procedure
I suggest:
    Test Procedure  (Refer to Section 26 of RFC 2544)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      Send traffic from port(s) Ap towards DUT at a constant load towards
                            ^^^
      IP prefixes (BpRN1 addresses) advertised by the test tool on the
                                                      ^^^^^^^^^
      receive ports, for a fixed time interval.
                                 ^^^^^^^^^^^^^

Section 6.1.1. Throughput for MPLS Label Imposition

in the Objective section, I suggest:
      To obtain the DUT's Throughput (as per RFC 2544) during label imposition
      (i.e. IP to MPLS) for a regular (IPv4 or IPv6) packet.

last sentence of Test Setup:
s/it's/its/

first line of Discussion:
s/contain non-reserved/contain a non-reserved/

2nd sentence of Discussion:
s/testool/test tool/

2nd sentence of Procedure:
s/must/MUST/

2nd sentence of Reporting format says:
      Results for each test SHOULD be in the form of a table with a row
      for each of the tested frame sizes. Additional columns SHOULD
      include: offered load and measured throughput.
An RFC 2544 Throughput benchmark *is* the offered load, the only
measurement on the output is a count to determine if loss occurred.
So, these two columns are the same thing, maybe something else would
be a better example.

Similar comments apply to sections 6.1.2 and 6.1.3 and 6.1.4

Section 6.2. Latency Measurement
in Procedure, I suggest:
s/RFC 2544/section 26.2 of RFC 2544/

Section 6.2. Frame Loss Measurement (FLR)
in Procedure, I suggest:
s/RFC 2544/section 26.3 of RFC 2544/

Section 9 References:
IMO, both RFC 2544 and RFC 1242 are Normative in this memo,
so should appear in the Normative section.

---

From: "Bill Cerveny" <[email-masked]>
Date: Fri, 24 Oct 2008 15:04:10 -0400

I-D Title(s): MPLS Forwarding Benchmarking Methodology
Filename(s): draft-ietf-bmwg-mpls-forwarding-meth-00.txt
Reviewer Name: Bill Cerveny
Date: Oct. 24, 2008

Review Summary:

Overall:

Yes

N/A

Yes

No

No

Yes and yes

Letter references to parts of the test layout are sometimes unclear, in
my opinion.

Not applicable

Yes

Clarity and Utility:

Yes. They seem reasonable.

Yes

Yes

Yes

Editorial Comments:

General:
- “test tool” frequently referenced as “testool”.  “Test tool” is
correct, in my opinion (although I’m also wondering if something else
such as “tester” may be even more correct).

- Use of “etc” should be avoided, in my  opinion, especially where text
says “for example, …”

- Terms test procedure and test setup should not be capitalized in
normal text.

Abstract: 
- s/The purpose of this draft is to describe/This document describes/

- s/devices. The scope of this benchmarking will be limited/devices,
limited/

4.1.2 Label Distribution Support

- S/includes when the label/including when the label/

- S/The most commonly used protocol is/The most commonly used protocols
are/

- Define acronyms RSVP-TE, MP-BGP and FEC on first usage

- S/This draft/This document/

4.1.3 Frame Sizes

- S/For better reference, the/The/

- Move “for IPv4” between “sizes” and “are”

- Define acronym IMIX and provide reference to its definition.

4.1.4 TTL or Hop Count

- Define acronym TTL

- Don’t capitalize hop count in text

4.1.5 Trial Duration

- S/The longer trial time for when/When/

- S/used for verifying/used, verify/

4.1.5.1 Traffic Verification

- S/In all cases the sent/In all cases, sent/

- S/The MPLS header presence/MPLS header presence/

- S/In general, many/Many/

- S/but that is considered/but this is/

5. Reporting Format

For table heading, “Unit” should be replaced with “Unit or example”

Remove “etc” in table

6. MPLS Forwarding Benchmarking tests (capitalize “tests” in title)

- S/MPLS is altogether a/MPLS is a/

- S/MPLS packet is likely to contain more than one MPLS headers/MPLS
packets are likely to contain more than one MPLS header/

- S/Such characteristics desire/Such characteristics/

- S/but not limited to/but is not limited to/

- S/It is recommended that a/A/

- Should what is meant by “regular traffic” be elaborated on?

6.2 Latency Measurement

- Is the description directly beneath title actually part of the
procedure?

- “The forwarding delay measurement requires the accurate propagation …"
Should "accurate propagation" be “accurate one-way propagation”?

On Wed, 24 Sep 2008 10:13:14 -0400, "Al Morton" <[email-masked]> said:
> (which replaced
> http://tools.ietf.org/id/draft-akhter-bmwg-mpls-meth-04.txt )
> 
> Al
> bmwg chair
>

---

From: Al Morton <[email-masked]>
Date: Sat, 25 Oct 2008 09:35:00 -0400

Bill,

Thanks for your comments, and for completing the
WGLC template.  I see your original message was
sent well before the deadline yesterday, (TMDA
challenge may have held it up, and we don't stand
on ceremony very often here in bmwg) so ...

Rajiv, Aamer,

Please address Bill's comments in the next revision.
If there are any unclear comments, or suggestions where
you would prefer to hear other opinions before implementing,
please raise those issues on the list.

Thanks to all who contributed to a productive last call!

Al
bmwg chair

At 03:04 PM 10/24/2008, Bill Cerveny wrote:

>Last Call Review Template ...

---

From: Al Morton <[email-masked]>
Date: Sat, 25 Oct 2008 09:39:02 -0400

At 09:35 AM 10/25/2008, Al Morton wrote:
>...If there are any unclear comments, or suggestions where

Just to clarify, this applies to all LC comments, not just Bill's!