Subject: [bmwg] WGLC: draft-ietf-bmwg-reset-00

From: Al Morton <[email-masked]>
Date: Wed, 16 Jun 2010 17:58:37 -0400

BMWG,

We've just added a Milestone to update 2544 Reset Benchmarking:
Sep 2010  Networking Device Reset Benchmark (Updates RFC 2544) to IESG Review

This message begins the First WG Last call on the corresponding draft:

http://tools.ietf.org/html/draft-ietf-bmwg-reset-00

The Last Call will end on July 21, 2010, which gives a little extra
time for Summer holidays.  Since this is the First WGLC, I ask folks
to consider items in the active review template as they review the
drafts.  Almost everyone of us has conducted this sort of testing,
and so many should have an opinion on the updated methods this draft
will bring to the industry.

The authors plan to revise the draft quickly and have an updated version
ready for discussion at IETF-78.

Please weigh-in on whether or not this Internet-Draft
should be given to the Area Directors and IESG for consideration and
publication as an Informational RFC.  Send your comments
to this list and/or [email-masked].

Al
bmwg chair

---

From: Al Morton <[email-masked]>
Date: Tue, 29 Jun 2010 15:32:15 -0400

At 05:58 PM 6/16/2010, Al Morton wrote:
>... This message begins the First WG Last call on the corresponding draft:
>
>The Last Call will end on July 21, 2010,

As Chair, I'm reminding active bmwg'ers to queue this up
on their reading list in the next few weeks and provide
feedback.

As a participant, my comments follow.

Good draft, a little more to do.
Al

>PS: this PS text should show-up in your mail reader as
>indented or quoted text, otherwise you'll have some trouble
>deciphering my comments below.

>1. Introduction
>...
>    In order to provide consistent and fairness while benchmarking a set
>    of different DUTs, the Network tester / Operator MUST (a) use
>    identical control and data plane information during testing, (b)
>    document & report any factors that may influence the overall time
>    after reset / convergence.

It's odd to have a requirement in the Intro, I suggest to move it to
a later section.

>    Some of these factors follow:

Suggest:
    Some of these factors include:

...
>      11. Performance - Interface encapsulation as well as achievable
>          NDR (non-dropping rate)

NDR not really defined in bmwg. Suggest:
      11. Performance - Interface encapsulation as well as achievable
          Throughput [RFC2544]
...
>    This document specifies a methodology for characterizing reset
>    during benchmarking of forwarding devices, and provides clarity and
>    consistency in reset procedures beyond what's specified in
>    [RFC2544]. These procedures may be used by other benchmarking
>    documents such as [RFC2544], [RFC5180], [RFC5695], etc.
>
>    This document updates Section 26.6 of [RFC2544].
>
>1.1. Scope
>
>    This document focuses on only the reset criterion of benchmarking,
>    and presumes that it would be beneficial to [RFC2544], [RFC5180],
>    [RFC5695], and other BMWG benchmarking efforts.

I really think all the above is part of the Scope, and suggest
(with modification to the first and last paragraphs):
1.1. Scope

    This document specifies a methodology for characterizing reset
    during benchmarking of forwarding devices, and provides clarity and
    consistency in reset procedures beyond what is specified in
                                                ^^
    [RFC2544]. These procedures may be used by other benchmarking
    documents such as [RFC2544], [RFC5180], [RFC5695], etc.

    This document updates Section 26.6 of [RFC2544].

    This document focuses on only the reset criterion of benchmarking,
    and presumes that it would be beneficial to [RFC5180],
    [RFC5695], and other BMWG benchmarking efforts.

------------------------------------

General comment on Section 3:

All the procedures end with something like:

>      Finally, the characterization is completed by measuring the frame
>      loss and recovery time from the moment the RP is re-initialized or
>      reinserted.

I think you are measuring loss, and inferring recovery time from the
packet rate, similar to the "loss-derived convergence time" used
in the IGP-Dataplane drafts.

If so, then you need to provide an equation or procedure to calculate
the inferred recovery time (or methods to measure a time interval with
explicit start and stop).

in section 3.2.2, the method gets closer to defining start and
stop times:

>      Finally, the characterization for each individual process is
>      completed by measuring the complete frame loss and recovery time
>      from the moment the reset instruction was given until the
>      Operating System finished the reload and re-initialization
>      (inferred by the re-establishing of traffic).

Also, for Multiple Trials, the reporting should make it clear that
an Average will be used, and also the number of trials in the average.

            Loss   (average)            Frames

            Time   (average)            Seconds, with sufficient resolution

            Number of trials            Integer count

Also" with sufficient resolution" to accomplish what?
Display all significant digits and at least two digits
that were variable over the trials...  ???

Another general point:
What do we report if the stimulus produces no loss?
IOW, recovery time is zero, but there may still be some
affect on traffic, such as transient delay variation or increase
in latency.

----------------------------------------------------

>3.1.1.2. RP Failure for a multiple-RP device (optional)
>...
>    Procedure
>
>      This test characterizes "RP Switchover". Many implementations
>      allow for optimized switchover capabilities that minimize the

Suggest this section be titled "RP Switchover for a multiple..."

...
>      Third, perform the primary Route Processor (RP) hardware reset at
>      this point. This entails for example physically removing the RP,
>      or triggering a hardware reset by other means (e.g., command line
>      interface, physical switch, etc.) Is up to the Operator to decide
>      if the active RP needs to be re-inserted after a grace period or
>      not.

Suggest some small edits here:

      Third, perform the primary Route Processor (RP) hardware reset at
      this point. This entails for example physically removing the RP,
      or triggering a hardware reset by other means (e.g., command line
      interface, physical switch, etc.) It is up to the Operator to decide
                                        ^^^^^
      if the primary? RP needs to be re-inserted after a grace period or
             ^^^^^^^^
      not.

>    Reporting format
>
>      The reset results are reported twice, one for the default
>      switchover behavior and the other for the non-default one. For

Sorry, I don't understand the above sentence.

>3.1.2. Line Card (LC) Removal and Insertion (mandatory)
>
>...
>
>      Third, perform the Line Card (LC) hardware reset at this point.
>      This entails for example physically removing the LC to later re-
>      insert it, or triggering a hardware reset by other means (e.g.,
>      command line interface (CLI), physical switch, etc.). However,
>      most accurate results will be obtained using the CLI or a physical
>      switch, and therefore these are RECOMMENDED. Otherwise, the time
>      spend trying to physically seat the LC will get mixed into the
>      results.

Suggest:
s/spend/spent/

>3.2.1. Operating System (OS) reset (mandatory)
>
>    Objective
>
>      To characterize the speed at which a DUT recovers from an
>      Operating System (OS) software reset.

Speed seems like the wrong word, how about:

      To characterize the time needed for a DUT to recover from an
      Operating System (OS) software reset.

(same comment for 3.2.2, Process Reset definition used "speed")

More in 3.2.2:

>      Such speed may depend upon the number and types of process running
>      in the DUT and which ones are tested. Different implementations of
>      forwarding devices include various common processes. A process
>      reset should be performed only in the processes most relevant to
>      the tester.

... and relevant to forwarding?  otherwise forwarding not affected?

>3.3.1. Power Interruption (mandatory)
>
>...
>
>      Third, interrupt the power (AC or DC) that feeds the corresponding
>      DUTs power supplies at this point. This entails for example
>      physically removing the power supplies in the DUT to later re-
>      insert them, or simply disconnecting or switching off their power
>      feeds (AC or DC as applicable). The actual power interruption
>      should last at least 15 seconds.
>
>      Finally, the characterization is completed by measuring the frame
>      loss and recovery time from the moment the power is restored or
>      the power supplies reinserted in the DUT.

Does the reported recovery time include the 15 seconds, or
can it be removed for easier comparison with other testing?

coffee time!