Subject: FW: Comments for Benchmarking Methodology for LAN Switching Devices

From: "Gene" <[email-masked]>
Date: Fri, 28 Jul 2000 18:36:20 -0700

After careful review, here are a few comments for the mswitch draft:

1) [3]   Bradner, S., Editor, "Benchmarking Methodology for Network
         Interconnect Devices", RFC 2544, March 2000.
                                          ^^^^^^^^^^^^^

This document was actually published in March 1999.

2) Technical problems
- Algorithms/pseudo code
Section 5.7.3 Procedure - this algorithm is invalid; it will not iterate.  I
think there must be a typo in the while statement...

Perhaps that pseudo-code would be more readable if it was verbalize, rather
than code-ized.

- Formulas
Appendix A-1 - the formula calculation for IBG is incorrect.  If the LOAD is
equal to 1, then we have a divide by zero condition.  In addition, this
formula is ethernet-specific, also true with respect to the other
definitions in
both appendices.

Also, TXTIME is again, ethernet-specific.

- Protocol references
In section 3 - Test Setup - there is a paragraph that describes learning.
If this set of methods are to be applied to both layer 2 & layer 3 devices,
then
some reference should be made to ARP/static ARP in addition to MAC learning.

Also, I don't think it is appropriate in this section to specify the rate at
which address learning frames must be offered, unless the wording is
modified to
refer to the number of frames determined by the address learning rate
methodology (section 5.8).

4)  General readability.
- This document is inconsistant in the mechanism of how it refers to itself.
  For example, in Section 1 Introduction:

This RFC primarily deals with devices which switch frames at the <snip>
     ^^^^

section 4.  Frame formats and sizes
   <snip>
   The length, contents and method of detection is not defined in this
   memo.
   ^^^^^^

There are other examples, however I think the point is that the document
should refer to itself in a consistant fashion, such as "this document"  or
"this
memo", as in the very first paragraph on the first page.

- There is so much copy & paste from w/in the document that several sections
are repeated over & over specifying the same basic details.  Having these
duplications makes this document unnecessarily lengthy & difficult to
follow.It would be very convenient if they could be generalize & made
applicable to
the entire document except where there is departure from the generalities.

For example:  Section 5.1.2 - this defines frames size, interframe gap,
duplex mode, burst size, addresses per port, and trial duration.  This same
set of
parameter descriptions may be found in sections 5.2.2, 5.2.4, 5.3.2, 5.4.2,
5.5.2, 5.6.2 (a partial list but still very similar), 5.10.2 (again, a
partial list
but mostly similar).

In addition, the frame size paramter is also defined in:
Section 4 -  Frame formats and sizes:

   "For frame sizes, refer to RFC 2544, section 9 [3]."

Another example of duplication is in section 5.2.4 - these 4 paragraphs
are repeated throughout the document:

   "Any frame originating from the DUT/SUT MUST not be counted as a
   received frame.  Frames originating from the DUT/SUT MAY be counted
   as flooded frames or not counted at all.

   Forwarding rate (FR) of the DUT/SUT SHOULD be reported as the number
   of test frames per second that the device is observed to successfully
   transmit to the correct destination interface in response to a
   specified Oload.  The Oload MUST also be cited.

   Forwarding rate at maximum offered load (FRMOL) MUST be reported
   as the number of test frames per second that a device can
   successfully transmit to the correct destination interface in
   response to the MOL as defined in section 3.6 [2]. The MOL MUST also
   be cited.

   Maximum forwarding rate (MFR) MUST be reported as the highest
   forwarding rate of a DUT/SUT taken from an iterative set of
   forwarding rate measurements.  The iterative set of forwarding rate
   measurements are made by adjusting Iload.  The Oload applied to the
   device MUST also be cited."

If these are not specifically terminology redefinitions from RFC 2285 (see
sections 3.5.x), then it would be more appropriate to define it once for the
entire document.

Section 5.2.3 - there are some superfluous 'the's in the second paragraph.
Also, the third paragraph is only one sentence, probably could be broken up.

- The sections that describe "Procedure" are more or less copy and paste; it
is not apparent why some of the procedure paragraphs can not be generalized
for the entire document.  Examples include:

section 5.2.3:
   "All ports on the tester MUST transmit test frames either in a Frame
   Based or Time Based mode (Appendix B).  Depending upon traffic
   direction, some or all of the ports will be transmitting.  All ports
   SHOULD start transmitting their frames within 1% of the trial
   duration.  For a trial duration of 30 seconds, all ports SHOULD have
   started transmitting frames within 300 milliseconds of each other."

from section 5.3.3:
   "All ports on the tester MUST transmit test frames either in a Frame
   Based or Time Based mode (Appendix B).  All ports SHOULD start
   transmitting their frames within 1% of the trial duration.  For a
   trial duration of 30 seconds, all ports SHOULD have started
   transmitting frames with 300 milliseconds of each other."

I don't see any significant difference in the meaning behind these
paragraphs; yet, they are not identical nor are they generalized in one
location for the
entire draft.

A similar argument may be applied to Measurements, Throughput, Forwarding
rate and Reporting Format.  All these definitions are defined again & again
throughout the document and are redundant.

- Construction/Grammar:
section 4.  Frame formats and sizes
   "The test frame format is defined in RFC 2544 section 8 [3] and MUST
   contain a unique signature field located in the UDP DATA area of the
   Test Frame (see Appendix C [3]).  The purpose of the signature field
   is filter out frames that are not part of the offered load."

The last sentence in this paragraph would be better worded as:
  The purpose of the signature field is to filter out received frames that
                                        ^^            ^^^^^^^^^
  are not part of the offered load.

section 4.  Frame formats and sizes
   "The signature field MUST be unique enough to identify the frames not
   originating from the DUT/SUT."

This would be better worded as:
   "The signature field MUST be unique enough to identify the frames
   originating from the test device."

section 4.  Frame formats and sizes
   "There are three possible frame formats for layer 2 Ethernet switches:
   standard MAC Ethernet frames, standard MAC Ethernet frames with
   vendor-specific tags added to them, and IEEE 802.3ac frames tagged to
   accommodate 802.1p&Q.  The two types of tagged frames may exceed the
   standard maximum length frame of 1518 bytes, and may not be accepted
   by the interface controllers of some DUT/SUTs. It is recommended to
   check the compatibility of the DUT/SUT with tagged frames before
   testing."

Several comments re: this paragraph -
first, is this document describing
only layer 2 devices?  Layer 3 devices may also be tagged; is the intention
to
not worry about that?

Second - reword the second sentence to read something like this:
  "The two types of tagged frames may exceed the
   standard maximum length frame of 1518 bytes, and may not be accepted
   by some DUT/SUTs."

Third - also, by stating that there are 'two types of tagged frames', we are
limiting ourselves to the concept that no other creative vendor will ever
come up w/another type of tag (what about MPLS, for example....?)

5.1.2 Setup Parameters
   "When offering full meshed traffic, the following parameters MUST be
   defined.  Each parameter is configured with the following
   considerations."

I think you meant to write "fully meshed" ...

section 5.5.1 Objective
   "To determine how a DUT handles congestion.  Does the device implement
   congestion control and does congestion on one port affect an
   uncongested port.  This procedure determines if Head of Line Blocking
   and/or Backpressure are present."

This is a poorly constructed paragraph.

section 5.7.4 Measurements
   "Whether the offered addresses per port was successful forwarded
   without flooding."

This is not a sentence.

section 5.8.4 Measurements
   "Whether the offered addresses per port were successful forwarded
   without flooding at the offered learning rate."

This is not a sentence.

5.9.4 is missing (I assume it would have been the measurement section..?)

"5.9.5 Reporting format

   For each of the error conditions in section 5.6.3, a "pass" or "fail"
   MUST be reported.  Actual frame counts MAY be reported for diagnostic
   purposes."

The reference to 5.6.3 should be 5.9.3.

Appendix B: Generating Offered Load

   "The unit of measurement for Oload is bits per second.  The two
   methods described here will hold one unit constant and let the
   DUT/SUT vary the other unit.  The traffic generator SHOULD specify
   which method it uses."

This paragraph is unclear.