Subject: Re: [bmwg] Planning for an Interim Meeting

From: Al Morton <[email-masked]>
Date: Thu, 08 Oct 2009 10:59:29 -0400

As an Update to the Working Group, we are now
looking at the fourth week of October,
on 26 Monday, 27 Tuesday or 30 Friday,
as dates for the Interim meeting.

I still need to hear from some authors, and any
participants who have a strong preference.

regards,
Al
bmwg chair

At 10:28 AM 9/21/2009, Al Morton wrote:
>BMWG,
>
>As discussed at the last session in Stockholm, both
>our AD-Advisor Ron and I have schedule conflicts with
>IETF-76 week, and are unable to travel to the meeting.
>We mentioned that BMWG could have an Interim meeting to
>progress work, likely a 2 hour conference call supported by
>a document conference system, such as webex.
>
>The weeks between IETF meetings are rapidly slipping by, and I
>want to investigate the possibility of a meeting during
>the 3rd week of October, on 19 Monday, 20 Tuesday or 23 Friday.
>The time would need to be friendly to both Europe and US
>(we don't have any active participants from other zones, AFAIK)
>so that means it would start at about 11AM US Eastern time,
>+/1 an hour or so.
>
>The main question at this point is to ALL DOCUMENT EDITORS:
>
>- Will there be an update of your draft ready for discussion
>   by the 2nd week of October, to give people a chance to read
>   it prior to the meeting?  If not, by when will an update be
>   available?
>
>I know of one such draft right now, a new work proposal from
>David Newman and Timmons Player:
>http://www.ietf.org/id/draft-player-dcb-benchmarking-00.txt
>
>If we have sufficient drafts to discuss, then we'll meet.
>If not, we'll look at the calendar again with the dates
>supplied by the editors.
>
>Please reply to the list.
>
>thanks and regards,
>Al
>bmwg chair
>
>_______________________________________________
>bmwg mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/bmwg

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Fri, 9 Oct 2009 13:18:47 +0200

Hi Al,

I have just submitted a new IP Flow monitoring draft
version.
It addresses comments/discussions from the Stockholm meeting and
finalises
the changes towards the subject clarity (hopefully) which did
not get into the draft in time for the last meeting.

Tx, Jan

The climate of Edinburgh is such that the weak
succumb young .... and the strong envy them.

                                 Dr. Johnson  

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of Al
> Morton
> Sent: 21 September 2009 15:28
> To: [email-masked]
> Subject: [bmwg] Planning for an Interim Meeting
> 
> BMWG,
>

---

From: Al Morton <[email-masked]>
Date: Fri, 16 Oct 2009 11:11:25 -0400

BMWG,

While we finalize the conference-oriented details
for the Interim meeting, I want to communicate the
consensus on the DATE, with the time being constrained
by the time zones involved. (There are a few who could not
make this date, but it seems to be the best for most).

UTC (GMT)  Friday, October 30, 2009 at 13:30:00 to 16:00:00
start times in
Geneva (Switzerland)    Friday, Oct 30, 2009 at 2:30:00 PM  UTC+1 hour CET
New York (U.S.A.)       Friday, Oct 30, 2009 at 9:30:00 AM  UTC-4 hours EDT
San Francisco (U.S.A.)  Friday, Oct 30, 2009 at 6:30:00 AM  UTC-7 hours PDT
Note that the USA will still be observing Daylight Savings Time,
while Europe and many other locations be on Standard Time.

The meeting will include a 30 minute warm-up, because we anticipate
the ability to use webex for this conference in addition to an
audio bridge, and it may take time to get everyone on-board.
Then we will have a full two hours to discuss drafts
and resolve comments.

So far, we have proposals to discuss the following drafts,
some of which are available now, and others which are planned
to be complete by the revised draft deadline of Oct 26th:

NEW WORK ITEM PROPOSAL: BENCHMARKS FOR DATA CENTER BRIDGING DEVICES
http://www.ietf.org/id/draft-player-dcb-benchmarking-00.txt

IP Flow Information Accounting and Export Benchmarking Methodology
http://tools.ietf.org/id/draft-novak-bmwg-ipflow-meth-04.txt

-=-=-=-=-=-=-

Benchmarking Terminology for Protection Performance
Methodology for benchmarking MPLS protection mechanisms
To be provided: Revised versions of one or both drafts
Benchmarking Link-State IGP Data Plane Route Convergence
To be provided: terms and meth drafts

Benchmarking Methodology for Content-Aware Network Devices
To be provided: draft-hamilton-bmwg-ca-bench-meth-(02)

(There has also been off-list discussion of the RESET work,
but no commitment to produce a -00 draft yet by the 19th.)

-=-=-=-=-=-=-

I've simply listed these in the order that authors responded,
with a preference to drafts that are available now, then the
committed drafts.

Authors: to get on the agenda (which will look roughly like the above
list), you should have revised drafts submitted by the regular IETF
deadlines, and slides to help organize discussion, raise issues, etc.
IF we have additional agenda time, we could
possibly take-up a slides-only discussion on issues with a
*working group chartered item*.

Questions, Comments, Bashing to me or the list, as appropriate.

regards,
Al
bmwg chair

---

From: Al Morton <[email-masked]>
Date: Fri, 16 Oct 2009 16:42:18 -0400

BMWG,

The IETF-wide announcement of our interim meeting has prompted some discussion
on the wg chairs and IAB lists, where several folks who are not
subscribed to [email-masked] cross-posted their comments.

As list moderator, I am about to approve all messages in the spirit of
open communications, but know in advance that:

The announcement was sent 2 weeks in advance of the meeting, as required:
http://www.ietf.org/iesg/statement/interim-meetings.html
Our AD Advisor approved the meeting (WG chair approval is
sufficient for teleconferences and jabber sessions).
The IESG was consulted because of the special circumstances
in our case, and did not object to BMWG's Interim meeting.

read on,
Al
bmwg chair

>UTC (GMT)  Friday, October 30, 2009 at 13:30:00 to 16:00:00
...

---

From: Al Morton <[email-masked]>
Date: Sun, 18 Oct 2009 16:35:16 -0400

BMWG,

In order to keep our Interim meeting completely within
the guidelines (agenda must be announced 1 week in advance),
I've published the 1st Draft agenda today on
a new web page dedicated to the meeting:
http://home.comcast.net/~acmacm/BMWG/Interim-Oct09.html

When additional information is available, I'll post it
and announce on the list.

comments on the draft agenda to me,
Al
bmwg chair

At 11:11 AM 10/16/2009, Al Morton wrote:
>To: [email-masked]
>From: Al Morton <[email-masked]>
>Subject: Re: [bmwg] Planning for an Interim Meeting
>
>BMWG,
>
...

---

From: Al Morton <[email-masked]>
Date: Fri, 23 Oct 2009 17:39:22 -0400

>not get into the draft in time for the last meeting.

Hi Jan and Benoit,

I have a few comments and suggestions on your draft, below.

Al (as a participant)

============================================================================

Some suggestions for:
>2.2.5 Flow Export Rate
>
>    Definition:
>       Number of Flow Records that expire from the Cache (as defined by
>       the Flow Expiration term) within a time interval and are exported
>       to the Collector.

Number of Flow Records that expire from the Cache (as defined by
the Flow Expiration term) and are exported to the Collector
within a time interval.

>    Discussion:
>
>       The Flow Export Rate is measured using Flow Export data collected

s/collected/observed/

>       at the Collector by counting the exported Flow Records

during the measurement interval.

>. The
>       obtained value is an average of the instant export rates over the
>       measurement interval (see sections 5.1.3 and 5.2.3).

The value obtained is an average of the instantaneous export rates
observed during the measurement interval (see sections 5.1.3 and 5.2.3).

>       ...The lowest
>       possible measurement interval (if the requirement is to measure
>       rather instant DUT load than average load) is limited by the
>       export capabilities of the particular Flow monitoring
>       implementation.

The smallest possible measurement interval (if attempting to measure
instantaneous export rate rather than average export rate on the DUT)
is limited by the export capabilities of the particular Flow monitoring
implementation.

====================================
Suggestion for:
>2.2.6 Cache Overflow
>
>    Definition:
>       Cache Overflow is a state of the Cache when all the available Flow
>       Record entries in the Cache are already used/occupied e.g. there is
>       no space in the Cache for new incoming Flow Records.
... and the new records will be discarded?

====================================
Suggestion for:
>3.  Flow Monitoring Performance Metric
>
>3.1 The Definition
>
>    Flow Monitoring Throughput
>
>    Definition:
>       The maximum Flow Export Rate the DUT can sustain without losing a
>       single Flow Record expired from the Cache and without dropping any
>       forwarded packets.

...and without dropping any packets in the forwarding plane (see Figure 1).

>       The measured Flow Export Rate MUST include the Control
>       Information.

The measured Flow Export Rate MUST include BOTH the Data stream and
the Control Information, as defined in section 2 of [RFC5470].

(maybe this last sentence should be in the Flow Export Rate Definition?)

===============================================
>3.3 Test Concept

(this is a very helpful section now)

>    Traffic in the Figure 1 represents the test traffic sent to the DUT
>    and forwarded by the DUT. When testing devices which do not act as
>    network devices (appliances - probes, sniffers, replicators) the
>    forwarding plane is missing.

suggest:
... the forwarding plane is simply an Observation Point,
as defined in section 2 of [RFC5470].

>    The Flow monitoring enabled (see section 4.3) on the DUT (and
>    represented in the Figure 1 by the Flow Monitoring Plane) uses the
>    traffic information provided by the Forwarding Plane and configured
>    Flow Keys to create the Flow Records representing the traffic
>    forwarded by the DUT.

... forwarded (or observed) by the DUT.

<stuff omitted>
>    The Flow Monitoring Plane handles Flow Records which reflect the
>    forwarded traffic. The metric that measures the Flow monitoring
>    activity is Flow Export Rate.

...performance is Flow Export Rate.

=======================================================
>3.4 The Measurement Procedure Overview
>
>    The measurement procedure is fully specified in sections 4, 5 and 6.
>    This section provides an overview of principles for the measurements.
>
>    The basic measurement procedure of performance characteristics of a
>    DUT with Flow monitoring enabled is a conventional non drop rate

s/non drop rate/Throughput/  (non drop rate is not a defined term in BMWG)

>    measurement with gradually increasing traffic rates until losses are
>    seen as fully specified by [RFC2544].

=============================================================
some typos below:
>3.4.2 Forwarding Plane Performance Measurement
>
>    The Forwarding Plane (see Figure 1) performance metrics are fully
>    specified by [RFC2544] and MUST be measured accordingly. A detailed
>    traffic analysis (see the bullets below) with relation to Flow
>    monitoring must be performed prior of any [RFC2544] measurement.

s/measurement/measurements/

>    Mmainly the Flow Export Rate caused by the test traffic during an

s/Mmainly/Mainly/

>    [RFC2544] measurement MUST be known and noted.

General comment on Forwarding Plane Measurement
(possibly addressed in later sections of the draft):
The test load used for flow monitoring stress
will likely be different from
that specified in RFC2544 - we need to take care that
the Throughput benchmark will still be meaningful,
and than may mean some compromises between traffic that
stresses the Flow Monitoring Plane and the Forwarding Plane.

(stopping at the end of 3.4 for now)

---

From: Al Morton <[email-masked]>
Date: Sun, 25 Oct 2009 17:18:19 -0400

>I have a few comments and suggestions on your draft, below.

4.1 Test Topology
...
    In the test topology with bidirectional traffic (Figure 3), the
    sender and receiver are basically on the same device, which generates
    the traffic to be sent through the DUT, and received back once it
    reaches the ?END?.

Bidirectional Traffic?  I don't remember seeing this in earlier
BMWG testing...

<skipped stuff>

7. Flow Monitoring Accuracy

    The pure Flow monitoring tests in section 5 provide the capability to
    verify the Flow monitoring accuracy in terms of the exported Flow
    Record data. Since every Flow Record created in the Cache is
    populated by just one packet, the full set of captured data on the
    Collector can be parsed (e.g. providing the exported Flow Record
    contents not only the Flow Record count) and each set of parameters
suggestion (if correct):
    ...(e.g. providing the exported Flow Record
    contains info beyond the Flow Record count)
    ^^^^^^^^^^^^^^^^^^^^
...
     The exported Flow Record is considered accurate if:

       a. all the Flow Record fields are present in each exported Flow
          Record
       b. all the Flow Record fields values match the value ranges
          as set by the traffic generator (for example an IP address
          falls within the range of the IP addresses increments on the
          traffic generator)
       c. all the possible Flow Record fields values as defined at the
          traffic generator have been found in the captured export data
          on the Collector. This check needs to be offset to potential
          detected packet losses at the DUT during the test

    If Packet Sampling is deployed then only verifications in point 1
    and 2 can be performed.

I think you meant a. and b. above.

8. Evaluating Flow Monitoring Applicability
...
    It needs to be kept in mind that the above is a very rough and
    averaged Flow activity estimate which cannot account for traffic
    anomalies like large number of for example DNS request packets which
    are typically small packets coming from many different sources and
    represent mostly just one packet Flow.
comment:
Models of Live Traffic have the goal of realism, but also raise
questions as to the meaning of the results...

9. Miscellaneous Tests

    This section lists the tests which could be useful to asses a proper
    Flow monitoring operation under various operational or stress
    conditions. These tests are not deemed suitable for any benchmarking
    for various reasons.
comment:
Then this should probably be an Appendix.

12. Security Considerations
Thanks for using the "standard" text here.

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Mon, 26 Oct 2009 11:38:19 +0100

Hi Al,

Thanks a lot for your review comments - most of them seem easy
to address, just two/three need some more thinking/discussion:

1) bidir traffic - it somehow naturally evolved from the way
   flow monitoring is used/tested - what would be your advice
   regarding that please ?? restrict it for only flow monitoring tests
   or drop it at all ??
  
2) cache overflow - there are few controversial points in the
   two cache modes tests - one of which you picked up by your
   comment :-) - (which gradually surfaced during some
   discussions about HW platform testing and testing of the
   normal cache mode) - recently the idea here actually was to simplify
   the whole thing and specify just one set of tests without 
   distinguishing the two modes - in principle all what needs to
   be seen is Flow Export data and when the DUT starts to lose
   some Flow records and not the internals under which loses happen
.....

3) your general comment - yes thanks - also realised later that section
   6 needs to reflect more of the discussion in section 3 and 
   the Stockholm meeting

Tx, Jan 

                                 Dr. Johnson  

> From: Al Morton [mailto:[email-masked]]
> Sent: 25 October 2009 21:18
> To: Jan Novak (janovak); [email-masked]; Benoit Claise (bclaise)
> Subject: Re: comments on draft-novak-bmwg-ipflow-meth-04
>

---

From: Benoit Claise <[email-masked]>
Date: Wed, 28 Oct 2009 16:35:55 +0100

Al,

Thanks for your review.
We'll address those comments in the next version of draft.
The following comment will require some more thoughts.

> stresses the Flow Monitoring Plane and the Forwarding Plane.
Regards, Benoit.

---

From: Al Morton <[email-masked]>
Date: Wed, 28 Oct 2009 17:11:33 -0400

>The following comment will require some more thoughts.

Maybe we can prompt some discussion of this on Friday,
at the meeting...
Al