Subject: Corrected BMWG minutes

From: [email-masked] (Kevin Dubray)
Date: Thu, 30 Apr 1998 13:24:13 -0400

Please find below a revised version of the BMWG minutes.

Changes to the original minutes can be summarized as follows:

  Section 2, paragraph 3:    

     Wording corrected to reflect the author's belief that the definition
     of firewall is complete.

  Section 2, paragraph 4:
   
     Corrected omission by presenting a clarifying thought as to why 
     "forwarding rate" differs in the context of the current firewall 
     draft vs. RFC 2285.

  Section 2, paragraph 9:

     Changed wording from "less vendor related" to "vendor-neutral."

-Kevin

----------------------------------------------------------------------------
Minutes of the Benchmarking Methodology Working Group (BMWG)

[These minutes have been edited.]

Reported by Kevin Dubray

Kevin Dubray opened the Benchmarking Methodology working group session.
Upon regarding the light turnout (about 30) he commented that holding a
session in the last meeting slot on the last day before the Plenary might
not be such a good thing...

Light turnout notwithstanding, Dubray presented the agenda for the
41st IETF BMWG meeting.
 
Agenda:

   1. Administration
         1.1  Agenda Bashing
         1.2  RFC 2285

   2. Benchmarking Terminology of Firewall Performance

   3. Terminology for IP Multicast Benchmarking

   4. Goals for Next Period.

1. Administration.

Dubray announced that this would be an abbreviated session since
unexpected scheduling changes forced some speakers to amend their plans.

David Newman asked for an item on the agenda to be added to ascertain
interest in a new work item.  The chair amended the agenda to reflect this
change.

The chair announced that the work formerly known as the LAN Switching
Benchmarking Terminology draft was approved by the IESG as an Informational
Request for Comments.  The RFC editor designated this work as RFC 2285.  
Dubray thanked the editor, Bob Mandeville, and the other people who had 
contributed to the draft.

 2. Benchmarking Terminology of Firewall Performance

David Newman was introduced to lead a discussion on the Firewall 
Performance Benchmarking draft.  David began with a quick overview
of the changes to the draft.  He describe the two major changes revolving
around the notion of:

      A. Connection
      B. Forwarding Rate

Newman then engaged in a detailed discussion of the changes to this 
latest version of the draft.  He noted that the term "connection" was
changed pursuant to the comments at the Washington DC IETF.  He noted 
that the term "session" was modified as well.  David noted that given
the relationship between the two terms, it might be a good idea to
combine the two at some point.

He went on to say that the concept of "external" was modified to reflect
the idea of an "unprotected" network segment.  David said the draft
also offered a definition for the term, "Firewall."  David conveyed that
because of value-added features sometimes supplemented to firewalls, the 
definition might not seem encompassing. These supplemental features
nothwithstanding, David feels the firewall definition is complete and 
historically accurate.

In presenting the term "forwarding rate," Newman articulated that
he believed it was significantly different than the term in RFC 2285.
A comment was offered that, at first glance, the only seemingly different
notions were that of the measurement units "bits" versus "frames" per 
second and that of the more specific term "firewall" in lieu of "DUT/SUT".
The follow-up question then asked how different were the notions of
"Forwarding Rate" in this draft over RFC 2285?  Newman thought that
they were substantially different - a frame-based metric is not 
meaningful in the context of an application layer device like a firewall.
A trailing statement remarked that the two terms were ambiguous outside 
the context of the separate discussion.

Newman acknowledged that the overlap was troubling.  He asked for 
suggestions.  One suggestion offered changing the name to something more
firewall specific - perhaps a "Firewall Forwarding Rate" metric.  Another 
suggestion was made to align the metric's title closer to the primary
difference, "bit forwarding."  This was met with general approval 
excepting a caveat that the term not necessarily be restricted to just 
firewalls.  David asked for clarification of what was the consensus, 
firewall forwarding rate or bit forwarding rate.  The generic 
bit-forwarding term was reaffirmed.

Upon visiting the term "Session," a discussion ensued regarding the
differentiation between the terms session and connection.  Dubray 
commented that earlier in the session, Newman referred to Session as a 
State - "data flowing" - though the term "state" was not expressly used
in the draft.  Dubray thought it might be useful to explicitly define
session as a state.  For example, "A state in which higher level
transactions can occur or is occurring between associated peers."
  
It was further noted that Newman verbally referred to a connection as a 
state, too.  The draft, however, refers to connection not as a state, but 
as a "path."  A question was posed as to whether the draft, in defining 
connection and session, was attempting to define a hierarchy between a 
data path and data transfer state. The example was offered that many 
sessions could be offered over a single connection. David reiterated his 
view that a "connection," as defined, was as state - communicating a 
hierarchy was not a goal.  David expressed that he viewed connection as a 
table entry in a device.

[At the meeting's conclusion, Kim Martin offered that a single, unifying 
concept - circuit, could be perhaps used.  It's historical use is 
consistent to the tone of the discussion.   This was noted by Mr. Newman
as a good idea.]

David then moved the discussion to the modifications surrounding stateful
inspection versus stateful packet filtering.  He noted the wording was
changed, per the DC meeting, to be vendor-neutral.

As the discussion moved on to security considerations, a question was
offered regarding the merits of assessing firewall functionality.  David 
strongly stated that is was not the draft's intention to provide device 
validation or other verification-style tests.  A voice was heard to
say that an understanding could not be gathered on whether performance 
may compromise functionality.

A general question was poised to Mr. Newman.  The query went along the
lines of what level did Mr. Newman perceived the end to end testing? 
Layer 3? Layer 4?  Firewall functionality vs. Forwarding path?  David
communicated, "all the above."

In closing the discussion, final questions were offered Mr. Newman.
In his opening remarks, Mr. Newman referred to two major metrics of
the draft: a) forwarding rate, and b) connection.  Forwarding rate
was straightforward, but how does one make a metric of a data path
or state?  David said that you could measure the number of concurrent
connections.  The person posing the question articulated that number of
connections seems more like a measurement unit than a metric.  Using
connection as a metric may be ambiguous.  David acknowledged the
comment.  David thanked the group for its input.

3. Terminology for IP Multicast Benchmarking

Kevin Dubray began the discussion of <draft-ietf-bmwg-mcast-03.txt> with
a quick overview of the draft's initiatives and areas of concentration.
He noted that the review at this meeting marked the coverage of all 
the metrics presented by the Multicast Terminology draft.
 
He then gave a high level summary of changes to the current draft.  Two
areas of change Dubray focused on was a) the notion of "interaction," 
and b) the fact that "Fairness" was removed from the draft.  In regards
to the latter, Dubray articulated that Fairness was removed from the
draft in a direct response to queries regarding QoS as being a possible 
area of future BMWG work.

The section on Interaction was added as a direct response to input from
the Washington, D.C. BMWG meeting and input from the BMWG mailing list.

Dubray presented the metric Encapsulation Throughput.  Dubray indicated 
that Decapsulation Throughput and Re-encapsulation throughput were defined
similarly.  When a question was posed as to why three separate metrics were
needed, Kevin pointed out that these individual metrics were presented as 
one metric, Translational Throughput in the last draft.  Dubray said it
was intended that the methodology be the defining factor.  Dubray indicated
that the working group thought it would be better to define three discrete
metrics.  The members of this session reaffirmed the previous wisdom. 

With old business behind, Dubray ensued in a discussion of the last 
sections of the draft:  Overhead, Capacity, and Interaction.

Dubray indicated that feedback from the mailing list indicated three
thoughts, not necessarily all predominant:

      A. Additional factors contributing to this metric should be 
         identified.
      B. Crisper definition as to start/stop clocking-type events.
      C. Join/Leave Delay metrics are more "interesting" in the presence 
         of other factors, such as Offered Load.
      D. The notion of "Time duration" is generically vague.

Dubray presented the wording to relieve the condition cited in A.  The
group offered no dissenting views.   Dubray communicated his thoughts 
that items B and D were better suited for the corresponding methodology 
document.  The group agreed.  He thought that the associated "burdened 
response" metric to be presented later would raise the interest index 
cited in condition C. 
 
On Group Join Delay (3.4.1), there was a question as to what constitutes
the successful transaction.  Kevin thought it was a good question - but,
again, thought it was a question better answered in the methodology
document.  The group agreed.

The group further concurred that the additions of Interaction (3.6) 
and Burdened Response (3.6.1) were welcome to the draft.

On the topic of Forwarding-Burdened Multicast Latency, there was
a question regarding the nature of the metric.  Was the goal to
parameterized the latency metric?  Dubray said yes, especially with
respect to load and associated forwarding rate.  An additional question
queried as to what should be gleaned from the metric.  Dubray indicated
the intent was not to limit the statistical information obtained from
the measurement.  So, from the set of n latencies, the appropriate
statistics could be collected.

There was little discussion on the remaining part of the presentation.

With that, Dubray announced that the major parts of the draft had been
presented and discussed.  He asked if there were any issues.  A question 
was raised on the Multicast Group Capacity metric.  While the metric
tried to identify the maximum number of multicast groups a device can
support, it did so with the qualification that traffic be correctly 
forwarded.  Specifically, what level/rate of traffic needed to be 
processed to indicate goodness.

Dubray indicated that he didn't know.  Moreover, he didn't think the 
actual rate was important.  He thought it was more important to
ascertain the correct delivery.  Regardless, he thought it was an issue 
best addressed by the corresponding methodology document.

Dubray asked that if the changes suggested were made, would the document
be ready for final review.  There were no dissenting reviews. 

Dubray then invited David Newman to present the new work item. 

David Newman said that he wished to make a proposal for Bob Mandeville,
who was unable to make the meeting.  David informed the group that Bob 
was hoping to do some work in the QoS area using latency as vehicle. 
David asked if there was interest.  The group indicated that there was.  
Someone suggested that the work would be more useful if it were
expanded beyond latency.  Others agreed.  The chair asked that
Mandeville submit a workplan for WG consideration.

Interest was again indicated in the area of the Cell/Call benchmarking
draft.  Dubray indicated that Raj Jain had hoped to attend, but he
was called away at last minute.  Dubray indicated that he anticipated
a rise in activity in this direction.  In the interim, he asked that
folks start to open the discussion on the BMWG mailing list.  

With no new business, Dubray reviewed the next period's goals:

    1.  Advance the Firewall Draft; get it in the shape for
        final review.

    2.  Prepare the Multicast draft for final review.

    3.  Deliver a LAN Switch Methodology draft to be a companion 
        document to RFC 2285.

    4.  Revise and resubmit a Call/Cell terminology draft.
  
    5.  Review a QoS benchmarking workplan, if submitted.