Subject: Re: [bmwg] Revised Draft Text for Re-Chartering

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Tue, 3 Apr 2018 17:56:35 +0000

Hi Warren,

The comment week closed on the 21st, and there were
no comments in overtime :-) so we can comfortably declare 
WG consensus on the revised text and milestones and
pass to you for AD review and Last Call/IESG.

<clean text file attached>

Thanks,
Al
bmwg co-chair

> -----Original Message-----
> From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: Wednesday, March 21, 2018 3:41 PM
> To: [email-masked]
> Subject: Re: [bmwg] Revised Draft Text for Re-Chartering
> 
> *** Security Advisory: This Message Originated Outside of AT&T ***.
> Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.
> 
> BMWG,
> 
> We have slightly revised the Proposed Milestones as part of
> Re-Charter Review in our IETF-101 session.
> 
> Revised proposal is below.
> 
> Final Comment period is one week, by March 21st, 2018, to the
> bmwg-list please.
> 
> regards,
> Al
> bmwg co-chair
> 
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> BMWG - Benchmarking Methodology Working Group  (as of March 20, 2018)
> 
> Description of Working Group:
> 
>   The Benchmarking Methodology Working Group (BMWG) will continue to
>   produce a series of recommendations concerning the key performance
>   characteristics of internetworking technologies, or benchmarks for
>   network devices, systems, and services. Taking a view of networking
>   divided into planes, the scope of work includes benchmarks for the
>   management, control, and forwarding planes.
> 
>   The scope of BMWG has been extended to develop methods for
>   virtual network functions (VNF) and their unique supporting
>   infrastructure (such as SDN Controllers and vSwitches).
>   Benchmarks for platform capacity and performance characteristics
>   of virtual routers, firewalls (and other security functions),
>   signaling control gateways, and other forms of gateways are included.
>   The benchmarks will foster comparisons between physical and virtual
>   network functions, and also cover unique features of
>   Network Function Virtualization systems. Also, with the emergence
>   of virtualized test systems, specifications for test system
>   calibration are also in-scope.
> 
>   Each recommendation will describe the class of network function, system,
>   or service being addressed; discuss the performance characteristics that
>   are pertinent to that class; clearly identify a set of metrics that aid
>   in the description of those characteristics; specify the methodologies
>   required to collect said metrics; and lastly, present the requirements
>   for the common, unambiguous reporting of benchmarking results.
> 
>   The set of relevant benchmarks will be developed with input from the
>   community of users (e.g., network operators and testing organizations)
>   and from those affected by the benchmarks when they are published
>   (networking and test equipment manufacturers). When possible, the
>   benchmarks and other terminologies will be developed jointly with
>   organizations that are willing to share their expertise. Joint review
>   requirements for a specific work area will be included in the
>   description of the task.
> 
>   To better distinguish the BMWG from other measurement initiatives in the
>   IETF, the scope of the BMWG is limited to the characterization of
>   implementations of various internetworking technologies
>   using controlled stimuli in a laboratory environment. Said differently,
>   the BMWG does not attempt to produce benchmarks for live, operational
>   networks. Moreover, the benchmarks produced by this WG shall strive to
>   be vendor independent or otherwise have universal applicability to a
>   given technology class.
> 
>   Because the demands of a particular technology may vary from deployment
>   to deployment, a specific non-goal of the Working Group is to define
>   acceptance criteria or performance requirements.
> 
>   An ongoing task is to provide a forum for development and
>   advancement of measurements which provide insight on the
>   capabilities and operation of implementations of inter-networking
>   technology. Ideally, BMWG should communicate with the operations
>   community through organizations such as NANOG, RIPE, and APRICOT.
> 
> 
> Goals and Milestones: (preliminary list, we have an active draft for all
> topics)
> 
>   Aug 2018 - Methodology for Next-Gen Firewall Benchmarking to IESG Review
>   Dec 2018 - Update to RFC2544 Back-to-back Frame Benchmarking to IESG
> Review
>   Dec 2019 - Methodology for EVPN Benchmarking to IESG Review
>   Dec 2019 - Considerations for Benchmarking Network Virtualization
> Platforms
>              to IESG Review
>   Dec 2019 - Draft on Network Service Layer Abstract Models applied to
>              Benchmarking to IESG Review
>   Dec 2019 - Draft on General VNF Benchmarking Automation to IESG Review
> 
> 
> > -----Original Message-----
> > From: bmwg [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C
> > (AL)
> > Sent: Thursday, March 1, 2018 10:13 AM
> > To: [email-masked]
> > Subject: [bmwg] Revised Draft Text for Re-Chartering
> >
> > *** Security Advisory: This Message Originated Outside of AT&T ***.
> > Reference http://cso.att.com/EmailSecurity/IDSP.html for more
> information.
> >
> > BMWG,
> >
> > We have slightly revised text to review during our
> > Virtual Interim Meeting today (the first draft text
> > was sent to bmwg-list in October, 2017).
> > I'll summarize the changes during the meeting.
> >
> > regards,
> > Al
> > bmwg-co-chair
> >
> > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >
> > BMWG - Benchmarking Methodology Working Group
> >
> > Description of Working Group:
> >
> >   The Benchmarking Methodology Working Group (BMWG) will continue to
> >   produce a series of recommendations concerning the key performance
> >   characteristics of internetworking technologies, or benchmarks for
> >   network devices, systems, and services. Taking a view of networking
> >   divided into planes, the scope of work includes benchmarks for the
> >   management, control, and forwarding planes.
> >
> >   The scope of BMWG has been extended to develop methods for
> >   virtual network functions (VNF) and their unique supporting
> >   infrastructure (such as SDN Controllers and vSwitches).
> >   Benchmarks for platform capacity and performance characteristics
> >   of virtual routers, firewalls (and other security functions),
> >   signaling control gateways, and other forms of gateways are included.
> >   The benchmarks will foster comparisons between physical and virtual
> >   network functions, and also cover unique features of
> >   Network Function Virtualization systems. Also, with the emergence
> >   of virtualized test systems, specifications for test system
> >   calibration are also in-scope.
> >
> >   Each recommendation will describe the class of network function,
> system,
> >   or service being addressed; discuss the performance characteristics
> that
> >   are pertinent to that class; clearly identify a set of metrics that
> aid
> >   in the description of those characteristics; specify the methodologies
> >   required to collect said metrics; and lastly, present the requirements
> >   for the common, unambiguous reporting of benchmarking results.
> >
> >   The set of relevant benchmarks will be developed with input from the
> >   community of users (e.g., network operators and testing organizations)
> >   and from those affected by the benchmarks when they are published
> >   (networking and test equipment manufacturers). When possible, the
> >   benchmarks and other terminologies will be developed jointly with
> >   organizations that are willing to share their expertise. Joint review
> >   requirements for a specific work area will be included in the
> >   description of the task.
> >
> >   To better distinguish the BMWG from other measurement initiatives in
> the
> >   IETF, the scope of the BMWG is limited to the characterization of
> >   implementations of various internetworking technologies
> >   using controlled stimuli in a laboratory environment. Said
> differently,
> >   the BMWG does not attempt to produce benchmarks for live, operational
> >   networks. Moreover, the benchmarks produced by this WG shall strive to
> >   be vendor independent or otherwise have universal applicability to a
> >   given technology class.
> >
> >   Because the demands of a particular technology may vary from
> deployment
> >   to deployment, a specific non-goal of the Working Group is to define
> >   acceptance criteria or performance requirements.
> >
> >   An ongoing task is to provide a forum for development and
> >   advancement of measurements which provide insight on the
> >   capabilities and operation of implementations of inter-networking
> >   technology. Ideally, BMWG should communicate with the operations
> >   community through organizations such as NANOG, RIPE, and APRICOT.
> >
> >
> > Goals and Milestones: (preliminary list, we have an active draft for all
> > topics)
> >
> >   Dec 2018 - Methodology for Next-Gen Firewall Benchmarking to IESG
> Review
> >   Dec 2018 - Update to RFC2544 Back-to-back Frame Benchmarking to IESG
> > Review
> >   Dec 2019 - Methodology for EVPN Benchmarking to IESG Review
> >   Dec 2019 - Considerations for Benchmarking Network Virtualization
> > Platforms
> >              to IESG Review
> >   Dec 2019 - Draft on Network Service Layer Abstract Models applied to
> >              Benchmarking to IESG Review
> >   Dec 2019 - Draft on Automated VNF Benchmarking to IESG Review
> >
> > _______________________________________________
> > bmwg mailing list
> > [email-masked]
> > https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ-
> >
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=lB6OG6NJlVPcNq_hFaAplcHHWP5AQ
> > v2kMcsxYDTMFDo&s=xE6icgQqHQn-lu-aS00Fu45JJF6kvx92rmWZFwP1XFM&e=
> 
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__www.ietf.org_mailman_listinfo_bmwg&d=DwICAg&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=C2slT6LxJ9UfLkLuYslijFwaU0UXD
> 4RGtuk9YWX1P9s&s=jfXw9LmZxlYJ12OfFN-txwQeLTKJVguCt4E24TqI8bs&e=