Subject: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

From: Simon Edwards <[email-masked]>
Date: Wed, 3 Jun 2020 09:10:10 +0000

Hi all,

In a number of sections, but specifically '7.8.3.2.  Test Equipment Configuration Parameters', there are requirements to measure with 50% of the maximum connections/ sec measured in the HTTP/S throughput tests

E.g. "Target objective for scenarios 1 and 2: 50% of the maximum connections per second measured in test scenario..."

I'm sure this 50% value is the product of much thought and discussion, rather than an arbitrary choice. Is anyone able to explain the reason for the specific '50%' value (as opposed to 25%, 75% or whatever) or could you please point to documentation around that decision made by the group?

I'm asking just to understand. I don't disagree with the decision : )

Very best wishes,
Simon

---

From: <[email-masked]>
Date: Wed, 3 Jun 2020 10:55:41 -0400

Simon,

This requirement was agreed to and adopted by the working group within
NetSecOPEN. It first appeared in the IETF individual draft on October 14,
2018.
(https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05). It
was clarified and expanded on March 5, 2019 in
https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00. It's form
has largely been unchanged since then.

I will leave it to the technical folks to expand on this more. However, I am
saying all of this because it has been in the position to be reviewed and
commented on by the BMWG community for awhile. Our assumption is that given
no one appears to have an issue with it that we hit the mark.

Brian

From: bmwg <[email-masked]> On Behalf Of Simon Edwards
Sent: June 3, 2020 5:10 AM
To: [email-masked]
Subject: [bmwg] Query about 50% values in [Benchmarking Methodology for
Network Security Device Performance draft 02]

Hi all,

In a number of sections, but specifically '7.8.3.2.  Test Equipment
Configuration Parameters', there are requirements to measure with 50% of the
maximum connections/ sec measured in the HTTP/S throughput tests

E.g. "Target objective for scenarios 1 and 2: 50% of the maximum connections
per second measured in test scenario..."

I'm sure this 50% value is the product of much thought and discussion,
rather than an arbitrary choice. Is anyone able to explain the reason for
the specific '50%' value (as opposed to 25%, 75% or whatever) or could you
please point to documentation around that decision made by the group?

Very best wishes,

Simon

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 3 Jun 2020 16:12:27 +0000

Thanks for your question, Simon, and the history, Brian.

I confess that this question (why 50%?) has occurred to me in other contexts, and it may help to add a sentence to two of rationale. So if the technical folks can help with suggestions, that would be great!

regards,
Al

From: bmwg [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Wednesday, June 3, 2020 10:56 AM
To: 'Simon Edwards' <[email-masked]>
Cc: [email-masked]
Subject: Re: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Simon,

This requirement was agreed to and adopted by the working group within NetSecOPEN. It first appeared in the IETF individual draft on October 14, 2018. (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>). It was clarified and expanded on March 5, 2019 in https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>. It's form has largely been unchanged since then.

I will leave it to the technical folks to expand on this more. However, I am saying all of this because it has been in the position to be reviewed and commented on by the BMWG community for awhile. Our assumption is that given no one appears to have an issue with it that we hit the mark.

Brian

From: bmwg <[email-masked]<mailto:[email-masked]>> On Behalf Of Simon Edwards
Sent: June 3, 2020 5:10 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Hi all,

Very best wishes,
Simon

---

From: Brian Monkman <[email-masked]>
Date: Wed, 3 Jun 2020 16:20:51 +0000

Hi Al,

Are there any other items that caused you pause that could be addressed the same way?

Brian

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: MORTON, ALFRED C (AL) <[email-masked]>
Sent: Wednesday, June 3, 2020 12:12:27 PM
To: [email-masked] <[email-masked]>; 'Simon Edwards' <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: RE: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

regards,

Al

Simon,

This requirement was agreed to and adopted by the working group within NetSecOPEN. It first appeared in the IETF individual draft on October 14, 2018. (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>). It was clarified and expanded on March 5, 2019 in https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>. It’s form has largely been unchanged since then.

Brian

Hi all,

Very best wishes,

Simon

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 3 Jun 2020 17:04:27 +0000

Hi Brian,  no other pauses or speed-bumps yet, but I need to complete my
review of the most recent version.

sorry, I know that doesn't help!
Al

From: Brian Monkman [mailto:[email-masked]]
Sent: Wednesday, June 3, 2020 12:21 PM
To: MORTON, ALFRED C (AL) <[email-masked]>; 'Simon Edwards' <[email-masked]>
Cc: [email-masked]
Subject: Re: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Hi Al,

Brian

Get Outlook for iOS<https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_o0ukef&d=DwMF-g&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=gwfQJfXOl1ZubaA9a-aKfQ8n5AY2sKsb3Ml9NAklRwQ&s=ai2z_d2vwJeZx_EibZkLkoE_ZWS26ZSoAJq9RZNf6bM&e=>
________________________________
From: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, June 3, 2020 12:12:27 PM
To: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>; 'Simon Edwards' <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
Subject: RE: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

regards,

Al

From: bmwg [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, June 3, 2020 10:56 AM
To: 'Simon Edwards' <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Simon,

Brian

Hi all,

Very best wishes,

Simon

---

From: <[email-masked]>
Date: Wed, 3 Jun 2020 13:34:39 -0400

It doesn’t hurt either. 😊

Should we hold off until you complete your review before we submit a new draft? I am hoping others weigh in on this as well.

Brian

From: MORTON, ALFRED C (AL) <[email-masked]> 
Sent: June 3, 2020 1:04 PM
To: Brian Monkman <[email-masked]>; 'Simon Edwards' <[email-masked]>
Cc: [email-masked]
Subject: RE: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

sorry, I know that doesn’t help!

Al

From: Brian Monkman [mailto:[email-masked]] 
Sent: Wednesday, June 3, 2020 12:21 PM
To: MORTON, ALFRED C (AL) <[email-masked] <mailto:[email-masked]> >; 'Simon Edwards' <[email-masked] <mailto:[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]> 
Subject: Re: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Hi Al,

Brian

Get Outlook for iOS <https://urldefense.proofpoint.com/v2/url?u=https-3A__aka.ms_o0ukef&d=DwMF-g&c=LFYZ-o9_HUMeMTSQicvjIg&r=_6cen3Hn-e_hOm0BhY7aIpA58dd19Z9qGQsr8-6zYMI&m=gwfQJfXOl1ZubaA9a-aKfQ8n5AY2sKsb3Ml9NAklRwQ&s=ai2z_d2vwJeZx_EibZkLkoE_ZWS26ZSoAJq9RZNf6bM&e=> 

  _____  

From: MORTON, ALFRED C (AL) <[email-masked] <mailto:[email-masked]> >
Sent: Wednesday, June 3, 2020 12:12:27 PM
To: [email-masked] <mailto:[email-masked]>  <[email-masked] <mailto:[email-masked]> >; 'Simon Edwards' <[email-masked] <mailto:[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]>  <[email-masked] <mailto:[email-masked]> >
Subject: RE: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02] 

regards,

Al

From: bmwg [mailto:[email-masked]] On Behalf Of [email-masked] <mailto:[email-masked]> 
Sent: Wednesday, June 3, 2020 10:56 AM
To: 'Simon Edwards' <[email-masked] <mailto:[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]> 
Subject: Re: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Simon,

This requirement was agreed to and adopted by the working group within NetSecOPEN. It first appeared in the IETF individual draft on October 14, 2018. (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05 <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=> ). It was clarified and expanded on March 5, 2019 in https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00 <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=> . It’s form has largely been unchanged since then.

Brian

From: bmwg <[email-masked] <mailto:[email-masked]> > On Behalf Of Simon Edwards
Sent: June 3, 2020 5:10 AM
To: [email-masked] <mailto:[email-masked]> 
Subject: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Hi all,

Very best wishes,

Simon

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 3 Jun 2020 18:51:34 +0000

It’s beginning to look like I won’t complete a review this week, and I have already promised to review a doc over the weekend before a Monday deadline. So, bottom line is that I’m with you, hoping others weigh-in with reviews to help!

Al

From: bmwg [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Wednesday, June 3, 2020 1:35 PM
To: MORTON, ALFRED C (AL) <[email-masked]>; 'Simon Edwards' <[email-masked]>
Cc: [email-masked]
Subject: Re: [bmwg] Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Brian

To: Brian Monkman <[email-masked]<mailto:[email-masked]>>; 'Simon Edwards' <[email-masked]<mailto:[email-masked]>>

To: MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>>; 'Simon Edwards' <[email-masked]<mailto:[email-masked]>>

Hi Al,

Brian

regards,

Al

Simon,

This requirement was agreed to and adopted by the working group within NetSecOPEN. It first appeared in the IETF individual draft on October 14, 2018. (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf..org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>). It was clarified and expanded on March 5, 2019 in https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf..org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>. It’s form has largely been unchanged since then.

Brian

Hi all,

Very best wishes,

Simon

---

From: Alex Samonte <[email-masked]>
Date: Wed, 3 Jun 2020 12:53:53 -0700

The main rationale is to see what the nominal latency is.

As we all know the latency, will rise dramatically (and be unpredictable)
when something gets close to bottlenecking.  That doesn't mean things will
fail, but you start getting ugly behavior.

from 0%-XX% the latency is fairly reliable (XX to be guessed at later).
This means the latency doesn't change dramatically as you increase that
percentage.  Usually it's something linear with a slope of much less than
1.    When you get to XX%-100%, the latency becomes unstable and it varies
wildly as various components of the system have a bottleneck.  When that
happens, other things compensate, and then they become a bottleneck (when
the first thing is not) and it oscillates all over the place.  This is what
leads to the unstable latency.  The difference of the latency at 1% vs XX%
is small compared to the latency at XX% and some of the unstable latency at
100%.

So the question is how to we determine XX%?   Unfortunately, all DUTs are
different, have different limits, bottlenecks, and features that may change
XX%.   It's not the same from vendor to vendor, nor even model to model
within the same vendor.

A couple big examples to point out.
1) multi core CPUs.  Because of the way that most stateful devices behave,
a single session will probably be handled by a single core/thread of a
CPU.  If a device has 8 cores, if there is only 1 session, 7 cores will
likely go unused.   If there are 9 sessions, 1 core will be handling 2
sessions, and the others will be handling 1 (50% difference).  If there are
8001 sessions, then it's 1001, and others 1000, and the uneveness does not
matter.   However, if one of those sessions is 'harder' or more difficult
to process.  Or the number of sessions on a single core may be harder to
process, that core may reach 100% capacity before the others do.   Lets'
say I have 100%, 50%, 50%, 50% in a 4 core case.   Here we will experience
unstable latency.  The sessions that go through the core at 100% will
experience additional latency, where the sessions going through the other 3
cores will not, leading to unstable latency.

2) internal buffers.  Depending on the protocol different protocols may
have different buffers associated with them, so when buffers fill up, more
latency is added, but because they can be differently allocated for
different protocols, you will see variable latency depending on which is
exhausted and which is not.

3) interface (internal or external) limits.   When you approach 100% of a
certain speed link (be it internal or external to the DUT) you can also
experience additional latency.  It's hard to determine what a particular
DUT's internal architecture is(nor do we really want to try to figure that
out), So we generally do not want to push it to 100% because we know we
will incur

So what we've said is that:
0%-XX% the latencies are reliable / not unstable
XX%-100% the latency are unstable

People looking to benchmark are generally interested in the largest stable
value (XX).   100% (or near it) is generally not a representative value of
the normal case of traffic.   0% (or 1%) is also generally not
representative of the normal case for traffic.

Ideally XX% is what we're trying to get, but it's different for each
model.   Since 0-XX% represents the stable side, i'd rather err on that
side, than the XX%-100%.

When doing the benchmark we're finding the various max values (without
failure), but that does not necessarily mean stable latency.   In order to
get ot that range we're going to back off from 100%.

Now what could that value be?  95%  75%  50%?   This was purposefully set
low so we would not encounter any other bottlenecks that would make the
latency unstable.

I've tested lots of different devices over the years, both my own and other
vendors, and the lower you make that number, the higher the number of
devices will stay in the 'stable' area of latency.

so 50% was chosen as the least likely to cause unstable latency, and still
representative of real customer usage (plenty of customers have no issue of
using a device that is at 50% capacity).   At 75% and much more so at 95%
customers are looking to change/upgrade their equipment so they are not
nearing the top of what the device is capable of.   So again it makes me
less interested in the latency there.

Hopefully that makes sense, and while it could be shorter in the
explanation, i'm not sure I could make it into a 2 sentence one.

-Alex

On Wed, Jun 3, 2020 at 9:12 AM MORTON, ALFRED C (AL) <[email-masked]>
wrote:

> I confess that this question (why 50%?) has occurred to me in other
> contexts, and it may help to add a sentence to two of rationale. So if the
> technical folks can help with suggestions, that would be great!
>
>
>
> regards,
>
> Al
>
>
>
> *From:* bmwg [mailto:[email-masked]] *On Behalf Of *
> [email-masked]
> *Sent:* Wednesday, June 3, 2020 10:56 AM
> *To:* 'Simon Edwards' <[email-masked]>
> *Cc:* [email-masked]
> *Subject:* Re: [bmwg] Query about 50% values in [Benchmarking Methodology
> for Network Security Device Performance draft 02]
>
>
>
> Simon,
>
>
>
> 2018. (
> https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05
> <https://urldefense..proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>).
> It was clarified and expanded on March 5, 2019 in
> https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>.
> It’s form has largely been unchanged since then.
>
>
>
> I will leave it to the technical folks to expand on this more. However, I
> am saying all of this because it has been in the position to be reviewed
> and commented on by the BMWG community for awhile. Our assumption is that
> given no one appears to have an issue with it that we hit the mark.
>
>
>
> Brian
>
>
>
> *From:* bmwg <[email-masked]> *On Behalf Of *Simon Edwards
> *Sent:* June 3, 2020 5:10 AM
> *To:* [email-masked]
> *Subject:* [bmwg] Query about 50% values in [Benchmarking Methodology for
> Configuration Parameters', there are requirements to measure with 50% of
> the maximum connections/ sec measured in the HTTP/S throughput tests
>
>
>
> E.g. "Target objective for scenarios 1 and 2: 50% of the maximum
> connections per second measured in test scenario..."
>
>
>
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg
>

-- 

Alex Samonte
Director Of Technical Architecture

<http://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=fortinet-logo>
E: [email-masked]
M: +1 408-475-8737
Skype: asamonteFN
www.fortinet.com
<https://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr-company-url&utm_campaign=sd-wan>
<https://twitter.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-twitter>
<http://www.linkedin.com/company/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-linkedin>
<http://www.facebook.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-facebook>
<https://www.instagram.com/behindthefirewall/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-instagram>
<https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog>[image:
Secure Remote Access for Your Workforce at Scale]
<http://signatures.fortinet.com/uc/5e868f6230170c1cb4a71ffb>

***  Please note that this message and any attachments may contain confidential and proprietary material and information and are intended only for the use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any review, use, disclosure, dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this email in error, please immediately notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed. Please also note that any views, opinions, conclusions or commitments expressed in this message are those of the individual sender and do not necessarily reflect the views of Fortinet, Inc., its affiliates, and emails are not binding on Fortinet and only a writing manually signed by Fortinet's General Counsel can be a binding commitment of Fortinet to Fortinet's customers or partners. Thank you. ***

---

From: Gábor LENCSE <[email-masked]>
Date: Wed, 3 Jun 2020 22:53:56 +0200

Dear Alex Samonte,

Is it absolutely necessary to use only a single value? Perhaps using 
multiple latency values at a few different working points (like 25%, 
50%, 75%) could give more information about the behavior of the system.

Best regards,

Gábor Lencse

On 6/3/2020 21:53, Alex Samonte wrote:
> The main rationale is to see what the nominal latency is..
>
> As we all know the latency, will rise dramatically (and be 
> unpredictable) when something gets close to bottlenecking. That 
> doesn't mean things will fail, but you start getting ugly behavior.
>
> from 0%-XX% the latency is fairly reliable (XX to be guessed at 
> later).  This means the latency doesn't change dramatically as you 
> increase that percentage.  Usually it's something linear with a slope 
> of much less than 1.    When you get to XX%-100%, the latency becomes 
> unstable and it varies wildly as various components of the system have 
> a bottleneck. When that happens, other things compensate, and then 
> they become a bottleneck (when the first thing is not) and it 
> oscillates all over the place.  This is what leads to the unstable 
> latency.  The difference of the latency at 1% vs XX% is small compared 
> to the latency at XX% and some of the unstable latency at 100%.
>
> So the question is how to we determine XX%?  Unfortunately, all DUTs 
> are different, have different limits, bottlenecks, and features that 
> may change XX%.   It's not the same from vendor to vendor, nor even 
> model to model within the same vendor.
>
> 1) multi core CPUs.  Because of the way that most stateful devices 
> behave, a single session will probably be handled by a single 
> core/thread of a CPU.  If a device has 8 cores, if there is only 1 
> session, 7 cores will likely go unused.   If there are 9 sessions, 1 
> core will be handling 2 sessions, and the others will be handling 1 
> (50% difference).  If there are 8001 sessions, then it's 1001, and 
> others 1000, and the uneveness does not matter.   However, if one of 
> those sessions is 'harder' or more difficult to process.  Or the 
> number of sessions on a single core may be harder to process, that 
> core may reach 100% capacity before the others do.   Lets' say I have 
> 100%, 50%, 50%, 50% in a 4 core case.   Here we will experience 
> experience additional latency, where the sessions going through the 
> other 3 cores will not, leading to unstable latency.
>
> 2) internal buffers.  Depending on the protocol different protocols 
> may have different buffers associated with them, so when buffers fill 
> up, more latency is added, but because they can be differently 
> allocated for different protocols, you will see variable latency 
> depending on which is exhausted and which is not.
>
> 3) interface (internal or external) limits.   When you approach 100% 
> of a certain speed link (be it internal or external to the DUT) you 
> can also experience additional latency.  It's hard to determine what a 
> particular DUT's internal architecture is(nor do we really want to try 
> to figure that out), So we generally do not want to push it to 100% 
> because we know we will incur
>
>
> People looking to benchmark are generally interested in the largest 
> stable value (XX).   100% (or near it) is generally not a 
> representative value of the normal case of traffic.   0% (or 1%) is 
> also generally not representative of the normal case for traffic.
>
> model.   Since 0-XX% represents the stable side, i'd rather err on 
> that side, than the XX%-100%.
>
> failure), but that does not necessarily mean stable latency.   In 
> order to get ot that range we're going to back off from 100%.
>
> Now what could that value be?  95%  75%  50%?   This was purposefully 
> set low so we would not encounter any other bottlenecks that would 
> make the latency unstable.
>
> I've tested lots of different devices over the years, both my own and 
> other vendors, and the lower you make that number, the higher the 
> number of devices will stay in the 'stable' area of latency.
>
> so 50% was chosen as the least likely to cause unstable latency, and 
> still representative of real customer usage (plenty of customers have 
> no issue of using a device that is at 50% capacity).   At 75% and much 
> more so at 95% customers are looking to change/upgrade their equipment 
> so they are not nearing the top of what the device is capable of.   So 
> again it makes me less interested in the latency there.
>
>
> On Wed, Jun 3, 2020 at 9:12 AM MORTON, ALFRED C (AL) 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     I confess that this question (why 50%?) has occurred to me in
>     other contexts, and it may help to add a sentence to two of
>     rationale. So if the technical folks can help with suggestions,
>     that would be great!
>
>     regards,
>
>     Al
>
>     *From:* bmwg [mailto:[email-masked]
>     <mailto:[email-masked]>] *On Behalf Of
>     [email-masked] <mailto:[email-masked]>
>     *To:* 'Simon Edwards' <[email-masked] <mailto:[email-masked]>>
>     *Cc:* [email-masked] <mailto:[email-masked]>
>     *Subject:* Re: [bmwg] Query about 50% values in [Benchmarking
>     Methodology for Network Security Device Performance draft 02]
>
>     Simon,
>
>     This requirement was agreed to and adopted by the working group
>     within NetSecOPEN. It first appeared in the IETF individual draft
>     on October 14, 2018.
>     (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05
>     I will leave it to the technical folks to expand on this more.
>     However, I am saying all of this because it has been in the
>     position to be reviewed and commented on by the BMWG community for
>     awhile. Our assumption is that given no one appears to have an
>     issue with it that we hit the mark.
>
>     Brian
>
>     *From:*bmwg <[email-masked] <mailto:[email-masked]>>
>     *On Behalf Of *Simon Edwards
>     *To:* [email-masked] <mailto:[email-masked]>
>     *Subject:* [bmwg] Query about 50% values in [Benchmarking
>     Methodology for Network Security Device Performance draft 02]
>
>     Hi all,
>
>     In a number of sections, but specifically '7.8.3.2.  Test
>     Equipment Configuration Parameters', there are requirements to
>     measure with 50% of the maximum connections/ sec measured in the
>     HTTP/S throughput tests
>
>     I'm sure this 50% value is the product of much thought and
>     discussion, rather than an arbitrary choice. Is anyone able to
>     explain the reason for the specific '50%' value (as opposed to
>     25%, 75% or whatever) or could you please point to documentation
>     around that decision made by the group?
>
>     [email-masked] <mailto:[email-masked]>
> FortinetLogosig1550861636.png 
> E: [email-masked] <mailto:[email-masked]>
> M: +1 408-475-8737
> Skype: asamonteFN
> www.fortinet.com 
> Fortinetsocialtwitternew1550866064.png 
> Fortinetsociallinkedinnew1550866125.png 
> Fortinetsocialfbnew1550866196.png 
> Fortinetsocialinstagramnew1550866291.png 
> Fortinetsocialblognew1550866342.png 
> <https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog> 
>
>
> Secure Remote Access for Your Workforce at Scale 
> *
> ------------------------------------------------------------------------
> *** Please note that this message and any attachments may contain 
> confidential and proprietary material and information and are intended 
> only for the use of the intended recipient(s). If you are not the 
> intended recipient, you are hereby notified that any review, use, 
> disclosure, dissemination, distribution or copying of this message and 
> any attachments is strictly prohibited. If you have received this 
> email in error, please immediately notify the sender and destroy this 
> e-mail and any attachments and all copies, whether electronic or 
> printed. Please also note that any views, opinions, conclusions or 
> commitments expressed in this message are those of the individual 
> sender and do not necessarily reflect the views of Fortinet, Inc., its 
> affiliates, and emails are not binding on Fortinet and only a writing 
> manually signed by Fortinet's General Counsel can be a binding 
> commitment of Fortinet to Fortinet's customers or partners. Thank you. 
> ***
> ------------------------------------------------------------------------
> *
>

---

From: "Van Den Breekel, Jurrie" <[email-masked]>
Date: Wed, 3 Jun 2020 21:22:51 +0000

Hi Gábor,

That is a good suggestion, the challenge however is that the latency tests are already 12 out of the total of 35 tests in the spec. The 12 tests for latency are Throughput and CPS  for both HTTP and HTTPS with 3 object sizes each. Relative to the other tests its already on the high side.

Best regards,
Jurrie

From: bmwg <[email-masked]> on behalf of Gábor LENCSE <[email-masked]>
Date: Wednesday, June 3, 2020 at 1:54 PM
To: "[email-masked]" <[email-masked]>
Subject: Re: [bmwg] [Newsletter] Re: Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

[Warning] This email comes from an external source. Be careful of any embedded links and attachments.

Dear Alex Samonte,

Is it absolutely necessary to use only a single value? Perhaps using multiple latency values at a few different working points (like 25%, 50%, 75%) could give more information about the behavior of the system.

Best regards,

Gábor Lencse

As we all know the latency, will rise dramatically (and be unpredictable) when something gets close to bottlenecking.  That doesn't mean things will fail, but you start getting ugly behavior.

from 0%-XX% the latency is fairly reliable (XX to be guessed at later).  This means the latency doesn't change dramatically as you increase that percentage.  Usually it's something linear with a slope of much less than 1.    When you get to XX%-100%, the latency becomes unstable and it varies wildly as various components of the system have a bottleneck.  When that happens, other things compensate, and then they become a bottleneck (when the first thing is not) and it oscillates all over the place.  This is what leads to the unstable latency.  The difference of the latency at 1% vs XX% is small compared to the latency at XX% and some of the unstable latency at 100%.

So the question is how to we determine XX%?   Unfortunately, all DUTs are different, have different limits, bottlenecks, and features that may change XX%.   It's not the same from vendor to vendor, nor even model to model within the same vendor.

A couple big examples to point out.
1) multi core CPUs.  Because of the way that most stateful devices behave, a single session will probably be handled by a single core/thread of a CPU.  If a device has 8 cores, if there is only 1 session, 7 cores will likely go unused.   If there are 9 sessions, 1 core will be handling 2 sessions, and the others will be handling 1 (50% difference).  If there are 8001 sessions, then it's 1001, and others 1000, and the uneveness does not matter.   However, if one of those sessions is 'harder' or more difficult to process.  Or the number of sessions on a single core may be harder to process, that core may reach 100% capacity before the others do.   Lets' say I have 100%, 50%, 50%, 50% in a 4 core case.   Here we will experience unstable latency.  The sessions that go through the core at 100% will experience additional latency, where the sessions going through the other 3 cores will not, leading to unstable latency.

2) internal buffers.  Depending on the protocol different protocols may have different buffers associated with them, so when buffers fill up, more latency is added, but because they can be differently allocated for different protocols, you will see variable latency depending on which is exhausted and which is not.

3) interface (internal or external) limits.   When you approach 100% of a certain speed link (be it internal or external to the DUT) you can also experience additional latency.  It's hard to determine what a particular DUT's internal architecture is(nor do we really want to try to figure that out), So we generally do not want to push it to 100% because we know we will incur

People looking to benchmark are generally interested in the largest stable value (XX).   100% (or near it) is generally not a representative value of the normal case of traffic.   0% (or 1%) is also generally not representative of the normal case for traffic.

Ideally XX% is what we're trying to get, but it's different for each model.   Since 0-XX% represents the stable side, i'd rather err on that side, than the XX%-100%.

When doing the benchmark we're finding the various max values (without failure), but that does not necessarily mean stable latency.   In order to get ot that range we're going to back off from 100%.

Now what could that value be?  95%  75%  50%?   This was purposefully set low so we would not encounter any other bottlenecks that would make the latency unstable.

I've tested lots of different devices over the years, both my own and other vendors, and the lower you make that number, the higher the number of devices will stay in the 'stable' area of latency.

so 50% was chosen as the least likely to cause unstable latency, and still representative of real customer usage (plenty of customers have no issue of using a device that is at 50% capacity).   At 75% and much more so at 95% customers are looking to change/upgrade their equipment so they are not nearing the top of what the device is capable of.   So again it makes me less interested in the latency there.

Hopefully that makes sense, and while it could be shorter in the explanation, i'm not sure I could make it into a 2 sentence one.

-Alex

On Wed, Jun 3, 2020 at 9:12 AM MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>> wrote:
Thanks for your question, Simon, and the history, Brian.

regards,
Al

From: bmwg [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of [email-masked]<mailto:[email-masked]>

Simon,

This requirement was agreed to and adopted by the working group within NetSecOPEN. It first appeared in the IETF individual draft on October 14, 2018. (https://tools.ietf.org/html/draft-balarajah-bmwg-ngfw-performance-05<https://urldefense..proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dbalarajah-2Dbmwg-2Dngfw-2Dperformance-2D05&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=8ElQf-XUyhkke33v7BvLLf5mEBDCEU2mlwlFDpxHJD8&e=>). It was clarified and expanded on March 5, 2019 in https://tools.ietf.org/html/draft-ietf-bmwg-ngfw-performance-00<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dietf-2Dbmwg-2Dngfw-2Dperformance-2D00&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=mylQC4CnyBjr1JS-Qbiw5d392Llnmp_6LxMRXJO8NVI&s=xbiPb0v60VZvEM2cumbN3J41IrOw2hMUK7HHaR-HN8o&e=>. It’s form has largely been unchanged since then.

Brian

Hi all,

Very best wishes,
Simon
_______________________________________________
bmwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bmwg

--

Alex Samonte

<http://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=fortinet-logo>
E: [email-masked]<mailto:[email-masked]>
M: +1 408-475-8737
Skype: asamonteFN
www.fortinet.com<https://www.fortinet.com/?utm_source=email-signature&utm_medium=sigstr-company-url&utm_campaign=sd-wan>              <https://twitter.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-twitter>      <http://www.linkedin.com/company/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-linkedin>      <http://www.facebook.com/fortinet?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-facebook>     <https://www.instagram.com/behindthefirewall/?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-instagram>      <https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog>
[Image removed by sender. Secure Remote Access for Your Workforce at                     Scale]<http://signatures.fortinet.com/uc/5e868f6230170c1cb4a71ffb>

bmwg mailing list

[email-masked]<mailto:[email-masked]>

---

From: Alex Samonte <[email-masked]>
Date: Wed, 3 Jun 2020 14:35:35 -0700

Certainly possible to use multiple values, but I'm not sure you would truly
gain that much more information.
We were not trying to increase the test load for latency (or anything else
that used a %age of previous traffic) by a factor of 3 (in this case).

But those numbers (25%, 50%, 75%) and the number of them (3) is just as
arbitrary as what we chose.   If someone wants to do a study (or can point
to one) where it would gain enough information to be useful we can consider
that for a future update.

-Alex

On Wed, Jun 3, 2020 at 1:53 PM Gábor LENCSE <[email-masked]> wrote:

> multiple latency values at a few different working points (like 25%, 50%,
> 75%) could give more information about the behavior of the system.
>
> Best regards,
>
> Gábor Lencse
> other vendors, and the lower you make that number, the higher the number of
>> *Subject:* [bmwg] Query about 50% values in [Benchmarking Methodology
> <https://www.fortinet.com/blog?utm_source=email-signature&utm_medium=sigstr&utm_campaign=social-blog> [image:
> * ------------------------------ *** Please note that this message and any
> attachments may contain confidential and proprietary material and
> information and are intended only for the use of the intended recipient(s).
> If you are not the intended recipient, you are hereby notified that any
> review, use, disclosure, dissemination, distribution or copying of this
> message and any attachments is strictly prohibited. If you have received
> this email in error, please immediately notify the sender and destroy this
> e-mail and any attachments and all copies, whether electronic or printed.
> Please also note that any views, opinions, conclusions or commitments
> expressed in this message are those of the individual sender and do not
> necessarily reflect the views of Fortinet, Inc., its affiliates, and emails
> are not binding on Fortinet and only a writing manually signed by
> Fortinet's General Counsel can be a binding commitment of Fortinet to
> Fortinet's customers or partners. Thank you. ***
> ------------------------------*
>
> bmwg mailing [email-masked]://www.ietf.org/mailman/listinfo/bmwg
>

-- 

Alex Samonte

---

From: "Ryan Liles (ryliles)" <[email-masked]>
Date: Thu, 4 Jun 2020 14:20:52 +0000

Jurrie has a valid point in that creating more data points, while it can be very beneficial, can start to exponentially increase the number of executed tests. This can lead to a huge bloat in the effort necessary to execute the complete draft methodology.

Thanks,

Ryan Liles

From: bmwg <[email-masked]> on behalf of "Van Den Breekel, Jurrie" <[email-masked]>
Date: Wednesday, June 3, 2020 at 4:22 PM
To: Gábor LENCSE <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [bmwg] [Newsletter] Re: Query about 50% values in [Benchmarking Methodology for Network Security Device Performance draft 02]

Hi Gábor,

Best regards,
Jurrie

Dear Alex Samonte,

Best regards,

Gábor Lencse

-Alex

regards,
Al

Simon,

Brian

Hi all,

Very best wishes,
Simon

--

Alex Samonte

bmwg mailing list