Subject: Re: [bmwg] [answer to Bill Cerveny] RE: New Version Notification for draft-janovak-bmwg-ipflow-meth-00 [answer to Bill Cerveny]

From: Bill Cerveny <[email-masked]>
Date: Wed, 16 Jul 2008 22:52:38 -0400

Comments below ...

Bill

On Jun 11, 2008, at 9:43 AM, Jan Novak (janovak) wrote:

> Hi Bill,
>
> I am sorry, was on holidays and missed your answer ....
>
>> devices, the throughput measurement wouldn't be appropriate. Perhaps
>> the scope of this document should be specifically on network devices
>> that transit network traffic?
>
> Yes, the intention was to make it specific for only network devices.
> Will review the text to make sure it is clear.

OK

>
>
>> Wouldn't a big consideration be the quality and accuracy of the flow
>> data?  How would the accuracy of the generated flow be evaluated?   
>> The
>
>> flow generating capabilities of a network device is certainly less
>> useful if the device's flow accuracy degrades when it is transiting
>> more traffic.
>
> I would consider this out of the scope since to check that the testing
> would
> have to involve some external collector and the way those handle data
> is even more complex than what happens on the network devices when
> just registering the traffic.
> I will think about it yet, but based on the testing we have done I  
> don't
> quite see how to do it.

Couldn't you do something like send a known set of network traffic  
(defined by number of packets, types of packets, packet sizes, etc)  
across the router?  If the collector's interpretation of of the flow  
data matches what was sent across the router, a statement could be  
made about flow accuracy. If the collector's interpretation of the  
flow data does not match what was sent across the router, then a  
statement of accuracy cannot be made.  If the collector is unable to  
keep up with the flow data, perhaps another way of verifying the  
accuracy of the flow may be recording the flow data and playing it  
back at a rate that a collector could understand.

I just think that making statements about a router's flow handling and  
reporting capabilities aren't complete if it can't be verified that  
the flow data reported is accurate.

>
>
>> What about flow sampling?  There is no reference to sampling in the
>> draft.
>
> Yes, that should not be difficult to discuss.
>
> Thank you for your comments.
>
> Jan
>
> The climate of Edinburgh is such that the weak
> succumb young .... and the strong envy them.
>
>                                 Dr. Johnson
>
>
>
> Jan,
>
> I think a document on benchmarking flow analysis and export could be
> valuable.
>
> Questions:
> - The draft assumes the device generating flow will be a transit
> device (such as a router).  What about flow generating devices that
> passively monitor the network and are not transit devices?  For such
> devices, the throughput measurement wouldn't be appropriate. Perhaps
> the scope of this document should be specifically on network devices
> that transit network traffic?
> - Wouldn't a big consideration be the quality and accuracy of the flow
> data?  How would the accuracy of the generated flow be evaluated?  The
> flow generating capabilities of a network device is certainly less
> useful if the device's flow accuracy degrades when it is transiting
> more traffic.
> - What about flow sampling?  There is no reference to sampling in the
> draft.
>
> Thanks,
>
> Bill Cerveny
>
>
>> -----Original Message-----
>> From: Jan Novak (janovak)
>> Sent: 21 May 2008 11:45
>> To: [email-masked]'
>> Subject: FW: New Version Notification for
>> draft-janovak-bmwg-ipflow-meth-00
>>
>> Hi,
>>
>> I have worked for over 3 years in the IP Flow (RFC5101) testing area
>> (RFC3954 - Netflow) and during that time we have got numerous  
>> requests
>> from various customers about Netflow performance. Every time
>> it started off a long discussion what was meant by the term
>> "netflow performance", how to measure it and deliver some
>> reproducible data.
>> We ended up compiling an internal document which we provided
>> to those asking the questions above. This IETF draft document
>> tries to specify IP Flow performance and some characteristics to
>> quantify and measure it in an implementation independent way.
>>
>> I would appreciate the opinion of this working group if this bit
>> of work is deemed useful and worthwhile of accepting as the WG
>> document.
>>
>> Thanks, Jan
>>
>> The climate of Edinburgh is such that the weak
>> succumb young .... and the strong envy them.
>>
>>                                 Dr. Johnson
>>
>>> -----Original Message-----
>>> From: IETF I-D Submission Tool [mailto:[email-masked]]
>>> Sent: 21 May 2008 11:34
>>> To: Jan Novak (janovak)
>>> Subject: New Version Notification for
>>> draft-janovak-bmwg-ipflow-meth-00
>>>
>>>
>>> A new version of I-D, draft-janovak-bmwg-ipflow-meth-00.txt
>>> has been successfuly submitted by Jan Novak and posted to the
>>> IETF repository.
>>>
>>> Filename:	 draft-janovak-bmwg-ipflow-meth
>>> Revision:	 00
>>> Title:		 IP Flow Information Accounting and
>>> Export Benchmarking Methodology
>>> Creation_date:	 2008-05-21
>>> WG ID:		 Independent Submission
>>> Number_of_pages: 19
>>>
>>> Abstract:
>>> This document provides methodology and framework for quantifying
>>> performance implications of enabling selective monitoring of
>>> IP flows on a network device and export of this information to
>>> a collector as specified in [RFC5101].
>>>
>>>
>>>
>>>
>>> The IETF Secretariat.
>>>
>>>
>>>
> _______________________________________________
> bmwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bmwg

_______________________________________________
bmwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bmwg

---

From: Jeff Byzek <[email-masked]>
Date: Thu, 17 Jul 2008 00:19:34 -0400

Bill Cerveny wrote:
>> I will think about it yet, but based on the testing we have done I don't
>> quite see how to do it.
>

This gets into the area of functionality verification and not
measurement, which seems to contradict the charter of the group (but I
could be misinterpreting).

Beyond this, there are many different variables involved, all of which
may have varying significance for different parties and can affect
accuracy in different ways.  It seems overly arduous to attempt to
identify a set (or multiple sets) of parameters that will satisfy all
(or even most) people's accuracy requirements for a production deployment.

-J

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Thu, 17 Jul 2008 12:16:21 +0200

Hi,

> the flow data reported is accurate.

The draft covers it to certain extent - checking the flow monitoring
counters to make sure that the expected flow conditions are met
- e.g. the network device reacts to and registers all the flows.

Something in terms of what you suggest is certainly possible, but the
biggest concern we always met was "what is flow monitoring performance
and where (which device in the network) can I afford to enable it".

The accuracy of the data comes always as second (when you know you
can actually do it :-) at all). It would be possible to check the
individual flows
counters on the network device in most of the cases described in the
draft (but not with cache overflow) but not as part of the performance
measurement since such a check would heavily load the CPU when
collecting
the necessary outputs. May be we can discuss this in Dublin ??

I am not in the position to suggest anything regarding external
collectors
since I am not familiar with them and never tested them and would like
to
keep this draft scope to just the network devices.

Jan

                                 Dr. Johnson  

> From: Bill Cerveny [mailto:[email-masked]] 
> Sent: 17 July 2008 03:53
> To: Jan Novak (janovak)
> Cc: [email-masked]
> Subject: Re: [bmwg] [answer to Bill Cerveny] RE: New Version 
> Notification for draft-janovak-bmwg-ipflow-meth-00 [answer to 
> Bill Cerveny]
> 
> Comments below ...
> 
> Bill
>

---

From: Bill Cerveny <[email-masked]>
Date: Sat, 26 Jul 2008 21:18:52 -0400

Jan and Jeff,

[I started with wanting to respond to both Jeff Byzek's and Jan  
Novak's e-mails separately, but didn't want to say "look at my other  
response" :-) So this is a combined response to both Jan and Jeff]

I concur that there can be a bit of complexity in verifying the flows  
and I'm interested in how the draft evolves. I understand that the  
draft is intended to measure performance and not necessarily standards  
compliance.

I also admit that I'm stuck on wanting to confirm the quality of the  
flow data in some manner before stating that a specific benchmark  
value was achieved (summarizing a paragraph I deleted in this e-mail  
where I essentially stated a subtle variation of what I've said in  
earlier e-mails :-) ).

As a compromise, perhaps the draft could state a mechanism whereas  
measurements values for which the accuracy of the flow data is somehow  
confirmed (such as in the DUT CPU) should be suffixed "with verified  
flow data" (or something like that).  Measurement values for which  
flow data was not confirmed should be suffixed "without verified flow  
data"

I won't be in Dublin, but I am planning on attending the meeting in  
Minneapolis.  I will attempt to attend the Dublin BMWG meeting  
virtually, I just won't be in the hallways (acknowledging that the  
jabber "hallway" isn't quite the same thing :-) ).

Bill

---

From: "Jan Novak (janovak)" <[email-masked]>
Date: Tue, 29 Jul 2008 16:23:47 +0200

Hi,

> and I'm interested in how the draft evolves. 

Great, thanks :-).

> flow data was not confirmed should be suffixed "without 

I noticed the other presenting guys specified even the reporting format
- no problem doing that as well.

The accuracy of the flow data could be checked to certain extent (for
the cache
overflow scenarios it could check the format of the Key Fields and if
the contents 
of them falls within certain range of values) on the DUT
by collecting the whole cache contents and checking it against sent 
traffic but it would have to be a separate test or performed after the
CPU measurement - would that fulfil your requirement a bit ??

> Minneapolis.  I will attempt to attend the Dublin BMWG meeting  

I have troubles with travelling, might manage next meeting in Europe but

doubt I could attend in the US - will see yet, if it improves over the
next
months, may be you could discuss this with my IPFIX colleagues.

Jan

                                 Dr. Johnson  

> Sent: 27 July 2008 02:19
> To: Jan Novak (janovak); Jeff Byzek (byzek)
> Cc: [email-masked]