Subject: [Gen-art] [Fwd: Re: Gen-Art IETF LC review: draft-ietf-ipfix-testing-04.txt]

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 02 May 2008 07:20:33 -0400

Since this did not make it to the spreadsheeting, I am forwarding the 
note again.
the -05 of the ipfix testing document addressed my concerns and as far 
as I can tell is now ready for publication as an Informational RFC.

Yours,
Joel

-------- Original Message --------
Subject: Re: [Gen-art] Gen-Art IETF LC review: 
draft-ietf-ipfix-testing-04.txt
Date: Fri, 18 Apr 2008 17:42:06 -0400
From: Joel M. Halpern <[email-masked]>
To: Benoit Claise <[email-masked]>
CC: Romascanu, Dan (Dan) <[email-masked]>, [email-masked], 
[email-masked], [email-masked], 
[email-masked]
References: 
<[email-masked]> 
<[email-masked]> <[email-masked]>

Thank you.  These changes sufficiently address my concerns.

(I rather like the one case where the text reads "the tester must check
and if necessary improve the software so that ...".  Why do I think that
this is something folks have tripped over before? :-)

I appreciate your efforts in addressing the issues.
Yours,
Joel M. Halpern

Benoit Claise wrote:
>   Dear Joel,
> 
> I posted a new version of the testing draft
> 
>     A new version of I-D, draft-ietf-ipfix-testing-05.txt has been successfuly submitted by Benoit Claise and posted to the IETF repository.
> 
>     Filename:	 draft-ietf-ipfix-testing
>     Revision:	 05
>     Title:		 Guidelines for IP Flow Information eXport (IPFIX) Testing
>     Creation_date:	 2008-04-14
>     WG ID:		 ipfix
>     Number_of_pages: 38
> 
>     Abstract:
>     This document presents a list of tests for implementers of IP Flow
>     Information Export (IPFIX) compliant Exporting Processes and
>     Collecting Processes.  This document specifies guidelines for a
>     series of tests that can be run on the IPFIX Exporting Process and
>     Collecting Process in order to probe the conformity and robustness of
>     the IPFIX protocol implementations.  These tests cover all important
>     functions, in order to gain a level of confidence in the IPFIX
>     implementation.  Therefore they allow the implementer to perform
>     interoperability or plug tests with other IPFIX Exporting Processes
>     and Collecting Processes.Conventions used in this document
> 
> We hope to have addressed your concerns successfully. Would you mind 
> checking this latest version.
> Note: we have addressed your point 10), as we thought we didn't want to 
> be that formal in our test description.
> 
> Regards, Benoit.
> 
> 
>>  I have been selected as the General Area Review Team (Gen-ART)
>> reviewer for this draft (for background on Gen-ART, please see
>> http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html ).
>>
>> Please resolve these comments along with any other Last Call comments
>> you may receive.
>>
>> Document: Guidelines for IP Flow Information eXport (IPFIX) Testing
>> Reviewer: Joel M. Halpern
>> Review Date: 15-Feb-2008
>> IETF LC End Date: 26-Feb-2008
>> IESG Telechat date: N/A
>>
>> Summary: This document needs some additional work before publication 
>> as an informational RFC.
>> I would particularly recommend considering addressing at least the 
>> first comment below prior to RFC publication.
>> I would also suggest that the test descriptions need some 
>> clarification as described in the technical section below, 
>> particularly items 5 and 6.
>>
>> Comments:
>>
>> Conceptual:
>> 1) While the document is being published as an information RFC, the 
>> wording of the abstract and introduction make it seem that this 
>> document is actually defining conformance to the IPFIX RFCs.  The IETF 
>> has generally carefully steered clear of defining such conformance.
>> So, while publishing a useful test suite is probably a good idea, I 
>> strongly recommend fixing the wording of at least the abstract and 
>> introduction to make it quite clear that these are not mandatory 
>> tests, and that these tests do not define conformance.
>> Related to this, please do not assert (in section 3) that passing this 
>> test suite constitutes conformance to the IPFIX architecture and 
>> protocol.  (Among other things,test suite passage proves nothing about 
>> architectural conformance.)
>>
>>
>> Technical:
>> 2) In the terminology section, an Observation Point is defined simply 
>> as a place where packets can be observed.  An Observation Domain is a 
>> collection of Observation points.  Then, in the middle of the 
>> definition of an Observation domain it say "In the IPFIX MEssage it 
>> generates..." but up till now none of the things that have been 
>> defined generate IPFIX messages.  It is possible that the "it" in the 
>> quote is supposed to be the "Metering Process" mentioned in passing 
>> earlier in the definition. But the English grammar does not lead the 
>> reader to such a conclusion. Later in that same definition, it beings 
>> to appear that an Observation Domain (which is a collection of points, 
>> not a process or entity) is supposed to generate IPFIX messages, since 
>> it is supposed to include a Domain ID in the messages it generates.  
>> This definition for an Observation Domain needs to be reworked, to 
>> avoid confusing the Domain with the Measurement Process which is 
>> running in / for / on the Domain.
>>
>> 3) The use of capital "MUST" in section 3.1 is almost certainly wrong. 
>> Firstly, what I think that section is saying is that being able to 
>> correctly perform the basic tests is a precondition for being able to 
>> perform further test successfully.  Thats a precondition, not a "MUST".
>> Of lesser significance, this document does not provide any description 
>> of what it means by "MUST".  We are usually careful about how such 
>> language is used in informational RFCs.  I think the meaning would be 
>> clearer if the real intent were stated.  I suspect that some readers 
>> of this review may find my concern here pedantic.  But the continual 
>> use of MUST in the document really, really bothers me. (I hope the 
>> next comment helps explain why it bothers me so much.)
>>
>> 4) Then, the test descriptions go on to keep using this language.  
>> This is a test suite description document.  Simply state how to run 
>> the test.  There is no need for "MUST".  Section 3 should indicate 
>> that the test descriptions describe the preconditions and steps that 
>> the tester goes through.  So section 3.1 would begin "The tester 
>> creates one Exporting Process and one collection process, configures 
>> the Exporting Process to ..."
>>
>> 5) It is not clear what test steps like ~The tester ensures that an 
>> SCTP association is established.~ (Or worse, the actual text which 
>> reads "the test MUST ensure that an SCTP association is established." 
>> are supposed to do.  Is this an instruction to the tester to use 
>> network management tools or CLI to verify a connection on both 
>> devices?  Is it an instruction to perform additional configuration?  
>> How does the tester "ensure".  A test suite should tell a tester what 
>> steps to undertake, and what observations to perform.  "Ensure" is not 
>> either one of those.
>> 5a) To elaborate on this issue, in the middle of the test step about 
>> ensuring that Data Records are actually exported, we finally get a 
>> testable instruction, to whit, use a packet sniffer and check that the 
>> packets are coming by.
>>
>> 6) I believe I understand how a tester would create templates, for the 
>> template test.  But how is the tester to create data sets.  
>> Particularly data sets with specific properties, such as the padding 
>> in section 3.2.3 and 3.2.4?  The best conclusion I can come to is that 
>> this is a collector test, and that it assumes a packet generator which 
>> can generate IPFIX packets.  Having such a device in a test setup 
>> makes sense.  But the test description does not say "configure a 
>> packet generator to generate an IPFIX packet with ..."  (There are 
>> other ways to say this, but there needs to be some description of how 
>> testers are expected to create data sets.)
>> 6a) Related to this, I find reading this document rather odd.  I have 
>> read many test suites for protocols and implementations of protocols. 
>> They generally focus on a Device (or implementation, or entity) Under 
>> Test, and the framing around that Device.  This suite appears to be 
>> trying to test two interacting devices simultaneously.  That is 
>> extremely difficult, and extremely confusing.  It is particularly hard 
>> because then the tester doesn't have enough points of control to 
>> perform the tests and observe the results meaningfully.  It is 
>> possible that this combined suite is right for this problem.  But if 
>> so, a lot of explanation of why it is done that way and how the tester 
>> is to accomplish his goals is needed.
>>
>> Minor:
>> 7) The abstract is worded as if one could not perform interoperability 
>> testing without first running the tests in this document.  While 
>> having run the tests in this document will presumably increase the 
>> chances of a successful interoperability test, they are not an 
>> inherent requirement for such testing.
>>
>> 8) I would probably be inclined to lighten up the Motivation section a 
>> bit.  Or even remove it.  I don't think we need to explain why test 
>> suites are useful.  If we really need a motivation section, then it 
>> should explain something about why it is particularly complex to test 
>> IPFIX implementations (if it is) and thus why the IETF feels it is 
>> particularly useful to publish a test suite ourselves in this case.
>>
>> 9) The definition of Transport Session is actually the definition of 
>> various kinds of transport sessions, and how they are identified.  
>> Could the definition start with an actual definition please. (I.e. the 
>> communication over time used to carry X between Y and Z?  Or something.)
>>
>> 10) As an editorial matter, most testers I have worked with strongly 
>> prefer if every step in a test is explicitly separate and named / 
>> numbered.  That way, they can check off each step as it goes.  So the 
>> beginning of 3.1.1 would be
>> i) Create One Exporting Process
>> ii) Create One Collection  Process
>> iii) Configure the Exporting Process ...
>>
>> 11) It is particularly odd to see a set of Stress/Load tests that 
>> simultaneously claim to be measuring conformance and to not specify 
>> the level of Stress / Load.  Having a description of how to perform 
>> load tests is useful.  But its relationship to the other tests is 
>> confusing.  (This obviously is helped once we no longer claim that 
>> this is a conformance test.)
>>
>>
> 
_______________________________________________
Gen-art mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/gen-art

_______________________________________________
Gen-art mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/gen-art

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Sun, 4 May 2008 15:38:36 +0200

Thanks, Joel. Based on your review and inputs the latest version is now
on the agenda of the IESG telechat this Thursday. 

Dan

> -----Original Message-----
> From: Joel M. Halpern [mailto:[email-masked]] 
> Sent: Friday, May 02, 2008 2:21 PM
> To: [email-masked]
> Cc: Romascanu, Dan (Dan)
> Subject: [Fwd: Re: [Gen-art] Gen-Art IETF LC review: 
> draft-ietf-ipfix-testing-04.txt]
> 
> Since this did not make it to the spreadsheeting, I am 
> forwarding the note again.
> the -05 of the ipfix testing document addressed my concerns 
> and as far as I can tell is now ready for publication as an 
> Informational RFC.
> 
> Yours,
> Joel
> 
> CC: Romascanu, Dan (Dan) <[email-masked]>, 
> [email-masked], [email-masked], 
> [email-masked],
> [email-masked]
> References: 
> (I rather like the one case where the text reads "the tester 
> must check and if necessary improve the software so that 
> ...".  Why do I think that this is something folks have 
> tripped over before? :-)
> 
> >     A new version of I-D, draft-ietf-ipfix-testing-05.txt 
> has been successfuly submitted by Benoit Claise and posted to 
> the IETF repository.
> > 
> > Note: we have addressed your point 10), as we thought we 
> didn't want 
> > to be that formal in our test description.
> > 
> > Regards, Benoit.
> > 
> > 
> >> document is actually defining conformance to the IPFIX RFCs.  The 
> >> IETF has generally carefully steered clear of defining 
> such conformance.
> >> Related to this, please do not assert (in section 3) that passing 
> >> this test suite constitutes conformance to the IPFIX 
> architecture and 
> >> protocol.  (Among other things,test suite passage proves nothing 
> >> about architectural conformance.)
> >>
> >>
> >> Technical:
> >> to appear that an Observation Domain (which is a collection of 
> >> points, not a process or entity) is supposed to generate IPFIX 
> >> messages, since it is supposed to include a Domain ID in 
> the messages it generates.
> >> Of lesser significance, this document does not provide any 
> >> description of what it means by "MUST".  We are usually 
> careful about 
> >> how such language is used in informational RFCs.  I think 
> the meaning 
> >> would be clearer if the real intent were stated.  I 
> suspect that some 
> >> readers of this review may find my concern here pedantic.  But the 
> >> continual use of MUST in the document really, really 
> bothers me. (I 
> >> hope the next comment helps explain why it bothers me so much.)
> >>
> >> steps to undertake, and what observations to perform.  "Ensure" is 
> >> not either one of those.
> >> testable instruction, to whit, use a packet sniffer and check that 
> >> the packets are coming by.
> >>
> >> 6) I believe I understand how a tester would create templates, for 
> >> the template test.  But how is the tester to create data sets.
> >> in section 3.2.3 and 3.2.4?  The best conclusion I can come to is 
> >> that this is a collector test, and that it assumes a 
> packet generator 
> >> which can generate IPFIX packets.  Having such a device in a test 
> >> setup makes sense.  But the test description does not say 
> "configure 
> >> a packet generator to generate an IPFIX packet with ..."  
> (There are 
> >> extremely difficult, and extremely confusing.  It is particularly 
> >> hard because then the tester doesn't have enough points of 
> control to 
> >> so, a lot of explanation of why it is done that way and how the 
> >> tester is to accomplish his goals is needed.
> >>
> >> Minor:
> >> 7) The abstract is worded as if one could not perform 
> >> interoperability testing without first running the tests in this 
> >> document.  While having run the tests in this document will 
> >> presumably increase the chances of a successful interoperability 
> >> test, they are not an inherent requirement for such testing.
> >>
> >> 8) I would probably be inclined to lighten up the 
> Motivation section 
> >> a bit.  Or even remove it.  I don't think we need to 
> explain why test 
> >> Could the definition start with an actual definition please. (I.e. 
> >> the communication over time used to carry X between Y and Z?  Or 
> >> something.)
> >>