Subject: Re: [sacm] Interop testing

From: [email-masked]
Date: Sun, 05 Aug 2012 15:22:32 -0500 (CDT)

David and all:

Interoperability of SACM products is of interest to the users and needs attention.

Very likely, interoperability testing will consist of two processes.

1.  Taking one validated product and running content from a reputable repository (such as the National Checklist Program Repository).

2.  Generating content from a validated product and running it on another and viceversa.

The second testing process will probably be needed to ensure that SCAP 1.2  Asset Management applications (that incorporate Vulnerability and Configuration reporting) work as intended.

David Oliva

On 08/03/12, Waltermire, David A.<[email-masked]> wrote:

Another thing we might want to address in the charter is some concept of interop testing as part of the I/D development process.  This has been discussed within the Security Automation community historically, but was not mentioned during the side meeting.

Dave

_______________________________________________
sacm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/sacm

---

From: <[email-masked]>
Date: Mon, 6 Aug 2012 10:55:37 -0500

All,

Interoperability testing is something we have wanted to do for quite a while. I have been a big proponent of it but, I am not sure it should be in the charter though. I would think this is something the working group could foster but would like to ask others on the list with more IETF experience if this is something that has been done in past charters?

I believe that a true interoperability test at this point would be if I as vendor A gave vendor B the content that I created and Vendor B processed it and then gave me the results to evaluate.  I would do the same with content they developed.  If there are deviations then the two vendors would need to figure out what is different in the interpretation / evaluation.  The problem is not evaluating content that exists in well known places but is created by guidance authors and vendors that are not well known and tested. The example you gave is lacking content for most UNIX and Linux distributions. Yes, interpretation of the well known content can be concerning but  many times that's due to what was not implemented.

The other issue is there would need to be a reasonably stable test environment that's configuration was known so that the results could be properly evaluated and relied on. That would mean multiple UNIX, Linux and Windows environments at a minimum.  The other issue this poses is a scalability issue.  If Vendor A needs to validate with 45+ other vendors, this becomes a very large effort that will take good deal of planning and potentially some automation to do a reasonable accurate evaluation of the generated results.

I could see an Information I-D being developed that described the results / lessons learned from interoperability testing but this type of testing requires that a reasonable set of vendors participate to be effective.  That would not be something a charter should mandate.

Thoughts?

Kent Landfield

McAfee | An Intel Company
Direct: +1.972.963.7096
Mobile: +1.817.637.8026
Web: www.mcafee.com<http://www.mcafee.com/>

From: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Date: Sunday, August 5, 2012 3:22 PM
To: David Waltermire <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [sacm] Interop testing

 David and all:

David Oliva

On 08/03/12, Waltermire, David A.<[email-masked]<mailto:[email-masked]>> wrote:

Dave

________________________________

_______________________________________________
sacm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/sacm