Subject: [sacm] OVAL/SACM IM mapping document posted

From: "Hansbury, Matt" <[email-masked]>
Date: Thu, 7 May 2015 14:20:01 +0000

All,

I'd like to call your attention to the following Internet Draft that I submitted on Tuesday evening:

https://datatracker.ietf.org/doc/draft-hansbury-sacm-oval-info-model-mapping/

This document discusses how the Open Vulnerability and Assessment Language (OVAL) can be leveraged to fulfill some aspects defined by the SACM Information Model.  Specifically, it discusses the various components of the Information Model where the OVAL Language could be used either fully or partially as a Data Model for those components.  Furthermore, it provides some lessons learned over the course of the past 10+ years by the OVAL community and some concrete recommendations on how best to utilize the different models of the OVAL Language within SACM.

We welcome your feedback and/or questions on the document and encourage you to review ahead of the virtual interim meeting next week, where we will spend a few minutes talking about the draft.

Thanks
Matt

============================================
Matthew Hansbury
J83D - Cyber Security Partnerships, Sharing, and Automation
The MITRE Corporation
Email: [email-masked]<mailto:[email-masked]>

---

From: Gunnar Engelbach <[email-masked]>
Date: Tue, 19 May 2015 10:37:56 -0400

Matt,  good job on this.  And I think your recommendations and lessons 
learned sections are spot on.

However, I also find it necessary to mention a few of the downsides I 
see to using OVAL:

1) The existing test types and some schemas are inconsistent.  That is 
to say, some are modeled on the data to collect, some are modeled on a 
particular API, and others are modeled on a particular collection 
methodology.

2) The OVAL SC model is pretty straightforward, but using OVAL 
definitions to drive collection is problematic to implement 
consistently.  This is largely due to ambiguities in the language that 
frequently are only resolved via an email thread archived on nabble.

3) An OVAL document doesn't scale particularly well.

But even with all that in mind, I haven't seen anything more suitable 
for SACM use.

--gun

On 5/7/2015 10:20 AM, Hansbury, Matt wrote:
>
> All,
>
> I’d like to call your attention to the following Internet Draft that I 
> submitted on Tuesday evening:
>
> This document discusses how the Open Vulnerability and Assessment 
> Language (OVAL) can be leveraged to fulfill some aspects defined by 
> the SACM Information Model.  Specifically, it discusses the various 
> components of the Information Model where the OVAL Language could be 
> used either fully or partially as a Data Model for those components. 
>  Furthermore, it provides some lessons learned over the course of the 
> past 10+ years by the OVAL community and some concrete recommendations 
> on how best to utilize the different models of the OVAL Language 
> within SACM.
>
> We welcome your feedback and/or questions on the document and 
> encourage you to review ahead of the virtual interim meeting next 
> week, where we will spend a few minutes talking about the draft.
>
> Thanks
>
> Matt
>
> Email: [email-masked] <mailto:[email-masked]>
>
>
>
> _______________________________________________
> sacm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sacm

---

From: David Ries <[email-masked]>
Date: Thu, 21 May 2015 15:02:35 -0500

Hi Matt,

I second Gunnar’s praise of your work here. It does generally seem spot-on to me as well.

I do have questions about two small points raised in the document that I’d love some feedback on.

1) Platform Schema Extension Challenges
4.5.  Collection Guidance: “Important Note: One of the key drawbacks to OVAL is that ... is not easy or scalable to create or update extensions for rapidly changing platforms and products in a timely manner.”

Doesn’t the NETCONF schema demonstrate that it is not always necessary to create a platform extension to support a new or modified platform? Is there any language-based obstacle to adding more protocol-oriented schemas if that is desirable?

When it is desirable to create or modify a platform-specific schema, I’ve found that the primary obstacle is speed of the review and release process. And, the community seems to be taking steps to change that. For example, the recently revised versioning policy now allows platforms to be rev’d independently of the core so they can be released more quickly.

2) Collection & Evaluation Must Be De-Coupled
6.2.1.  Lesson: “the close coupling found in OVAL between the OVAL Object and OVAL State ... does not easily allow for dynamic querying of previously collected data, nor does it easily allow for efficiencies in data collection.”

I’ve heard this a few times in discussions of OVAL, but I’ve never really understood the criticism. It seems like this might be a misconception based on specific implementations of OVAL-based solutions, but not actually a limitation of OVAL itself.  

I see an OVAL document as a policy statement of sorts and not a high-level programming language. The document specifies data that needs to be collected and (optionally) makes assertions about that data. But, is there a requirement to “run the whole document” in one go when a scan is performed?

To be more specific, I don’t see any obstacle to an enterprise solution storing the collected data and re-querying it later. Can’t you create definitions designed to collect posture attributes that include objects without states and store the collected data? Inventory definitions are another example of using OVAL to collect data that (presumably) is stored and queried later.

Additionally, I don’t see inherent obstacles to optimizing data collection for multiple objects. Is there a requirement that each object be collected individually? Can’t the collector analyze the data required by each of the objects contained in the superset of definitions applicable to a managed device and identify strategies for optimizing collection? For example, if there’s an object that specifies files in directory X and another that specifies files in directories X and Y, I’d expect the collector to only scan directory X once.

To give a slightly more complete example, an OVAL-based enterprise solution could do something like this:
Retrieve large numbers of OVAL definitions (i.e. ~25k vulnerability and inventory definitions from Mitre, the Cisco IOS advisory definitions from Cisco, etc.)
Analyze the definitions applicable to each platform in order to identify all of the data that must be collected from a managed device in order to resolve all the variables, generate items, etc. and evaluate the applicable definitons
Collect and store that data
Refresh that data when required by the organizations requirements for data freshness
Evaluate any or all of the definitions on a schedule completely disconnected from the collection schedule

In such a solution, evaluating new or modified definitions may be possible based entirely on previously collected data.

Best,
David

> On May 7, 2015, at 9:20 AM, Hansbury, Matt <[email-masked]> wrote:
> 
> All,
>  
> I’d like to call your attention to the following Internet Draft that I submitted on Tuesday evening:
>  
> https://datatracker.ietf.org/doc/draft-hansbury-sacm-oval-info-model-mapping/ <https://datatracker.ietf.org/doc/draft-hansbury-sacm-oval-info-model-mapping/>
>  

-David

--
David E. Ries
Partner
Farnam Hall Ventures 

farnamhallventures.com <http://farnamhallventures.com/>

---

From: David Ries <[email-masked]>
Date: Thu, 21 May 2015 15:28:50 -0500

Comments inline below:

> On May 19, 2015, at 9:37 AM, Gunnar Engelbach <[email-masked]> wrote:
> 
> 
> Matt,  good job on this.  And I think your recommendations and lessons learned sections are spot on.
> 
> However, I also find it necessary to mention a few of the downsides I see to using OVAL:
> 
> 1) The existing test types and some schemas are inconsistent.  That is to say, some are modeled on the data to collect, some are modeled on a particular API, and others are modeled on a particular collection methodology.

Is this a downside or an example of the flexibility and power of OVAL? If there was a clear consensus that it is advisable to take or avoid one of these approaches, the community could certainly adopt that as a best practice and develop schemas and content that way. As things are, I take the fact that these three approaches are used to signify that the community finds them each useful and is taking advantage of OVAL’s ability to support them all.

> 
> 2) The OVAL SC model is pretty straightforward, but using OVAL definitions to drive collection is problematic to implement consistently.  This is largely due to ambiguities in the language that frequently are only resolved via an email thread archived on nabble.

I agree! The documentation should be improved on an ongoing basis to incorporate these resolutions.

> 3) An OVAL document doesn't scale particularly well.

Could you clarify what you mean by this? Are you saying that OVAL imposes some specific burden on implementations that hinder the implementations from scanning at scale? If so, we have not found that to be the case. For example, an instance of the Joval agent-less scanner running on commodity hardware can easily scan 300 devices concurrently, generating results in a few seconds or a few minutes per device depending on whether you’re running 1 definition or thousands. That means 3k+ devices per hour per instance. And, you can add instances to scan arbitrarily large numbers of devices in scan windows as short as several minutes.

> 
> 
> But even with all that in mind, I haven't seen anything more suitable for SACM use.
> 
> 
> --gun
> 
> 
>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/sacm <https://www.ietf.org/mailman/listinfo/sacm>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/sacm <https://www.ietf.org/mailman/listinfo/sacm>

-David

--
David E. Ries
Partner
Farnam Hall Ventures

---

From: Gunnar Engelbach <[email-masked]>
Date: Fri, 22 May 2015 00:39:31 -0400

On 5/21/2015 4:28 PM, David Ries wrote:
> Comments inline below:
>
>> On May 19, 2015, at 9:37 AM, Gunnar Engelbach 
>> <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>
>> 1) The existing test types and some schemas are inconsistent.  That 
>> is to say, some are modeled on the data to collect, some are modeled 
>> on a particular API, and others are modeled on a particular 
>> collection methodology.
>
> Is this a downside or an example of the flexibility and power of OVAL? 
> If there was a clear consensus that it is advisable to take or avoid 
> one of these approaches, the community could certainly adopt that as a 
> best practice and develop schemas and content that way. As things are, 
> I take the fact that these three approaches are used to signify that 
> the community finds them each useful and is taking advantage of OVAL’s 
> ability to support them all.

The flexibility of OVAL is that it is still being defined and revised.  
But when something is added that creates inherent and unnecessary 
limitations, I would not point to it as an example of flexibility.

>> consistently.  This is largely due to ambiguities in the language 
>> that frequently are only resolved via an email thread archived on nabble.
>
> Could you clarify what you mean by this? Are you saying that OVAL 
> imposes some specific burden on implementations that hinder the 
> implementations from scanning at scale? If so, we have not found that 
> to be the case. For example, an instance of the Joval agent-less 
> scanner running on commodity hardware can easily scan 300 devices 
> concurrently, generating results in a few seconds or a few minutes per 
> device depending on whether you’re running 1 definition or thousands. 
> That means 3k+ devices per hour per instance. And, you can add 
> instances to scan arbitrarily large numbers of devices in scan windows 
> as short as several minutes.

>>>  Furthermore, it provides some lessons learned over the course of 
>>> the past 10+ years by the OVAL community and some concrete 
>>> recommendations on how best to utilize the different models of the 
>>> OVAL Language within SACM.
>>> Email:[email-masked] <mailto:[email-masked]>
>>>
>>>
> /Partner/
> farnamhallventures.com <http://farnamhallventures.com>
>

---

From: Jerome Athias <[email-masked]>
Date: Fri, 22 May 2015 12:04:26 +0300

Hi David,

regarding your 2 specific questions, while I'm not an expert in
NETCONF/YANG (but know a little bit about it to understand how it is
interesting in the Domain of: network management - only for now);

in short, I would say that an approach similar to CybOX [1] (clear and
somehow in-depth pre-definitions of objects (static description for
collection)) would greatly help for fast and proper evaluation (with
proper Guidance which is another subject).
Note that while this could be seen as a huge effort (and it was for
OVAL), I do believe it is the best approach (controlled definitions
via a Domain Driven Design, controlled enumerations/vocabularies for
standardization, interoperability and automation).

If review and release processes take time, when it is done, it is
done, and IMHO it is better for long term strategy (than quick and
dirty, because "too fast, too bad"). Furthermore, I'm currently
observing in terms of implementation a lot of benefits from well
defined models (e.g. XML, UML) including easy and fast translation to
modern transport/exchange of data (e.g.: JSON) and an appreciated
relative stability and scalability.

[1] http://cybox.mitre.org/

My 2c

2015-05-21 23:02 GMT+03:00 David Ries <[email-masked]>:
> 4.5.  Collection Guidance: “Important Note: One of the key drawbacks to OVAL
> is that ... is not easy or scalable to create or update extensions for
> rapidly changing platforms and products in a timely manner.”
>
> Doesn’t the NETCONF schema demonstrate that it is not always necessary to
> create a platform extension to support a new or modified platform? Is there
> any language-based obstacle to adding more protocol-oriented schemas if that
> is desirable?
>
> When it is desirable to create or modify a platform-specific schema, I’ve
> found that the primary obstacle is speed of the review and release process.
> And, the community seems to be taking steps to change that. For example, the
> recently revised versioning policy now allows platforms to be rev’d
> independently of the core so they can be released more quickly.
>
> 6.2.1.  Lesson: “the close coupling found in OVAL between the OVAL Object
> and OVAL State ... does not easily allow for dynamic querying of previously
> collected data, nor does it easily allow for efficiencies in data
> collection.”
>
> I’ve heard this a few times in discussions of OVAL, but I’ve never really
> understood the criticism. It seems like this might be a misconception based
> on specific implementations of OVAL-based solutions, but not actually a
> limitation of OVAL itself.
>
> I see an OVAL document as a policy statement of sorts and not a high-level
> programming language. The document specifies data that needs to be collected
> and (optionally) makes assertions about that data. But, is there a
> requirement to “run the whole document” in one go when a scan is performed?
>
> To be more specific, I don’t see any obstacle to an enterprise solution
> storing the collected data and re-querying it later. Can’t you create
> definitions designed to collect posture attributes that include objects
> without states and store the collected data? Inventory definitions are
> another example of using OVAL to collect data that (presumably) is stored
> and queried later.
>
> Additionally, I don’t see inherent obstacles to optimizing data collection
> for multiple objects. Is there a requirement that each object be collected
> individually? Can’t the collector analyze the data required by each of the
> objects contained in the superset of definitions applicable to a managed
> device and identify strategies for optimizing collection? For example, if
> there’s an object that specifies files in directory X and another that
> specifies files in directories X and Y, I’d expect the collector to only
> scan directory X once.
>
> Retrieve large numbers of OVAL definitions (i.e. ~25k vulnerability and
> inventory definitions from Mitre, the Cisco IOS advisory definitions from
> Cisco, etc.)
> Analyze the definitions applicable to each platform in order to identify all
> of the data that must be collected from a managed device in order to resolve
> all the variables, generate items, etc. and evaluate the applicable
> definitons
> In such a solution, evaluating new or modified definitions may be possible
> based entirely on previously collected data.
>
> Best,
> David
>
>
>
>
> This document discusses how the Open Vulnerability and Assessment Language
> (OVAL) can be leveraged to fulfill some aspects defined by the SACM
> Information Model.  Specifically, it discusses the various components of the
> Information Model where the OVAL Language could be used either fully or
> partially as a Data Model for those components.  Furthermore, it provides
> some lessons learned over the course of the past 10+ years by the OVAL
> community and some concrete recommendations on how best to utilize the
> different models of the OVAL Language within SACM.
>
>
>
> We welcome your feedback and/or questions on the document and encourage you
> to review ahead of the virtual interim meeting next week, where we will
> spend a few minutes talking about the draft.
>
>
>
> Thanks
>
> Matt
>
>
>
>
>
> Email: [email-masked]
>
>
>
> farnamhallventures.com

---

From: Jerome Athias <[email-masked]>
Date: Fri, 22 May 2015 12:24:56 +0300

Hi,

Regarding 1) and 2) I see an opportunity to save a lot of time for
SACM, and meantime enhance OVAL System Characteristics Model.
I get your point and do agree that OVAL SC Model is not perfect.
Now, are you motivated to tackle this with us?

2015-05-19 17:37 GMT+03:00 Gunnar Engelbach <[email-masked]>:
> 1) The existing test types and some schemas are inconsistent.  That is to
> say, some are modeled on the data to collect, some are modeled on a
> 2) The OVAL SC model is pretty straightforward, but using OVAL definitions
> to drive collection is problematic to implement consistently.  This is
> largely due to ambiguities in the language that frequently are only resolved
> via an email thread archived on nabble.
>

---

From: Jerome Athias <[email-masked]>
Date: Fri, 22 May 2015 13:06:17 +0300

Hi,

after first review. Few comments:

First thing first, Thank you! ...
Lessons learned and gaps will be useful

I do appreciate honesty of recommending not to use OVAL Results Model,
and highlighting known identified limitations of the OVAL language.

I concur and appreciate the lesson learned regarding SMEs involvement.
(it is needed and critical while 'this thing' is just BIG, and could
allow Domain-driven design [1], consensus-per-domain). And we should
not limit innovation (Ref. 6.6.1)

KISS principle is good [2]. And I would add to this that it starts
with a proper and intelligent Taxonomy (while manipulated concepts
could seem obvious, it is not the case. e.g.: Event [6], system,
endpoint, ...)

Following Gunnar comments and my modest point of view on the current
OVAL System Characteristics Model (and few other MITRE models), I
would be happy to share my comments/arguments for some enhancements
(e.g.: regarding the class types). [5]
(6.3.1 "it will always require a frequent revision cycle")
(and that could help fighting against the "OVAL content is costly to
create" ref. 6.9.1 statement)

Regarding the pure wording of the document, in case you decide to update it;
3.2.
I suggest adding the concept of controlled vocabularies close to the
enumerations concept
I suggest adding "reducing redundancy" + "and providing better
interoperability and automation"

4.1.
regarding "The OVAL Language does not provide any mechanism for
instructing tools where to send collected data..."
why not introducing CAESARS? [3]

4.5.
"the OVAL Definitions Model could be modified to support monitoring of events."
Please kindly introduce this to STIX community asap! [4]

"The OVAL System Characteristics Model ... can be used to implement a
data format for sharing collected data."
Please kindly come discuss this in the STIX community.

"a standardized format is required"
I could try to sell you some XORCISM / IVIL here [5]

4.3.
"time interval"
This is discussed in STIX community also

"event"
see https://cybox.mitre.org/about/faqs.html#B15
(Object/Observable and State...)

6.3.2.
"SACM should focus on providing the foundational building blocks"
My lesson learned in addition: Taxonomy, definitions coming from the
SMEs are needed.
+
6.4.2.
SMEs should define the objects and enumerations/vocabularies (while
finding the appropriate "balance", ref. 6.6.2.)

6.5.2.
I suggest thinking about the concept of Maturity Levels (or Levels of
requirements)

6.7.2.
I do agree that Context is critical to capture.

6.8.1.
The concept of Confidence (as opposed to Fidelity) is important in
XORCISM [5] and currently under discussion in the STIX community.

4.2.
typo:
"security posture of one more endpoints"
should be
"security posture of one or more endpoints"

[1] https://en.wikipedia.org/wiki/Domain-driven_design

[2] https://en.wikipedia.org/wiki/KISS_principle

[3] http://www.dhs.gov/continuous-asset-evaluation-situational-awareness-and-risk-scoring-reference-architecture-report

[4] http://stix.mitre.org/

[5] http://www.frhack.org/research/xorcism.php
https://www.cupfighter.net/2010/10/ivil-an-xml-schema-to-exchange-vulnerability-information

[6] https://cybox.mitre.org/about/faqs.html#B15

---

From: Jerome Athias <[email-masked]>
Date: Fri, 22 May 2015 19:44:26 +0300

I would be interested, if possible, to have a feedback or lesson
learnt regarding SWID.
Thanks