Subject: [sacm] SACM data and packaging it for transmission

From: "Wolfkiel, Joseph L CIV DISA ID (US)" <[email-masked]>
Date: Fri, 14 Aug 2015 19:02:27 +0000

To follow up on a discussion at the endpoint ID meeting this morning, I would like to share the following discussion and definitions I had with my developers over the past few weeks.  The issue is how to most accurately describe the content of the different inventory and compliance reports that are sent between our sensors and the central repository.  I am in the process, now, of trying to move from a model where every time one of our sensors sends in data they have to send in either a complete set of inventory information or a complete set of compliance results.    In the future, the model is to send in an initial baseline of compliance and inventory information, then send only changes, with periodic data re-synchronization jobs to guard against data corruption.

To support machine processing of future and legacy publishing formats and content, we are building manifest descriptions in the WSNotification Header that is put on messages sent by our sensors so we can route received data to the "correct" processing logic.

*** The painful detail follows.  Please let me know if this is useful.  I have fairly complete design documents for data collection, transmission schema bindings for publication, proposed database schemas and processing logic, plus design documents for many of the sensors and data collection tools that we've had to build to automate the end-to-end security configuration data collection, analysis, and presentation processes.  If I don't hear back that it's useful, I won't spam the list with more e-mails like this.*** 

As a background, I provided informal definitions for the two types of assertions that are included in our data reports about what was discovered or computed by our sensors.  First, you need to define the data that is being collected.  Because I am primarily interested in assessments of the security posture of DoD assets, my focus is primarily on states of assets.  That definition follows:

STATE: Attributes associated with an object, properties of an object, or evaluations of an object, particularly with respect to its compliance with a policy

There are two ways to report on state that I am interested in.  Status and state change.  Those definitions follow:

STATUS:  A measurement of attributes associated with an object, properties of an object, or evaluations of an object's compliance with a policy at a given point in time
STATE CHANGE: A new attribute is associated with an object, a property of an object is added, changed, or deleted, or the compliance of an object with a policy is newly assessed or assessed to be different than it was previously

To be able to report status and state changes, we created a series of transport data models governing how we would package the state and status data, along with necessary provenance information.

Parameter definitions for the topic follow:

Publishing source: ACAS, HBSS, STIG_VIEWER, BLADE_LOGIC, SEP, (others as identified)
- Identification of the repository or sensor type that is transmitting the information

Data type list: SOFTWARE, BENCHMARK, OPERATIONAL_ATTRIBUTES, MANAGED_DEVICES, UNMANAGED_DEVICES
Data type definition:  The information contained in the message.

Format List: DETAILS, RESULTS, INVENTORY, CHANGES
Format Definition: The level and of detail contained in the report.
- DETAILS: Attributes about the assessed object, are contained in an object model (e.g. information about a device is associated with a JSON or XML
schema that represents information commonly collected about a device)
- RESULTS: Compliance assessment results containing items that were assessed against a criterion and return pass/fail/not checked/fixed results from the
XCCDF results enumeration.
- INVENTORY: A listing of items found (i.e. software and patches)
- CHANGES: A listing of new states associated with inventory or compliance results, timestamped as to when they occurred 

Population List: COMPLETE, COUNT, INCREMENTAL, DELTA, BASELINE
Population Definition: The set of assessment data included in the report, with respect to what portion of the entire population it represents
- COMPLETE: All results for every object/device in the population
- COUNT: Counts of how many of each inventory item or different compliance results there are for the referenced population **without reference to individual
objects/devices**
- INCREMENTAL: Subsets of the population are included in defined increments so that data publishing is distributed across a fixed time period
- DELTA: Only new data received since the last publishing job for the given population is included
- BASELINE: Establishes a new data set of a given device/object from which deviations will be tracked.  It notifies the consumer that the publisher has discarded historical data and will be ignoring any data sent prior to the new "baseline" data set.

Filter List: FILTERED, UNFILTERED
Filter Definition: A statement of what data and data elements that will be included or excluded from the published data set
- FILTERED: A filter has been used to limit data published
- UNFILTERED: No filter has been used to limit data published.  All available, relevant data is sent

Data Provider List: HCEE, PA, or SC (for benchmarks), ACCM or SCCM for software, HBSS for device details -- The data provider list should not be a fixed enumeration
Data Provider Definition: The application that collects the data from the object being assessed

It may also make sense to add a character that denotes the break between a topic descriptor and its parameters (such as benchmark name or publishing interval).  Maybe "?" or "#" would work. 

For example: 
 "HBSS.BENCHMARK?WINDOWS7_HIPS_RULES.CHANGES.DELTA.HCEE" would describe a change event-based data set with only changes occurring since the last publishing job for the WINDOWS7 HIPS RULES benchmark.

"HBSS.SOFTWARE.CHANGES.COMPLETE?7DAYS_DAY=2.UNFILTERED.ACCM" would describe an ACCM change event-based publishing job from HBSS that is on the second day of an incremental publishing job that will provide full software details over a 7 day period.

The text after the parameter delimiter wouldn't be parsed and understood by the consumer, but it would be available for human review to assist in troubleshooting.

It would really make more sense to break the topic up into elements, but we have combined them all into a single line because the topic is saved as part of the file name when the data is persisted to disk. Having a single line of text is easier to parse out and create in a file name.

Joseph L. Wolfkiel
SCM Engineering Lead
DISA ID52
Work: (301) 225-8820
Gov Cell: (571) 814-8231
[email-masked]

-----Original Message-----
From: Haynes, Dan [mailto:[email-masked]] 
Sent: Friday, August 14, 2015 11:13 AM
To: Wolfkiel, Joseph L CIV DISA ID (US)
Subject: Definition of a Data Model

Hi Joe,

Here is a link to the draft that Ira and Dave mentioned on the call that discusses information models and data models (http://www.rfc-editor.org/rfc/rfc3444.txt).

Regarding the different things that you are working on, it sounds like you are doing a lot of interesting work.  If you have any feedback on capabilities and data that you need supported (I will capture your need for hardware, BIOS, identifiers, etc.), lessons learned, or have solution contributions (data models, bindings, protocols) that you would like to see in SACM, I would encourage you to share it with the list.  I think that would be very valuable for the group given that you have been on the frontlines trying to operationalize a lot of what SACM is trying to accomplish.

Let me know if you have any questions or if there is anything else I can help you with.  Have a great weekend.

Thanks,

Danny

---

From: "Haynes, Dan" <[email-masked]>
Date: Fri, 21 Aug 2015 11:16:02 +0000

Hi Joe,

Sorry just getting to this now.  I find messages like this useful - so keep sending them :)!  In my opinion, they serve as experience and lessons learned that the group can draw on.  It may even cause members of the group to think of things differently or not previously considered.

A couple quick comments/questions inline below (in blue).

Thanks,

Danny

-----Original Message-----
From: sacm [mailto:[email-masked]] On Behalf Of Wolfkiel, Joseph L CIV DISA ID (US)
Sent: Friday, August 14, 2015 3:02 PM
To: [email-masked]
Subject: [sacm] SACM data and packaging it for transmission

[danny]: I definitely like the model of only sending the changes.  Please keep us updated on how things go with moving to this model.  From hearing everyone's experiences with OVAL Results, sending all that data for each endpoint every time over the network is just not scalable.  In this model, when you send changes, are you planning to implement functionality in your sensors that will detect changes and send them to the central repository?  Or, are you planning to just do full assessments, compare it to previous assessments, and then just send information over to the central repository when something has changed?

[danny]:  +1 to enabling the support of both future and legacy formats and content.  I think it is critical for SACM's success and adoption.  We learned from OVAL that not all vendors are going to just drop what they are doing and switch to our data model.  By supporting arbitrary data models which may be open or proprietary, we have the best chances of adoption.  The one thing that we must require, however, is that all data models must be properly documented so that we know how to parse/process them prior to storing data in a repository.  Luckily a lot of data models already have this.  We will also need to figure out which ones are mandatory to implement, but, we can do that as we get solutions into SACM.

What do others think about this?  I know at IETF 92 in Dallas, we discussed this idea of supporting arbitrary data models in a SACM working session.

STATUS:  A measurement of attributes associated with an object, properties of an object, or evaluations of an object's compliance with a policy at a given point in time STATE CHANGE: A new attribute is associated with an object, a property of an object is added, changed, or deleted, or the compliance of an object with a policy is newly assessed or assessed to be different than it was previously

Data type list: SOFTWARE, BENCHMARK, OPERATIONAL_ATTRIBUTES, MANAGED_DEVICES, UNMANAGED_DEVICES Data type definition:  The information contained in the message.

Format List: DETAILS, RESULTS, INVENTORY, CHANGES Format Definition: The level and of detail contained in the report.

- DETAILS: Attributes about the assessed object, are contained in an object model (e.g. information about a device is associated with a JSON or XML schema that represents information commonly collected about a device)

- RESULTS: Compliance assessment results containing items that were assessed against a criterion and return pass/fail/not checked/fixed results from the XCCDF results enumeration.

Population List: COMPLETE, COUNT, INCREMENTAL, DELTA, BASELINE Population Definition: The set of assessment data included in the report, with respect to what portion of the entire population it represents

objects/devices**

Data Provider List: HCEE, PA, or SC (for benchmarks), ACCM or SCCM for software, HBSS for device details -- The data provider list should not be a fixed enumeration Data Provider Definition: The application that collects the data from the object being assessed

For example:

Joseph L. Wolfkiel

SCM Engineering Lead

DISA ID52

Work: (301) 225-8820

Gov Cell: (571) 814-8231

[email-masked]<mailto:[email-masked]>

-----Original Message-----

Hi Joe,

Thanks,

Danny

---

From: "Wolfkiel, Joseph L CIV DISA ID (US)" <[email-masked]>
Date: Fri, 21 Aug 2015 13:11:30 +0000

For the question directed to me, here's the response:

Response:  We have some of each.  Our agent-based SCAP tool vendor has developed has two major versions of their product on DoD networks.  The older major version sends in full results after every scan, but the more recent version only sends in the OVAL thin or full results when a test comes back with a different result than the last audit.  We have also a tool that we developed called the Asset Configuration Compliance Module (ACCM) that is going through a similar evolution.  In the 2.X series, each endpoint collected a full installed software and patch inventory, which was sent to the central controller.  At the central controller, the application did a diff against the database to see what was newly installed and what was deleted.  In ACCM 3.0 and later, that functionality has been moved to the endpoint.  At the endpoint, now, ACCM has a local SQL Lite database of the software and patch inventory from the last scan that it does a local diff against.  Once the diff is complete, it 
 prepares a message to the central repository with the information that it conducted a scan at a particular time and includes a list of any installs or deletions detected.  Using this methodology, we've been able to reduce the message size from 40-100KB down to 2-3KB.

We also have the problem of getting data between our agent controller and our central continuous monitoring repository.  For this case, we're doing a similar process.  However, since the central repository receives data asynchronously instead of collecting/computing results locally, the business logic and timestamps required are slightly different.  You end up with timestamps that may say that a change happened last Thursday, the agent collected the data last Friday, and the agent controller received the results Monday.

There are also the two cases where nothing has changed that have to be accounted for.  1) I ran a scan and didn't find anything new to send; and 2) I didn't receive any new data from my agents, so I need to let the continuous monitoring repository know that nothing has changed.  

Joseph L. Wolfkiel
SCM Engineering Lead
DISA ID52
Work: (301) 225-8820
Gov Cell: (571) 814-8231

-----Original Message-----
From: Haynes, Dan [mailto:[email-masked]] 
Sent: Friday, August 21, 2015 7:16 AM
To: Wolfkiel, Joseph L CIV DISA ID (US); [email-masked]
Subject: RE: SACM data and packaging it for transmission

Hi Joe,

Thanks,

Danny

-----Original Message-----
From: sacm [mailto:[email-masked]] On Behalf Of Wolfkiel, Joseph L CIV DISA ID (US)
Sent: Friday, August 14, 2015 3:02 PM
To: [email-masked]
Subject: [sacm] SACM data and packaging it for transmission

Response:  We have some of each.  Our SCAP tool vendor has developed has two major versions of their product on DoD networks.  The older major version sends in full results after every scan, but more recent version only sends in the OVAL thin or full results when a test comes back with a different result than the last audit.  We have a tool that we developed called the Asset Configuration Compliance Module (ACCM) that is going through a similar evolution.  In the 2.X series, each endpoint collected a full installed software and patch inventory, which was sent to the central controller.  At the central controller, the application did a diff against the database to see what was newly installed and what was deleted.  In ACCM 3.0 and later, that functionality has been moved to the endpoint.  At the endpoint, now, ACCM has a local SQL Lite database of the software and patch inventory from the last scan that it does a local diff against.  Once the diff is complete, it prepares a message to
  the central repository with the information that it conducted a scan at a particular time and includes a list of any installs or deletions detected.  Using this methodology, we've been able to reduce the message size from 40-100KB down to 2-3KB.

objects/devices**

For example: 

Joseph L. Wolfkiel

SCM Engineering Lead

DISA ID52

Work: (301) 225-8820

Gov Cell: (571) 814-8231

[email-masked] <mailto:[email-masked]> 

-----Original Message-----

From: Haynes, Dan [mailto:[email-masked] <mailto:[email-masked]> ]

Hi Joe,

Here is a link to the draft that Ira and Dave mentioned on the call that discusses information models and data models (http://www.rfc-editor.org/rfc/rfc3444.txt <http://www.rfc-editor.org/rfc/rfc3444.txt> ).

 

 

Thanks,

Danny

---

From: "Haynes, Dan" <[email-masked]>
Date: Fri, 21 Aug 2015 17:43:59 +0000

Thanks for the clarification Joe!

-Danny

-----Original Message-----
From: Wolfkiel, Joseph L CIV DISA ID (US)
[mailto:[email-masked]] 
Sent: Friday, August 21, 2015 9:12 AM
To: Haynes, Dan <[email-masked]>; [email-masked]
Subject: RE: SACM data and packaging it for transmission

[danny]: I definitely like the model of only sending the changes.  Please
keep us updated on how things go with moving to this model.  From hearing
everyone's experiences with OVAL Results, sending all that data for each
endpoint every time over the network is just not scalable.  In this model,
when you send changes, are you planning to implement functionality in your
sensors that will detect changes and send them to the central repository?
Or, are you planning to just do full assessments, compare it to previous
assessments, and then just send information over to the central repository
when something has changed? 

Response:  We have some of each.  Our agent-based SCAP tool vendor has
developed has two major versions of their product on DoD networks.  The
older major version sends in full results after every scan, but the more
recent version only sends in the OVAL thin or full results when a test comes
back with a different result than the last audit.  We have also a tool that
we developed called the Asset Configuration Compliance Module (ACCM) that is
going through a similar evolution.  In the 2.X series, each endpoint
collected a full installed software and patch inventory, which was sent to
the central controller.  At the central controller, the application did a
diff against the database to see what was newly installed and what was
deleted.  In ACCM 3.0 and later, that functionality has been moved to the
endpoint.  At the endpoint, now, ACCM has a local SQL Lite database of the
software and patch inventory from the last scan that it does a local diff
against.  Once the diff is complete, it 
 prepares a message to the central repository with the information that it
conducted a scan at a particular time and includes a list of any installs or
deletions detected.  Using this methodology, we've been able to reduce the
message size from 40-100KB down to 2-3KB.

We also have the problem of getting data between our agent controller and
our central continuous monitoring repository.  For this case, we're doing a
similar process.  However, since the central repository receives data
asynchronously instead of collecting/computing results locally, the business
logic and timestamps required are slightly different.  You end up with
timestamps that may say that a change happened last Thursday, the agent
collected the data last Friday, and the agent controller received the
results Monday.

There are also the two cases where nothing has changed that have to be
accounted for.  1) I ran a scan and didn't find anything new to send; and 2)
I didn't receive any new data from my agents, so I need to let the
continuous monitoring repository know that nothing has changed.  

Joseph L. Wolfkiel
SCM Engineering Lead
DISA ID52
Work: (301) 225-8820
Gov Cell: (571) 814-8231

-----Original Message-----
From: Haynes, Dan [mailto:[email-masked]] 
Sent: Friday, August 21, 2015 7:16 AM
To: Wolfkiel, Joseph L CIV DISA ID (US); [email-masked]
Subject: RE: SACM data and packaging it for transmission

Hi Joe,

Sorry just getting to this now.  I find messages like this useful - so keep
sending them :)!  In my opinion, they serve as experience and lessons
learned that the group can draw on.  It may even cause members of the group
to think of things differently or not previously considered.

Thanks,

Danny

-----Original Message-----
From: sacm [mailto:[email-masked]] On Behalf Of Wolfkiel, Joseph L
CIV DISA ID (US)
Sent: Friday, August 14, 2015 3:02 PM
To: [email-masked]
Subject: [sacm] SACM data and packaging it for transmission

To follow up on a discussion at the endpoint ID meeting this morning, I
would like to share the following discussion and definitions I had with my
developers over the past few weeks.  The issue is how to most accurately
describe the content of the different inventory and compliance reports that
are sent between our sensors and the central repository.  I am in the
process, now, of trying to move from a model where every time one of our
sensors sends in data they have to send in either a complete set of
inventory information or a complete set of compliance results.    In the
future, the model is to send in an initial baseline of compliance and
inventory information, then send only changes, with periodic data
re-synchronization jobs to guard against data corruption.

[danny]: I definitely like the model of only sending the changes.  Please
keep us updated on how things go with moving to this model.  From hearing
everyone's experiences with OVAL Results, sending all that data for each
endpoint every time over the network is just not scalable.  In this model,
when you send changes, are you planning to implement functionality in your
sensors that will detect changes and send them to the central repository?
Or, are you planning to just do full assessments, compare it to previous
assessments, and then just send information over to the central repository
when something has changed? 

Response:  We have some of each.  Our SCAP tool vendor has developed has two
major versions of their product on DoD networks.  The older major version
sends in full results after every scan, but more recent version only sends
in the OVAL thin or full results when a test comes back with a different
result than the last audit.  We have a tool that we developed called the
Asset Configuration Compliance Module (ACCM) that is going through a similar
evolution.  In the 2.X series, each endpoint collected a full installed
software and patch inventory, which was sent to the central controller.  At
the central controller, the application did a diff against the database to
see what was newly installed and what was deleted.  In ACCM 3.0 and later,
that functionality has been moved to the endpoint.  At the endpoint, now,
ACCM has a local SQL Lite database of the software and patch inventory from
the last scan that it does a local diff against.  Once the diff is complete,
it prepares a message to
  the central repository with the information that it conducted a scan at a
particular time and includes a list of any installs or deletions detected.
Using this methodology, we've been able to reduce the message size from
40-100KB down to 2-3KB.

To support machine processing of future and legacy publishing formats and
content, we are building manifest descriptions in the WSNotification Header
that is put on messages sent by our sensors so we can route received data to
the "correct" processing logic.

[danny]:  +1 to enabling the support of both future and legacy formats and
content.  I think it is critical for SACM's success and adoption.  We
learned from OVAL that not all vendors are going to just drop what they are
doing and switch to our data model.  By supporting arbitrary data models
which may be open or proprietary, we have the best chances of adoption.  The
one thing that we must require, however, is that all data models must be
properly documented so that we know how to parse/process them prior to
storing data in a repository.  Luckily a lot of data models already have
this.  We will also need to figure out which ones are mandatory to
implement, but, we can do that as we get solutions into SACM.  

What do others think about this?  I know at IETF 92 in Dallas, we discussed
this idea of supporting arbitrary data models in a SACM working session.

*** The painful detail follows.  Please let me know if this is useful.  I
have fairly complete design documents for data collection, transmission
schema bindings for publication, proposed database schemas and processing
logic, plus design documents for many of the sensors and data collection
tools that we've had to build to automate the end-to-end security
configuration data collection, analysis, and presentation processes.  If I
don't hear back that it's useful, I won't spam the list with more e-mails
like this.*** 

As a background, I provided informal definitions for the two types of
assertions that are included in our data reports about what was discovered
or computed by our sensors.  First, you need to define the data that is
being collected.  Because I am primarily interested in assessments of the
security posture of DoD assets, my focus is primarily on states of assets.
That definition follows:

STATE: Attributes associated with an object, properties of an object, or
evaluations of an object, particularly with respect to its compliance with a
policy

There are two ways to report on state that I am interested in.  Status and
state change.  Those definitions follow:

STATUS:  A measurement of attributes associated with an object, properties
of an object, or evaluations of an object's compliance with a policy at a
given point in time STATE CHANGE: A new attribute is associated with an
object, a property of an object is added, changed, or deleted, or the
compliance of an object with a policy is newly assessed or assessed to be
different than it was previously

To be able to report status and state changes, we created a series of
transport data models governing how we would package the state and status
data, along with necessary provenance information.

Publishing source: ACAS, HBSS, STIG_VIEWER, BLADE_LOGIC, SEP, (others as
identified)

- Identification of the repository or sensor type that is transmitting the
information

Data type list: SOFTWARE, BENCHMARK, OPERATIONAL_ATTRIBUTES,
MANAGED_DEVICES, UNMANAGED_DEVICES Data type definition:  The information
contained in the message.

Format List: DETAILS, RESULTS, INVENTORY, CHANGES Format Definition: The
level and of detail contained in the report.

- DETAILS: Attributes about the assessed object, are contained in an object
model (e.g. information about a device is associated with a JSON or XML
schema that represents information commonly collected about a device)

- RESULTS: Compliance assessment results containing items that were assessed
against a criterion and return pass/fail/not checked/fixed results from the
XCCDF results enumeration.

- CHANGES: A listing of new states associated with inventory or compliance
results, timestamped as to when they occurred 

Population List: COMPLETE, COUNT, INCREMENTAL, DELTA, BASELINE Population
Definition: The set of assessment data included in the report, with respect
to what portion of the entire population it represents

- COUNT: Counts of how many of each inventory item or different compliance
results there are for the referenced population **without reference to
individual

objects/devices**

- INCREMENTAL: Subsets of the population are included in defined increments
so that data publishing is distributed across a fixed time period

- DELTA: Only new data received since the last publishing job for the given
population is included

- BASELINE: Establishes a new data set of a given device/object from which
deviations will be tracked.  It notifies the consumer that the publisher has
discarded historical data and will be ignoring any data sent prior to the
new "baseline" data set.

Filter Definition: A statement of what data and data elements that will be
included or excluded from the published data set

- UNFILTERED: No filter has been used to limit data published.  All
available, relevant data is sent

Data Provider List: HCEE, PA, or SC (for benchmarks), ACCM or SCCM for
software, HBSS for device details -- The data provider list should not be a
fixed enumeration Data Provider Definition: The application that collects
the data from the object being assessed

It may also make sense to add a character that denotes the break between a
topic descriptor and its parameters (such as benchmark name or publishing
interval).  Maybe "?" or "#" would work. 

For example: 

 "HBSS.BENCHMARK?WINDOWS7_HIPS_RULES.CHANGES.DELTA.HCEE" would describe a
change event-based data set with only changes occurring since the last
publishing job for the WINDOWS7 HIPS RULES benchmark.

"HBSS.SOFTWARE.CHANGES.COMPLETE?7DAYS_DAY=2.UNFILTERED.ACCM" would describe
an ACCM change event-based publishing job from HBSS that is on the second
day of an incremental publishing job that will provide full software details
over a 7 day period.

The text after the parameter delimiter wouldn't be parsed and understood by
the consumer, but it would be available for human review to assist in
troubleshooting.

It would really make more sense to break the topic up into elements, but we
have combined them all into a single line because the topic is saved as part
of the file name when the data is persisted to disk. Having a single line of
text is easier to parse out and create in a file name.

Joseph L. Wolfkiel

SCM Engineering Lead

DISA ID52

Work: (301) 225-8820

Gov Cell: (571) 814-8231

 

-----Original Message-----

Hi Joe,

Here is a link to the draft that Ira and Dave mentioned on the call that
discusses information models and data models
(http://www.rfc-editor.org/rfc/rfc3444.txt
<http://www.rfc-editor.org/rfc/rfc3444.txt> ).

Regarding the different things that you are working on, it sounds like you
are doing a lot of interesting work.  If you have any feedback on
capabilities and data that you need supported (I will capture your need for
hardware, BIOS, identifiers, etc.), lessons learned, or have solution
contributions (data models, bindings, protocols) that you would like to see
in SACM, I would encourage you to share it with the list.  I think that
would be very valuable for the group given that you have been on the
frontlines trying to operationalize a lot of what SACM is trying to
accomplish.

Let me know if you have any questions or if there is anything else I can
help you with.  Have a great weekend.

Thanks,

Danny