Subject: Re: [sacm] [Non-DoD Source] Comments to SACM Vulnerability Assessment Scenario

From: "Haynes, Dan" <[email-masked]>
Date: Mon, 7 Dec 2015 01:47:21 +0000

Hi Joe,

Thanks for the feedback!

Comments inline below.  Let me know if I am misunderstanding anything.

Thanks,

Danny

-----Original Message-----
From: Wolfkiel, Joseph L CIV DISA ID (US)
[mailto:[email-masked]] 
Sent: Thursday, November 19, 2015 4:43 PM
To: [email-masked]
Cc: Haynes, Dan <[email-masked]>
Subject: RE: [Non-DoD Source] [sacm] Comments to SACM Vulnerability
Assessment Scenario

Here are my comments to the scenario document:

Section 1, 2nd paragraph: " whether or not the flaw is present within an
enterprise, when

compared to information about the state of the enterprise's   endpoints"

Comment: Does it say which vulnerabilities were discovered on what software
on which endpoints?  There's not a definition or example of what
"vulnerability report" means.

[danny]: The input vulnerability report would likely tell you what software,
version, on which platform was affected and any other applicable data for
any endpoint.  After the assessment, you will know which vulnerabilities in
which software on which endpoints.

Section 2, bulleted list:

"The document begins with the assumption that the enterprise has

      received a vulnerability report , and that the report data has

      already been processed into a format that the enterprise's

      security software tools can understand and use.  In particular,

      this document:"

Comment: The text alludes to a requirement that the report be machine
readable, but never states it.

[danny]: The document does not specify whether or not the vulnerability
report is in a machine-readable format just that the necessary information
has been extracted and put in a format tool's can use.  With that said, I
think it would be great if there was some machine-readable format for
vulnerability reports.

Section 2, bulleted list:

"Does not discuss how the enterprise identifies a potentially

         relevant  vulnerability report."

Comment: It's not clear what a "potentially relevant" report is.

[danny]: This refers to the initial process of determining which
vulnerability reports are applicable to an enterprise and assumes:

    * an enterprise obtains applicable vulnerability reports in some manner
as they become available

    * non-applicable vulnerability reports are filtered out in some fashion

    * this is done prior to this scenario beginning

Given this, we should update the assumption to state "The document begins
with the assumption that the enterprise has received a vulnerability report
that is applicable to their environment, and the report data has already.".

Section 2, bulleted list:

" Does not discuss how the enterprise collects that report"

Comment: Does this address how the report was acquired, or how the data in
the report was acquired?

[danny]: No, it does not address how the enterprise collects the
vulnerability report from the publishing vendor.

Section 2, bulleted list:

   o  The document assumes that the enterprise has a means of

      identifying enterprise endpoints.  This could mean identifying

     endpoints as they join the network, actively scanning for

      connected endpoints, passive scanning of network traffic to

      identify connected endpoints, or some other method of accounting

     for the presence of all endpoints in the enterprise.

Comment: Does the statement about endpoint identification imply that the
report contains data about endpoints?

[danny]: No, this just means the enterprise has some way to know what
endpoints are on its network.  The vulnerability report only provides
information about what software and configurations result in a vulnerable
state.  The vulnerability report could then be used as input into an
assessment to determine if it is in fact vulnerable.

Section 2, bulleted list:

*  Does not specify how extracted endpoint information and

         vulnerability alert information are normalized to be

         compatible.

Comment: What are alerts normalized to be compatible with?

[danny]: Compatible here just means that the information about an endpoint
as well as the relevant data from the vulnerability report are normalized in
some way such that the data is comparable and an assessment can be
performed.

Section 2, bulleted list:

o  The document assumes that the enterprise has a policy by which

      assessment of endpoints based on vulnerability reports  are

      prioritized.  The document:

Comment: It appears that an organization will "assess" something on receipt
of a vulnerability report.  What does it assess?

[danny]: Yes. Based on the vulnerability report, an enterprise will
determine if the vulnerability report is applicable to an endpoint (not to
be confused with applicable to the enterprise during initial report
retrieval) and then carry out an assessment. The priority of that assessment
would be determined by the enterprise policy.  The priority would likely
affect the timeliness of the assessment, and even how often any
reassessments occur.  The priority of the assessments is given a nod in the
section titled Priority, but is not discussed as part of the overall
scenario.

Section 3, first paragraph:

"The first step in this scenario involves identifying endpoints and

collecting basic system information from them.  This occurs prior to

   the receipt of any specific vulnerability report and is part of the

   regular, ongoing monitoring of endpoints within an enterprise.  This

   process is not meant to report on, or gather data for any specific

   vulnerabilities.  The information gathered during this step could be

   applied in many enterprise automation efforts.  Specifically, in

   addition to vulnerability management, it could be used by

   configuration and license management tasks.  All of the information

   collected during this step is stored in a central location such as a

  CMDB ."

Comment: Please spell out "CMDB" on first use and provide a reference that
explains what the authors think a CMDB would contain.  A configuration
management database (CMDB) isn't necessarily assumed to contain a
comprehensive device inventory.  Arguably, it stores baseline configurations
and deviations to them.

[danny]: We just used this as a general place where we can store/retrieve
collection data (including inventory), guidance, assessment results
(including baseline configurations and deviations), etc.  It sounds like we
may need to provide a little more context on this.  We may also want to call
it something other than CMDB to avoid any of these assumptions (e.g.
repository with data stores which can hold all this information).

Section 3.1, first paragraph:

Unique endpoint IDs are used to

   allow for systems to be tracked over time  and also allow for proper

   counts of assets during inventories and other similar collections.

Comment: Is this the scope of how/what endpoint IDs are used for?  Are IDs
used to correlate information between sensors?

[danny]: This text was not meant to exclude between sensors.  It would be
fair to include across sensors as well.

Section 3.1, second paragraph:

This sub-step aligns with the Endpoint Discovery, Endpoint

  Characterization (automated collection data only) , and Endpoint

   Target Identification building block capabilities.

Comment: Please provide rationale on why manually collected data is
prohibited

[danny]: At the moment, I don't recall exactly why we included that
restriction :).  Let me find out and follow up on this one.

Section 3.3, bulleted list:

Suggest changing " Endpoint type- e.g., standard computer, printer, router,
mobile

      device, tablet, etc." to read " Endpoint function or general role -
e.g., standard computer, printer, router, mobile

     device, tablet, etc."

Comment: "type" is weakly defined and ambiguous.

[danny]: Another way we might want to classify endpoint type is by
traditional (workstation, server, etc.), mobile (cell phones, laptops,
tablets, etc.), network infrastructure (switches, routers, etc.), and
constrained (IoT, ICS, printer, etc.).

Section 3.3, bulleted list:

o  Hardware version/firmware

Comment: version or firmware of what?  The BIOS, cellular modem, hard drive?

[danny]: We did not specify anything in particular.  It was just an example
of the type of information an enterprise might want to collect about an
endpoint in its pre-assessment data collection activities.  An enterprise
could potentially collect all of the information you listed above.  With
that said, we think it would be fair to include BIOS versions and firmware
revisions to make it a little more clear.  The main point here, however, is
that the scenario is only talking about basic information about the
endpoint.

Section 3.3, bulleted list:

"o  Open ports/running services 

   o  Operating system optional component inventory - some OS' have

      optional components that can be installed which may not show up as

      separate pieces of software (e.g., web and ftp servers, demo web

      pages, shared libraries, etc.).  Note that this could also occur

      within third-party applications as well. "

Comment:  Suggest that neither of these bullets describes "basic" data,
since they are long, detailed lists that aren't specifically required to
identify or characterize a device.

[danny]: Seems like a reasonable suggestion.  We could either remove the
"basic" qualifier or separate them out into two different lists ("basic" and
"advanced").  Do you have a preference?

Section 3.3, bulleted list:

"Human-assigned endpoint attributes are yet another type of data item

   that could be collected here.  This would include any attributes that

   cannot be collected programmatically.  The attributes could be

   manually entered into a CMDB by a human after the initial data

   collection occurs.   The attributes could help with system

   categorization and may have influence on later prioritizations.  The

   human-assigned attributes could include:"

Comment: Recommend consideration of placing attributes on the endpoints so
they can be discovered by multiple sensors and also viewed and edited
locally.  This would enhance accuracy and ensure a common assessment by all
sensors collecting data from an endpoint.

[danny]: I could potentially see this as being an implementation detail.
Can your provide a little more information on how you envision this working?
Would organizations just place whatever human-assigned data they want on the
endpoint wherever they want?  Would there be mandated data models and
locations where the human-assigned information would be stored?  I am just
trying to think through the situation as this information would likely be
organization specific and unless an organization is developing its own
collection tools there is a good chance COTs collection tools would not know
about this information nor how to collect it.  Anyways, I think we can add
some more text to discuss this topic once we have a better understanding of
what SACM needs here.

Section 3.3, 5th paragraph:

" There is direct alignment with the Endpoint Component

   Inventory, Posture  Attribute Identification, and Posture Attribute

   Value Collection building block capabilities since the purpose of

this sub-step is to perform an initial inventory of the endpoint and

   collect basic attributes and their values."

Comment: Please define "posture"

[danny]: From the Terminology document (
<https://github.com/sacmwg/draft-ietf-sacm-terminology/blob/master/draft-iet
f-sacm-terminology.md>
https://github.com/sacmwg/draft-ietf-sacm-terminology/blob/master/draft-ietf
-sacm-terminology.md), "posture" is defined in the terminology document as:

"Defined in {{RFC5209}} as "configuration and/or status of hardware or
software on an endpoint as it pertains to an organization's security
policy." This term is used within the scope of SACM to represent the
configuration and state information that is collected from a target endpoint
in the form of endpoint attributes (e.g. software/hardware inventory,
configuration settings, dynamically assigned addresses). This information
may constitute one or more posture attributes."

Also, "posture attribute" is defined as:

"Defined in {{RFC5209}} as "attributes describing the configuration or
status (posture) of a feature of the endpoint. A Posture Attribute
represents a single property of an observed state. For example, a Posture
Attribute might describe the version of the operating system installed on
the system."  Within this document this term represents a specific assertion
about endpoint configuration or state (e.g. configuration setting, installed
software, hardware) represented via endpoint attributes. The phrase
"features of the endpoint" highlighted above refers to installed software or
software components."

Section 4, first paragraph:

"These stored vulnerability reports can be used and compared

   with later vulnerability reports for the purpose of duplicate

   detection and in some cases guidance on how to handle similar issues."

Comment: storing all individual reports in a data store and trying to
dynamically combine and collate them at run time shouldn't be a prescribed
behavior.

[danny]: Agree, we think this is an implementation detail.  An enterprise
could store all the reports in a single CMDB or they could store them as
part of the device history.  This could further be broken up into external
vulnerability report received from the vendor as well as the final format
used by tools.  We will rework this to make it clear that this is an
implementation detail. 

Section 4, second paragraph:

   All vulnerability reports should be assigned an internal tracking ID

   by the enterprise as a first step.  Incoming vulnerability reports

might not have a global identifier when they are received, and might

   never have one assigned.

Comment: Suggest reports should be assigned a unique ID by the transmitter
so the receiver isn't tasked with providing the ID

[danny]: It is true that many larger vendors use an ID (bug or even
vulnerability) to identify their vulnerability reports.  However, there are
many smaller vendors who do not provide an ID.  As well, some vendors may
not even acknowledge a vulnerability report generated by a third-party
security researcher.  In addition, there would be no consistency when using
the IDs across multiple vendors and third-parties.  The enterprise would
need to assign some kind of ID (maybe a GUID) within their CMDB to track the
vulnerability reports internally.  Other IDs assigned by third-parties
(e.g., CVE) could also be associated and used as needed.

Section 4, third paragraph, bulleted list:

o  Ingest Date - the date that the report was received by the enterprise .

Comment 1: Add "and time" so if two reports are received on the same day,
you can tell which one was first

[danny]: Agree, this change makes sense.

Comment 2: Is "enterprise" the same as "CMDB", since the report is
apparently machine readable and received by an application?

[danny]: I don't think this necessarily synonymous with a CMDB, but, an
enterprise could assign the ingest date and time when the report is added to
the CMDB.  With that said, I don't think an enterprise is required to do it
then.  An enterprise could decide to assign it in some other way.

Section 4, third paragraph, bulleted list:

Date and Report Release

Comment: Change to read " Date and time of Report Release"

[danny]: Agree, this change makes sense.

Section 4, third paragraph, bulleted list:

identify the period when a particular endpoint was first vulnerable

Change to read: " identify the period when a particular endpoint was first
assessed as vulnerable" - because you only know the first time a sensor
detected a vulnerability, which is dependent on someone discovering the
vulnerable state and a sensor looking for it on a given endpoint.  The
endpoint may have been vulnerable for years prior to the detection.

[danny]: Agree, this change makes sense.

Section 4, third paragraph, bulleted list:

o  Severity Score (if available) - these may be useful for later mitigation
prioritization.

Comment: Recommend that severities not be transmitted with the vulnerability
since they change over time and should be obtained from a continuously
updated lookup source

[danny]: Maybe there is a way a vulnerability report could externally
reference a vulnerability score somewhere so they can change as needed and
over time?  With that said, if we do something like this or out-of-band,
there is the potential we will lose the ability track the severity scores
over time.  We should discuss this further.  

Section 5, first paragraph:

If this is not sufficient to determine endpoint applicability, a secondary
data

  collection for additional data and attributes may be performed to

   determine status with regard to the vulnerability report.

Comment: This doesn't consider current remediation methodologies where
remediation packages that contain multiple patches are sent out and
determine whether/how to "fix" an endpoint at runtime.

[danny]: All we were trying to saying here is that there may be cases where
you need to do further assessment beyond what is possible with
pre-assessment collected data in order to determine if an endpoint is
vulnerable.  We also did not consider remediation because it is
out-of-scope.

Section 5.1, first paragraph:

This data would  already have been collected in the pre-assessment data
collection.

Change to read: " This data may have been collected in the pre-assessment
data collection."  There is no guarantee this has happened.

[danny]: Agree, this change makes sense.

Section 5, second paragraph, bulleted list:

o  Date of assessment - The date that the assessment was performed.

Change to read: " o  Date and time  of assessment - The date that the
assessment was performed."

[danny]: Agree, this change makes sense.

Section C.2 "Endpoint" bulleted list:

Endpoint type - the device type of the endpoint (e.g., standard

      computer, printer, router, mobile device, tablet, etc.)

Comment: This is ambiguous.  Typically a "tablet" is a type of "mobile
device", that may also function as a WIFI router when used as a hot spot.  I
think this list has a bunch of different, incompatible attributes jumbled
together.  Suggest it be broken out into multiple lists, each describing the
same "type" of 

characteristic (e.g. functionality, color, portability, form factor, etc)

[danny]: Another way we might want to classify endpoint type is by
traditional (workstation, server, etc.), mobile (cell phones, laptops,
tablets, etc.), network infrastructure (switches, routers, etc.), and
constrained (IoT, ICS, printer, etc.).

Section C.2 "Endpoint" bulleted list:

o  Hardware version/firmware - the hardware or firmware version (if
applicable)

Comment: This is meaningless without describing which hardware component
you're collecting this data for

[danny]: This is just intended to be a general definition and placeholder
for hardware/firmware information.  However, we can provide a couple basic
examples to make it more clear.

Section C.2 "Endpoint" bulleted list:

o  Operating system - Operating system name

Comment: Please state which part of the operating system descriptive text
you consider to be the "name" (e.g. Windows?  Windows 10?  Windows 10 Home?)

[danny]: This is just a high-level definition place holder for now.  If we
try to create data models and protocols around this work, we will definitely
need to be more specific here.

Section C.2 "Endpoint" bulleted list:

o  Installed software name  - List of all installed software packages

      (i.e., software inventory).  May or may not include software

      installed by the operating system.

Comment: Same issue for software names as for OS names

[danny]: Same answer as OS names :).

Section C.2 "Endpoint" bulleted list:

o  Open ports/services - Listening network ports (TCP and UDP ) and

      running services

Comment: Why is this list constrained to TCP and UDP?  I would expect GRE,
IPsec, and other protocols to also be of interest

[danny]: This wasn't intended to be an exhaustive list.  TCP and UDP were
just provided as common examples.

Section C.2 "Endpoint" bulleted list:

   o  Operating system optional component inventory - Operating system

      specific components and software (when NOT already included in the

      general software inventory)

Comment: Give examples or define a list.  The OS is made up of thousands of
"components"

[danny]: We provided a couple of examples on page 6 of the document "(e.g.,
web and ftp servers, demo web

      pages, shared libraries, etc.)", but, we will restate them here for
clarity.

Section C.2 "Endpoint" bulleted list:

  o  Location  - The physical location of an enterprise endpoint

      (department, room, etc.)

Comment: Describe how the location would be quantified.  Is it GPS
coordinates, is it a region name, is it a name pulled from an enumeration
defined at a URL?

[danny]: Like some of the other examples this was more of a general place
holder.  What this actually means will likely be determined in a data model
that would contain location information.

Section C.2 "Endpoint" bulleted list:

o  Role - The function  that the endpoint serves within the enterprise

      (e.g., end user system, database server, public web server, etc.)

Comment: Defining a role as a function is circular and simply makes the two
words synonyms.  I suggest keeping "function" as a separate item from role.
For example, a role may be "workstation", while the functions would be
office automation, video playback, e-mail client, etc.

[danny]: This term is just thinking along the lines of the enterprise.
Maybe we can just use "purpose" instead of "function" to make sure that it's
not confused with any SACM terms (e.g. SACM Function)?  We may also want to
think about "role" since there is a "SACM Role".

Section C.2 "Endpoint" bulleted list:

o  File system attributes - Attributes that describe the state of a

      file or directory (e.g., versions, size, write date, modified

      date, checksum, etc.)

Comment: Remove "the state of" since other attributes that aren't
state-based are also of interest

[danny]: Agree, this change makes sense.

Section C.2 "External Vulnerability Report"

Comment: Prior to listing properties of an external vulnerability report,
please describe what you mean.  Is a report "external" because it was
created by someone outside of your organization?  Is it a report about a
newly discovered vulnerability, or a vulnerability discovered on one of your
endpoints by an external entity?

[danny]: Sure, we will work to improve this.

Overall Comment: Please provide a description of the different "reports"
described in this paper in the introduction.  After reading through twice, I
think you are using vulnerability "report" and vulnerability "definition" to
mean the same thing.  A vulnerability assessment is apparently the process
of detecting vulnerabilities on specific endpoints leading to a
"vulnerability assessment report".  This is never explicitly defined or
addressed, so it is confusing to try to build a mental model of the ideas in
the scenario.

[danny]: We will work to clarify this.

I also found a few other comments not listed here that were in the original
word document.  Here are responses to those comments.

Section 3.3 Paragraph 4 bulleted list "Criticality"

Comment: Possibly broken out by C/I/A or MAC and Conf levels.

[danny]: Yes, those are all possibilities.  In this document, we didn't try
to dive down to that level of detail.  However, if the WG uses this document
to drive the development of data models and protocols to support this
scenario, we will need to discuss options and these should be considered.

Appendix A. Continuous Vulnerability Assessment

Comment: You generally can't fix vulnerabilities in a BYOD environment, so
It's not clear what value knowing about them brings.

[danny]: Yeah, this is likely dependent on an organization's BYOD policy.
An organization might not have access or as part of the BYOD policy require
users' provide access to BYOD devices.  We can better figure this out
depending on whether or not we want BYOD to be in scope for this first
iteration of work.

Appendix B. Priority

Comment: For instance, a vulnerability affecting an operating system or
application used through the enterprise would likely be prioritized higher
than a vulnerability in an application which is used only on a few,
low-criticality endpoints.

[danny]: I think a system administrator would be able to do this with a
general understanding of what operating systems and software they use.  For
example, they probably know if Windows 7, RHEL 6, Microsoft Office, Apache
Webserver, etc. are widely deployed throughout their enterprise.  This might
be good for a high-level triage of vulnerability reports.  With that said,
you would probably need to look to the pre-collection data in order to
determine a more granular and comprehensive priority.  The severity score of
the vulnerability report could also be used in determining priority.

Section C.2 "Criticality - An enterprise-defined rating (possibly a score)
that helps determine the criticality of the endpoint.  If this endpoint is
attacked or lost, what is the impact to the overall enterprise?"

Comment: "possibly a score" should be "set of scores".

[danny]: Agree, this change makes sense.

Section C.2 "External Vulnerability Report"

Comment: So these are required attributes of an external vulnerability
report?

[danny]: Yes, these are the information needs (defined in Section 4) that
would be useful for a vulnerability report.  Is there anything that you
think needs to be added or subtracted?

Section C.2 "Assessment Report"

Comment: I would think a vulnerability report is a type of assessment
report.

[danny]: I think there may be some confusion due to us using the term
"report".  Vulnerability report is essentially a security advisory where the
information necessary to assess an endpoint to see if it is vulnerable is
available.  Assessment report is the results of the assessment possibly in
human form.  Based on the IETF 94 Information Model Update session, there
seemed to be some consensus that "reports" are out-of-scope for SACM.  Given
this, we probably want to reconsider this terminology to eliminate
confusion.

---END OF RESPONSES TO COMMENTS---

Joseph L. Wolfkiel

SCM Engineering Lead

DISA ID52

Fort Meade DISA Acquisiton Bldg Cube A4A58E

Work: (301) 225-8820

Gov Cell: (571) 814-8231

 <mailto:[email-masked]> [email-masked]

-----Original Message-----

From: sacm [ <mailto:[email-masked]> mailto:[email-masked]] On
Behalf Of Wolfkiel, Joseph L CIV DISA ID (US)

Sent: Tuesday, November 17, 2015 9:26 AM

To:  <mailto:[email-masked]> [email-masked]

Cc: Haynes, Dan

Subject: [Non-DoD Source] [sacm] Comments to SACM Vulnerability Assessment
Scenario

All active links contained in this email were disabled.  Please verify the
identity of the sender, and confirm the authenticity of all links contained
within the message prior to copying and pasting the address to a Web
browser.  

 

----

I sent Danny Haynes comments on the vulnerability assessment document at:
Caution-https://datatracker.ietf.org/doc/draft-coffin-sacm-vuln-scenario/
Since they may be relevant to some of our discussions, I'm attaching a
document with the comments.  With my apologies, it was easier to insert
comments in a word document for review, so I'm attaching that.

Joseph L. Wolfkiel

SCM Engineering Lead

DISA ID52

Fort Meade DISA Acquisiton Bldg Cube A4A58E

Work: (301) 225-8820

Gov Cell: (571) 814-8231

 <mailto:[email-masked]> [email-masked]