Subject: [IDMEF][Issue 10] Severity scale too narrow

From: Herve Debar <[email-masked]>
Date: Fri, 09 Jan 2004 10:07:31 +0100

From Krzysztof Zaraska:
severity scale (3 grades only) is too narrow; for example, there is an
idea floating around of expressing the danger associated with the 
attack as exp(severity*vulnerabity_level), but you can't go far this 
way with 3 level severity scale only.

Hervé: we use 5 internally. It seems to be standard practice within 
the telco industry:
1) = critical
2) = warning
3) = minor
4) = normal
5) = unclassified/unclassifiable
This wouldn't solve your graduation issue, but it actually is enough 
for an operator. We also use another scale which has 3 levels,(solved 
within 1/2 hour), (solved within 1/2 day), (solved within 1 day), but 
that's maybe too operations-oriented.

IIRC, it is about the same with the Tivoli framework. I think there is 
a 0) = FATAL (i.e. system stops) in addition.

Possibilities:
- keep 3.
- move to five (and get appropriate definitions)
- something else

Please voice your opinion.

Hervé
-- 
Hervé Debar             <mailto:[email-masked]>
Tel: +33 (0)2 31 75 92 61            GSM: +33 (0)6 74 09 09 66
France Télécom R&D                   Fax: +33 (0)2 31 75 93 13
42 rue des Coutures  (--)  BP 6243  (--)  F-14066 Caen Cedex 4

---

From: Krzysztof Zaraska <[email-masked]>
Date: Sun, 11 Jan 2004 00:33:16 +0100 (CET)

> - keep 3.

I oppose :)

> - move to five (and get appropriate definitions)

Good idea, especially if it can be kept consistant with other things...
For example, syslog has own severity scale as well...

> - something else

I was thinking about a float number within 0 - 1 range (0 - harmless, 1 -
top privilege level compromise). Very flexible, but unfortunately
a little fuzzy...

// Krzysztof Zaraska * kzaraska (at) student.uci.agh.edu.pl
// http://mops.uci.agh.edu.pl/~kzaraska/ * http://www.prelude-ids.org/
// A dream will always triumph over reality, once it is given the chance.
//		-- Stanislaw Lem

---

From: Greg Shipley <[email-masked]>
Date: Sat, 10 Jan 2004 22:09:34 -0600 (CST)

I'm not sure if it's appropriate for me to comment on the severity topic
as I'm clearly not working on any of the drafts / part of the working
group, but on the off chance the following insight might be useful, here
goes:

The severity topic is a can of worms, but I believe the adoption of "SIM"
solutions (Security Information Management) in operational environments is
a testament to the need for more flexible classification models.  Many of
today's commercial SIM solutions will take in data from a security device
(firewall, NIDS, etc.) and attach a weight to it based on other data
points (asset rankings, vulnerability informations, etc.).

Security analysts can then use that weighting to help prioritize which
events they choose to investigate.  In a world where it's not uncommon for
a security analyst to be presented with 10k+ events in any given day, the
event weighting system *must* support a granular enough system to allow
effective prioritization.  (Obviously three choices won't help much with
prioritizing 10k events/alerts)

Now, again, I'm not entirely sure that the need for weighting is relevant
to the discussion at hand, but I know the limitations of syslog have also
been a problem in operational security units, and I'd hate to see those
mistakes duplicated again here.

Taking this from another angle, if you have an alert generated from a
device such as a NIDS sensor, the sensor itself might not be completely
capable of determining the alert level without further data points.  For
example, assume that device A is a NIDS sensor, and device B is a
vulnerability scanner.  While each device has its own view of the world,
their combined view can be far more accurate.  Extending the example, if
Device A detects an "IIS UNICODE" BO-based attack against system with IP
address y, and Device B knows that system y is vulnerable to such an
attack, the severity rating of that event may be weighted as z by a third
sensor/device/aggregator.  (By the same token, if Device B knows that
system y is NOT vulnerable to the attack it might provide information to
downgrade the weighting)

Bottom line: All this to say that I don't know what the ideal solution is
for severity rankings, but I am quite sure that our world will be moving
away from "high, medium, and low" rankings.  If anything is to be done
with severity rankings, it might be worth considering using an integer.
That way, at least the decision of severity could be determined by
endusers, implementors, data aggregation mechanisms, etc., and not be as
limited.

This is what I'm seeing done "in the field" today, anyway.

Hope this helps,

-Greg

---

From: Dan White <[email-masked]>
Date: Sun, 11 Jan 2004 10:02:37 -0500 (EST)

I agree with Greg,

SIM or Security Event Management products have many rules based
approaches that assign criticality. The real measure of severity
is the susceptibility to the enterprise of the attack. I.e. 
Does the vulnerabity that would allow an attack ssignature to be succssfull
exist?

Severity that is on a 1-10 scale with user configuration to change seems to be a reasonable suggestion?

---

From: Ozakil Azim <[email-masked]>
Date: Sun, 11 Jan 2004 17:47:05 -0500

>points (asset rankings, vulnerability informations, etc.).
Having worked on a SIM product in an earlier life, I tend to agree with 
Greg on this. Aggregating events and aggregating the impact of events 
was a problem we tried to solve by using a weighted system for risk 
assessment that took into consideration asset values, exposures, and 
perceived threats amongst other variables. An event reported by a 
'standalone' IDS was reevaluated in the context of the security of the 
whole enterprise.

 As per my reading of the draft, the 'severity' is an attribute of the 
'Impact' class that is meant to deliver the evaluated impact of the 
event as seen by the analyzer.  Even if this evaluation was used in a 
purely subjective manner, three grades of severity are too narrow. Given 
that the 'impact', as assessed by the 'analyzer',  may be further 
reassessed by higher order 'analyzers', we need to use some scale that 
can useful for further evaluation/aggregation. The question is whether 
one universal scale would suffice.

Analyzers assessing alerts from other analyzers need to take at least 
the following two aspects into consideration:
1) What is the trust relationship with the 'other' analyzer? Do I know 
this analyzer? What is my history with this analyzer? What is the role 
of this analyzer in the security posture of the enterprise? What is its 
rank in the security organization?  (my analogy drives from the P.o.V of 
a general gathering intelligence from a field agent :)
2) How severe does the analyzer think the event is? What is 'its' rating 
system? What are its grades?

Aspect 1) 'trust' between analyzers, I guess is out of scope of IDMEF. 
and aspect 2) seems to be what we are discussing.

My opinion is that the analyzer generating the alert knows best. If an 
analyzer decides based on its detection/analysis algorithms to have only 
three grades, that's okay.
If the analyzers decides to grade from 0-10,000 or 0.00 - 1.00, that's 
fine too. It would be great if the analyzer could specify the range of 
its 'severity' values on request
(or with each message - I may be wrong, but I don't seem to recall a 
query mechanism in the IDMEF draft). That way, each analyzer uses what 
it is most comfortable with, and leaves the 'normalizing' to the higher 
order 'analyzer'.

>endusers, implementors, data aggregation mechanisms, etc., and not be as
I agree that this decision needs to be re-determinable at all the way up 
the chain till the 'human'(A.I?) operator.

-azim

---

From: "David A. Curry" <[email-masked]>
Date: Sun, 11 Jan 2004 21:06:13 -0500

>>>>> On Fri, 9 Jan 2004, Herve Debar, identified as "Herve" below, wrote:

  Herve> From Krzysztof Zaraska: severity scale (3 grades only) is too narrow;
  Herve> for example, there is an idea floating around of expressing the danger
  Herve> associated with the attack as exp(severity*vulnerabity_level), but you
  Herve> can't go far this way with 3 level severity scale only.

I am way skeptical of any scheme that purports to assign numerical values to
this type of data and claim that they are "accurate" in any meaningful way.

As I recall, the group was similarly skeptical when the idea was proposed in
the past (and it was proposed more than once).

  Herve> Hervé: we use 5 internally. It seems to be standard practice within
  Herve> the telco industry:  1) = critical 2) = warning 3) = minor 4) = normal
  Herve> 5) = unclassified/unclassifiable This wouldn't solve your graduation
  Herve> issue, but it actually is enough for an operator. We also use another
  Herve> scale which has 3 levels,(solved within 1/2 hour), (solved within 1/2
  Herve> day), (solved within 1 day), but that's maybe too operations-oriented.

This seems reasonable to me.  It's fairly standard in other areas as well,
e.g. Microsoft alerts (I think), FS/ISAC alerts, and others.

Although we could argue for hours about whether the numbers should be in
ascending or descending order (proposal: Herve's choice :-).

--Dave

---

From: Krzysztof Zaraska <[email-masked]>
Date: Mon, 12 Jan 2004 16:26:19 +0100 (CET)

> this type of data and claim that they are "accurate" in any meaningful way.

Just to clarify: I was not proposing including this kind of metric in
IDMEF, I was merely giving an example of situation where 3 level severity
scale is too narrow.

---

From: Greg Shipley <[email-masked]>
Date: Mon, 12 Jan 2004 16:50:24 -0600 (CST)

On Sun, 11 Jan 2004, David A. Curry wrote:

> I am way skeptical of any scheme that purports to assign numerical
> values to this type of data and claim that they are "accurate" in any
> meaningful way.
>
> As I recall, the group was similarly skeptical when the idea was
> proposed in the past (and it was proposed more than once).

If it's a done deal, it's a done deal, and please view my ramblings as
nothing more than comments from the peanut gallery.  (and please excuse me
and my monkey wrench!) I just know from EXPERIENCE that simple
classification metrics (e.g. 5 ratings) are not going to cut it in many
real world scenarios.  For example, when we were doing our SIM testing
last year in conjunction with Syracuse University we had days when we hit
2600 events PER SECOND from our IDS/FW sensors.  Without some ability to
classify/rank those events more granularly you can forget about ever
really sifting through the data, much less acting on it.

IMO, granular classification mechanisms are mandatory when dealing with
large data volumes.  Now, whether or not that classification falls under
the IDWG charter is, well, for you folks to decide!

As for your skepticism, I totally agree with your concern about accuracy.
However, we're watching the NIDS space evolve right under our noses, and
I'm going to speculate that confidence levels will be appearing in
commercial products in less than 12 months.  Take, for example,
SourceFire's RNA product; they are using passive vulnerability
identification techniques to gather data for alert classifications.
(e.g.  That MS RPC/DCOM alert is critical because the target is indeed
vulnerable to that MS RPC attack)  I believe SourceFire's RNA is shipping,
TODAY, and other vendors are not far behind.

(Perhaps one leaves the classification system intact, and inserts an
additional/optional confidence variable, instead?)

How does this relate to the IDWG?  Again, that's for you folks to decide,
and I really don't intend to cause problems here.  But, IMO, I'm guessing
that granular classification systems and/or confidence levels will be a
factor for commercial implementations.  The commercial market is already
moving there NOW.  (And I'd be curious to hear what any of them think, if
they are on this list, but I digress...)

My .02,

-Greg

P.S. As for the ascending vs. descending, as long as its documented and
consistent, I don't think it matters...  (Herve's choice?  :)