Subject: IDMEF Assessment, Impact, Action, Confidence

From: Cynthia McLain <[email-masked]>
Date: Tue, 6 Nov 2001 21:05:33 -0500 (EST)

Hi all,

A proposal that attempted to address some of the
issues with impact and confidence was sent to the
mailing list on Sunday, Nov 4. To date we've
received only a deafening silence in reply. In
the case of impact, there are three options
available, so feedback on which of the options is
most acceptable to the community at large is
desirable.

In order to keep the publication of the new IDMEF
draft on track we'd like to get your comments by
Thursday noon EST. 

Thanks in advance for your feedback,
--Cindy
------------------------------------------------------------
Cynthia D. McLain                 M.I.T. Lincoln Laboratory
Information Systems Technology    244 Wood Street
phone:  781-981-3943              Lexington, MA 02420-9108
e-mail: [email-masked]       Fax: 781-981-0186
------------------------------------------------------------

---

From: "Chris M. Lonvick" <[email-masked]>
Date: Wed, 07 Nov 2001 07:17:12 -0600

Hi Cynthia,

I am still altogether opposed to having a Confidence
parameter.  In case you didn't receive my original
post on the subject, you may find it in the archive:
  http://www.semper.org/idwg-public/archive/0415.html

Thanks,
Chris

---

From: "David A. Curry" <[email-masked]>
Date: Wed, 07 Nov 2001 09:15:57 -0500

>>>>> On Wed, 7 Nov 2001, Chris M. Lonvick, identified as "Chris" below, wrote:

  Chris> I am still altogether opposed to having a Confidence parameter.  In
  Chris> case you didn't receive my original post on the subject, you may find
  Chris> it in the archive: http://www.semper.org/idwg-public/archive/0415.html

Chris,

I agree with your objections that (paraphrasing) there is no way to know
which analyzer to believe if they report the same event with different
confidence values (as well as a host of other similar problems).  And
you're right, providing the level of guidance to make it all perfect is
a major project.

But I also note that there has been fairly consistent desire from members
of the group to have a way to represent confidence values.  Enough so, I
think, that it's going to be done whether IDMEF supports it or not, with
the result being multiple different representations under AdditionalData.

Given that we "know" confidence is going to be shipped around in IDMEF, I
think it's better to provide a single way to do it that is usable by all
these different groups, rather than leaving it for AdditionalData (which
is intended more for the things we *haven't* thought of yet).

Would it ease your concerns if we were to add something like the following
to the description of the element:

  Caution:  It should be noted that different types of analyzers compute
    confidence values in different ways.  In general, confidence values
    from different types of analyzers cannot be compared to each other.
    At most, comparisons may only be meaningful among values from the
    same type of analyzer.  And in some cases, depending on the analyzer
    configuration, comparisons may only be meaningful among values from
    a single analyzer.  Care should be taken when implementing systems
    that process confidence values (such as event correlators) not to
    make comparisons or assumptions that cannot be supported by the
    system's knowledge of the environment in which it is working.

(or something like that; feel free to rewrite it).  At least this serves to
make people aware of the problem, even if we don't solve it in this version
of the specification.

--Dave

---

From: Mike Erlinger <[email-masked]>
Date: Wed, 7 Nov 2001 10:15:44 -0800

As far as having the Confidence parameter, I agree with Dave's approach
of documenting the parameter.  I believe that the WG wants the confidence
parameter and having it in IDMEF is much better than many different
representations in AdditionalData.  A long time ago, we had a similar
discussion in RMON about 'utilization', for which many algorithms exist.
We ended up with a similar 'Caution' statement.

mike

-- 
Mike Erlinger, Professor and Chair Computer Science
www:      http://www.cs.hmc.edu/~mike
email:    [email-masked]
smail:    Computer Science Dept., Harvey Mudd College,
          301 E. 12th Street, Claremont, CA, 91711 
          909-621-8912,   FAX: 909-621-8465

---

From: "Chris M. Lonvick" <[email-masked]>
Date: Wed, 07 Nov 2001 12:55:38 -0600

Hi Dave,

How about, then, going with the suggestion from David Chizmadia
about the {high|medium|low} values along with the Caution.

Caution:  It should be noted that different types of analyzers compute
     confidence values in different ways.  In general, confidence values
     from different types of analyzers cannot be compared to each other.
     At most, comparisons may only be meaningful among values from the
     same type of analyzer.  Care should be taken when implementing 
     systems that process confidence values (such as event correlators) 
     not to make comparisons or assumptions that cannot be supported by 
     the system's knowledge of the environment in which it is working.

I'd suggest dropping the sentence, 
     And in some cases, depending on the analyzer configuration, 
     comparisons may only be meaningful among values from a single 
     analyzer.
(Too nitty.)

Does that work?

Thanks,
Chris

---

From: Andy Walther <[email-masked]>
Date: Wed, 7 Nov 2001 11:05:18 -0800 (PST)

I think the current plan on the table is to do the following:

----------------------------------------------------------------------
15. The "Confidence" element represents the analyzer's best estimate of
the
    validity of its analysis:

       <!ENTITY % attvals.rating  "
           ( low | medium | high | numeric )
         ">

       <!ELEMENT Confidence (#PCDATA | EMPTY)>
       <!ATTLIST Confidence
           rating       %attvals.rating;         'numeric'
-----------------------------------------------------------------------

So the (low | medium | high ) concept is taken into account. We can't get
rid of numeric as an option though or we make correlation/weighing of
alerts within a given Analyzer that computes numeric confidence values
meaningless.

The caution is a good idea, and it seems that our hand is forced to go
that route...

Andy

---

From: "David A. Curry" <[email-masked]>
Date: Wed, 07 Nov 2001 14:18:03 -0500

>>>>> On Wed, 7 Nov 2001, Chris M. Lonvick, identified as "Chris" below, wrote:

  Chris> How about, then, going with the suggestion from David Chizmadia about
  Chris> the {high|medium|low} values along with the Caution.

Works for me.

As Andy has already pointed out, low/medium/high was already in Cindy's
proposal for those analyzers that don't compute "precise" numbers.

I have no problem with removing the "too nitty" sentence.

--Dave

---

From: Lansing Sloan <[email-masked]>
Date: Wed, 07 Nov 2001 12:54:58 -0800

Hi,

    This is just a nit.

    In the "caution" paragraph quoted below, I suggest deleting the
sentence that begins "At most," since it seems redundant with the
preceding sentence.

    If you keep the sentence, I think "At most" is not quite correct.
I'd say "In general" would be a correct replacement for "At most".
Or simply delete "At most".

    (If you think "At most" is correct, consider a thought
experiment: replace "At most," with "It is
impossible for a situation to be better than this:".  Then see if you
really believe "impossible" is correct.)

    The sentence that Chris Lonvick said is "too nitty" seems less
redundant.  Possibly the 2nd, 3rd, and 4th sentences could all be
replaced by something like: "In many cases, confidence values from
different analyzers cannot be compared to each other (for example,
if the analyzers are of different types or are of the same type but
differently configured)."

    I agree that the caution paragraph is warranted, and I haven't seen
any unacceptable proposal yet, so your best judgement should be fine.

-- Lansing Sloan

---

From: "David A. Curry" <[email-masked]>
Date: Wed, 07 Nov 2001 16:08:17 -0500

>>>>> On Wed, 7 Nov 2001, Lansing Sloan, identified as "Lansing" below, wrote:

  Lansing> I agree that the caution paragraph is warranted, and I haven't seen
  Lansing> any unacceptable proposal yet, so your best judgement should be
  Lansing> fine.

Actually, I like your sentence best, except that I joined it to the previous
one which (to me, anyway) improves the flow.  So here's how it reads now:

    Caution:  It should be noted that different types of analyzers compute
      confidence values in different ways and that in many cases, confidence
      values from different analyzers cannot be compared to each other (for
      example, if the analyzers are of different types or are of the same type
      but differently configured).  Care should be taken when implementing
      systems that process confidence values (such as event correlators) not
      to make comparisons or assumptions that cannot be supported by the
      system's knowledge of the environment in which it is working.

Comments?

--Dave

---

From: "John C. C. White" <[email-masked]>
Date: Thu, 08 Nov 2001 07:37:18 -0500

OK, you asked, so I will throw a monumental nit-pick. Namely, in the
proposed sentence, the phrase "cannot be compared to each other" should
be replaced by "cannot be compared with each other." The rationale is as
follows, quoted from the American Heritage Book of English Usage:

"Compare usually takes the preposition to when it refers to the activity
of describing the resemblances between unlike things: He compared her to
a summer day. Scientists sometimes compare the human brain to a
computer. It takes with when it refers to the act of examining two like
things in order to discern their similarities or differences: The police
compared the forged signature with the original. The committee will have
to compare the Senate’s version of the bill with the version that was
passed by the House."

-John-