Subject: Re: Alarm #40 - another thought

From: "Tom Petch" <[email-masked]>
Date: Thu, 16 May 2002 20:27:24 +0100

David

Thanks for the comprehensive reply on the difficulties of creating an
alarm system.

I am thinking of the bumblebee which, until very recently,
mathematicians had proved could not fly.  Except when I look in my
garden I see them getting from flower to flower without touching the
ground, or anything else. I think I can rely on them continuing to
pollinate my garden.

So when I use OpenView, despite all the difficulties, it offers me
something which has much in common with the definitions of alarms as
used in disman.  And since those difficulties, eg of correlating
trap/notifications to determine the start and end of an alarm, are
still extant at last call, I wondered if the bumblebee, in the shape
of OpenView, might enlighten.  But the absence of any other postings
suggests not.

Tom Petch, Network Consultant (and sometime mathematician).
[email-masked]
-----Original Message-----
From: David T. Perkins <[email-masked]>
To: Tom Petch <[email-masked]>
Cc: disman <[email-masked]>
Date: 09 May 2002 20:54
Subject: Re: Alarm #40 - another thought

>HI,
>
>Tom - haven't used HPOV for a couple of years, so they may have
improved
>it by now.
>
>The problems with using SNMP notifications (sent via an SNMP trap or
>inform message), are:
>1) many of the definitions of notifications require that the
>   implementation in the managed system throttle the sending,
>   and, thus, there may be outstanding fault conditions
>   and cleared fault conditions for which a notification
>   is never sent.
>2) SNMP trap messages are not confirmed (the receiver does not
>   send a reply). Thus, these messages can be dropped by the
>   network without anyone knowing! The result is again
>   outstanding or cleared fault conditions for which the
>   manager has received no message.
>3) SNMP inform messages are confirmed with a response. Even so,
>   sending is configured so that if no response is received
>   after a configured number of retries, then no further attempt
>   is made to resend. Note that an inform or response SNMP message
>   may not be received due to network conditions, overload of
>   the receiver, or the receiver being rebooted. Again the
>   result is outstanding or cleared fault conditions for which
>   the manager has received no message.
>
>Several of us are still working on fine tuning the definition
>of an alarm. However, in the current I-D, an alarm is defined
>as a persistent indication of a lasting error or warning condition,
>so lets use it.
>
>An important question to consider is the following:
>Can the existence of all alarms (using above definition) and the
>clearing of all previous alarms be determined by processing
>a notification stream from a managed system?
>For me, given the above examples showing that there is no
>guarantee that a manager will even get all notifications,
>the answer is no. Even if you could assume that the
>notification stream between a managed system and receiver
>had no drops, you still could not answer the question with
>a "yes" response because internally, some raise and clear
>notification will never be generated due to "deep throttling".
>That is, the internal APIs to tell an SNMP agent to generate
>a notification will never be called. One example of deep
>throttling is specified in RFC 2863, last paragraph of
>section 3.1.15, that suggests that when an interface is
>rapidly oscillating between the up and down states
>that notifications not be generated.
>
>Lets assume that there is no deep throttling and no drops
>between the managed system and the receiver. Is this enough
>to guarantee that the receiver of the notifications will
>be able to correctly determine the fault and warning conditions
>on a managed device? My answer for this is a "maybe". It
>is "maybe" because it is dependent on the definition of
>the notifications! The classic example is the linkUp and
>linkDown notifications define in RFC 1215. In the definion,
>the variables don't include the value of ifOperStatus and
>ifAdminStatus. They are important in determining whether
>or not there is a warning or fault condition is because
>setting ifAdminStatus to "down" results in generation of
>a linkDown notification. However, this is not an error
>or warning condition. The latest definition of linkDown
>in RFC 2863 states that it is generated when the value
>for operStatus "is about to enter the down state from some
>other state (but not from the notPresent state). " This
>definition misses two important error conditions, which
>are:
>1) when adminStatus is changed from down to up, but operStatus
>   does not "follow" to up, due to a lasting error condition.
>2) when an interface card is inserted, and the interfaces
>   on it have a lasting error condition.
>
>Are there many notifications that have definitions that do
>not help in reporting the beginning or ending of a lasting
>error or warning condition (that is an alarm raise or clear)?
>I don't know the answer, but have a hunch that a large
>percentage of the definitions are problematic.
>
>One one concern is that many managers like to classify alarms
>with a severity. This is so that the most important problems
>can be addressed first, followed by not so important problems.
>In almost every system that I've seen, the rules for severity
>assignment are "static". That is, the rules do not change the
>classification due to current usage patterns. For example,
>if a new interface is added to a router, and that interface
>is not yet sending traffic, then that it has a problem is
>less severe than problem on a high speed interface that is
>running at near 100% utilization. Also, problems can have
>relations between each other. For example, consider a chassis
>based system that supports 16 interface cards and three power
>supplies. Each power supply can support up to eight interface
>cards. If all three power supplies are present, and one fails,
>the other two have enough capacity to keep the system running.
>However, if two fail, then only eight interface cards can function.
>Say this happens and there are 10 interface cards present
>in the system. When the second power supply dies, say the
>system can shut down two of the interface cards so that
>the remaining 8 can function. What alarms should be present
>on the system, and what should be the severity of each.
>It is relatively easy to answer with dynamic alarm rules,
>but hard, if not impossible, to describe with static rules.
>
>At 02:53 PM 5/9/2002 +0100, Tom Petch wrote:
>>When I use OpenView (sorry Works Right Now Network Node Manager :-))
>>with an SNMPv1 network (my usual kind :-(), then it performs
>>correlation for me, it knows without being told the trap that clears
>>the 'alarm state' that a previous trap created, as in eg updating
its
>>log or changing the colour of the objects on the map.
>>
>>So there is an algorithm that works (at least for my scenarios);
>>anyone got any idea how that is done?  Anything we could learn from
>>that?
>>
>>Tom Petch, Network Consultant
>>[email-masked]
>
>Regards,
>/david t. perkins
>

---

From: "David T. Perkins" <[email-masked]>
Date: Thu, 16 May 2002 13:10:11 -0700

HI,

Oh come on....

In HPOV, if something doesn't work, an operator can go and fix it.
Also, don't know how much background polling is going by HPOV
to recover from the situations that I described. And there are
two - missing that a monitored system is in an "alarm state",
and missing that an alarm state has been cleared on a monitored
system.

Regards,
/david t. perkins