Subject: Re: Alarm #40 - another thought

From: "Tom Petch" <[email-masked]>
Date: Thu, 9 May 2002 14:53:49 +0100

When I use OpenView (sorry Works Right Now Network Node Manager :-))
with an SNMPv1 network (my usual kind :-(), then it performs
correlation for me, it knows without being told the trap that clears
the 'alarm state' that a previous trap created, as in eg updating its
log or changing the colour of the objects on the map.

So there is an algorithm that works (at least for my scenarios);
anyone got any idea how that is done?  Anything we could learn from
that?

Tom Petch, Network Consultant
[email-masked]

---

From: "David T. Perkins" <[email-masked]>
Date: Thu, 09 May 2002 12:56:03 -0700

HI,

Tom - haven't used HPOV for a couple of years, so they may have improved
it by now. 

The problems with using SNMP notifications (sent via an SNMP trap or
inform message), are:
1) many of the definitions of notifications require that the
   implementation in the managed system throttle the sending,
   and, thus, there may be outstanding fault conditions
   and cleared fault conditions for which a notification
   is never sent.
2) SNMP trap messages are not confirmed (the receiver does not
   send a reply). Thus, these messages can be dropped by the
   network without anyone knowing! The result is again
   outstanding or cleared fault conditions for which the
   manager has received no message.
3) SNMP inform messages are confirmed with a response. Even so,
   sending is configured so that if no response is received
   after a configured number of retries, then no further attempt
   is made to resend. Note that an inform or response SNMP message
   may not be received due to network conditions, overload of
   the receiver, or the receiver being rebooted. Again the
   result is outstanding or cleared fault conditions for which
   the manager has received no message.
 
Several of us are still working on fine tuning the definition
of an alarm. However, in the current I-D, an alarm is defined 
as a persistent indication of a lasting error or warning condition,
so lets use it.

An important question to consider is the following:
Can the existence of all alarms (using above definition) and the
clearing of all previous alarms be determined by processing
a notification stream from a managed system?
For me, given the above examples showing that there is no
guarantee that a manager will even get all notifications,
the answer is no. Even if you could assume that the
notification stream between a managed system and receiver
had no drops, you still could not answer the question with
a "yes" response because internally, some raise and clear
notification will never be generated due to "deep throttling".
That is, the internal APIs to tell an SNMP agent to generate
a notification will never be called. One example of deep
throttling is specified in RFC 2863, last paragraph of
section 3.1.15, that suggests that when an interface is
rapidly oscillating between the up and down states
that notifications not be generated. 

Lets assume that there is no deep throttling and no drops
between the managed system and the receiver. Is this enough
to guarantee that the receiver of the notifications will
be able to correctly determine the fault and warning conditions
on a managed device? My answer for this is a "maybe". It
is "maybe" because it is dependent on the definition of
the notifications! The classic example is the linkUp and
linkDown notifications define in RFC 1215. In the definion,
the variables don't include the value of ifOperStatus and
ifAdminStatus. They are important in determining whether
or not there is a warning or fault condition is because
setting ifAdminStatus to "down" results in generation of
a linkDown notification. However, this is not an error
or warning condition. The latest definition of linkDown
in RFC 2863 states that it is generated when the value
for operStatus "is about to enter the down state from some
other state (but not from the notPresent state). " This
definition misses two important error conditions, which
are:
1) when adminStatus is changed from down to up, but operStatus
   does not "follow" to up, due to a lasting error condition.
2) when an interface card is inserted, and the interfaces
   on it have a lasting error condition.

Are there many notifications that have definitions that do
not help in reporting the beginning or ending of a lasting
error or warning condition (that is an alarm raise or clear)?
I don't know the answer, but have a hunch that a large
percentage of the definitions are problematic. 

One one concern is that many managers like to classify alarms
with a severity. This is so that the most important problems
can be addressed first, followed by not so important problems.
In almost every system that I've seen, the rules for severity
assignment are "static". That is, the rules do not change the
classification due to current usage patterns. For example,
if a new interface is added to a router, and that interface
is not yet sending traffic, then that it has a problem is
less severe than problem on a high speed interface that is
running at near 100% utilization. Also, problems can have 
relations between each other. For example, consider a chassis
based system that supports 16 interface cards and three power
supplies. Each power supply can support up to eight interface
cards. If all three power supplies are present, and one fails,
the other two have enough capacity to keep the system running.
However, if two fail, then only eight interface cards can function.
Say this happens and there are 10 interface cards present
in the system. When the second power supply dies, say the
system can shut down two of the interface cards so that
the remaining 8 can function. What alarms should be present
on the system, and what should be the severity of each.
It is relatively easy to answer with dynamic alarm rules,
but hard, if not impossible, to describe with static rules.

>[email-masked]

Regards,
/david t. perkins