Subject: [BEHAVE] Major NAT MIB Issue: Notifications and processor requirements at the agent

From: Tom Taylor <[email-masked]>
Date: Sun, 26 Oct 2014 18:35:27 -0400

A little terminology to start with:

1) A realm defines an addressing space. Simple NATs may support just two 
realms: internal and external. This is all that RFC 4008 supports. In 
more complex situations the NAT will support several or many realms. 
Individual realms in that case can be internal from the point of view of 
some hosts served by the NAT and external from the point of view of others.

2) An address mapping is an assignment of an address in a selected 
external realm to the address of a host in its internal realm.

3) An address and port mapping is a mapping between the address and port 
of a host in its internal realm for a given protocol to an address and 
port in a selected external realm. For mappings triggered by packets 
outgoing from the internal host, the external realm is selected based on 
the packet destination. For mappings triggered by packets incoming from 
an external realm, the external realm is the one from which the packet 
originated.

4) An address pool is a separately-managed set of addresses and 
ports/ICMP identifiers in a particular realm, available for assignment 
to the 'external' portion of a mapping. Pools can be used to ration the 
available external addresses to different realms and classes of 
subscribers. Where more than one pool has been assigned to the realm, 
policy determines which subscribers and/or services are mapped to which 
pool.

The NAT MIB (draft-ietf-behave-nat-mib-11) currently has five 
notifications, which can be enabled and disabled through their 
respective threshold settings:

- Address pool low threshold undershot
   -- reports under-utilization of an address pool

- Address pool high threshold exceeded
   -- reports potential over-utilization of an address pool

- Address mapping threshold exceeded
   -- reports potential exhaustion of addresses
      for the NAT instance as a whole

- Address and port mapping threshold exceeded
   -- reports potential exhaustion of address plus port
      combinations for the NAT instance as a whole

- Per-subscriber address and port mapping threshold exceeded
   -- can be used for troubleshooting (enabled for specific
      subscriber) or provisioning (enabled for all subscribers)

Just as a preliminary, my personal view is that the notification 
relating to low pool utilization should not be there, and if the 
information is needed, it should be handled by the management 
application. Hence this one should be omitted from the discussion which 
follows.

David Harrington in his review was concerned that these notifications 
involved calculations at the agent that were excessively demanding of 
processor capacity. This is especially true of the MIB's current 
algorithm for address pool utilization. More on that in a moment.

Just for background, the MIB supports the following read-write hard 
limits, beyond which incoming packets are dropped rather than translated:

- Maximum number of active address mappings
- Maximum number of active address and port mappings
- Maximum number of fragments awaiting reassembly
- Maximum number of subscribers with active address and port mappings

The resources that need monitoring in the NAT are NAT memory, address 
pool utilization, and processor capacity. The only item in the above 
list of notifications and limits that at least partially addresses 
processor capacity is the last limit, on number of active subscribers. 
Possibly, a notification and limit could be introduced based on the rate 
of change of the number of active mappings.

The problem with NAT pool monitoring is that the pool actually contains 
two resources, addresses and ports, and how they get used up depends on 
RFC 4787 pooling behaviour.

For the RECOMMENDED pooling behaviour of 'paired', port exhaustion 
occurs when all the ports for the desired protocol available to the 
subscriber on a mapped external address are used up, the NAT cannot (for 
reasons of policy or availability) assign additional ports to that 
subscriber on that same address, and the NAT cannot assign an additional 
address containing free ports to that subscriber. (This last condition 
is subject to discussion: is it port exhaustion or address exhaustion?)

For a pooling behaviour of 'arbitrary', port exhaustion occurs when 
there is no available port for the desired protocol at any of the 
addresses in the pool. Clearly port exhaustion will be a rarer event in 
the 'arbitrary' case.

draft-ietf-behave-nat-mib-11 triggers notifications based on the ratio 
of active address and port mappings using a given pool to the total 
number of address-port combinations made available by the pool, 
expressed as a percentage. As David Harrington pointed out, this makes a 
lot of demands on the agent processor. We have some alternatives:

1) No notification: leave any processing to the management application.

2) Trigger the high usage notification based on the rate of occurrence 
of out-of-port events, with the same threshold value for all pools. This 
is reasonably valid statistically for a low threshold (i.e., taking the 
view that any port exhaustion at all is actionable), but not terribly 
reliable in the 'paired' case, in that a single subscriber can dominate 
the results.

3) Trigger the high usage notification based on the rate of occurrence 
of out-of-port events, with a higher threshold value based on pool size 
(addresses x ports) and sharing ratio. This requires some mathematical 
analysis and per-pool configuration.

What do people recommend?

Tom Taylor

---

From: Simon Perreault <[email-masked]>
Date: Mon, 27 Oct 2014 09:35:33 -0400

On Sun, Oct 26, 2014 at 6:35 PM, Tom Taylor <[email-masked]>
wrote:

> realms: internal and external. This is all that RFC 4008 supports. In more
> complex situations the NAT will support several or many realms. Individual
> realms in that case can be internal from the point of view of some hosts
> served by the NAT and external from the point of view of others.
>
> of a host in its internal realm for a given protocol to an address and port
> in a selected external realm. For mappings triggered by packets outgoing
> from the internal host, the external realm is selected based on the packet
> destination.
>

While this is often true in practice, in theory the NAT could pick the
external realm based on whatever arbitrary rule with which it is
configured. For example, software NATs (e.g., Linux's Netfilter) allow a
lot of flexibility here.

> For mappings triggered by packets incoming from an external realm, the
> external realm is the one from which the packet originated.
>
> 4) An address pool is a separately-managed set of addresses and ports/ICMP
> identifiers in a particular realm, available for assignment to the
> 'external' portion of a mapping. Pools can be used to ration the available
> external addresses to different realms and classes of subscribers. Where
> more than one pool has been assigned to the realm, policy determines which
> subscribers and/or services are mapped to which pool.
>
>
> notifications, which can be enabled and disabled through their respective
> threshold settings:
>
> Just as a preliminary, my personal view is that the notification relating
> to low pool utilization should not be there, and if the information is
> needed, it should be handled by the management application. Hence this one
> should be omitted from the discussion which follows.
>

Yes, I agree that that can work in practice. However I don't see the
difference in reasoning between this notification and the others. Couldn't
the others also be handled by the management application?

> processor capacity. This is especially true of the MIB's current algorithm
> for address pool utilization. More on that in a moment.
>
> The resources that need monitoring in the NAT are NAT memory, address pool
> utilization, and processor capacity. The only item in the above list of
> notifications and limits that at least partially addresses processor
> capacity is the last limit, on number of active subscribers. Possibly, a
> notification and limit could be introduced based on the rate of change of
> the number of active mappings.
>

Agreed. And such rate-limiting is required for CGNs per REQ-4-c and REQ-5-b
from RFC6888:

      c.  Additionally, it is RECOMMENDED that the CGN include
          administrator-adjustable thresholds to prevent a single
          subscriber from consuming excessive CPU resources from the CGN
          (e.g., rate-limit the subscriber's creation of new mappings).

[...]

      b.  Additionally, it SHOULD be possible to limit the rate at which
          memory-consuming state elements are allocated.

The problem with NAT pool monitoring is that the pool actually contains two
> resources, addresses and ports, and how they get used up depends on RFC
> 4787 pooling behaviour.
>

Well, yes, but I don't see how this is a problem.

Keep in mind that some NATs don't bother with ports and allocate full
addresses. That is supported with the current pool and mapping models.

For the RECOMMENDED pooling behaviour of 'paired', port exhaustion occurs
> when all the ports for the desired protocol available to the subscriber on
> a mapped external address are used up, the NAT cannot (for reasons of
> policy or availability) assign additional ports to that subscriber on that
> same address, and the NAT cannot assign an additional address containing
> free ports to that subscriber. (This last condition is subject to
> discussion: is it port exhaustion or address exhaustion?)
>

It depends on what you mean by "cannot". A "paired" NAT may decide to
allocate a mapping from any other available address and not even consider
this an exhaustion condition. It may decide to replace the current address
mapping with this new address. Or it may consider this a "temporary address
mapping" to be reverted eventually. Or it may not record this new mapping
at all, and just allocate mappings from any random address as long as the
primary address is exhausted. Or it may refuse to create a new mapping, as
you seem to imply.

We should not "think" too much about NAT implementation... What matters is
observable behaviour, and in particular observable behaviour that is
general enough that it can be usefully modelled in a MIB. Whatever triggers
an exhaustion event is not absolutely important. We just need to be able to
report it. It will be up to the NAT administrator to determine, based on
the particular NAT implementation, what exactly exhaustion means. Some NATs
are very configurable in this matter.

For a pooling behaviour of 'arbitrary', port exhaustion occurs when there
> is no available port for the desired protocol at any of the addresses in
> the pool. Clearly port exhaustion will be a rarer event in the 'arbitrary'
> case.
>

Again, it depends on the implementation.

draft-ietf-behave-nat-mib-11 triggers notifications based on the ratio of
> active address and port mappings using a given pool to the total number of
> address-port combinations made available by the pool, expressed as a
> percentage. As David Harrington pointed out, this makes a lot of demands on
> the agent processor. We have some alternatives:
>

The problem with this is that the management application would effectively
have to bulk fetch the full mapping table to be able to compute pool usage.
That could be much more taxing on the agent than computing pool usage by
itself.

2) Trigger the high usage notification based on the rate of occurrence of
> out-of-port events, with the same threshold value for all pools. This is
> reasonably valid statistically for a low threshold (i.e., taking the view
> that any port exhaustion at all is actionable), but not terribly reliable
> in the 'paired' case, in that a single subscriber can dominate the results.
>
> 3) Trigger the high usage notification based on the rate of occurrence of
> out-of-port events, with a higher threshold value based on pool size

I think these two options are non-starters because they rely on heuristics
rather than hard metrics. Plus, maintaining a "rate of occurence" is not
much less involved than computing pool usage in the first place.

Suggestion: We need to see what current NATs report, especially CGNs. For
many vendors the documentation is public...

Simon

---

From: Tom Taylor <[email-masked]>
Date: Mon, 27 Oct 2014 21:22:50 -0400

Leaving aside the notification question for research, the discussion 
below suggests that the boundary between port exhaustion and address 
exhaustion is somewhat indeterminate. The MIB currently has two counters 
(showing up in various tables, another issue), OutOfPortErrors and 
ResourceErrors, with the latter supposedly picking up address 
exhaustion. I'd suggest we make the first one OutOfAddressPortErrors and 
thereby eliminate the ambiguity. I'd also call these Drops rather than 
Errors to make the consequences clearer.

Tom Taylor

> On Sun, Oct 26, 2014 at 6:35 PM, Tom Taylor <[email-masked]>
...
>> the pool. Clearly port exhaustion will be a rarer event in the 'arbitrary'
...

---

From: Simon Perreault <[email-masked]>
Date: Tue, 28 Oct 2014 09:57:59 -0400

On Mon, Oct 27, 2014 at 9:22 PM, Tom Taylor <[email-masked]>
wrote:

> Leaving aside the notification question for research, the discussion below
> suggests that the boundary between port exhaustion and address exhaustion
> is somewhat indeterminate. The MIB currently has two counters (showing up
> in various tables, another issue), OutOfPortErrors and ResourceErrors, with
> the latter supposedly picking up address exhaustion. I'd suggest we make
> the first one OutOfAddressPortErrors and thereby eliminate the ambiguity.
> I'd also call these Drops rather than Errors to make the consequences
> clearer.

Works for me!

Thanks,
Simon

---

From: JF Tremblay <[email-masked]>
Date: Tue, 28 Oct 2014 12:00:39 -0400

> On Oct 27, 2014, at 9:22 PM, Tom Taylor <[email-masked]> wrote:
> 
> Leaving aside the notification question for research, the discussion below suggests that the boundary between port exhaustion and address exhaustion is somewhat indeterminate. The MIB currently has two counters (showing up in various tables, another issue), OutOfPortErrors and ResourceErrors, with the latter supposedly picking up address exhaustion. I'd suggest we make the first one OutOfAddressPortErrors and thereby eliminate the ambiguity. I'd also call these Drops rather than Errors to make the consequences clearer.

+1 
I believe it’s useful for a NAT operator to differentiate between these types of errors. 

As Tom mentioned, we shouldn’t dwelve* in implementation-specific details, but the MIB should provide enough flexibility to express real-life error conditions in a meaningful way. Where meaningful == clear enough for an operation person not to have to dig the MIB definition at 2:00 in the morning to figure out which problem needs fixing. 

This begs the question: are these objects sufficient and aptly named in the MIB to reach the goal above? 

Let’s take a few examples of real-life CGN error cases (they are fairly common in my experience, but your-experience-may-vary-of-course): 
1. Out-of-ports because an administrative limit of port-per-user is reached. 
2. Out-of-ports because a single address doesn’t have enough available ports and address “pairing” is enforced. 
3. Out-of-ports/addresses because all addresses in all configured pools do not have enough available ports. 
4. Out-of-addresses because an administrative limit of users per address is reached on all addresses of all configured pools.   
(in the above, “user” is loosely used as meaning a single inside source address)

MIB behaviour (in my opinion): 
1. natQuotaDrops incremented
2. Is natQuotaDrops or natOutOfPortErrors incremented? The MIB prohibits incrementing both. natOutOfPortsError would probably be preferable. 
3. natOutOfAddressPortErrors incremented
4. natQuotaDrops incremented (probably)

Note that natResourceErrors isn’t used above, but it’s probably useful to keep it as it is now. 

Possible corrective actions: 
a) Add addresses to currently configured pools: cases 3 and 4
b) Review the administrative boundaries policies (long term action): cases 2 and 4

In this light, Tom’s suggestion of adding natOutOfAddressesPortErrors is definitely useful, as it covers case 3 well and provides a way to differentiate it from case 1. However the current fields might not be sufficient to cover all cases, like differentiating cases 1 and 4 above. This might be left to implementation-dependant enterprise MIBs, but it would have been nice to have enough coverage in the standard MIB for most common cases. 

Suggestion: 
natQuotaDrops could be split in natAddressQuotaDrops and natPortsQuotaDrops. It would make it somewhat clearer that a "per port” or a “per address” policy is involved. 

My 2.302 cents (the exchange rate isn’t what it used to be) 

/JF

* I just learned that “dwelve” isn’t a real word after all. Interesting, I’ll go to sleep a little less idiot tonight (loose translation of a local expression).

---

From: Tom Taylor <[email-masked]>
Date: Tue, 28 Oct 2014 15:53:05 -0400

At the end with [PTT].

> My 2.302 cents (the exchange rate isn’t what it used to be)
[PTT] Looks to me like the solution might be to have a counter per 
administrative limit, plus one each for can't allocate an address 
mapping, can't allocate an address-port mapping. Then a general "other 
resources" counter. That's not distinguishing your cases 2 and 3, but 
based on what Simon was saying this seems less tied to implementation 
than trying to make that distinction. Case 3 should make itself evident 
in other ways (resource pool utilization).
> * I just learned that “dwelve” isn’t a real word after all. Interesting, I’ll go to sleep a little less idiot tonight (loose translation of a local expression).
[PTT] You've mingled two applicable words. You could have said "dwell on 
implementation details" or "delve into implementation details".
>
> _______________________________________________
> OPSAWG mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/opsawg
>

---

From: JF Tremblay <[email-masked]>
Date: Tue, 28 Oct 2014 16:34:37 -0400

> On Oct 28, 2014, at 3:53 PM, Tom Taylor <[email-masked]> wrote:
> At the end with [PTT].
> 
> [PTT] Looks to me like the solution might be to have a counter per administrative limit, plus one each for can't allocate an address mapping, can't allocate an address-port mapping.

Is that 4 counters? {admin | !admin} x { addresses | addresses-ports }

For example: 
natAdminOutOfAddressesDrops
natAdminOutOfAddressesPortsDrops
natOutOfPortsDrops
natOutOfAddressesPortsDrops

Not sure natAdminOutOfAddressesPortsDrops above makes any sense. An administrative policy will be on ports only. No implementation will specify a limit in address-ports (even if this is what it is, actually). 

Suggestion: 
natAdminOutOfAddressesDrops
natAdminOutOfPortsDrops
natOutOfPortsDrops
natOutOfAddressesPortsDrops
natResourceErrors

Then the MIB behavior according to the scenarios above become: 

1. natAdminOutOfPortsDrops++
2. natOutOfPortsDrops++
3. natOutOfAddressesPortsDrops++
4. natAdminOutOfAddressesDrops++

Makes sense?

> Then a general "other resources" counter.

Ok. (added above)

> That's not distinguishing your cases 2 and 3, but based on what Simon was saying this seems less tied to implementation than trying to make that distinction. Case 3 should make itself evident in other ways (resource pool utilization).

Agreed, “utilization” information would provide an indication of a problem, but it will not provide an indication of how severe the problem is or has been. By looking at a counter, an operator can get additional information (how many drops actually happened). 

JF

---

From: Tom Taylor <[email-masked]>
Date: Tue, 28 Oct 2014 16:47:35 -0400

Response with [PTT2].

>>> (in the above, “user” is loosely used as meaning a single inside source address)
...>>>
> 4. natAdminOutOfAddressesDrops++
[PTT2] It works. If we keep to the MIB in its current form, we need two 
more administrative counters because we have two more limits: on 
fragments, and on number of subscribers with active mappings.