Subject: RE: [NSIS] NATFW NSLP Issue 66: Possibility of NOTIFY storms

From: "Hancock, Robert" <[email-masked]>
Date: Sun, 19 Feb 2006 22:09:25 -0000

hi all,

a couple of comments from the GIST perspective:

*) [maybe not directly relevant here but worth mentioning]: 
rate control applies to all GIST messages, not just the C-mode ones.
It's just that the D-mode rate control is more primitive (it's
non-adaptive, for example). Part of the rationale for the 
existence of GIST to ensure that no signalling application
can cause this sort of problem for other users of the network,
no matter how it is designed.

*) [different from what I said in Vancouver, but hey ...]:
it's true that dropping out the C-mode connection could be
taken as a signal that a node was shutting down (which would
avoid the storm altogether). There's nothing to stop cooperating
implementations adopting this convention; however, I think it
would probably need some non-obvious changes to the GIST API
to achieve it (because essentially you want the signalling
application to start manipulating GIST MAs).

Fundamentally what one wants is for the NATFW NSLP to be able
to send a message to a specific *peer* (rather than about a
specific *flow*). In theory, that's tricky because GIST messages
are usually routed on MRI a.k.a. flow-id. However, it is possible
for an NSLP to bypass GIST routing and just say 'send the message
to that person'; this capability was introduced to allow state
teardown on an old path, but is equally applicable here. The
NATFW NSLP can define a shutdown message, which would be sent
with a zero SID and an MRI describing the flows which are affected
(typically but not necessarily all of them). See sections 5.2.1
and 7.1.4 for more discussion.

The difference between this functionality and NSLPs randomly choosing
which peers they want to talk to is that GIST only allows this to
be done in relation to state which has previously been created;
the NSLP gets told an opaque ID for the peer which accepts the
state, and can explicitly route a message back to that peer.
AFAICT GIST already has precisely the right functionality here,
packaged in precisely the right way (which is quite reassuring).

r.

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Martin Stiemerling
> Sent: 18 February 2006 11:37
> To: nsis
> Subject: [NSIS] NATFW NSLP Issue 66: Possibility of NOTIFY storms
> 
> 
> Hi all,
> 
> 
> Elwyn has raised an issue at the last IETF meeting about the 
> possibility of notification storms. See also issue 66 in the 
> issue tracker at 
> https://kobe.netlab.nec.de/roundup/nsis-natfw-nslp/issue66. 
> 
> 
> The issue has been described with pictures in the slide set 
> shown at the last NSIS session in Vancouver (IETF-64):
> http://www.stiemerling.org/ietf/ietf64/ietf64_nsis_natfw_nslp_v1.ppt
> 
> 
> Short summary of this issue:
> - A given NATFW NSLP forwarder (NF1) has installed NATFW 
> stated for many flows
> - NF2 is a upstream forwarded of NF1
> - NF2 fails in operation and drops out (for whatever reason)
> 
> 
> +------+       +------+        +------+
> |      |       |      |        |      |
> | NIs  |=======| NF1  |--------| NF2  |---->
> |      |       |      |        |      |
> +------+       +------+        +------+
> 
> 
>                       <~~~~~~~~ "Storm"
> 
> 
> In the case NF2 is shut down/drops out, but is still able to 
> generate NATFW NSLP messages, NF1 will receive, possibly, as 
> many NOTIFY messages as session are installed between NF1 and 
> NF2. This is what Elwyn calls a "notification storm".
> 
> 
> Things to know about the "relationship" of NF1 and NF2: The 
> NATFW NSLP requires using the C-MODE of the NTLP. The C-MODE 
> offers reliable, congestion controlled, transport of NTLP and 
> NSLP messages between two adjacent peers. Further, NATFW NSLP 
> MAY send NOTIFY messages but the MUST not send them, i.e., 
> NSLP nodes can send them but the do not need to send them.
> 
> 
> My conclusions about this issue, which has partially been 
> discussed with Robert and Andrew at the last meeting:
> Yes, there is a risk that a node is generating many NOTIFY 
> messages upon an error condition 
> (either node failure or an upstream link is going down). But 
> there are several, already existing, measures to avoid a 
> storm towards upstream nodes:
> - Using the C-MODE: The transport protocol (currently TCP) 
> takes care of some sort of rate limiting, since the receiver 
> (here NF1) would not be able to read all NOTIFY messages at 
> once. NF2 would be blocked by the TCP socket in sending new 
> messages, unless NF1 has read more from its sockets (I do not 
> go into TCP details for that).
> - This could lead to a growing NSLP sending queue at NF2, 
> which in turn gets stuck. In the case, the node fails anyway, 
> this queue will lengthen the shutdown process or the queue 
> will be flushed by the shutdown process. 
> - NF2 does not need to send all NOTIFY messages, due to the 
> fact, that NOTIFY MAY be send. Therefore, in such an error 
> condition, NF can refrain from sending so many NOTIFY 
> messages (see also below for the countermeasure section)
> 
> 
> 
> 
> Countermeasure:
> A good way to avoid a NF generating many NOTIFY messages is 
> to not generate these messages ;-) Instead of generating 
> these messages, the NF can solely close the TCP connection 
> (here NF2 is closing the connection) and does not need to 
> send as many NOTIFY messages as sessions. NF1 must interpret 
> the closing of the TCP connection as NOTIFY messages for all 
> the associated sessions. 
> 
> 
> This behaviour has not yet been described in the draft, but 
> if accepted, it would go in an applicability statement on how 
> to treat this case.
> 
_______________________________________________
nsis mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nsis

---

From: Martin Stiemerling <[email-masked]>
Date: Mon, 20 Feb 2006 11:57:53 +0100

Hi Robert, all,

> no matter how it is designed.

Good job! :-) That is good to remember, since there will be no harm
whatever mode is finally used.

> application to start manipulating GIST MAs).

Good point raised. Especially, tearing down an MA could hurt other
NSLPs which probably have no problem, since they could share
the same MA.

> packaged in precisely the right way (which is quite reassuring).

I have to check sections 5.2.1 and 7.1.4 and will come back. Right
now, this looks like an excellent way to do the complete teardown.

To the QoS people: Does this notification storm apply to the QoS NSLP
as well?

Thanks,

   Martin

---

From: Martin Stiemerling <[email-masked]>
Date: Tue, 21 Feb 2006 21:17:03 +0100

> NSLPs which probably have no problem, since they could share

Robert's proposal works and here is the text proposal:

    NATFW NSLP forwarders, keeping multiple signaling sessions at the
    same time, can experience problems when shutting down service
    suddenly.  This sudden shutdown can be result of node locale  
failure,
    for instance, due to a hardware failure.  This NF generates NOTIFY
    messages for each of the signaling sessions and tries to send them
    upstream.  Due to the amount of NOTIFY messages the shutdown of the
    node may be unnecessarily prolonged, since not all messages can be
    send at the same time.  This case can be described as a NOTIFY  
storm,
    if a multitude of signaling sessions is involved.

    To avoid the need of generating per signaling session NOTIFY message
    in such a described or similar scenario, NFs SHOULD follow this
    procedure: The NF uses the NOTIFY message with the session ID in the
    NTLP set to zero and with the MRI completely wildcarded.  The
    upstream NF receiving this type of NOTIFY immediately regards all
    signaling sessions bound to the messaging association where this
    NOTIFY has been received as null and void.

   Martin

---

From: Martin Stiemerling <[email-masked]>
Date: Wed, 22 Feb 2006 09:03:50 +0100

Hi,

I have noticed that the references to the appropriate sections in the  
NTLP were missing, after sending the text proposal for issue 66.

Here is the reworked proposal:

    To avoid the need of generating per signaling session NOTIFY message
    in such a described or similar scenario, NFs SHOULD follow this
    procedure: The NF uses the NOTIFY message with the session ID in the
    NTLP set to zero, with the MRI completely wildcarded, and the
    'explicit routing' as described in Sections 5.2.1 and 7.1.4. in [1].
    The upstream NF receiving this type of NOTIFY immediately regards  
all
    signaling sessions bound to the messaging association where this
    NOTIFY has been received as void.

[1] refers to the NTLP.

Regards,

   Martin