Subject: Re: [Lsr] WG Adoption Call for draft-li-lsr-dynamic-flooding-02 + IPR poll.

From: Huaimo Chen <[email-masked]>
Date: Wed, 6 Mar 2019 05:04:54 +0000

Hi Tony,

> From: Tony Li [mailto:[email-masked]]
> Sent: Wednesday, February 27, 2019 2:07 AM
> To: Huaimo Chen <[email-masked]>
> Cc: Peter Psenak <[email-masked]>; Christian Hopps <[email-masked]>; [email-masked];
> lsr- [email-masked]; [email-masked]
> Subject: Re: [Lsr] WG Adoption Call for draft-li-lsr-dynamic-flooding-02 + IPR poll.
>
>
> Hi Huaimo,
>
> > > 1)           There is no concrete procedure/method for fault tolerance
> > > to multiple failures. When multiple failures happen and split the
> >>  flooding topology, the convergence time will be increased
> > > significantly without fault tolerance. The longer the convergence
> >>  time, the more the traffic lose.
> >
> > there is a solution for multiple failures - see section 6.7.11.
> >

> > Section 6.7.11 just briefly mentions that the edges of split parts will determine
> > and repair the split after the split of the flooding topology happens. However,
> > there is not any details or description on how to determine or repair the split.
> > This is not useful for implementers.

> I’m sorry that you don’t find it useful. Determining the split is trivial: when you receive an IIH,
> it has a system ID of the another system in it. If that other system is not currently part of the
> flooding topology, then it is quite clear that it is disconnected from the flooding topology.
> Repairing the split is done by enabling temporary flooding on the new link.

For an adjacency between two nodes is up, the Hello packets exchanged between them will not change node/system IDs in them.
How do you determine that other system is not currently part of the flooding topology?

> There is an issue here that we have not yet resolved, which is the rate that new links should be
> temporarily added to the flooding topology.  Some believe that adding any new link is the
> correct thing to do as it minimizes the recovery time. Others feel that enabling too many links
> could cause a flooding collapse, so link addition should be highly constrained. We are still
> discussing this and invite the WG’s opinions.

The issue is resolved by the solutions in draft-cc-lsr-flooding-reduction.
One solution is below, where the given distance can be adjusted/configured.
If we want every node to flood on all its links, we let the given
distance to a big number. If we want the nodes within 2 hops to a failure
to flood on all their links, we set the given distance to 2.
   “In one way, when two or more failures on the current flooding
   topology occur almost in the same time, each of the nodes within a
   given distance (such as 3 hops) to a failure point, floods the link
   state (LS) that it receives to all the links (except for the one from
   which the LS is received) until a new flooding topology is built.”

Another solution is just adding minimum links temporarily on the flooding
topology to repair the split flooding topology until a new flooding topology
is built.

> > > 2)           The extensions to Hello protocols for enabling “temporary
> > flooding” over a new link is not needed.
> >
> > not if you do flooding on every link that comes up. If you want to be smarter, then you need to
> > selectively enable flooding only under specific conditions and that must be done from both sides of
> > the new link.

> > There are only a limited number of conditions (or cases).  In each condition/case, it is
> > deterministic whether we need to enable “temporary flooding” for a new link when it
> > is up.  Thus there is no need for any extensions to Hello protocols for enabling
> > “temporary flooding” on a new link.

> We know of only two cases: (1) the neighbor is not part of the flooding topology and we feel
> that we can add more temporary flooding. (2) The neighbor is not part of the flooding topology
> and we cannot add more temporary flooding.

> Obviously, in the case where we want to add temporary flooding, that TLV is needed in the IIH.

> > For example, suppose that we have a current flooding topology containing all live
> > nodes in an area, when a new link comes up, we may just have two conditions/cases.
> > One condition/case is that the new link is attached to a new node not on the current
> > flooding topology. In this condition/case, the new link needs to be enabled for
> > “temporary flooding” after it is up.

> Agreed, which is why we need the TLV.

The link can be enabled for “temporary flooding” by the node without using any TLV or Hello with the TLV.

> >The other condition/case is that the new link is attached to nodes on the current
> >flooding topology. In this condition/case, there is no need to enable “temporary
> > flooding” on the link.

>Agreed.

>Note that there are some additional corner cases.  Since the two neighbors may not have the
>exact same information, one may consider the other to be on the flooding topology when in fact
> it is not.  This might happen in the case of a node reboot. The IIH TLV gives us an explicit way
> of signaling, rather than simply guessing and sometimes getting it wrong.

The TLV in Hello packet just requests for adding “temporary flooding” on the link. The other information is accessed by the node locally. The TLV in Hello packet does not help for corner case. In the case where a node is rebooted, a new link attached to a new node may apply.

>> > 3)           The extensions to Hello protocols for requesting/signaling
>> > “temporary flooding” for a connection does not work.
>>>
>>> sorry, but if you see a problem, please provide details, saying above is
>>> simply unproductive.

>> “The nodes … will try to repair the flooding topology locally by enabling
>>temporary flooding towards the nodes that they consider disconnected from the
>>flooding topology ...”

>>The above quoted text is from draft-li-lsr-dynamic-flooding-02, where
>> “enabling temporary flooding towards the nodes” is to request/signal
>> “temporary flooding” for a connection to connect partitioned/disconnected
>>flooding topology into one through the extensions to Hello protocols described
>>in draft-li-lsr-dynamic-flooding-02. Right?

>>The extensions to Hello protocols for requesting/signaling “temporary
>>flooding” for a connection to connect partitioned/disconnected flooding
>>topology into one does not work since the connection may have two or more
>>hops and a Hello packet may get lost.

>All adjacencies are a single hop in both IS-IS and OSPF.  Yes, Hello packets may be lost.
>Fortunately, they are periodically transmitted, thus the next transmission will also contain the
> TLV.  If IIH’s are getting lost at a significant rate, then the adjacency will not (and should not)
>come up.  Thus, the request for temporary flooding will propagate to the neighbor in all cases
>that matter.

It takes too long when Hello packet is lost. Repairing split flooding topology needs to be fast.

>>It is not convenient for a user/operator to configure on an area leader since the
>>leader is dynamically selected. How do you address this?

>No configuration is required.  The election algorithm selects the area leader.  The rules are in
>the draft.  An implementation may have a default priority and a default algorithm setting, so no
>configuration is mandatory.  If the operator desires a specific node to become area leader, then
>configuration may be required to adjust the priority.  FWIW, we have this already working in
>our implementation.  It Just Works.

It does not mean that a user/operator configures/select an area leader. It means that a user/operator configures other things such as indicating an algorithm or selecting the centralized mode on the area leader.

>>After the user/operator does some configurations on the (designated) leader, will the
>>backup leader takes over the configurations after the designated leader is down?

>There is no need for a backup leader.  If the area leader is partitioned from the topology, then
> leader election is repeated, resulting in a new leader.  Again, no configuration is required.

The above does not talk about topology split, but about the leader down. After a user/operator has configured some things on the leader, and the leader has got them and distributed them in some form, and then some time later, the leader goes down, a new leader is selected. In this case, will the new leader take and maintain the configurations or the information derived from the configurations done on the old leader.

Best Regards,
Huaimo

>Tony

---

From: Christian Hopps <[email-masked]>
Date: Wed, 6 Mar 2019 08:28:54 -0500

Let's use new email threads for suggested changes. One thread per suggested change/problem to keep things easier to follow.

Thanks,
Chris.

---

From: [email-masked]
Date: Wed, 6 Mar 2019 07:45:17 -0800

Hi Huaimo,

> How do you determine that other system is not currently part of the flooding topology?

The IIH includes the system ID.  See ISO 10589 v2, section 9.7, field “source Id”.  The local system will have
a copy of the flooding topology and can easily see if the neighbor was present as of the last FT computation.  If not, then it should be
added (modulo rate limiting). The local system can also examine it’s own LSDB.  If there is no LSP for the neighbor, then it would seem
highly likely that there is a disconnect and the neighbor should again be added (modulo rate limiting).

We are not requiring it, but a system could also do a more extensive computation and compare the links between itself and the neighbor
by tracing the path in the FT and then confirming that each link is up in the LSDB.

>    which the LS is received) until a new flooding topology is built.”

As we have discussed, this is not a solution. In fact, this is more dangerous than anything else that has been proposed and
seems highly likely to trigger a cascade failure. You are enabling full flooding for many nodes.  In dense topologies, even
a radius of 3 is very high.  For example, in a LS topology, a radius of 3 is sufficient to enable full flooding throughout the
entire topology. If that were stable, we would not need Dynamic Flooding at all.

> topology to repair the split flooding topology until a new flooding topology

Agreed.  Which links constitute the minimum?  In a general topology, with arbitrary failures that are not distributed globally,
how do we make a distributed decision about which links to enable? This is the problem that we are trying to solve. And
we have no oracle to tell us The Right Answer.

> The link can be enabled for “temporary flooding” by the node without using any TLV or Hello with the TLV.

There are cases where it is far easier for the neighbor to realize that it is disconnected than for the local system to realize
that the neighbor is disconnected.  Thus, it is easier to allow one system to request temporary addition. 

> The TLV in Hello packet just requests for adding “temporary flooding” on the link. The other information is accessed by the node locally. The TLV in Hello packet does not help for corner case. In t…

If the node that rebooted has 1000 interfaces, which interfaces should be temporarily added?  Adding all of them is likely to trigger a cascade failure.  The TLV allows us to signal which ones should be enabled.

> It takes too long when Hello packet is lost. Repairing split flooding topology needs to be fast.

Fortunately, lost hello packets are a relatively rare occurrence.  While repairing the flooding topology needs to be done expediently, attempting to do so and triggering a cascade failure of the network is counter-productive. Given this alternative, a bit of extra delay when adding a new system to the network, or trying to recover from multiple failures seems wise. Rushing and making things worse does not.  The first
priority must remain network stability.

> It does not mean that a user/operator configures/select an area leader. It means that a user/operator configures other things such as indicating an algorithm or selecting the centralized mode on th…

In an implementation, centralized mode and algorithm selection can be the defaults.  In fact, in our implementation, the only required configuration is to enable dynamic flooding. Everything else is automatic.

Regards,
Tony

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Wed, 6 Mar 2019 18:29:47 +0000

[Les:] Just to put this minor issue to rest…
It would also be possible for the node making the temporary flooding request to send hellos at a faster rate (say once a second rather than once every 10 seconds) until it sees that the neighbor is sending LSP/SNP updates.
This just isn’t a significant issue.

   Les

---

From: Huaimo Chen <[email-masked]>
Date: Mon, 11 Mar 2019 17:08:09 +0000

Hi Tony,

    In summary for multiple failures, two issues below in draft-li-lsr-dynamyic-flooding are discussed:

1)      how to determine the current flooding topology is split; and

2)      how to repair/connect the flooding topology split.
For the first issue, the discussions are still going on.
For the second issue, repairing/connecting the flooding topology split through Hello protocol extensions does not work.  When a “backup path”/connection of multiple hops is needed to connect/repair the flooding topology split, Hello can not go beyond one hop, thus can not repair the flooding topology split in this case.

>From: Tony Li [mailto:[email-masked]] On Behalf Of [email-masked]
>Sent: Wednesday, March 6, 2019 10:45 AM
>Cc: Christian Hopps <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
>Subject: Multiple failures in Dynamic Flooding
>
>Hi Huaimo,
>

>by tracing the path in the FT and then confirming that each link is up in the LSDB.

It normally takes a long time such as more than ten minutes to age out and remove an LSP/LSA for the neighbor from the LSDB even though the neighbor is disconnected physically.
How can you decide quickly in tens of milliseconds that the flooding topology is disconnected?

>entire topology. If that were stable, we would not need Dynamic Flooding at all.

This full flooding is enabled only for a very short time.
How do you get that this is more dangerous than anything else and seems highly likely to trigger a cascade failure? Can you give some explanations in details?

>we have no oracle to tell us The Right Answer.

We can discuss this after the first method is discussed.

Best Regards,
Huaimo

---

From: Peter Psenak <[email-masked]>
Date: Mon, 11 Mar 2019 18:21:25 +0100

Hi Huaimo,

On 11/03/2019 18:08 , Huaimo Chen wrote:
> Hi Tony,
>
>
>
>     In summary for multiple failures, two issues below in
> draft-li-lsr-dynamyic-flooding are discussed:
>

there is no need to do that. The recovery mechanism will repair the 
split topology if there is a way to do that.

> 2)      how to repair/connect the flooding topology split.

6.7.11.  Recovery from Multiple Failures

    "The nodes that remain active on the edges
    of the flooding topology partitions will recognize this and will try
    to repair the flooding topology locally by enabling temporary
    flooding towards the nodes that they consider disconnected from the
    flooding topology until a new flooding topology becomes connected
    again."
> For the second issue, repairing/connecting the flooding topology split
> through Hello protocol extensions does not work.  When a “backup
> path”/connection of multiple hops is needed to connect/repair the
> flooding topology split, Hello can not go beyond one hop, thus can not
> repair the flooding topology split in this case.

there is no need to send anything multi-hop.

thanks,
Peter

>
>
>
>>*From:* Tony Li [mailto:[email-masked]] *On Behalf Of
> [email-masked]
>>*Sent:* Wednesday, March 6, 2019 10:45 AM
>>*To:* Huaimo Chen <[email-masked]>
>>*Cc:* Christian Hopps <[email-masked]>; [email-masked];
> [email-masked]; [email-masked]
>>*Subject:* Multiple failures in Dynamic Flooding
>
>>
>
>>Hi Huaimo,
>
>>
>
>>>For an adjacency between two nodes is up, the Hello packets exchanged
> between them will not change node/system IDs in them.
>
>>a copy of the flooding topology and can easily see if the neighbor was
> present as of the last FT computation.  If not, then it should be
>
>>added (modulo rate limiting). The local system can also examine it’s own
> LSDB.  If there is no LSP for the neighbor, then it would seem
>
>>We are not requiring it, but a system could also do a more extensive
> computation and compare the links between itself and the neighbor
>
> It normally takes a long time such as more than ten minutes to age out
> and remove an LSP/LSA for the neighbor from the LSDB even though the
> neighbor is disconnected physically.
>
>>As we have discussed, this is not a solution. In fact, this is more
> dangerous than anything else that has been proposed and
>
>>seems highly likely to trigger a cascade failure. You are enabling full
> flooding for many nodes.  In dense topologies, even
>
>>a radius of 3 is very high.  For example, in a LS topology, a radius of
> 3 is sufficient to enable full flooding throughout the
>
> How do you get that this is more dangerous than anything else and seems
> highly likely to trigger a cascade failure? Can you give some
> explanations in details?
>
>>Agreed.  Which links constitute the minimum?  In a general topology,
> with arbitrary failures that are not distributed globally,
>
>>how do we make a distributed decision about which links to enable? This
> is the problem that we are trying to solve. And
>
> _______________________________________________
> Lsr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lsr
>

---

From: [email-masked]
Date: Mon, 11 Mar 2019 10:41:08 -0700

Hi Huaimo,

> For the second issue, repairing/connecting the flooding topology split through Hello protocol extensions does not work.  When a “backup path”/connection of multiple hops is needed to connect/repair…

You do not try to repair things remotely, they are always repaired locally.  If there are multiple failures in the flooding topology and it is partitioned, then it follows that there are multiple remaining connected components of the flooding topology.  Nodes that are adjacent to the failures will update their LSPs and flood them throughout their connected component.  Each component will see at least two link failures if there is a partition of the FT and each node in the component can detect that the FT has partitioned.  Each node is then capable of enabling temporary flooding on one or more links that will traverse the partition, thereby restoring a functioning FT.  The Area Leader then recomputes and redistributes the revised FT.

To put it yet another way, repair is fully distributed.  You should like that.  :-)

> How can you decide quickly in tens of milliseconds that the flooding topology is disconnected?

You do not wait for LSP/LSA removal.  You look for link changes in the LSPs that you do get, or local link changes.

> This full flooding is enabled only for a very short time.

All it takes is enabling it at sufficient density to create a cascade failure.  Milliseconds are sufficient for a collapse.

> How do you get that this is more dangerous than anything else and seems highly likely to trigger a cascade failure? Can you give some explanations in details?

Again, we do not have absolute metrics on what triggers a cascade failure today.  We have several data points of several different implementations at different points in time.  We know that in the early ‘90s, a full mesh of 20 neighbors running L1L2 was sufficient.  Obviously things have changed somewhat, but even more modern implementations have had problems.  This is why the MSDC went to BGP.

As a result, we need to be very conservative about what flooding we temporarily enable.  We do not want to walk anywhere near the cliff, as the cascade failure is fatal to the network.

Tony