Subject: [Pce] Virtual Network Topology (VNT) changes and IP layer routing loops

From: Greg Bernstein <[email-masked]>
Date: Mon, 05 Jun 2006 09:17:24 -0700

Hi folks, in drafts draft-ietf-ccamp-gmpls-mln-reqs-00.txt and 
draft-ietf-pce-inter-layer-frwk-00.txt the concepts of multi-layer and 
multi-region networking are defined and discussed. 

An important case is when the IP layer rests directly upon a "virtual 
network topology" layer that we will desire to change via GMPLS 
signaling. Changes to the IP layer VNT can produce "transient IP layer 
routing loops" also known as "micro loops" which can last 100's of mili 
seconds or more.  Hence any time we change a connection between IP layer 
routers via GMPLS we can cause significant outages compared to the 
recovery times that we set out to achieve with GMPLS fast reroute and 
other mechanisms such as SDH APS (rings and linear).

At the Routing working group, Alex's draft 
draft-ietf-rtgwg-microloop-analysis-01.txt provides some analysis and a 
method to reduce the impact (duration too) of the transient loops.  More 
recently the drafts:
(1) draft-bryant-shand-lf-applicability-01.txt
(2) draft-bryant-shand-lf-conv-frmwk-02.txt
(3) draft-francois-ordered-fib-01.txt
Address this problem more generally including in (3) a method that 
guarantee loop free convergence.

The problem is that these have generated very little interest at the RTG 
WG and may not move forward.  This area is not within PCE or CCAMPs 
charter but can have an impact on the adoption of GMPLS in 
multi-layer/region networking. If you're interested please take a look 
and comment to the RTG WG.  Note I wasn't involved with writing these, 
but came across them when considering the effects of GMPLS changes on 
the IP layer VNT.

Thanks

Greg B.

-- 
===================================================
Dr Greg Bernstein, Grotto Networking (510) 573-2237

_______________________________________________
Pce mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/pce

---

From: "Don Fedyk" <[email-masked]>
Date: Mon, 5 Jun 2006 14:38:34 -0400

Hi Greg

Whoa.... 

> -----Original Message-----
> From: Greg Bernstein [mailto:[email-masked]] 
> Sent: Monday, June 05, 2006 12:17 PM
> To: [email-masked]; [email-masked]
> Subject: [Pce] Virtual Network Topology (VNT) changes and IP 
> layer routingloops
> 
> Hi folks, in drafts draft-ietf-ccamp-gmpls-mln-reqs-00.txt 
> and draft-ietf-pce-inter-layer-frwk-00.txt the concepts of 
> multi-layer and multi-region networking are defined and discussed. 
> 
> An important case is when the IP layer rests directly upon a 
> "virtual network topology" layer that we will desire to 
> change via GMPLS signaling. Changes to the IP layer VNT can 
> produce "transient IP layer routing loops" also known as 
> "micro loops" which can last 100's of mili seconds or more.  
> Hence any time we change a connection between IP layer 
> routers via GMPLS we can cause significant outages compared 
> to the recovery times that we set out to achieve with GMPLS 
> fast reroute and other mechanisms such as SDH APS (rings and linear).

1) These virtual topologies are for control traffic and they are solid
most of the time.
2) The existing data traffic is not disrupted when the control traffic
has microloops. 
3) There may be a delay in signaling of new or rerouted connections when
the control traffic has microloops. This is analogous to Graceful
recovery of the control plane where the data plane is momentarily unable
to adapt to new changes.
4) Many of the optical technologies we have do not need response to
signaling in less than 100s of milliseconds so the delay is not
critical.
5) We need a clear separation of protection which can be locally driven
and fast (50 msec) and routing and rerouting which are slower
restoration mechanisms. As long as protection mechanisms are independent
of the control plane primary protection is safe IMHO.

> draft-ietf-rtgwg-microloop-analysis-01.txt provides some 
> analysis and a method to reduce the impact (duration too) of 
> the transient loops.  More recently the drafts:
> Address this problem more generally including in (3) a method 
> that guarantee loop free convergence.
> 

Loop Free Alternates are a good thing.  But with more advanced
mechanisms it is very hard to determine if the cure is worse than the
symptoms.

> The problem is that these have generated very little interest 
> at the RTG WG and may not move forward.  This area is not 
> within PCE or CCAMPs charter but can have an impact on the 
> adoption of GMPLS in multi-layer/region networking. If you're 
> interested please take a look and comment to the RTG WG.  
> Note I wasn't involved with writing these, but came across 
> them when considering the effects of GMPLS changes on the IP 
> layer VNT.

There was a lot of interest and a genuine amount of push back.
Personally I would be very concerned about tight coupling of a control
plane convergence to impacts on a GMPLS control data plane. 

Regards,
Don

---

From: Greg Bernstein <[email-masked]>
Date: Mon, 05 Jun 2006 13:10:45 -0700

Hi Don, I'm using the term "Virtual Network Topology" related to the 
data plane not the control plane.  For example, when I interconnect IP 
routers via a WDM or SDH switching layer the resulting IP layer topology 
is termed a "virtual network topology" since the connectivity amongst 
the IP routers is not necessarily the same as the fiber topology. It is 
these changes to the IP layer topology that can cause the traffic 
disrupting micro loops.  Note that other ways that micro loops occur 
are: (1) during IGP link weight adjustments and (2) maintenance operations.

The connection to GMPLS/PCE is we are trying to make setting up these 
virtual topologies quicker and potential for traffic engineering 
purposes. There have been some good papers in the literature on using 
GMPLS and advanced algorithms for the VNT design problem.

Greg B.

Don Fedyk wrote:
> [Snip]

---

From: Kohei Shiomoto <[email-masked]>
Date: Wed, 07 Jun 2006 08:11:35 +0900

Hi Greg

Thank you for your kind notice.

I think that the avoiding micro-loop is an important work. Micro-loop
could unnecessarily amplify traffic load as much as 128 times, which
should be a serious problem for today's wire-speed packet forwarding at
high-speed link.

So far micro-loop avoidance has been discussed in the context of
maintenance operations including link install/removal, IGP link cost
change, etc. As you pointed out, micro-loop may form in the process of
VNT reconfiguration if IP datagram is forwarded directly on top of VNT
(if we use explicit-routed LSP, we could avoid micro-loop). Micro-loop
avoidance should also be considered in the context of VNT reconfiguration.

Thank you again for your pointer.
--
Kohei

Greg Bernstein wrote:

>> has microloops. 3) There may be a delay in signaling of new or 
>> rerouted connections when
>>> draft-ietf-rtgwg-microloop-analysis-01.txt provides some analysis and 
>>> a method to reduce the impact (duration too) of the transient loops.  
>>> More recently the drafts:
>>> The problem is that these have generated very little interest at the 
>>> RTG WG and may not move forward.  This area is not within PCE or 
>>> CCAMPs charter but can have an impact on the adoption of GMPLS in 
>>> multi-layer/region networking. If you're interested please take a 
>>> look and comment to the RTG WG.  Note I wasn't involved with writing 
>>> these, but came across them when considering the effects of GMPLS 
>>> changes on the IP layer VNT.
>>>     
>>
>>

-- 
Kohei Shiomoto
NTT Network Service Systems Laboratories

---

From: Greg Bernstein <[email-masked]>
Date: Tue, 06 Jun 2006 17:54:58 -0700

Hi Kohei,  For the folks who haven't read the "micro-loop" drafts. These 
describe transient loops that can occur at the IP layer or in MPLS with 
LDP. In the GMPLS world we may also use the term LSP somewhat loosely to 
apply to SDH, WDM, G.709 connections. Since these are all explicitly 
routed (and due to their circuit switched nature too) we won't get loops 
at these layers.

However, when we use GMPLS to set up a new SDH, WDM, G.709, etc... 
connection (or tear down a connection) between IP routers (modifying the 
IP layer VNT) this looks like a link up or down "maintenance" action to 
the IP layer and hence the issue with micro-loops so well described in 
the drafts.

Regards

Greg B.

---

From: Kohei Shiomoto <[email-masked]>
Date: Wed, 07 Jun 2006 11:31:20 +0900

Hi Greg

Thank you for your reply.

> routed (and due to their circuit switched nature too) we won't get loops 

I just meant to explicit-routed LSP in MPLS domain over VNT in the
previous email.

> the IP layer and hence the issue with micro-loops so well described in 

Yes. I called it VNT reconfiguration in the previous email. Either IP
datagram or MPLS labeled packet can be forwarded over the VNT. If we use
the explicit routed LSP to forward the MPLS labeled packet, we can
control the route to avoid micro-loop before the VNT changes.

Of course we need a micro-loop avoidance mechanism when we carry IP
datagram packet directly over the VNT. Such issue need to be addressed
in the future revision in the MLN docs and the Inter-layer PCE/VNT
manager docs.

Thanks,
--
Kohei

>>> data plane not the control plane.  For example, when I interconnect 
>>> IP routers via a WDM or SDH switching layer the resulting IP layer 
>>> topology is termed a "virtual network topology" since the 
>>> connectivity amongst the IP routers is not necessarily the same as 
>>> the fiber topology. It is these changes to the IP layer topology that 
>>> can cause the traffic disrupting micro loops.  Note that other ways 
>>> that micro loops occur are: (1) during IGP link weight adjustments 
>>> and (2) maintenance operations.
>>>
>>>>> draft-ietf-rtgwg-microloop-analysis-01.txt provides some analysis 
>>>>> and a method to reduce the impact (duration too) of the transient 
>>>>> loops.  More recently the drafts:
>>>>> The problem is that these have generated very little interest at 
>>>>> the RTG WG and may not move forward.  This area is not within PCE 
>>>>> or CCAMPs charter but can have an impact on the adoption of GMPLS 
>>>>> in multi-layer/region networking. If you're interested please take 
>>>>> a look and comment to the RTG WG.  Note I wasn't involved with 
>>>>> writing these, but came across them when considering the effects of 
>>>>> GMPLS changes on the IP layer VNT.
>>>>>             
>>>>

-- 
Kohei Shiomoto
NTT Network Service Systems Laboratories
3-9-11 Midori, Musashino, Tokyo 180-8585, Japan

---

From: JP Vasseur <[email-masked]>
Date: Wed, 7 Jun 2006 10:20:22 -0400

Hi Greg,

Thanks for your note. I just wanted to add that this is no different  
from IGP Link weight tuning and off-line algorithms/heuristics  
computing link costs already take such constraint into account. One  
can expect an PCE implementation for VNT computation to also take  
such constraint into account.

Cheers,

JP.

On Jun 5, 2006, at 12:17 PM, Greg Bernstein wrote:

> draft-ietf-pce-inter-layer-frwk-00.txt the concepts of multi-layer  
> and multi-region networking are defined and discussed.
> "virtual network topology" layer that we will desire to change via  
> GMPLS signaling. Changes to the IP layer VNT can produce "transient  
> IP layer routing loops" also known as "micro loops" which can last  
> 100's of mili seconds or more.  Hence any time we change a  
> connection between IP layer routers via GMPLS we can cause  
> significant outages compared to the recovery times that we set out  
> to achieve with GMPLS fast reroute and other mechanisms such as SDH  
> APS (rings and linear).
>
> At the Routing working group, Alex's draft draft-ietf-rtgwg- 
> microloop-analysis-01.txt provides some analysis and a method to  
> reduce the impact (duration too) of the transient loops.  More  
> recently the drafts:

---

From: "Don Fedyk" <[email-masked]>
Date: Fri, 9 Jun 2006 15:48:22 -0400

Hi  Greg

Sorry for the delay on this. 
> Hi Don, I'm using the term "Virtual Network Topology" related 
> to the data plane not the control plane.  For example, when I 
> interconnect IP routers via a WDM or SDH switching layer the 
> resulting IP layer topology is termed a "virtual network 
> topology" since the connectivity amongst the IP routers is 
> not necessarily the same as the fiber topology. It is these 
> changes to the IP layer topology that can cause the traffic 
> The connection to GMPLS/PCE is we are trying to make setting 
> up these virtual topologies quicker and potential for traffic 
> engineering purposes. There have been some good papers in the 
> literature on using GMPLS and advanced algorithms for the VNT 
> design problem.

OK In as much as these topologies represent nodes and links, some how
distributed, I would agree that: 

1) Proactive topology changes should be dealt with in a graceful manner.
One way is to have loop free alternates activated then make the topology
changes.  This may result in slowly stepwise shifting the topology when
turning on/off resources. 
More elaborate schemes can add coverage in this case. But you can just
as well design mechanisms for this particular case that explicit versus
algorithmic. 

2) When it comes to reactive topology changes that is where it gets a
bit trickier. You should have have loop free and perhaps a few more
advanced IPFRR techniques. But most of IPFRR can only protect single
failure protection, node or links. There are provisions for shared
resource correlation to maximize picking independent alternates still
biased by a single failure. 

While the simple IPFRR techniques are a no brainer I'm not convinced
that Virutal network topology is an agrument for more complex IPFRR. 

Regards,
Don 
<snip>

---

From: Olivier Bonaventure <[email-masked]>
Date: Sat, 10 Jun 2006 20:25:46 +0200

Don,
> 1) Proactive topology changes should be dealt with in a graceful manner.

I think that everyone agrees on this. Some operators have procedures to
deal with planned link shutdowns by first setting the OSPF/ISIS weight
of those links to the maximum metric value. This is a good way to move
traffic before shutting down the link, but this technique may lead to
transient loops during the OSPF/ISIS convergence.

> changes.  

Loop free alternates have one drawback. Only about 80% of the links in
common ISP topologies are protected by loop-free alternates. How should
the operator deal with links without LFA ?

I'm not convinced that operators would like solution like the following ;
- check whether the link is protected by LFA
- if the link is protected by LFA, do something
- otherwise do another check and do something else

I guess operators will prefer a solution where the router recognise
proactive changes and deal with them automatically.

> This may result in slowly stepwise shifting the topology when turning on/off resources. 

There is one simple solution to deal with proactive topology changes :
order the FIB updates on the routers. This ordering guarantees that the
network will converge without any transient loop and thyus without
packet losses. This ordering can be easily added to link state protocols
as shown in the following drafts :

http://www.ietf.org/internet-drafts/draft-francois-ordered-fib-01.txt
http://www.ietf.org/internet-drafts/draft-bonaventure-isis-ordered-00.txt

Furthermore, we recently did simulations to evaluate the convergence
time of IS-IS with the proposed extension. For these simulations, we
used as parameters the measured time to perform SPF, FIB updates, etc.
on Cisco 12k as discussed in
http://www.info.ucl.ac.be/people/OBO/papers/ccr403.pdf

The simulations were performed by considering a European network and a
Tier-1 ISP with presence in Europe, USA and Asia. In the Tier-1 ISP, the
paper mentionned above showed that sub-second convergence with regular
IS-IS can be achieved. Our recent simulations with the ordering of FIB
updates indicate that sub-second *ordered* convergence is also achieved
in this Tier-1 ISP.

Basically, in this Tier-1 ISP, our simulations indicate that for a
single link shutdown, regular IS-IS converges within 200- 300
milliseconds depending on the link being shutdown. During this regular
IS-IS convergence, there are transient loops that lead to packet losses.
With the ordered-ISIS, the convergence time is not larger than 750
milliseconds. During this convergence, there are *NO* packet losses in
contrast to regular IS-IS. Thus, in contrast to popular belief, ordered
FIB updates do not lead to slow network convergence...

Olivier