Subject: [Isis-wg] Comments on draft-xu-isis-flooding-reduction-in-msdc-00

From: Erik Auerswald <[email-masked]>
Date: Tue, 18 Apr 2017 21:27:23 +0200

Hi,

after reading the openfabric draft I looked at other data
center related IS-IS drafts. I would like to comment on
draft-xu-isis-flooding-reduction-in-msdc-00.

I like the idea of using what the draft calls "controllers" to
regulate link-state information dissemination. This architecture
shows similarities to using route reflectors for BGP (RFC 4456) and
to Google's Firepath routing solution (see Jupiter Rising: A Decade of
Clos Topologies and Centralized Control in Google's Datacenter Network,
https://research.google.com/pubs/pub43837.html).

This idea can be realized using existing implementations, with additional
possible optimizations via IS-IS extensions. In fact I built a lab today
to play with the idea, using a couple of switches, one of which was used
as controller, and the others as fabric switches.

To prevent the "management network" of the draft to be used for data
forwarding, high link costs on these interfaces, both on controller and
fabric (i.e. non-controller) switches, suffices. This can and should be
symmetrical, there is no need for asymmetrical link costs as proposed
by e.g. openfabric. Another possibility is to set the overload bit for
the controllers.

RFC 2973, IS-IS Mesh Groups, can be used to prevent flooding on the
data path links.

The management network as described by the draft should not be interpreted
as a virtual router or VRF. Thus the (often single) management port
found on data center switches cannot be used for this purpose. Front
ports need to be used instead. The Dell Z9100-ON switch, for example,
provides two 1G/10G ports (in addition to 32 100G ports) that could be
used for redundant controller connections.

The controllers can be implemented using switches, or using x86 servers
running a routing daemon (e.g. BIRD, Quagga, ...). Small data center
networks could use two switches as controllers with a direct connection
from each data plane switch to each controller switch. If the fabric
comprises more switches than can be connected to a single controller
switch, two separate Ethernet networks comprised of standard switches can
be used to connect two servers acting as controllers. Each controller and
fabric switch could be connected to two independent Ethernet networks in
a resiliency model similar to Fibre Channel networks (SAN A & SAN B).
These simple Ethernet networks would not even need to rely on STP to
provide redundant controller connections. Each fabric switch would use
one VLAN to connect to management Ethernet one, and a different VLAN
to connect to management Ethernet two. If each management Ethernet is
comprised of just one controller switch connecting to all fabric switches,
one VLAN per controller switch suffices. The two controller switches
would not be connected. If two servers are used as controllers, each
could connect one routed port to each of the two management Ethernets.

The basic architecture of sending link-state information via controller
lends itself to further optimizations, e.g. suppressing link-state if
and only if a default or summary route pointing towards all connected
spines suffices. This would open the doors for traffic engineering via
controllers as well.

I'd say draft-xu-isis-flooding-reduction-in-msdc-00 shows a lot of
potential. It even allows experimentation today using readily available
parts (switches and open source routing daemons).

Thanks,
Erik
-- 
Dipl.-Inform. Erik Auerswald         http://www.fg-networking.de/
[email-masked] T:+49-631-4149988-0 M:+49-176-64228513

Gesellschaft für Fundamental Generic Networking mbH
Geschäftsführung: Volker Bauer, Jörg Mayer
Gerichtsstand: Amtsgericht Kaiserslautern - HRB: 3630

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 25 Apr 2017 08:46:06 +0000

Hi Erik,

Thanks a lot for your valuable comments. Please see my response inline.

> -----邮件原件-----
> 发件人: Erik Auerswald [mailto:[email-masked]]
> 发送时间: 2017年4月19日 3:27
> 收件人: Xuxiaohu; [email-masked]
> 主题: Comments on draft-xu-isis-flooding-reduction-in-msdc-00
> 
> Hi,
> 
> after reading the openfabric draft I looked at other data center related IS-IS
> drafts. I would like to comment on draft-xu-isis-flooding-reduction-in-msdc-00.
> 
> I like the idea of using what the draft calls "controllers" to regulate link-state
> information dissemination. This architecture shows similarities to using route
> reflectors for BGP (RFC 4456) and to Google's Firepath routing solution (see
> Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's
> Datacenter Network, https://research.google.com/pubs/pub43837.html).

Your observation is absolutely correct. It could be looked as an IETF standards compliant alternative to the Firepath to some extent.

> possible optimizations via IS-IS extensions. In fact I built a lab today to play with
> the idea, using a couple of switches, one of which was used as controller, and
> the others as fabric switches.

It would be great if you could share more information about your experiment:)

> forwarding, high link costs on these interfaces, both on controller and fabric (i.e.
> non-controller) switches, suffices. This can and should be symmetrical, there is
> no need for asymmetrical link costs as proposed by e.g. openfabric. Another
> possibility is to set the overload bit for the controllers.

Right, another way to achieve the above goal is to use the overload bit. I will add it in the next version.

> RFC 2973, IS-IS Mesh Groups, can be used to prevent flooding on the data path
> links.

Could you explain more about how to apply the ISIS mesh group in the CLOS topology?

> as a virtual router or VRF. Thus the (often single) management port found on
> data center switches cannot be used for this purpose. Front ports need to be
> used instead. The Dell Z9100-ON switch, for example, provides two 1G/10G
> ports (in addition to 32 100G ports) that could be used for redundant controller
> connections.

Your understanding of the management network is correct. Thanks a lot for sharing the valuable information.

> running a routing daemon (e.g. BIRD, Quagga, ...). Small data center networks
> could use two switches as controllers with a direct connection from each data
> plane switch to each controller switch. If the fabric comprises more switches
> than can be connected to a single controller switch, two separate Ethernet
> networks comprised of standard switches can be used to connect two servers
> acting as controllers. Each controller and fabric switch could be connected to
> two independent Ethernet networks in a resiliency model similar to Fibre
> Channel networks (SAN A & SAN B).

Yes, the controller here plays the similar role as a BGP route server. Hence, it could be implemented on servers or switches.

> These simple Ethernet networks would not even need to rely on STP to provide
> redundant controller connections. Each fabric switch would use one VLAN to
> connect to management Ethernet one, and a different VLAN to connect to
> management Ethernet two. If each management Ethernet is comprised of just
> one controller switch connecting to all fabric switches, one VLAN per controller
> switch suffices. The two controller switches would not be connected. If two
> servers are used as controllers, each could connect one routed port to each of
> the two management Ethernets.
> 
> The basic architecture of sending link-state information via controller lends itself
> to further optimizations, e.g. suppressing link-state if and only if a default or
> summary route pointing towards all connected spines suffices. This would open
> the doors for traffic engineering via controllers as well.

Exactly. In fact, we have been considering applying Fibbing (http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p43.pdf) into our approach for traffic-engineering purpose.

> I'd say draft-xu-isis-flooding-reduction-in-msdc-00 shows a lot of potential. It
> even allows experimentation today using readily available parts (switches and
> open source routing daemons).

Thanks again for your valuable comments.

Best regards,
Xiaohu

---

From: Erik Auerswald <[email-masked]>
Date: Tue, 25 Apr 2017 12:26:17 +0200

Hi Xiaohu,

please see my response inline (content not relevant to this
response removed).

> It would be great if you could share more information about your experiment:)

Nothing big, I just took four Extreme Networks X460[-G2] switches from
the company lab, connected three switches in a row to build the "fabric"
(you could think of the middle one as "spine" and the two at the ends
as "leafs"), and then connected the fourth switch to all three "fabric"
switches to act as "controller". I then configured IS-IS on the switches,
configured a high metric on the links to/from the "controller" switch,
and used the "meshBlocked" attribute from RFC 2973 to restrict flooding
inside the "fabric." To do this I configured "meshBlocked" on all links
between "fabric" switches.

Show outputs for transmitted LSPs showed flooding reduction (e.g. when
adding a new loopback interface on a "leaf" switch), with full IP
connectivity over the fabric. The "management" links to the "controller"
were not used for "leaf" to "leaf" traffic due to the higher metric.

> Could you explain more about how to apply the ISIS mesh group in the CLOS topology?

Please see above. RFC 2973 describes a method to configure individual links
to inhibit flooding of LSPs using the "meshBlocked" attribute.

Thanks,
Erik

---

From: Hannes Gredler <[email-masked]>
Date: Wed, 26 Apr 2017 02:54:23 +0200

+1

i'd like to add to erik's observations that mesh-group support is
available in contemporary router software since 15+ years

/hannes

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 26 Apr 2017 01:18:15 +0000

Hi Hannes,

That's great! It would make the hybrid routing protocol (i.e., mix of centralized link-state dissemination and distributed SPF calculation) as described in this draft more deployable in MSDC networks.

Best regards,
Xiaohu

> -----邮件原件-----
> 发件人: Hannes Gredler [mailto:[email-masked]]
> 发送时间: 2017年4月26日 8:54
> 收件人: Erik Auerswald; Xuxiaohu
> 抄送: [email-masked]
> 主题: Re: [Isis-wg] Comments on draft-xu-isis-flooding-reduction-in-msdc-00
> 
> +1
> 
> i'd like to add to erik's observations that mesh-group support is available in
> contemporary router software since 15+ years
> 
> /hannes
> 
> > please see my response inline (content not relevant to this response
> > removed).
> >
> >>> This idea can be realized using existing implementations, with
> >>> additional possible optimizations via IS-IS extensions. In fact I
> >>> built a lab today to play with the idea, using a couple of switches,
> >>> one of which was used as controller, and the others as fabric switches.
> >>
> > switches to act as "controller". I then configured IS-IS on the
> > switches, configured a high metric on the links to/from the
> > "controller" switch, and used the "meshBlocked" attribute from RFC
> > 2973 to restrict flooding inside the "fabric." To do this I configured
> > "meshBlocked" on all links between "fabric" switches.
> >
> > Please see above. RFC 2973 describes a method to configure individual
> > links to inhibit flooding of LSPs using the "meshBlocked" attribute.
> >
> > Thanks,
> > Erik
> >