Subject: [GROW] comments on draft-lapukhov-bgp-routing-large-dc-06

From: Jon Mitchell <[email-masked]>
Date: Thu, 5 Sep 2013 22:45:23 -0400

GROW WG - 

We would like to solicit any further comments on
draft-lapukhov-bgp-routing-large-dc-06.  Originally this draft was
presented by Petr in Vancouver in both IDR and GROW and we feel this
draft is useful to the IETF community as it documents a working
scalable design for building large DC's.

Thanks,

Ariff, Petr, and Jon

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 08 Sep 2013 13:50:40 +0100

> scalable design for building large DC's.

i think this draft is really interesting.  A couple of things come to mind:

- default route origination is a real pain in larger scale networks for
obvious reasons, not least because you can often end up with non-optimal
egress choice (e.g. the RR's choice of default route).  You also need to
take special care to ensure that the scope of the default route is limited
to only router sets that actually need it.  I prefer the idea of using a
floating anycast default route.  i.e. you inject an anycast prefix from all
or a selection of all routers which can handle default free traffic flow,
then on small-fib routers you can install a static default pointing to this
floating default and depend on recursive route lookup to figure out the
best path (original idea from Saku Ytti - http://goo.gl/Nj69OZ).  This
allows much tighter control of default route propagation, with better
egress choice characteristics.

- in virtualised networks hosting third party tenancies, it is iften useful
to extend L3 to the hypervisor.  With current tech, running thousands of
vms per cabinet is not unrealistic, and this number will undoubtedly
increase over time.  This brings up the issue of both how to handle address
assignment in an optimal manner and also how to be able to assign one or
more IP addresses per vm or vm cluster, without the problems associated
with flat shared lans, without using NAT and its consequent problems, but
also without wasting precious public IP addresses on link networks which
typically have abysmal assignment efficiency (e.g. 50% for /30, 75% for
/29, etc).  There are models out there which suggest using routing
protocols, but BGP may not always be the best choice due to scalability
issues: if you have a ToR switch, it will be pretty limited in terms of the
number of bgp sessions it can handle.  Some people have approached this by
using RIP to the client VMs.  The advantage of this is that RIP is fully
stateless, whereas BGP has session overhead.  On a small ToR box with an
itty bitty RP, the overhead associated with high hundreds hundreds or even
low thousands of BGP sessions may be too much.  I'm not sure if this is in
the scope of this draft though.

- i think you skimp over the problems associated with bgp session failure
detection / reconvergence.  Mandating ebgp will get rid of the problems
associated with the traditional loopback-to-loopback configuration of most
ibgp networks, but there are still situations where dead link detection is
going to be necessary using some form of keepalive mechanism which works
faster than ebgp timers.  It would be good to have a little more discussion
about bfd - if we can point vendors to good use cases here, they will be
more inclined to support bfd on tor boxes.

- typo: s/it's/its/g

Nick

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 9 Sep 2013 13:52:10 +0200

Hi Nick,

Some observations to your comments ...

> - default route origination is a real pain in larger scale networks

True statement in general. But I think DC fabric which is really the
application area of this draft IMHO have slight different
characteristics from large SP transit networks.

Injecting default (or as you/Saku correctly hint anycast default) from
ASBRs seems much more contained and secured as such default will only
reach DC fabric nodes .. precisely the place where it is needed
(assume flat routing).

Building WAN interconnect as POP (standalone cluster) yet in the
East-West traffic pattern seems good idea.

> vms per cabinet is not unrealistic, and this number will undoubtedly

Great point !

Let me mention how we are handling this - by creating an local and/or
global overlay.

For local overlay openstack neutron configures all VMs networking and
has complete knowledge of the tenant facing IP addressing. Those are
advertised via single IBGP sessions to TORs with specific host's next
hops. No need to build 1000s of BGP sessions from TOR to each physical
machine.

Going further if one would like to save BGP control plane in the
fabric itself there is simple and proven technology already in place
where you treat tenant networks as a VPN and send over the DC fabric
to ASBRs (or further if needed :). Data plane is still IP and only
carries sufficient reachability to BGP next hops.

> faster than ebgp timers.

One solution is to use LAGs and lower layer OEM. That is I think going
to be the most common one.

Shifting the detection to BFD for parallel links may be a bit a
challenge today. Correct me if I am missing some way to configure this
today, but you can run BFD today over p2p EBGP peering going over
single link or you can run multihop BFD over N links. In the latter
case you use most likely EBGP multihop or disable-connected-check
trick. How BFD is going to help there to detect one of N links down in
the latter case ? For former I think you are not recommending N of
EBGP sessions each p2p over each link ?

Cheers,
R.

---

From: "Russ White" <[email-masked]>
Date: Mon, 9 Sep 2013 10:47:45 -0400

> about bfd - if we can point vendors to good use cases here, they will be
more
> inclined to support bfd on tor boxes.

Another point to consider is you're counting on there being an alternate
path always available to bring convergence to down detection time. If you
run eBGP in this situation, and there are multiple hops, any given AS will
only advertise one path to any destination to any other given AS; if that
path fails, you must eBGP converge to get back to forwarding, which is then
going to depend on route flap dampening (assuming this wouldn't be
configured here) and min route advertisement interval (assuming this would
be 0 here) --but either way, convergence isn't going to be in the msec. Note
this isn't local convergence, it's remote convergence --it's not reaction to
a link failure between the two peers, it's reaction to the link failure
several hops (eBGP wise) away.

I think I had planned to think through this with Jon at one point, but we
never sat down and worked it out on paper, so I could just missing something
that prevents eBGP convergence time from coming into play here (?). 

iBGP would be faster to converge, given you're either using something like
add-paths at the RR, or something else, to improve convergence. We can't
fall back on "the igp will keep our bgp session up through some other path,"
because there's no igp underneath.

Russ

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 9 Sep 2013 17:19:30 +0200

Hi Russ,

> so I could just missing something

As you see in this specific topology in the draft there are 16/32/64
ECMP paths so single or maybe even double failures of any node/link
should not result in significant outage. It's still local repair.

Best,
R.

On Mon, Sep 9, 2013 at 4:47 PM, Russ White <[email-masked]> wrote:
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow

---

From: Petr Lapukhov <[email-masked]>
Date: Mon, 9 Sep 2013 16:07:26 +0000

Hi Nick,

Thanks for your feedback! For your questions

1) In the proposed design, the network topology is symmetric and homogenous, and the default route is simply relayed from the "WAN-facing" border routers. It is only used to steer traffic to destinations outside of data-center, and the issues of "WAN default routing" are outside of the scope of the document. Keep in mind that the default route is supplied is in addition to the full routing information for all in-data-center destinations. The FIB problem is generally not an issue in DC's with modern merchant silicon switches, even with data-centers as large as 200K bare metal servers. However, if needed, simple virtual aggregation could take care of it, thanks to shared next-hop sets among large groups of prefixes.

2) Server virtualization is outside of the scope of the document. However, if required, overlaying techniques could be used, to isolate tenant IP addressing/signaling from the fabric. Hypervisors may participate in an overlay control plane, but we avoided this topic on purpose, since it's a much broader discussion scope. Let's put it this way - the proposed design is for the "underlay" bare metal network :)

3) The major idea for failure detection was relying on p2p nature of interconnection links and leveraging optical layer for failure detection. BFD could be an option, though many merchant silicon solutions do not support hardware generation of BFD packets, and as such they are not much different from various control-plane keep-alives, since we won't be sharing it across multiple upstream protocols. LACP and other technologies could be used to add an extra layer of health probing, though this does not change the whole fault-detection logic... 

Regards,

Petr

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Nick Hilliard
Sent: Sunday, September 8, 2013 5:51 AM
To: Jon Mitchell
Cc: [email-masked]
Subject: Re: [GROW] comments on draft-lapukhov-bgp-routing-large-dc-06

- default route origination is a real pain in larger scale networks for obvious reasons, not least because you can often end up with non-optimal egress choice (e.g. the RR's choice of default route).  You also need to take special care to ensure that the scope of the default route is limited to only router sets that actually need it.  I prefer the idea of using a floating anycast default route.  i.e. you inject an anycast prefix from all or a selection of all routers which can handle default free traffic flow, then on small-fib routers you can install a static default pointing to this floating default and depend on recursive route lookup to figure out the best path (original idea from Saku Ytti - http://goo.gl/Nj69OZ).  This allows much tighter control of default route propagation, with better egress choice characteristics.

- in virtualised networks hosting third party tenancies, it is iften useful to extend L3 to the hypervisor.  With current tech, running thousands of vms per cabinet is not unrealistic, and this number will undoubtedly increase over time.  This brings up the issue of both how to handle address assignment in an optimal manner and also how to be able to assign one or more IP addresses per vm or vm cluster, without the problems associated with flat shared lans, without using NAT and its consequent problems, but also without wasting precious public IP addresses on link networks which typically have abysmal assignment efficiency (e.g. 50% for /30, 75% for /29, etc).  There are models out there which suggest using routing protocols, but BGP may not always be the best choice due to scalability
issues: if you have a ToR switch, it will be pretty limited in terms of the number of bgp sessions it can handle.  Some people have approached this by using RIP to the client VMs.  The advantage of this is that RIP is fully stateless, whereas BGP has session overhead.  On a small ToR box with an itty bitty RP, the overhead associated with high hundreds hundreds or even low thousands of BGP sessions may be too much.  I'm not sure if this is in the scope of this draft though.

- i think you skimp over the problems associated with bgp session failure detection / reconvergence.  Mandating ebgp will get rid of the problems associated with the traditional loopback-to-loopback configuration of most ibgp networks, but there are still situations where dead link detection is going to be necessary using some form of keepalive mechanism which works faster than ebgp timers.  It would be good to have a little more discussion about bfd - if we can point vendors to good use cases here, they will be more inclined to support bfd on tor boxes.

- typo: s/it's/its/g

Nick

---

From: Jon Mitchell <[email-masked]>
Date: Mon, 9 Sep 2013 15:35:50 -0400

Nick - thanks for your comments, [JM] inline...

> egress choice characteristics.

[JM] Petr and Robert already responded and in this draft and
in most of our environments the default comes from one set of border
routers as decribed but I actually think this is an interesting idea
and could be useful in some other cases depending on how you connect
these large DC fabrics together, right now we are carefully limiting
default propogation in the odd case where it's necessary via policy.
I think it's worth noting that even if we include such an idea, it
would be limited in discussion since the primary focus of this
document is covering a known working deployed design.

> the scope of this draft though.

[JM] I think this is worth a whole different doc or WG even :-)

> more inclined to support bfd on tor boxes.

[JM] I agree, we could definitely look at using BFD and it is
mentioned in the doc, most appropriately draft-ietf-bfd-on-lags-01
when all the smaller vendors are likely to support it and we have done
good testing to make sure it's not causing more problems than adding
value.  Again, the focus of the document has been what's actually been
shown to have value today in actual deployments.  BFD as Petr
mentioned is not providing a lot of value over LACP detection,
although it validates the L3 path, it's session only gets pinned to
one member.  It still may be slightly better than fast fallover mixed
with adjusted BGP timers but we focused on reduction of protocols and
until a majority of vendors have a better BFD implemenations (also
supporting IPv6 draft).  Right now we've choosen to use the limited
centralized control planes on these platforms for BGP and LACP session
maintenance.

> - typo: s/it's/its/g

[JM] Next rev, thanks!