Subject: [Idr] I-D Action: draft-ietf-idr-rs-bfd-02.txt

From: [email-masked]
Date: Sat, 11 Mar 2017 06:32:51 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Inter-Domain Routing of the IETF.

        Title           : Making Route Servers Aware of Data Link Failures at IXPs
        Authors         : Randy Bush
                          Jeffrey Haas
                          John G. Scudder
                          Arnold Nipper
                          Thomas King
	Filename        : draft-ietf-idr-rs-bfd-02.txt
	Pages           : 14
	Date            : 2017-03-11

Abstract:
   When route servers are used, the data plane is not congruent with the
   control plane.  Therefore, the peers on the Internet exchange can
   lose data connectivity without the control plane being aware of it,
   and packets are dropped on the floor.  This document proposes the use
   of BFD between the two peering routers to detect a data plane
   failure, and then uses a newly defined BGP SAFI to signal the state
   of the data link to the route server(s).

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-idr-rs-bfd/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-idr-rs-bfd-02

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-idr-rs-bfd-02

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Randy Bush <[email-masked]>
Date: Mon, 13 Mar 2017 14:01:43 +0900

note that this version is rather different and warrants reading.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 13 Mar 2017 10:51:23 +0100

Hi Randy & authors of this document,

Major comment:

Let's observe that the ultimate objective of this draft (and in fact much
more) can be achieved by assuring that clients receive more then one path
to given bgp destinations at normal operation.

That direction has been proven to work much more effectively in all other
scenarios which require fast connectivity restoration as opposed to
counting on control plane "convergence".

The following alternative techniques can be used to distribute more then
just best path between BGP clients:

*A* Use of add-paths RFC7911 from RS to clients (with eBGP support
extension)

*B* Use of diverse-path RFC6774 from RS to clients

*C* Direct EBGP peering between clients with the help of BGP Auto Discovery
https://goo.gl/KKgqfU  (now with merged work of Automagic peering at IXPs:
https://goo.gl/RJIKYM)

Let's also notice that in IX folks peering via RS have open or selective
peering policy and they send at most their clients + internal routes.
Therefor for any given b_net there will be no more then few paths. There is
no Transit Internet open peering via RS or very often even via IX.

This will allow all clients to quickly switch over to backup path which
would be with PIC already present in their data plane ahead of failure.

Moreover this will reduce churn and convergence delay when say 1000 clients
in one shot is bombarding RS using new proposed SAFI of the very same
client router going down.

Also let's keep in mind that modern routing stacks allow to much more
intelligent routing by actively and passively monitoring all alternative
paths to your interesting destinations and based on the quality of the data
plane via any peer choose the best path. That with just getting a single
path at a time is simply not possible as those "interesting destinations"
are not BGP next hops, but applications servers sitting one or more ASes
far from client.

Minor comment:

Today BFD sessions last time I checked must be bounded to a protocol. In
current IX+RS model there is no direct protocol between clients. Hence BFD
implementations on the clients needs to be augmented to auto peer with all
BGP next hops received over eBGP sessions to RS. Not impossible and in fact
perhaps good feature on its own .. but needs to be supported on all clients
just like the new SAFI draft proposes.

Kind regards,
Robert

On Mon, Mar 13, 2017 at 6:01 AM, Randy Bush <[email-masked]> wrote:

> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 13 Mar 2017 10:31:57 +0000

Robert Raszuk wrote:
> Let's observe that the ultimate objective of this draft (and in fact
> much more) can be achieved by assuring that clients receive more then
> one path to given bgp destinations at normal operation. 

This is a fundamental misunderstanding.  The aim of this draft is to
ensure that traffic isn't blackholed if there is a layer 2 incongruity
on the IXP which causes connectivity to be non commutative.

Given the following topology, A's connectivity to RS works fine, and B's
connectivity to RS works fine, but A cannot communicate with B:

>    +-----+    +-----+
>    |     +--> |     |
>    |  A  |    |  B  |
>    |     | <--+     |
>    +-----+    +-----+
>       |           |
>       | +------+  |
>       +-+      +--+
>         |  RS  |
>         |      |
>         +------+

What's needed is some protocol running between A and B to ensure that
they can still talk to each other, i.e. BFD.

No amount of alternative paths will save you in this situation because
the control plane (RS) has connectivity to both data plane routers.

Nick

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 13 Mar 2017 11:45:25 +0100

Nick,

I am afraid you have completely missed my point.

I never said clients must not detect other clients liveness before using it
for best path selection and in their local data planes.

I said RS does not need to bothered with that information.

In fact I explicitely said that modern clients will detect health of the
path well beyond local peer's next hop.

Cheers,
R.

On Mar 13, 2017 11:32 AM, "Nick Hilliard" <[email-masked]> wrote:

Nick

---

From: Randy Bush <[email-masked]>
Date: Mon, 13 Mar 2017 21:00:11 +0900

> to given bgp destinations at normal operation.

let's break tradition and actually see what the document says.

Abstract

and, if you wander over to the referenced route server docco, you'll see
that it already deals with alternate paths.

rady

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 13 Mar 2017 13:33:25 +0100

> let's break tradition and actually see what the document says.

​Let's also break tradition and become a bit open minded to look at bigger
picture vs just consider on one specific solution to the problem.

>    and packets are dropped on the floor.

​That not correct if you do it right. If you can reach A via NH1 and NH2
and RS propagates both of those NHs to clients - it is up to clients to
validate those NHs and choose one which they prefer for forwarding at a
given point of time.

RS does not need to be part of that game at all.

> ​   ​
> This document proposes the use
>    failure,

​BFD can be extended to work without protocol, object tracking ​can used,
passive or active probes traversing next hops etc ... but it's a local
matter on the client.

In fact it remains interesting to see what timers are required to handle
BFD to say 1000 peers from a single box.

Now comes BFD security with maintaining different MD5 keys across 1000
peers ! The draft says read RFC5880 and that's what is there.

> ​a​
> nd then uses a newly defined BGP SAFI to signal the state

​Signalling anything to route servers is not needed. It puts us all back to
late 80s' or early 90s' on how networks were running at that time.

> that it already deals with alternate paths.

​It's not about "dealing with them" ... it is about distributing them to
the clients so clients can use them.

Thx,
Robert​

---

From: Randy Bush <[email-masked]>
Date: Tue, 14 Mar 2017 14:32:57 +0900

> validate those NHs and choose one which they prefer for forwarding at a

why do folk use RS?  to offload the bgp decision process.

randy

---

From: Gert Doering <[email-masked]>
Date: Tue, 14 Mar 2017 09:07:32 +0100

Hi,

On Tue, Mar 14, 2017 at 02:32:57PM +0900, Randy Bush wrote:
> > ???That not correct if you do it right. If you can reach A via NH1 and NH2

Mmmmh, I'm not sure I agree to that - we use the RS because it saves
us the hassles of maintaining 300+ BGP sessions with lots of small peers,
so "human life time" plus "router CPU" is more a factor.  

Offloading the BGP decision process is a side effect - with sometimes 
undesired results, in which case we add direct sessions to reclaim
control.  But usually, for those 300 smaller peers, "it just gets things
done".

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 10:45:59 +0100

> why do folk use RS?  to offload the bgp decision process.

​Wrong completely !

RS are used to establish one (or two) eBGP session and get all the juice. ​

The more juice you get the better and in fact you need more then you can
swallow at a time to choose which one tastes better at a given point of
time.

---

From: Randy Bush <[email-masked]>
Date: Tue, 14 Mar 2017 18:52:31 +0900

> RS are used to establish one (or two) eBGP session and get all the

that is one.  my case is one.  and there are others.

folk are at large exchanges with small routers; some can handle about
one view.  they do not have the horsepower to make best next hop choices
from large sets.  we should not disenfranchise them.

randy

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 11:05:05 +0100

Randy ..

Any ASBR (even software running in a VM or container) today used as peering
router can handle Internet table. That is 600K+ nets.

IXes give you tiny subset of it via open or selective policy. Moreover as I
mentioned in my original comment - over RS you get local customer and
internal routes and no more. So for any net it will be usually one or two
paths anyway.

So if someone has one path (and client has no way of knowing) signalling to
RS that next hop is gone is not helpful as there is no backup path RS could
recompute and send to such client.

Learning two paths for given net with different NHs (even if RS has more
then two) seems to provide sufficient robustness and enables you to measure
quality to the destinations as well as do things line multipath TCP

= =  =

Little suggestion:

Now if someone is really so weak on the edge and attaches to IX to only
peer locally we should perhaps invent co-located with RS pair of data plane
boxes where such weak guys would for forwarding just default to such data
plane gateway. That way those two forwarders could have as many paths as RS
feeds it with yet all weak clients are happy to forward and reach all open
peers of a given IX.

In fact most IX switches while running in L2 mode could be configured for
L3 as well so no need to even invest new $$$ needed.

Kind regards,
Robert.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 14 Mar 2017 11:14:15 +0000

Robert Raszuk wrote:
> Randy Bush wrote: 

A bit less URRONG might be productive on this point.  As an operator,
I'm aware of both rationales being used in production.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 14 Mar 2017 11:30:38 +0000

Robert,

speaking with my ixp operator hat on:

Robert Raszuk wrote:
> Any ASBR (even software running in a VM or container) today used as
> peering router can handle Internet table. That is 600K+ nets.

this statement is incorrect on production networks.

> IXes give you tiny subset of it via open or selective policy. 

this statement is incorrect on production ixps.

> Moreover
> as I mentioned in my original comment - over RS you get local customer
> and internal routes and no more. So for any net it will be usually one
> or two paths anyway. 

this statement is incorrect on production ixps.

> So if someone has one path (and client has no way of knowing) signalling
> to RS that next hop is gone is not helpful as there is no backup path RS
> could recompute and send to such client. 
>
> then two) seems to provide sufficient robustness and enables you to
> measure quality to the destinations as well as do things line multipath TCP

this statement is both incorrect and wildly unrealistic on real world
networks, even from the most rarified theoretical point of view.

> peer locally we should perhaps invent co-located with RS pair of data
> plane boxes where such weak guys would for forwarding just default to
> such data plane gateway. That way those two forwarders could have as
> many paths as RS feeds it with yet all weak clients are happy to forward
> and reach all open peers of a given IX. 
> 
> In fact most IX switches while running in L2 mode could be configured
> for L3 as well so no need to even invest new $$$ needed. 

this statement is incorrect at production ixps.

I would be happy to provide multiple counterexamples to all these points.

This draft is attempting to solve a real world problem on real world
networks.  It would be nice for ixp participants to have ideally
specified equipment providing service for clients which all had working,
network-aware multipath implementations which could react immediately to
blackholing events, but this doesn't even begin to approximate reality.
Production IXPs need to be able to provide baseline services to a
hodge-podge of different types of equipment serving the great unwashed
of the internet.  There are very few assumptions you can make about
equipment specification that will turn out to be realistic.

Nick

---

From: Gert Doering <[email-masked]>
Date: Tue, 14 Mar 2017 12:32:03 +0100

Hi

> router can handle Internet table. That is 600K+ nets.

*cough*

Please enlighten yourself what people use "as peering router" today :-)

> IXes give you tiny subset of it via open or selective policy. 

... and about the amount of prefixes you can receive on a reasonably-sized
IXP.

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 13:27:42 +0100

Nick,

If someone is using weak BGP stacks which can not handle two BGP paths per
net .. sorry that to me does not justify inventing new SAFI.

Moreover the idea behind the new SAFI is flawed since it assumes RS always
would have a secondary path to send to client. Otherwise noise of 1000
peers bombarding RS with message "NH-X is not reachable" is pretty useless
if given net has only one NH on the RS.

Moreover those weak clients which can not run best path for RS received
nets are now subject to handle 1000 BFD sessions ... it really does not
sound right that an edge can do the latter just fine and not the former.

Last to address real world problem - I proposed very easy solution which
you as an IXP operator can enable today via simple config on your switch
and treat such switch (or pair of switches) as default forwarder. Done !
Fully backward compatible with existing IXP deployment too.

Thanks,
R.

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 13:54:00 +0100

​> Please enlighten yourself what people use "as peering router" today :-)

Care to enlighten me on or offline ? Which BGP stacks can not handle full
table ?

> IXP.

AMS-IX ... 160K v4 prefixes ..
Ref: https://ams-ix.net/technical/statistics/route-server-stats

LINX (all combined RSes) - 120K v4 prefixes
https://stats.linx.net/rs/

JPIX - 20K v4 routes ... avrage grow 4K per year
https://www.slideshare.net/apnic/the-trend-stats-of-routing-table-at-jpix-route-servers

Is this overwhelming ? Please notice that not all of them would churn on a
daily basis too so your weak edge may choke only upon initial dump. No
public data on how many paths each of those nets have.

R.

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 14 Mar 2017 12:56:00 +0000

Robert,

your proposal is not even wrong.

Nick

Robert Raszuk wrote:
> If someone is using weak BGP stacks which can not handle two BGP paths
> per net .. sorry that to me does not justify inventing new SAFI. 
> 
> Moreover the idea behind the new SAFI is flawed since it assumes RS
> always would have a secondary path to send to client. Otherwise noise of
> 1000 peers bombarding RS with message "NH-X is not reachable" is pretty
> useless if given net has only one NH on the RS. 
>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 14:30:23 +0100

Clearly it requires a bit out of the box perspective ;-)

For those interested it does borrow a bit from RFC6769
<https://goo.gl/zOk1fy> as well as from the notion of BGP Virtual Next Hop
proposal/patent.

Best,
R.

---

From: [email-masked] (Niels Bakker)
Date: Tue, 14 Mar 2017 15:43:30 +0100

>quality to the destinations as well as do things line multipath TCP

This would require a major rearchitecting of pretty much all network 
components out there because we'd be doing away with the concept of 
having one best path that's chosen by each hop handling a packet.

>L3 as well so no need to even invest new $$$ needed.

As Nick said, not even wrong.

Route servers decouple the forwarding plane from the control plane 
in an additional way beyond what already happens inside routers for 
hardware design reasons.  This draft seeks to add a mechanism 
to signal any dislocations, improving robustness of existing 
deployments.

Regards,

	-- Niels (former IXP operator, current IXP participant hat)

--

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 16:04:15 +0100

Not sure if we should continue this thread  ....  I think all what needs to
be said has been said already regarding proposed draft. But just to clarify
on your questions:

>> measure
> components out there because we'd be doing away with the concept of having
> one best path that's chosen by each hop handling a packet.

​When IX client advertises​ a route to Route Server it will still advertise
best path only as he sets nhs. When RS today already import to per client
bRIB all paths (if available) and selects best then advertises it to all
peers (subject to policy).

Only change is needed is to advertise with either add-paths or diverse-path
second best. No major change at all. One line config on existing routers.

​N​
>> ow if someone is really so weak on the edge and attaches to IX to only
>> plane
> Route servers decouple the forwarding plane from the control plane in an
> additional way beyond what already happens inside routers for hardware
> design reasons.

​I am not saying that Route Servers should be doing forwarding ... if this
is what you are trying to address by the above comment.

RS would send all paths they know ​about to device which will act as a
forwarder in the middle of IX. Sitting close or being a function of L3
forwarding on the switch allows to have all weak clients go via such pair
of routers. No suboptimal routing as packets go via the switch anyway, no
1000 x 1000 BFD sessions between all clients, quick local repair in case of
path failure ... and such device could be used only for those open policy
destinations which do have more then one path in the first place.

Yes this is not the model IX do today ... but is this sufficient reason to
dismiss it - provided above basic solution of sending more then best path
would perhaps be not feasible.

Thx,
R.

> This draft seeks to add a mechanism to signal any dislocations, improving
> robustness of existing deployments.
>
> Regards,
>
>

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 14 Mar 2017 20:31:23 +0000

> Clearly it requires a bit out of the box perspective ;-)

Blue-sky thinking has its place, but we're discussing a tweak of a
corner case of an existing well-established system, not a complete core
rethink based on ideas which have not been exposed to the realities of
production environments.  If we could return to discussing this draft,
that would be great.

Nick

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 14 Mar 2017 16:42:12 -0400

Robert,

I'll let the other comments in this thread stand on their merit.  However, I
did want to respond to one specific point here:

> I said RS does not need to bothered with that information.

The relevant bit of procedure within this document to run BFD toward an eBGP
nexthop and use it in the local decision process is of potential value even
without the RS SAFI part of the protocol.  So, in this respect, I agree.

The authors had previously discussed extracting this procedure from the
document and are fine with doing so if it makes sense.

The one "common" use case where this may be of benefit outside of a
route-server environment is BGP "VPNs" that are constructed using IPSEC
tunnels with the network effectively an NBMA subnet.  

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 21:45:28 +0100

​Hi Jeff,​

> without the RS SAFI part of the protocol.  So, in this respect, I agree.

​Completely agree !

In fact to me this is basic 4271 as you should not use a path (even single
path) if you can not validate if the next hop to it is reachable.

I think we all in BGP did not pay that much attention to reachability on
multiaccess LANs ​as most peering happens on p2p links where the same
problem does not happen.

> document and are fine with doing so if it makes sense.

Makes sense. It should be done by default one way or another. /* Not sure
if BFD considering those "weak customer peering routers" is the best idea.
*/

> tunnels with the network effectively an NBMA subnet.

​See all SD-WANs today need more then one path such that they make
forwarding decision by probing all alternative overlay links and based on
the applied logic switch over to better or lower cost or less delay path.
If you give them just single path this is all no longer possible. That is
what worries me in this the most if you would like to utlize this new NH
NOTIFCATION SAFI for WAN VPNs.

​Thx,
R.​

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 14 Mar 2017 17:05:56 -0400

> multiaccess LANs ​as most peering happens on p2p links where the same

For first party nexthops, BGP timers can address this to some extent.

BGP timers are too long, hence BFD.

For third party nexthops, you can have persistant bad forwarding.  Normal
BGP procedure doesn't help you here.

So, we agree that the case isn't given good attention in the core specs.

> NOTIFCATION SAFI for WAN VPNs.

The new SAFI is targeted specifically for the RS case and hasn't received
specific consideration for other use cases yet.  Feel free to kick off that
as a topic, although I'd suggest doing so separably from the rs-bfd thread.

-- Jeff

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 14 Mar 2017 21:08:24 +0000

Jeff, 

Is the assumption here that the client routers have routing view limited to what’s provided by the Route Server? If not, then wouldn’t Client Routers benefit from having to invalidate the path learned from the remote client router as soon as the connectivity check failed?

Of course, Client Routers conveying the lack of NLRI reachability per NH to the Route Server, and expecting Route Server to provide a different NHs of the NLRIs, and expecting it to be functional, while still attracting the traffic for unreachable destinations since the Loc-RIB is still pointing to the unreachable NH for the affected NLRIs.

I wonder whether  https://tools.ietf.org/html/draft-ietf-idr-bgp-bestpath-selection-criteria be useful here.

-- 
Cheers,
Rajiv Asati
Distinguished Engineer, Cisco

-----Original Message-----
From: Idr <[email-masked]> on behalf of Jeffrey Haas <[email-masked]>
Date: Tuesday, March 14, 2017 at 4:42 PM
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [Idr] I-D Action: draft-ietf-idr-rs-bfd-02.txt

    Robert,

    

    
    -- Jeff

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 14 Mar 2017 21:11:20 +0000

>   BGP procedure doesn't help you here.

Agreed. Unless BGP bestpath selection considers (event-driven) NH liveliness check, the blackholing problem would continue.

-- 
Cheers,
Rajiv 

https://tools.ietf.org/html/draft-ietf-idr-bgp-bestpath-selection-criteria-06

-----Original Message-----
From: Idr <[email-masked]> on behalf of Jeffrey Haas <[email-masked]>
Date: Tuesday, March 14, 2017 at 5:05 PM
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: [Idr] I-D Action: draft-ietf-idr-rs-bfd-02.txt

    

    

    -- Jeff

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 14 Mar 2017 21:21:18 +0000

the text states:

> 5.1.  Route Server Client Procedures for NHIB Changes
> 
>    When entries are added to the a route server client's Adj-NHIB-In for
>    a route server peering session, it will then attempt to verify
>    connectivity to the BGP nexthop for that entry.  The procedure
>    described in this specification utilizes BFD; other mechanisms are
>    permitted but are out of scope of this document.

It might be an idea to explicitly acknowledge that some clients may want
to selectively filter out NHIB on the basis that they don't want to
establish bfd sessions to those particular NHs.  There could be many
reasons for this, e.g. reliability of bfd on the remote side,
reliability of bfd on the local side.  This would require some selection
mechanism to be implemented on the client.

A couple of other things:

1.  if a route server client stops announcing prefixes to the route
server but still accepts prefixes, do all bfd sessions to it also get
torn down?  Or is the purpose of the RS Adj-NHIB-In to ensure stickiness
of bfd sessions?  If this is the case, it should be mentioned explicitly
because otherwise the feedback loop looks ... well, a bit peculiar.

2.
>                    It is incorrect provisioning for an IXP client which
>    is using a Route Server to have a BGP session with another IXP
>    client.

lolno, if I'm reading this correctly.  It's quite usual to have
bilateral + multilateral sessions configuration between router pairs.

Nick

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 14 Mar 2017 17:36:07 -0400

Rajiv,

> Is the assumption here that the client routers have routing view limited to what’s provided by the Route Server? If not, then wouldn’t Client Routers benefit from having to invalidate the path lear…

This is what I believe the procedure says.  See section 6.

> Of course, Client Routers conveying the lack of NLRI reachability per NH to the Route Server, and expecting Route Server to provide a different NHs of the NLRIs, and expecting it to be functional,…

The thing that is somewhat different for a IXP environment running a route
server than normal eBGP is the low (to zero) likelihood of having a backup
path.  If 10/8 was learned from the route server for nexthop 192.0.2.1, and
you stop being able to reach that nexthop, removing it from your forwarding
(unreachable) is your only choice.

You *might* have a source of that path internally.  In that case, you can
use it.

But more importantly, you'll stop sending it toward your own peering and
attracting blackholed traffic.

> I wonder whether  https://tools.ietf.org/html/draft-ietf-idr-bgp-bestpath-selection-criteria be useful here.

In a sense of good timing, John Scudder had brought this to my attention
about an hour ago. 

RS-BFD basically reinvents the same procedure in your draft, simply being
specific about the use of BFD as the dataplane liveness check mechanism.
However, as you note in the RS-BFD draft, we do leave the option for other
mechanisms.

I suspect the other authors of RS-BFD aren't particular where we pick up our
text for the resolvability condition.  However, if the suggestion is to make
a reference to your draft, you'll need to bring it back from zombie state
and progress it. :-)

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 14 Mar 2017 17:48:17 -0400

Nick,

> mechanism to be implemented on the client.

We had some text in earlier internal versions of the document that tried to
make this explicit.  I believe the text that remains is appropriately
permissive but maybe not as explicit as you might like:

:   At the route server, the Adj-NHIB-Out for each client is populated
:   with the next hops from its Loc-RIB.  If the BGP capabilities learned
:   during BGP session setup identify a next hop as compatible with this
:   proposal, this is reflected in the NHIB.  Initially, it is assumed
:   that the client router is able to reach its next hops which is stored
:   in the NHIB.  If a next hop is added to the NHIB for a particular
:   client, a route SHOULD be added to the router server's Adj-NHIB-Out.

The SHOULD is the relevant point.  This didn't make it into the sections for
explicit NHIB behavior, and probably could use some emphasis in a future
revision.

>From the client procedure side:
:   If the client can not establish a BFD session with an entry in its
:   NHIB, the next hop is put it in the Adj-NHIB-Out for backward
:   compatibility.

> because otherwise the feedback loop looks ... well, a bit peculiar.

Section 5.2:

:   In the event that a given client that supports this feature does not
:   provide any routes containing BGP next hops that would be used to
:   populate an Adj-NHIB-Out entry, the route server SHOULD advertise an
:   entry for such a router using the provided self-originated entry.
:   This permits the provisioning of BFD peering sessions for continuity
:   check when route exchange via the route server is asymmetric and one
:   client has routes from a second client, but not vice-versa.

> bilateral + multilateral sessions configuration between router pairs.

I suspected the sanity check might bounce here. :-)

We'll update the wording accordingly.  The main consideration is the new
SAFI doesn't get negotiated and start transitively propagating the state.

It might be worth adding verbiage about ensuring RS-Reachable SAFI have an
AS_PATH that is consistent with directly connected peers instead.

-- Jeff

---

From: Job Snijders <[email-masked]>
Date: Tue, 14 Mar 2017 22:48:32 +0100

On Tue, Mar 14, 2017 at 05:36:07PM -0400, Jeffrey Haas wrote:
> > Is the assumption here that the client routers have routing view
> > limited to what’s provided by the Route Server? If not, then
> > wouldn’t Client Routers benefit from having to invalidate the path
> > learned from the remote client router as soon as the connectivity
> > check failed?
> 
> > Of course, Client Routers conveying the lack of NLRI reachability
> > per NH to the Route Server, and expecting Route Server to provide a
> > different NHs of the NLRIs, and expecting it to be functional, while
> > still attracting the traffic for unreachable destinations since the
> > Loc-RIB is still pointing to the unreachable NH for the affected
> > NLRIs.
> 
> The thing that is somewhat different for a IXP environment running a
> route server than normal eBGP is the low (to zero) likelihood of
> having a backup path. If 10/8 was learned from the route server for
> nexthop 192.0.2.1, and you stop being able to reach that nexthop,
> removing it from your forwarding (unreachable) is your only choice.
> 
> You *might* have a source of that path internally.  In that case, you
> can use it.

I might be misunderstanding you, but a route server is considered (at
best) a supplementary partial view on the default-free zone. If the
route server doesn't have the route, you will have learned an
alternative route through either bilateral sessions on the IXP or
through other sources such as transit sessions, or through ebgp paths
distributed over ibgp. Any path that does not exist in the 'DFZ', but
_does_ exist on the route server, is either an artifact of hyper local
traffic engineering through deaggregation, or is a bgp hijack for the
purpose of spamming a specific subset of route server participants.

I'm somewhat inclined to accept that the route server itself may not
have alternative paths for anything it received from one of its
participants. however the likelihood of having an alternative path might
increase as the route server's constituency increases (a growth
trajectory which in turn might go hand in hand with increased chances of
intra-fabric partial unreachability between participants of the fabric).

either way, since we accepted that path hiding is an issue, we've
implicitly accepted that backup paths may exist.

> But more importantly, you'll stop sending it toward your own peering
> and attracting blackholed traffic.

i'm not sure i follow this scenario

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 14 Mar 2017 18:05:05 -0400

On Tue, Mar 14, 2017 at 10:48:32PM +0100, Job Snijders wrote:
> distributed over ibgp.

The RS client having a source of a path via its iBGP from its internal
network is exactly what I had in mind.

RS clients having bilateral sessions with other IXP clients was somewhat
less than common when I last operated a route server, so forgive my myopia
on that point. :-)

>  Any path that does not exist in the 'DFZ', but

I would also caution against Internet-operations focused myopia as well
here.  RSes are deployed in environments that aren't IXPs.

For the IXP case, I tend to agree with you.  (And have been half following
the GROW threads about having the RS participate in route validation.)

> i'm not sure i follow this scenario 

Receiving a path from the RS that is blackholing and propagating it
internally via iBGP.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 23:02:02 +0100

​Hi Job,​

> ​
> purpose of spamming a specific subset of route server participants.

​Interesting ... why would you think so ?

Imagine I connect to an IX and also have direct Transit ​with NTT. To the
same IX also customers of NTT connect such that I can use IX switch to get
to them without going via longer path of transit provider.

To me their direct routes are better (BGP path selection wise) and I go via
IX to reach them (or they go via IX to reach my content).

> participants.

​Well that is indeed very important to verify. I am also inclined to the
same. But have no data to validate how many paths for all nets RSes carry
in major IXes.

Moreover IXes normally use two RS today so it is trivial to make sure one
RS sends ​first best the other second best to the clients (if they have
more then one). Signalling client to RS that NH is down seems at best
redundant.

​Thx,
r.

---

From: Job Snijders <[email-masked]>
Date: Tue, 14 Mar 2017 23:21:24 +0100

> less than common when I last operated a route server, so forgive my myopia

Glad we uncovered this discrepancy, it is safe to assume that in many
cases (i say 'many' because we have no well established metric system
for this), both bilateral and multilateral sesions exist. Any
specification will need to be engineered with this in mind.

If someone is looking to receive as many BGP routes and paths as
possible, you'll set up sesions with whoever wants to, this includes
both direct and route servers. With the rise of network automation based
on public peering directories such as peeringdb.com, this process
becomes even more streamlined. The process effort of denying bilateral
peering requests because you can see the remote network through the
route server, or tuning the route server to not be used when there is a
bilateral session, would outweight just setting up bgp indiscriminately. 

> here.  RSes are deployed in environments that aren't IXPs.

This is new information for me.

> For the IXP case, I tend to agree with you.  (And have been half
> following the GROW threads about having the RS participate in route
> validation.)
> 

gotcha, yes.

---

From: Job Snijders <[email-masked]>
Date: Tue, 14 Mar 2017 23:33:33 +0100

On Tue, Mar 14, 2017 at 11:02:02PM +0100, Robert Raszuk wrote:
> > Any path that does not exist in the 'DFZ', but _does_ exist on the
> > route server, is either an artifact of hyper local traffic
> > engineering through deaggregation, or is a bgp hijack for the
> Interesting ... why would you think so ?

It is my observation that some networks will announce for instance a /16
to their upstream providers, but a /17 to their 'settlement free peers'
(which could be found at an IXP).

While there is no one single version of view of the Default-Free Zone,
we do have a an understanding of what global reachability means vs local
visibility. Through services such as http://stat.ripe.net/ you can see
instances where the /16 enjoys good visiblity on the RIS/routeviews
collectors but that more specifics also exist, which are not seem on all
collectors.

While the practise of announcing more specifics to certain classes of
BGP neighbors does not have my preference, I can follow the rationale.

The hijack case is discussed here https://ripe72.ripe.net/archives/video/203/ -
https://ripe72.ripe.net/presentations/165-invisiable-hijacking-follow-up.pdf 

To summarize: if a route server has a unique route (either by virtue of
being a more-specific of something learned over ibgp, or by being a
hijack), that route in and of itself might not be of any value.

I'd argue that for the public internet use case, it is fine to design a
solution which assumes that route servers never contain 'unique'
information, and that any participanting autonomous system will have
alternative paths either to the same prefix or a less specific.

> Imagine I connect to an IX and also have direct Transit with NTT. To
> the same IX also customers of NTT connect such that I can use IX
> switch to get to them without going via longer path of transit
> provider.

This is a valid and common scenario.

> To me their direct routes are better (BGP path selection wise) and I
> go via IX to reach them (or they go via IX to reach my content).

Correct.

> Well that is indeed very important to verify. I am also inclined to
> the same. But have no data to validate how many paths for all nets
> RSes carry in major IXes.
>
> RS sends first best the other second best to the clients (if they have

I believe it is a common practise amongst IXP Route Server operators to
let the two (or more) route servers operate entirely _independent_ from
each other, e.g. with no BGP sessions amongest the RS's themselves.

Kind regards,

Job

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 14 Mar 2017 23:46:45 +0100

​Job,​

> each other, e.g. with no BGP sessions amongest the RS's themselves.

​Since clients will likely eBGP peer to both route servers and send pretty
much the same UPDATES to both you do not need any session between RSes to
choose best one one and second best on the other towards the clients :) In
fact in RS case tie break may be rtr_id :)  Very easy to set such
preference in cli.

= = =

All,

And just for the record here the idea for NH SAFI has been proposed by Ilya
& myself in 2012 ...

https://tools.ietf.org/html/draft-ietf-idr-bgp-nh-cost-00

Best,
R.

​

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 14 Mar 2017 18:58:55 -0400

> https://tools.ietf.org/html/draft-ietf-idr-bgp-nh-cost-00

rs-bfd started with that as a likely way to handle carrying the RS state.
However, we reached two conclusions:
- the draft was inactive.
- and even if we used it, it was an awkward fit for the mechanism.

https://www.ietf.org/proceedings/93/slides/slides-93-idr-3.pdf

Similarly, BGP-LS was given a try and decided to be too heavy weight.

So, we're closer to where we started. :-)

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 15 Mar 2017 00:08:50 +0100

Ok so we have history covered :)

- - -

Apart of that the more I think about this the more reasons pop-up which
question both justification and value in informing RS about next hop
reachability from clients to clients of Internet Exchanges.

Let's summarize those again:

- RS may not have other paths

- RS may send more then best path to clients

- RS come in pairs and it is trivial to send different paths from each of
the RS with zero upgrade to clients

- RS clients are grouped per their policies .. per client RIB requires a
lot of overhead on route servers to address the case where only one client
in the group suffers from some unreachable next hop

- BFD sessions to next hops will require per next hop MD5 for security

- IX have often more then one lan ... say 1500 and 9000 MTU. Next hops
again may be different on both.

- BFD session timers can't be unified on the clients as some clients are
local and some remote. Remote both in the case of distributed IX as well as
in the case of IX customers just connecting to IX with fiber without any
*local* metal.

- Clients will likely report churn of the same next hop being unreachable
reasonable in the same time

- When client reports unreachable next hop to RS .. it removes the path
from client's RIB. But when does it add it again ? Note eBGP to RS are all
fine .. would clients now not only keep BFD sessions to active peers but
also trying to establish BFD to unreachable peers then notify RS that the
peer is back ?

- BFD on clients needs to work in non protocol associated mode.

... and I am pretty sure this is just the starter.

r.

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 14 Mar 2017 20:03:26 -0400

> the RS with zero upgrade to clients

But then you're not using a secondary RS for redundancy of your best path.
(That may be fine.)

> in the group suffers from some unreachable next hop

RS operators will have to speak to what sort of policies they want, but this
*does* impact what a RS chooses to do with a declaration of unreachability
from one peer in a peer group:
- Does it try to split the group for that peer? (Unrealistic, IMO.)
- Does it de-pref routes with that nexthop for *all* peers on the
  presumption that something bad has happened that the other peers haven't
  noticed yet?

Etc.

Good topic for further discussion.

> - BFD sessions to next hops will require per next hop MD5 for security

The use of BFD security mechanisms in general is an interesting question.
Thankfully the suggested BFD timers here are very slow which helps the
session scaling.

> reasonable in the same time

A significant enough fabric event will simply lose the peering sessions to
the RS in the first place.  But losing entire "sides" of clients happens.  

> also trying to establish BFD to unreachable peers then notify RS that the

This is no worse than an IGP event.

> ... and I am pretty sure this is just the starter.

And good fodder for discussion from IX operators.

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 15 Mar 2017 10:13:29 +0100

> This is no worse than an IGP event.

Well that in fact brings another alternative ... why not to simply run a
separate instance of ISIS over IX open LAN fabric interfaces ?

DIS could be on RS and even with basic hellos both RS and remote clients
will know about client being unreachable. If someone wants you can run BFD
to speed things up as well.

That addresses the reported problem without need for RS SAFI and for those
who do not want more then single path speeds up connectivity restoration
etc ... Even for those who have more then one path if ISIS timers are
enough they do not need to run BFD any more to all next hops.

Moreover *modern* IX operators will soon start adding value add services
(example: DDoS protection) for their customers for a little extra fee per
month. That in turn will be much simpler to be deployed with service chain
and IGP local flooding of for example SIDs.

Has that approach been considered here ? Again just like with data plane
gateway proposal this requires zero change on the client side and is just a
control plane config touch.

Best,
r.

On Wed, Mar 15, 2017 at 1:03 AM, Jeffrey Haas <[email-masked]> wrote:

> > > > And just for the record here the idea for NH SAFI has been proposed
> by
> > > Ilya
> > > > & myself in 2012 ...
> > > >

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 15 Mar 2017 14:55:06 +0000

> permissive but maybe not as explicit as you might like:

it's definitely permissive, but sometimes it's good to make things easy
for the reader.  FWIW, I think you're having some reverse bikeshedding
here, and part of the reason is a lack of broad brush strokes to make it
easy for the casual reader to follow what's going on in the heads of the
authors.

> :   client has routes from a second client, but not vice-versa.

Again, to make it easier for the reader, I'd suggest adding in some
helicopter-view text to explain the rationale behind this feedback
mechanism.  It's quite a nice way of handling bfd session stability,
btw, but it really is not obvious from a first (or even second) reading
what the overall picture is.

> I suspected the sanity check might bounce here. :-)

"This text was included intentionally to see if people were paying
attention".

> SAFI doesn't get negotiated and start transitively propagating the state.

Right, I hadn't considered that.  That would cause a pile of hilarity,
no doubt about it.

> AS_PATH that is consistent with directly connected peers instead.

ew, ugly hack. Please don't.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 15 Mar 2017 15:13:09 +0000

Robert Raszuk wrote:
> Has that approach been considered here

I have seen people attempt IGPs at IXPs before.  It was hilarious.

It wasn't intended to be hilarious.

Nick

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 15 Mar 2017 16:25:30 +0100

Nick,

One thing is to try to use IGP to distribute routes. Completely another
thing is to use it as topology discovery and next hop liveness signalling.

I am talking about the latter ... not the former.

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 15 Mar 2017 11:35:34 -0400

> easy for the casual reader to follow what's going on in the heads of the

This is more the issue of three different authors wordsmithing the document
with different styles.  This round had enough bits rewritten that the focus
was on correctness rather than ease of reading.  Next round will hopefully
address that presuming the technical content is accepted.

> what the overall picture is.

Agreed.

> "This text was included intentionally to see if people were paying

:-)

> Right, I hadn't considered that.  That would cause a pile of hilarity,

Exactly.  This state, for want of better terminology, is session local.
However, most implementations of BGP use generic RIB plumbing that makes
redistribution a likely accident.

> ew, ugly hack. Please don't.

Why not? It's the closest thing we have to "only accept stuff from the
adjacent peer".  We don't currently have a "verify that no-advertise did its
job" feature.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 15 Mar 2017 15:50:50 -0400

> It wasn't intended to be hilarious.

The reasons for the hilarity might not be obvious to some here.  Some of
them include:
- Effectively an NBMA environment that most software wants to pick up as a
  broadcast environment... and the protocols break when the L2 goes awry.
- IXPs tend to get a rather entertaining mish-mash of equipment... and thus
  bugs.
- IGPs really like to have a consistent map and aren't happy when the LSDB
  isn't consistent for various reasons. (See some above.)
- Security of IGP with multiple participant? Um, no.

Please share others. :-)

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 15 Mar 2017 21:09:11 +0100

> - IXPs tend to get a rather entertaining mish-mash of equipment... and
thus bugs.
+
> Please enlighten yourself what people use "as peering router" today :-)

While this is GROW question ... perhaps it would be very helpful to get
some more real data on what type of routers customers of the IX use to peer
within IX ? Why such equipment is considered at best as 2nd class citizen ?

For my BB IX peerings I would in fact put the best routers to peer outside
of my network on the edge - not just some junk or homebrew bgp/igp.

It is anyway interesting that now fear of IGP bugs is used a motivation for
BGP protocol extension.

My personal problem with proposed RS SAFI is that it delays even further
both RS support and IX operators support for ability to send multiple paths
to me as client of the IX. And I need those multiple paths for already
stated reasons.

One would hope that we learn something from years of PIC experience ...

Regards,
Robert.

---

From: Gert Doering <[email-masked]>
Date: Wed, 15 Mar 2017 22:26:56 +0100

Hi,

> within IX ? Why such equipment is considered at best as 2nd class citizen ?

"What people can afford, and which can barely keep up with the demand".

So, you'll still find Cisco Sup720 or Juniper MX80 as peering routers -
reliable workhorses as far as forwarding capacity is concerned, but CPU 
starved (= keep the number of BGP sessions down if you want any
semblance of stability), and unfortunately also with centralized BFD
(at least on the Cisco).

> of my network on the edge - not just some junk or homebrew bgp/igp.

If you keep buying "best of breed" boxes every 3 years, while the 
competition keeps chugging along with half the CAPEX needed, your
customers will wander over, and you'll have to reconsider.

For a "smallish" ISP over here, a "fat pipe" to an IXP is 10Gbit/s or
maybe 20 Gbit/s - so if you need to by a US$ 50k router for that,
every 3 years, it does quite add to the peering costs.

We typically have budget to forklift IXP routers every 8-10 years...
bandwidth growth isn't that fast anymore, so "nice and shiny" today
will be "old and crappy" in 6 years, but as long as it keeps working,
it will be kept around.

It's not like "peering routers are treated second-class", more like
"the market expects that this all costs nothing" - so *inside* the
networks, "proper routers" get replaced by "dumb and fast" P-routers
with little/no BGP, to save even more CAPEX.

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 15 Mar 2017 22:58:14 +0100

Thank you Gert for the info !

Like Job observed for some customers peering to/via IX is just an
additional optimization hence indeed some may not always put most expensive
and latest gear there.

Actually after this discussion I am quite curious how many nets on the
routes servers really have more then one bgp path. If not much then this
entire effort here may not be practically useful.

Now if you say that the equipment is rather dated on the customer edge how
and when do you think it will get upgraded with brand new images (provided
it is not already EOE/EOL) to even support what is being discussed here ?
Shouldn't we at least try to come up with solutions which require no new
upgrades for IX customers ? At most few lines of configuration of existing
equipment ?

Cheers,
r.

---

From: Job Snijders <[email-masked]>
Date: Wed, 15 Mar 2017 23:22:56 +0100

> entire effort here may not be practically useful.

Did some small crude sampling by asking around:

AMS-IX RS indicated that one out of three routes have an alternative path
DE-CIX RS 1 out of 2 routes have an alternative path
INEX RS seemed to have virtually no alternative paths for the routes

Kind regards,

Job

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 15 Mar 2017 23:38:31 +0100

Thank you Job !

So this means that every third or second customer in first two IXes
respectively connect to exchange with at least two edge routers or it could
be the same CE router with two interfaces :).

Now this also means that each such edge router is connecting to both
independent route servers.

Based on the above just making sure that those two route servers pick paths
with different next hop as best may already address the problem to improve
robustness with just putting only a negligible additional load on the
customer edge keeping current hardware and software as is.

Kind regards,
Robert.

---

From: Gert Doering <[email-masked]>
Date: Thu, 16 Mar 2017 08:37:53 +0100

Hi,

> be the same CE router with two interfaces :).

There's large ISPs connecting with up to 4 routers to the IXP (so, same
peer AS has multiple possible nexthops).

There's also smaller ASes buying upstream from two or more ISPs that are
all connected to the same IXP (so, same origin AS has paths through
different neighbouring ASes at the same IXP).

Optimizing the "route server shall distribute multiple paths" is only
half of the solution.  More interesting for me is "where do I need to
point my vendors to, to get them to implement proper black-hole
detection in the forwarding path, so the BGP NH can be invalidated?"
(I'm not aware of any gear that will do this right now, as in "if
IPv4 ARP or IPv6 ND fails, mark NH as invalid and look for other paths
in BGP" - but I'm always willing to learn)

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 16 Mar 2017 10:29:29 +0100

Hi Gert,

> different neighbouring ASes at the same IXP).

Well ​for one I just went through talking to multiple large ISPs to get
transit via IXes and the answer is NO. They all (except one) said only
direct peering with fiber. And the one who accepts to come via IX clearly
requires direct eBGP session.

So even if ISP is connecting to IX he is not offering free transit so his
routes will not be downloadable from RS. That is the main point here.

> in BGP" - but I'm always willing to learn)

​Cisco IOS classic/XE + NX object tracking is a cool feature quite unknown
as no marketing force was promoting it. You can run it even on Sup720 and
set frequency of probing accordingly :) ​I use it every day.

Juniper have had similar tool, but less flexible.

Asking vendors (or extend BFD spec) to support BFD livness detection to set
of destinations (say your current active next hops) without protocol may be
a wise move too.

But if we know that in real deployments only 0%, 33% or 50% nets have path
redundancy on RS sending information there in the new SAFI .. such that
single path nets lost reachability to next hops on all or some clients will
do no good.

​Thx,
r.​

---

From: Gert Doering <[email-masked]>
Date: Thu, 16 Mar 2017 11:04:44 +0100

Hi,

On Thu, Mar 16, 2017 at 10:29:29AM +0100, Robert Raszuk wrote:
> Well ???for one I just went through talking to multiple large ISPs to get

Terminology seems to be funky here.

"transit" is, for me, "I can use you to access the full Internet", not
"I can use your network to reach your customers" - which would be "peering".

So, except for 6939, I wouldn't expect anyone ever to give me "free transit",
no matter whether on an exchange or otherwise.

Now, at an IXP, there are participants (of course) that consider me too
small to peer with them - that's a political and marketing decision, but
not truly relevant for this discussion.  At DECIX, there are something
like 300+ ASes connected, some with multiple routers, and many of them
are quite happy to peer with the RS, offering connectivity to their
customer ASes.

Whether or not a few of them are not using the RS, or not sending routes
to the RS, can hardly be considered "the main point here", if we're
discussing enhancing RS functionality.

> ???Cisco IOS classic/XE + NX object tracking is a cool feature quite unknown
> set frequency of probing accordingly :) ???I use it every day.

Oh, I can, and I can tie other stuff into it (like, static routes).  But 
does *BGP* know, and invalidate its nexthops, if object tracking tells
it "does not ping anymore"?  Can I tell BGP "please run this automatically 
for all nexthops seen on this interface"?

Manually configuring this for each peer address I *might* receive over the
RS is operationally insane - 600+ devices on DECIX, with next-hop addresses
I might not even know about until I receive a route for it, from the RS.

> of destinations (say your current active next hops) without protocol may be

This is why I'm asking for "do we have a document to point vendors so,
so this magic would be available for me to use"?

Until then, the proposed draft is a good start.

Gert Doering
        -- NetMaster

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 16 Mar 2017 10:54:26 +0000

Robert Raszuk wrote:
> Based on the above just making sure that those two route servers pick
> paths with different next hop as best may already address the problem to
> improve robustness with just putting only a negligible additional load
> on the customer edge keeping current hardware and software as is.

just to be clear, 1. this does not fix the problem we're trying to
address here and even if it did, 2. ixp operators are not going to
impose their own routing tweaks on their participant base by choosing
different NHs.  IXP participants demand consistency and if ixp operators
don't provide this, or start interfering in the routing decisions of
their stakeholders, then the ixp participants will go elsewhere.

Nick

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 16 Mar 2017 11:01:12 +0000

> - Security of IGP with multiple participant? Um, no.

"Is that your anycast DNS IP address I see there? LOL, this should be fun."

No-one ever said IXPs were hygienic.

Nick

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 16 Mar 2017 13:12:48 +0100

> for all nexthops seen on this interface"?

​You can today say run it for all the peers which match a route map. It is
just a little extension to BGP Next Hop Tracking feature (which already
have table of all next hops) to actually also automate tracking next hops.

In fact one customer in the past requested that already .. but I don't know
if this was implemented in IOS already. I am sure folks from C vendor will
tell us :)​

> I might not even know about until I receive a route for it, from the RS.

​No manual stuff.

All you need is a single knob which will force only /32 next hop for
resolution (already available) + do the tracking for all next hops received
over given eBGP peer. ​

> Until then, the proposed draft is a good start.

​The feature request is one liner stated above. I am not sure if IETF
document is needed for that​. IMHO it is more of topic for Nanog/RIPE.

This is rather very much implementation specific.

Cisco will tell we will extend object + next hop tracking, Juniper will do
nice trick with rib-groups, linux based vendors will recommend to base nh
liveness detection based on arp tracking which is already in all linux
kernels.

The fundamental question is should it be unidirectional (no session
established of any sort between any to any) or we would like to keep the
state associated with tracked object (BFD, symmetric or asymmetric
echo/seamless etc ...).

​r.​

---

From: Gert Doering <[email-masked]>
Date: Thu, 16 Mar 2017 13:16:59 +0100

Hi,

> state associated with tracked object (BFD, symmetric or asymmetric

Both would be vastly superiour to what we have now - "no lively detection".

As for "just ask your vendor about it" - yeah, been doing that, since
years, for stuff that should have been obvious.  But the standard reply
is "nobody else is asking for it" and "how many new boxes are you going
to buy?" - and since we're small, the latter argument is not convincing.

Having an IETF recommendation how to make indirect peerings more robust
by doing x, y or z and tieing that into BGP NH feasibility decision 
would defuse the "nobody else is asking for it" argument thoroughly :)

Gert Doering
        -- NetMaster

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 16 Mar 2017 14:44:41 +0100

> Both would be vastly superiour to what we have now - "no lively detection".

​Ok .. let me take a stub on this with existing ​equipment after IETF. Will
report findings either offline or online. If for nothing else then to just
document what automation is already possible today in this space.

> would defuse the "nobody else is asking for it" argument thoroughly :)

​Well this to me sounds like a perfect GROW draft at most. ​

But let's face it ... for all major vendors IX based revenue does not even
fit to annual balance sheet .... so I am pretty sure any enhancement in
this space will be done only after good amount of convincing arguments even
if all IXPs sign an open letter "We need it".

Cheers,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 20 Mar 2017 16:04:15 -0400

> detection in the forwarding path, so the BGP NH can be invalidated?"

As noted earlier, the BFD component of the draft is separable and may be
worth a split at some later point if necessary.  It's also the easy part of
the draft.

> in BGP" - but I'm always willing to learn)

Having been at the wrong end of quirks of ND/ARP issues, I don't recommend
this as a sufficiently useful mechanism.  

I also suspect the IXPs wouldn't want their customers participating in a
layer 2 OAM mechanism.  This leaves layer 3, which moves us largely to BFD.

Or alternatively ping. Which is gross, but would totally work.

-- Jeff

---

From: Gert Doering <[email-masked]>
Date: Tue, 21 Mar 2017 09:26:22 +0100

Hi,

> On Thu, Mar 16, 2017 at 08:37:53AM +0100, Gert Doering wrote:
[..]
> this as a sufficiently useful mechanism.  

Well, it won't catch all possible faults - but generally having a feedback
mechanism from the L2 address resolution layer about "no idea where to
send this packet to!" would catch the easy ones, that is: total blackholing
due to fabric issues and "peer botched their IPv6 ACLs and ND breaks".

> Or alternatively ping. Which is gross, but would totally work.

Indeed :-)

Gert Doering
        -- NetMaster