Subject: [Idr] Debugging accepted routes from BGP speakers

From: Jared Mauch <[email-masked]>
Date: Mon, 18 Nov 2019 03:54:27 -0500

Greetings,

As a network operator we often don’t know if our ISPs are accepting the routes we send to them.  This makes it difficult at times for operators to debug things.  This is a first stab at a draft to try and propose a method to have the remote device echo back some route counts (and possibly NLRI) that were both accepted and rejected by the remote speaker.

It’s got a few things that I know need to be fixed at minimum, but I wanted to share this more broadly now.

I’m interested in feedback and ready to accept your arrows.

- Jared

> URL:            https://www.ietf.org/internet-drafts/draft-mauch-bgp-accepted-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-mauch-bgp-accepted/
> Htmlized:       https://tools.ietf.org/html/draft-mauch-bgp-accepted-00
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-mauch-bgp-accepted
> 
> Abstract:
>   This document defines a method to receive accepted and rejected NLRI
>   over a BGP peering session.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 18 Nov 2019 10:28:25 +0100

Hi Jared,

As you know what you are asking for has been covered by previous work and
even IDR WG document:

https://tools.ietf.org/html/draft-ietf-idr-operational-message-00

So as you are defining new BGP message anyway I take there are two options:

*A*  You think  that what we specified in operational message is too much

*B* You think that we will never need more information to be exchanged in
an informational manner between eBGP peers

If *A* would you think that clearly making all of those "additional
information" as optional would fix the problem and we could proceed with
the IDR WG draft in terms of asking for implementation ?

As to your document - few questions/observations:

* new message likely will require new capability.

* "Unknown IANA Considerations" is a new one :)

* How much does it help to know that peer's ASBR accepted your prefix but
it got filtered on egress of peering AS ?

* What is the definition of rejected ? Dropped by policy ? Not eligible for
best path ? Not inserted into RIB as better AD is there ? Marked badly by
origin validation ?

Thx,
Robert.

On Mon, Nov 18, 2019 at 9:54 AM Jared Mauch <[email-masked]> wrote:

> Greetings,
>
> As a network operator we often don’t know if our ISPs are accepting the
> routes we send to them.  This makes it difficult at times for operators to
> debug things.  This is a first stab at a draft to try and propose a method
> to have the remote device echo back some route counts (and possibly NLRI)
> that were both accepted and rejected by the remote speaker.
>
> > URL:
> https://www.ietf.org/internet-drafts/draft-mauch-bgp-accepted-00.txt
> > Status:
> https://datatracker.ietf.org/doc/draft-mauch-bgp-accepted/
> > Htmlized:
> https://datatracker.ietf.org/doc/html/draft-mauch-bgp-accepted
> >
> > Abstract:
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr
>

---

From: Job Snijders <[email-masked]>
Date: Mon, 18 Nov 2019 09:39:34 +0000

Dear Robert,

On Mon, Nov 18, 2019 at 9:28 AM Robert Raszuk <[email-masked]> wrote:
> As you know what you are asking for has been covered by previous work and even IDR WG document:
> *B* You think that we will never need more information to be exchanged in an informational manner between eBGP peers
>
> If *A* would you think that clearly making all of those "additional information" as optional would fix the problem and we could proceed with the IDR WG draft in terms of asking for implementation ?

You raise good points, I'll park this architectural question for
broader discussion on where we as a group want to go.

> * "Unknown IANA Considerations" is a new one :)

Yup, that is plumbing that needs to happen. We'll get a revision out
to address these points so a more complete proposal is on the table.

> * How much does it help to know that peer's ASBR accepted your prefix but it got filtered on egress of peering AS ?

In Internet routing operations, the common occurrence of the debugging
action we are trying to optimize is "are you accepting my route" and
much rarer "are you exporting it in the right places". While the
latter question is interesting, I don't see answering that question as
important as the first one. The latter one is oftentimes easily
validated by Internet-wide looking glasses, where as the first
instance has a stronger dependency on the adjacent network providing
(public) looking glass services.

> * What is the definition of rejected ? Dropped by policy ? Not eligible for best path ? Not inserted into RIB as better AD is there ? Marked badly by origin validation ?

Yes, good point, this should be clarified. The intention here is at
least "rejected by policy" and "failed to pass origin validation check
(which is a variant of 'rejected by policy'). However, it would be
good to hear more feedback on whether routes that are not considered
in the best path selection process due to an incorrect NEXT_HOP (or
AS_PATH loop, or whatever) should be communicated back as part of this
feature or not

Kind regards,

Job

---

From: Jared Mauch <[email-masked]>
Date: Mon, 18 Nov 2019 17:47:35 +0800

There are varying desires from operators. It may be as simple as a count to the list of accepted or rejected NLRI. 

If you look at the text the desire is to know is this path eligible for selection. That is accepted. 

Sent from my iCar

> On Nov 18, 2019, at 5:28 PM, Robert Raszuk <[email-masked]> wrote:
> 
> 
> Hi Jared,
>

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 18 Nov 2019 10:50:24 +0100

Hey Job,

> The latter one is oftentimes easily validated by Internet-wide looking
glasses

Hmmmm I must say that IMHO both latter and former could be addressed by
looking-glass. In fact when I read this draft that was my first question -
why not to just look at peer's looking glass ?

So perhaps we should simply issue a BCP to say that each AS should run a
looking glass server holding all paths and declare victory ? And that could
be all GROW WG thing too :)

I already see a bunch of new things we could accomplish in the Internet if
we would have those in place consistently everywhere - at least for each
transit AS.

Thx,
R.

On Mon, Nov 18, 2019 at 10:39 AM Job Snijders <[email-masked]> wrote:

> > *B* You think that we will never need more information to be exchanged
> in an informational manner between eBGP peers
> >
> > * How much does it help to know that peer's ASBR accepted your prefix
> but it got filtered on egress of peering AS ?
>
> > * What is the definition of rejected ? Dropped by policy ? Not eligible
> for best path ? Not inserted into RIB as better AD is there ? Marked badly
> by origin validation ?
>

---

From: Jared Mauch <[email-masked]>
Date: Mon, 18 Nov 2019 05:49:59 -0500

> On Nov 18, 2019, at 4:50 AM, Robert Raszuk <[email-masked]> wrote:
> 
> Hey Job,
> 
> > The latter one is oftentimes easily validated by Internet-wide looking glasses  
> 
> Hmmmm I must say that IMHO both latter and former could be addressed by looking-glass. In fact when I read this draft that was my first question - why not to just look at peer's looking glass ? 

I may have more than one session with the provider and need to know on a per-session basis.

> So perhaps we should simply issue a BCP to say that each AS should run a looking glass server holding all paths and declare victory ? And that could be all GROW WG thing too :) 

Sure, I’m sure the GROW chairs would welcome a draft.

> I already see a bunch of new things we could accomplish in the Internet if we would have those in place consistently everywhere - at least for each transit AS. 

I welcome you to open issues/suggestions (Job poked me about the existing GitHub page we have) here:

https://github.com/bgp/draft-mauch-idr-accepted-prefixes/issues

I can think of several things that people have suggested that I add including:

ack on a per-prefix/update basis
dump list of prefixes accepted
dump list of prefixes rejected
dump just counts of accepted/rejected
or perhaps variants of all of these.  (i also heard “i want to know if it’s the best-path and installed in fib”).

I think when we’re trying to write this, we should be focusing on what is minimally possible to do vs trying to enumerate them all. 

My intent here is to just know “does this pass the policy engine”, not did it win best path, etc.  We (akamai) see our ISP providers not always accept our prefixes but it’s not clear until you’re already in that failure case.  We would like to improve the ability to QA on our routing, which since each LG has a different API makes things often infeasible to scale.

- Jared

---

From: Job Snijders <[email-masked]>
Date: Mon, 18 Nov 2019 11:00:21 +0000

> Hmmmm I must say that IMHO both latter and former could be addressed by looking-glass. In fact when I read this draft that was my first question - why not to just look at peer's looking glass ?

Many networks unfortunately do not make BGP Looking Glasses available,
nor is there any standardized interface/method/design/approach for BGP
Looking Glasses. So solely relying on Looking Glasses for this
functionality has proven to be insufficient.

> So perhaps we should simply issue a BCP to say that each AS should run a looking glass server holding all paths and declare victory ? And that could be all GROW WG thing too :)

That is an interesting idea, but in my mind not the exclusive viable solution.

> I already see a bunch of new things we could accomplish in the Internet if we would have those in place consistently everywhere - at least for each transit AS.

Agreed - it would be a nicer world. Through the MANRS initiative I've
pitched the idea to provide more encouragement for networks to provide
looking glasses to the public, but arguably their availability is not
ubiquitous.

Another observation is that in the "IP Transit Carrier" segment of the
market we see BGP Looking Glasses from time to time, but we rarely see
similar functionality offered by Cloud/CDN providers. Perhaps the
latter category is not interested in running & maintaining looking
glasses, or perhaps there are other constraints that prevent them from
exposing this information via suchs tools. My hope is that by creating
a feedback mechanism in BGP we create more opportunity to share
debugging information specific to EBGP sessions between different
orgs.

Kind regards,

Job

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Tue, 19 Nov 2019 05:19:13 +0000

In https://tools.ietf.org/html/rfc4271#section-9.1.2
the AS loop is broken at the receiver.
Nowhere does it say that the sender must break the AS loop.
Split horizon filtering is a common practice, but nowhere
is it mandated. At least I could not find it.

If the receiver of your route were to send you back its
best path, even if it's your route, then you have your
information.

We could invent an address-family specific capability
to indicate that you wish your route to be echoed back.

Regards,
Jakob.

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Job Snijders
Sent: Monday, November 18, 2019 3:00 AM
To: Robert Raszuk <[email-masked]>
Cc: IDR <[email-masked]>
Subject: Re: [Idr] Debugging accepted routes from BGP speakers

Kind regards,

Job

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 19 Nov 2019 13:41:18 +0800

+1 Jakob

Sometime ago, validating exactly that (across 4 vendors/6 OS's), I observed  difference in behaviors across vendors and in 1 instance between different OS’s of the same vendor

Cheers,
Jeff

---

From: Zhuangshunwan <[email-masked]>
Date: Tue, 19 Nov 2019 05:57:51 +0000

+1
[inline] with [Shunwan].

Thanks,
Shunwan

-----Original Message-----
From: Idr [mailto:[email-masked]] On Behalf Of Jakob Heitz (jheitz)
Sent: Tuesday, November 19, 2019 1:19 PM
To: Job Snijders <[email-masked]>; Robert Raszuk <[email-masked]>
Cc: IDR <[email-masked]>
Subject: Re: [Idr] Debugging accepted routes from BGP speakers

In https://tools.ietf.org/html/rfc4271#section-9.1.2
the AS loop is broken at the receiver.
Nowhere does it say that the sender must break the AS loop.
Split horizon filtering is a common practice, but nowhere is it mandated. At least I could not find it.
[Shunwan] When we figure out how to Enhanced AS Loop Detection for BGP (https://datatracker.ietf.org/doc/draft-chen-grow-enhanced-as-loop-detection/), we also notice the different behaviors between different Network OSes and cannot find which rfc has documented it.

If the receiver of your route were to send you back its best path, even if it's your route, then you have your information.

We could invent an address-family specific capability to indicate that you wish your route to be echoed back.
[Shunwan] I think this is a good idea.

Regards,
Jakob.

-----Original Message-----
From: Idr <[email-masked]> On Behalf Of Job Snijders
Sent: Monday, November 18, 2019 3:00 AM
To: Robert Raszuk <[email-masked]>
Cc: IDR <[email-masked]>
Subject: Re: [Idr] Debugging accepted routes from BGP speakers

> Hmmmm I must say that IMHO both latter and former could be addressed by looking-glass. In fact when I read this draft that was my first question - why not to just look at peer's looking glass ?

Many networks unfortunately do not make BGP Looking Glasses available, nor is there any standardized interface/method/design/approach for BGP Looking Glasses. So solely relying on Looking Glasses for this functionality has proven to be insufficient.

> So perhaps we should simply issue a BCP to say that each AS should run 
> a looking glass server holding all paths and declare victory ? And 
> that could be all GROW WG thing too :)

> I already see a bunch of new things we could accomplish in the Internet if we would have those in place consistently everywhere - at least for each transit AS.

Agreed - it would be a nicer world. Through the MANRS initiative I've pitched the idea to provide more encouragement for networks to provide looking glasses to the public, but arguably their availability is not ubiquitous.

Another observation is that in the "IP Transit Carrier" segment of the market we see BGP Looking Glasses from time to time, but we rarely see similar functionality offered by Cloud/CDN providers. Perhaps the latter category is not interested in running & maintaining looking glasses, or perhaps there are other constraints that prevent them from exposing this information via suchs tools. My hope is that by creating a feedback mechanism in BGP we create more opportunity to share debugging information specific to EBGP sessions between different orgs.

Kind regards,

Job

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Nov 2019 10:18:19 +0100

Hey Jakob,

But this is not what is being asked here.

There is need to validated if routes received on *specific* eBGP session
were accepted by bgp inbound policy.

What you suggesting is also quite useful but IMO 100% orthogonal to the
topic.

Reason being that in your scenario router would advertise the best path to
a peer back which may not be local eBGP route. Worse I may have N eBGP
session to a given peer's ASBR and I would like to make sure all of my
paths has passed his policy.

Moreover as Jared said he is not so much concern of overall
reachability validation. He is concerned when his most preferred path goes
down does his peer has backup path(s) to him.

Kind regards,
R.

On Tue, Nov 19, 2019 at 6:19 AM Jakob Heitz (jheitz) <[email-masked]>
wrote:

> > I already see a bunch of new things we could accomplish in the Internet
> if we would have those in place consistently everywhere - at least for each
> transit AS.
>

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Tue, 19 Nov 2019 18:29:39 +0000

Robert,

You are right about the echoed path being a different path.
I forgot that ebgp does next-hop-self, so the receiver of the
echo cannot be sure it is his own echo.

Easy fixed. Do next-hop-unchanged on an echo path.

Another fix would be to add a community.
However, that would be harder to implement.
Ease of implementation is important if you want your
feature implemented.

The draft mentions nothing about backup.
I do remember some years ago, Ignas asked for this kind of feedback,
so he could know whether he should install a backup on the sender end.
Suppose R1 sends a route to R2. If R2 accepts the route,
R1 would like to install a backup in its FIB. Since FIB space
is precious, R1 does not want to install a backup if R2 does not
use R1's route.

Regards,
Jakob.

From: Robert Raszuk <[email-masked]>
Sent: Tuesday, November 19, 2019 1:18 AM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Job Snijders <[email-masked]>; IDR <[email-masked]>
Subject: Re: [Idr] Debugging accepted routes from BGP speakers

Hey Jakob,

There is need to validated if routes received on *specific* eBGP session were accepted by bgp inbound policy.

What you suggesting is also quite useful but IMO 100% orthogonal to the topic.

Reason being that in your scenario router would advertise the best path to a peer back which may not be local eBGP route. Worse I may have N eBGP session to a given peer's ASBR and I would like to make sure all of my paths has passed his policy.

Moreover as Jared said he is not so much concern of overall reachability validation. He is concerned when his most preferred path goes down does his peer has backup path(s) to him.

Kind regards,
R.

On Tue, Nov 19, 2019 at 6:19 AM Jakob Heitz (jheitz) <[email-masked]<mailto:[email-masked]>> wrote:

Regards,
Jakob.

-----Original Message-----
From: Idr <[email-masked]<mailto:[email-masked]>> On Behalf Of Job Snijders
Sent: Monday, November 18, 2019 3:00 AM
To: Robert Raszuk <[email-masked]<mailto:[email-masked]>>
Cc: IDR <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Idr] Debugging accepted routes from BGP speakers

Kind regards,

Job

_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 19 Nov 2019 19:43:18 +0100

Hi Jakob,

> Easy fixed. Do next-hop-unchanged on an echo path.

I don't think the above will be sufficient the way it is expected to work.
If you still want to stick it into existing AFI/SAFI and send it on the
same session the only feasible option I could see is to actually send all
paths back with ADD-PATHs on the eBGP.

However I am not sure if we would want to go there.

I am suggesting to resurrect BGP Operational Message IDR WG draft and do
this properly. At least now we seems to be having some operational support
:).

Best,
R.

PS. And I agree with the other part of your comments. But as Jared & Job
explained this is not about data plane ... this is just to make sure my
path was accepted to your BGP.

On Tue, Nov 19, 2019 at 7:29 PM Jakob Heitz (jheitz) <[email-masked]>
wrote:

> *From:* Robert Raszuk <[email-masked]>
> *Sent:* Tuesday, November 19, 2019 1:18 AM
> *To:* Jakob Heitz (jheitz) <[email-masked]>
> *Cc:* Job Snijders <[email-masked]>; IDR <[email-masked]>
> *Subject:* Re: [Idr] Debugging accepted routes from BGP speakers
>
>
>
> Hey Jakob,
>
>
>