Subject: Re: [GROW] BGP Looking Glass Capability

From: "Rayhaan Jaufeerally (IETF)" <[email-masked]>
Date: Sun, 25 Jul 2021 21:24:14 +0200

Dear GROW WG,

Following the feedback previously on my looking glass draft proposal, I've
updated the looking glass draft (
https://datatracker.ietf.org/doc/draft-jaufeerally-bgp-lg-cap/), to move
away from a capability mechanism to a new AFI which uses the MP_REACH and
MP_UNREACH messages in the BGP multiprotocol specification (
https://datatracker.ietf.org/doc/html/rfc4760) to carry an administrative
message. This administrative message is then defined to have a payload type
that carries a URL for the looking glass (LG), as well as the ASN that is
operating the LG.

The advantage of this approach is that it allows for propagating the LGs of
upstream peers so leaf ASes can check prefix reachability in peers not
directly connected.

Looking at the development of the idea, I think it could make sense to
split the document into one for the administrative message and AFI, and
another for the looking glass message type within that.

I will present an overview of the draft and the expected use cases during
the upcoming GROW meeting at IETF111, so feel free to either comment here
or ask a question in the session if you have any feedback.

Thanks,
Rayhaan

On Tue, Apr 27, 2021 at 1:29 PM Job Snijders <[email-masked]> wrote:

> On Mon, Apr 26, 2021 at 01:16:55AM +0200, Rayhaan Jaufeerally (IETF) wrote:
> > > Consistent API that serves RIB data
> >
> > Initially I tried to avoid defining the exact API of the looking glass by
> > pointing to https://tools.ietf.org/html/rfc8522, but unfortunately it
> does
> > not strictly define the response format instead leaving it up to the
> > implementation what to return to queries, which IMO is not very useful
> for
> > automation.
> >
> > As y'all have pointed out there is a wealth of implementations out there
> > that have tried to address this (and adding some others I've found):
> >
> >    - Paolo's pmacct looking glass document:
> >
> https://github.com/pmacct/pmacct/blob/master/docs/LOOKING_GLASS_FORMAT,
> >    - Birdseye https://github.com/inex/birdseye
> >    - CAIDA looking glass API
> >    https://www.caida.org/tools/utilities/looking-glass-api/
> >    - DE-CIX: https://www.de-cix.net/en/resources/looking-glass (which is
> >    using https://github.com/alice-lg/alice-lg)
> >    - RIPEStat's API, e.g.
> >
> https://stat.ripe.net/data/looking-glass/data.json?preferred_version=2.1&resource=2a0d:d740::/48
>
> Just to add to the list: there is yet another API-based Looking Glass
> targetting OpenBGPD: https://github.com/cjeker/bgplgd
>
> A demo is here:
>     http://165.22.27.105/bgplgd/summary
>     http://165.22.27.105/bgplgd/rib?as=22512
>
> Kind regards,
>
> Job
>

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 25 Jul 2021 22:58:21 +0200

Dear Rayhaan,

Thank you for working on this draft.

As far as your suggestion to use new SAFI to auto discover presence and
addresses of Looking Glass servers across ASN boundaries I have
nothing against.

As you know the alternative to exchange this type of information would be
via defining a new BGP Message (just like we ddi this in the past in case
of BGP Operational Message). But here perhaps you want to propagate this
beyond single ASN peer so new SAFI may work better.

I however have one major concern on how are Looking Glasses going to help
to address your original problem ie. to check if my prefix was announced
and accepted by peering AS.

Note that ASBRs normally send only best paths. If best path was learned
over IBGP (tie breaker was Local Pref or MED) ASBR may receive and accept
your route, but that route would not be advertised to LGs.

The natural solution here is use of best external and add-paths to make
sure LGs really get all received and accepted paths for a given net - but I
am not sure how common that is deployed today on sessions between ASBRs and
LGs. Last time I checked LGs get peering with RRs and get RR paths. Even
with add-path there and no best external your original issue would not be
solved.

So I would recommend that we review this point before focusing more on LG
auto discovery mechanics. Btw this draft if progressing should be IMO
discussed in IDR and changed to Standards Track. But GROW is an excellent
forum to first make sure if the entire idea proposed here is sound enough.

Best,
R.

On Sun, Jul 25, 2021 at 9:24 PM Rayhaan Jaufeerally (IETF) <[email-masked]>
wrote:

> The advantage of this approach is that it allows for propagating the LGs
> of upstream peers so leaf ASes can check prefix reachability in peers not
> directly connected.
>
> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
>

---

From: "Rayhaan Jaufeerally (IETF)" <[email-masked]>
Date: Sun, 25 Jul 2021 23:58:21 +0200

Thanks for the reply and the great points Robert,

> addresses of Looking Glass servers across ASN boundaries I have

Ah I just realized that you mention SAFI here, but when writing the draft I
was thinking about an AFI. Now that you mention it though, I was initially
thinking how a peer with a multiprotocol session would know which address
family the looking glass is for, if there are multiple. My reasoning here
was that the looking glass itself would allow for selecting the AFI through
its interface, so it does not need to be advertised in band. Now you
mentioned it though, it seems that actually using the proper AFI set (e.g.
IPv4 or IPv6 etc) and then using a new SAFI to carry the looking glass
address would be an elegant design too.

> beyond single ASN peer so new SAFI may work better.

Yeah I took a look at the operational message and it is one of the things I
took inspiration from, to make the mechanism extensible to carry other
administrative payloads in the future. I think going down the path
attribute approach would be easier to implement since it reuses the
existing semantics of the multiprotocol path attributes. For example, I can
imagine looking glass reachability can use MP_REACH and then when the
looking glass is no longer reachable it can be withdrawn with MP_UNREACH,
without needing to add that complexity into the looking glass message
itself.

> your route, but that route would not be advertised to LGs.

Interesting, I hadn't considered this type of architecture where there is a
centralized looking glass that is fed from the border routers. From what I
had seen on some looking glasses like https://lg.he.net/ or
https://lg.twelve99.net/ they let you choose which PoP to perform the
lookup in. From that I had assumed that the LG URL provided in the
administrative message would be customized to be a looking glass for the
ASBR that the BGP session is with. Maybe one way to deal with this would be
to use the query parameter router field in RFC8522 section 2.2:
https://datatracker.ietf.org/doc/html/rfc8522#section-2.2, and pre-populate
that URL with the query parameter in the admin message.

When I do a BGP route lookup on for example the HE looking glass, I get
some output like

1 Prefix: 193.36.104.0/24, Rx path-id:0x00000000, Tx path-id:0x00000000,
rank:0x00000001, Status: BI, Age: 15d1h2m45s
NEXT_HOP: 91.206.52.201, Metric: 17, Learned from Peer: 216.218.252.82
(6939)
LOCAL_PREF: 100, MED: 0, ORIGIN: igp, Weight: 0, GROUP_BEST: 1
AS_PATH: 210036
COMMUNITIES: 6939:7215 6939:8756 6939:9002
2 Prefix: 193.36.104.0/24, Rx path-id:0x00000000, Tx path-id:0x000a0001,
rank:0x00000002, Status: I, Age: 15d1h2m45s
NEXT_HOP: 37.49.238.32, Metric: 127, Learned from Peer: 216.218.252.184
(6939)
LOCAL_PREF: 100, MED: 0, ORIGIN: igp, Weight: 0, GROUP_BEST: 0
AS_PATH: 210036
COMMUNITIES: 6939:7055 6939:8250 6939:9002
... truncated ... full version: https://pastebin.com/SxukdS4L

Which shows all the accepted routes, and then the announcing router can
look through that for it's own address as the nexthop? I agree though that
rfc8522 does not specify this and maybe a more structured output format
would be desirable to allow for parsing.

> with add-path there and no best external your original issue would not be

I think using the router query parameter / nexthop lookup would solve this,
but I may have misunderstood the setup / problem, so please do correct me
if this would not solve the issue.

> forum to first make sure if the entire idea proposed here is sound enough.

Sounds good, I'm learning a lot through discussing the proposal here and
I'm thankful for the first rounds of feedback to get the document into a
better state before discussing it more broadly. I'll defer to the chairs to
determine if / when this should be changed to standards track, as I think
there are some prerequisites for that?

Thanks,
Rayhaan

>
> Best,
> R.
>
>
>
>
>
>
>
>
> On Sun, Jul 25, 2021 at 9:24 PM Rayhaan Jaufeerally (IETF) <
> [email-masked]> wrote:
>
>> Dear GROW WG,
>>
>> Following the feedback previously on my looking glass draft proposal,
>> I've updated the looking glass draft (
>> https://datatracker.ietf.org/doc/html/rfc4760) to carry an
>> administrative message. This administrative message is then defined to have
>> a payload type that carries a URL for the looking glass (LG), as well as
>> the ASN that is operating the LG.
>>

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 26 Jul 2021 01:04:30 +0200

Hi,

Just few comments:

> was thinking about an AFI.
>

To define a new MP_REACH you need both AFI (two octets) + SAFI (1 octet)
RFC 4760 section 2.

interface, so it does not need to be advertised in band. Now you mentioned
> it though, it seems that actually using the proper AFI set (e.g. IPv4 or
> IPv6 etc) and then using a new SAFI to carry the looking glass address
> would be an elegant design too.
>

You mean that address of the same LG would be send over AFI=1/SAFI=LG to
send IPv4 address and AFI=2/SAFI=LG to send given LG's IPv6 address ? Looks
ok if you want to really negotiate two capabilities for that.

>  For example, I can imagine looking glass reachability can use MP_REACH
> and then when the looking glass is no longer reachable it can be withdrawn
> with MP_UNREACH, without needing to add that complexity into the looking
> glass message itself.
>

Typically in BGP we try to shield the world from say switch flap given
server is connected to.  Sending host routes all over the place - while
doable - I am not sure will be best for protocol stability.

I am still not sure what is wrong using DNS for this type of discovery. If
I were to tackle this problem I would start with DNS.

Say your draft will define a well known name  ASN-LG (example: 6939-lg) .
Then each AS willing to play the game will just push the current IP
address(es) into such well known DNS record so anyone can get seamlessly to
their looking glasses pool. Done.

And this does not require any extension to BGP :) Moreover such draft can
be worked on and published by GROW WG too.

> centralized looking glass that is fed from the border routers.
>

Well I am sure there is a lot of such collectors in the wild. Maybe we also
have as you say just type of jump hosts where under the hood when you query
LG a box get's to production box to get the info live. And even so maybe it
would log into POP's RR not ASBR so the requirement of best external still
applies.

Thx,
R.

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 26 Jul 2021 10:30:59 +0100

Robert Raszuk wrote on 26/07/2021 00:04:
> I am still not sure what is wrong using DNS for this type of discovery. 
> If I were to tackle this problem I would start with DNS.

^^^ this, but I'm not convinced that dynamic discovery of LG endpoints 
is a pressing issue to start with.  What's more important is a 
functional API which can retrieve the rib info dynamically (netconf or 
otherwise).

Nick

---

From: "Rayhaan Jaufeerally (IETF)" <[email-masked]>
Date: Mon, 26 Jul 2021 14:02:58 +0200

I totally agree that an API for programmatically querying the LG would be
the next step on the road to implementing the overall goal, but I picked
this smaller task of discovering the looking glass first, before designing
the broader ecosystem.

I thought of the following concerns when evaluating the DNS approach (I
think it's been discussed earlier too):

   - You need to have some DNS zone for the AS, and there's no guarantee
   that AS 65534 can get a domain like as65534.net
   - You could use something like reverse DNS zones in in-addr.arpa for
   example to map the IP of the ASBR to an SRV record or something, but
      - That requires a coordination between the DNS zone management and
      looking glass / BGP operations, and in larger organizations it might be
      more work to set that up which I fear will impede adoption,
   - In the DNS approach you don't know to which peers a route announcement
   was forwarded to, so how would you then know which ASes to look up the
   looking glass for? The in-band approach solves this by having the BGP
   speaker forward the LG address of peers that announcements have been
   forwarded to. That way the downstream AS can discover which upstream peers
   exist and query the LG and probe reachability that way,

I think the last point is the most concerning for me, the fact that it's
all self contained is more of a nice to have, but I think not knowing
programatically what peers announcements are being forwarded to would limit
the usefulness of the DNS approach, I'm happy to be convinced otherwise
though :).

Cheers,
Rayhaan

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 26 Jul 2021 14:49:12 +0200

Hi,

I think Nick meant API to query the local BGP RIB of the router itself not
the LG.

For DNS domain there are couple of options. For example we could
register new root domain .bgp which would be reserved to be allocated
automatically with well known names like ASN.bgp by RIR which allocates
ASN.

Then lg.[ASN].bgp would be what you are looking for.

> The in-band approach solves this by having the BGP speaker forward the LG
address of peers that announcements have been forwarded to.

Sorry but I assume you know your peers and their ASNs. So not sure I get
the benefit of same session (not even guaranteed) autodiscovery. Besides in
looking glass you normally do not have access to specific ASBR's RIB ...
maybe POP. If so I assume you know which POP to check.

Do you rather mean that you want to autodiscover your peer's POP and in the
case of LG per POP you want to get specific Looking Glass server address
serving that POP ?

Keep in mind that accepting the announcement does not mean it is going to
be used in forwarding or propagated by BGP upstream. If you are looking for
the assurance of the latter I would rather use data plane tools with wisely
chosen src address of say enhanced trace to make sure what you announced is
installed and used by the peer.

Cheers,
R,

On Mon, Jul 26, 2021 at 2:03 PM Rayhaan Jaufeerally (IETF) <[email-masked]>
wrote:

>    - In the DNS approach you don't know to which peers a route
>    announcement was forwarded to, so how would you then know which ASes to
>    look up the looking glass for? The in-band approach solves this by having
>    the BGP speaker forward the LG address of peers that announcements have
>    been forwarded to. That way the downstream AS can discover which upstream
>    peers exist and query the LG and probe reachability that way,
>

---

From: "Rayhaan Jaufeerally (IETF)" <[email-masked]>
Date: Mon, 26 Jul 2021 19:54:41 +0200

Hi,

> I think Nick meant API to query the local BGP RIB of the router itself not

I was under the impression that the LG would provide a view of the RIB of
the ASBR?

> Then lg.[ASN].bgp would be what you are looking for.

That would be a neat solution and I can imagine other uses for such a
domain name, e.g. RPKI certificate hosting, peering policy etc, so it might
be worth looking into as a broader effort. It still requires coordination
with some kind of authoritative DNS service though so it would not be
standalone from the point of the BGP speakers.

>
>
> > The in-band approach solves this by having the BGP speaker forward the
> LG address of peers that announcements have been forwarded to.
>

Yes you know the direct peers that you have established sessions with, but
whom they decide to forward announcements to may also be of interest.
Consider a leaf AS which buys IP transit from a local ISP, which then has a
mix of 3 tier-1 upstreams. I think it would be useful for the leaf AS to
automatically discover what tier-1 ASes the announcement gets forwarded to,
so that they can check for reachability in them to make sure they are not
filtered out due to having a misisng IRR or bad RPKI config, or whatever
other policy reason.

>
> Do you rather mean that you want to autodiscover your peer's POP and in
> the case of LG per POP you want to get specific Looking Glass server
> address serving that POP ?
>

Yes that is what I was trying to do with the router query parameter from
RFC8522, and once you know which "router" or PoP to query, my assumption
would be that if you peer with any ASBR in that PoP then the looking glass
should, if the route was accepted, display it when you look the prefix up?

> installed and used by the peer.

Sure of course there's no guarantee that it will be used in the FIB, but my
rationale was, if it's not in the RIB then it's for sure not in the FIB, so
let's check that the desired routes are in the RIB by looking through the
LG.

Cheers,
Rayhaan

>
> Cheers,
> R,
>
>
> On Mon, Jul 26, 2021 at 2:03 PM Rayhaan Jaufeerally (IETF) <
> [email-masked]> wrote:
>

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 27 Jul 2021 12:03:12 +0100

Rayhaan Jaufeerally (IETF) wrote on 26/07/2021 18:54:
> would be that if you peer with any ASBR in that PoP then the looking 
> glass should, if the route was accepted, display it when you look the 
> prefix up?

Hi Rayhaan,

Can you provide a clear explanation of what your overall end goal is here?

Nick

---

From: "Rayhaan Jaufeerally (IETF)" <[email-masked]>
Date: Tue, 27 Jul 2021 15:56:58 +0200

Hi Nick,

The overall goal I'm trying to achieve is to know in an automated fashion
whether the routes being announced from a BGP speaker are being accepted by
peers (and their upstreams where applicable).

Thanks,
Rayhaan

---

From: Thomas Mangin <[email-masked]>
Date: Tue, 27 Jul 2021 22:25:24 +0100

Hello everyone,

While quite a few drafts have been using attributes to carry weird
information into BGP, this one proposes to use MP.
I can see how one may think it would be helpful and reduce implementation
burgen but I am not sure it is wise and I believe it goes beyond what
AFI/SAFI are for.

Also this reminds me very much of
https://datatracker.ietf.org/doc/html/draft-ietf-idr-operational-message
which I implemented but never saw traction.
So while I can see why this would benefit operational matters, I do not
believe the RFC as proposed should be accepted.

Sorry for being such a party pooper !

Thomas

On Sat, 24 Apr 2021 at 14:38, Rayhaan Jaufeerally (IETF) <[email-masked]>
wrote:

> Dear GROW chairs and participants,
>
> I would like to propose draft-jaufeerally-bgp-lg-cap-00 (
> https://datatracker.ietf.org/doc/draft-jaufeerally-bgp-lg-cap/) as a
> mechanism for in-band dissemination of looking glass endpoints in BGP,
> using a new OPEN message capability.
>
> The rationale behind this is to facilitate automation around eBGP peering,
> for example  to make it possible to automatically detect if the peer has
> accepted some routes which are expected to be accepted.
>
> I'm aware that the underlying RFC8522 is an informational RFC and leaves
> some details unspecified for the response format (i.e. a schema for the
> queries/responses) but I believe that can be further refined in other works
> independent to this proposal.
>
> I would like to hear what the WG thinks, if this is a reasonable proposal
> which fits into the broader charter of GROW?
>
> Thanks,
> Rayhaan

---

From: "Rayhaan Jaufeerally (IETF)" <[email-masked]>
Date: Wed, 28 Jul 2021 01:01:15 +0200

Thanks for the feedback here and in the session,

I agree that using AFI / MP is a hack and would be better served by an
operational style message. I'm not sure how to take that work and adapt it
to carry the information for communicating route rejection based on policy.
I guess it would be a new draft and cite the operational message in the
acknowledgements?

My key takeaways from the discussion in the meeting:

   - The problem that is to be solved is: "Did the peer accept a route
   which a speaker sent".
      - Explicitly say forwarding routes to other peers is out of scope
      because that depends on other factors that not all networks want to share
      - A looking glass does not actually show the state of the RIB of the
      peering router, because provider architectures send the routes from the
      ASBR through other internal mechanisms before showing up on the looking
      glass, and as such looking at the LG, it's not clear or not
visible at all
      what happened to a particular route.
      - It's cleaner to flip the problem statement around: "Did a route
      that a peer sent get rejected (i.e. not imported to the local RIB or
      considered as a viable path) for some policy reason?".
   - There is also value in collecting looking glass addresses, but that's
   out of scope of what this discussion is about, maybe some kind of DNS SRV
   record on the peering router's IP or something but look into that
   separately.

Cheers,
Rayhaan

On Tue, Jul 27, 2021 at 11:25 PM Thomas Mangin <
[email-masked]> wrote:

>> The rationale behind this is to facilitate automation around eBGP
>> peering, for example  to make it possible to automatically detect if the
>> peer has accepted some routes which are expected to be accepted.
>>

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 28 Jul 2021 01:13:17 +0200

Rayhaan,

Thank you for the presentation.

Homework from your session and discussion @ GROW (including some side
discussions) is that I will follow up with Operational Message and
perhaps try to get 2nd implementation for it. With that it may just catch
momentum and your and few other needs may start to see the light at the end
of the tunnel.

Cheers,
R.

On Wed, Jul 28, 2021 at 1:01 AM Rayhaan Jaufeerally (IETF) <[email-masked]>
wrote:

>       - A looking glass does not actually show the state of the RIB of
>       the peering router, because provider architectures send the routes from the
>       glass, and as such looking at the LG, it's not clear or not visible at all
>    - There is also value in collecting looking glass addresses, but
>    that's out of scope of what this discussion is about, maybe some kind of
>    DNS SRV record on the peering router's IP or something but look into that
>    separately.
>
> Cheers,
> Rayhaan
>
>>> I would like to hear what the WG thinks, if this is a reasonable
>>> proposal which fits into the broader charter of GROW?
>>>
>>> Thanks,
>>> Rayhaan