Subject: Re: [GROW] Limiting AS path length?

From: John Kristoff <[email-masked]>
Date: Mon, 16 Sep 2019 08:18:47 -0500

On Mon, 16 Sep 2019 08:21:08 +0000
Iljitsch van Beijnum <iljitsch=[email-masked]> wrote:

> Dear Global Routing Operators,

And medium-size academic netop.

> My question: is rejecting excessively long AS paths something we want
> to do?

It is something I've decided to do.  When I looked at the paths we were
getting a few months back I decided to drop at 20.  I've pointed people
to my template doc so it is possible that others may have done this as
well.

  <https://github.com/jtkristoff/junos/blob/master/sanitize.conf>

John

---

From: Job Snijders <[email-masked]>
Date: Mon, 16 Sep 2019 16:24:57 +0300

> to my template doc so it is possible that others may have done this as

Can we articulate what problem is solved by limiting the AS_PATH length?

Kind regards,

Job

---

From: David Farmer <[email-masked]>
Date: Mon, 16 Sep 2019 10:18:09 -0500

> Can we articulate what problem is solved by limiting the AS_PATH length?

I'm aware of a Netflow tool that crashes when it receives BGP paths that
are excessively long.  Furthermore, excessively long paths will use more
memory which could cause stability issues in some situations.

Having a sanity limit on path length isn't a bad idea. Personally, I think
20 a little on the short side, but 40 or 50 seems like a reasonable limit.
Anything beyond that is most definitely excessive, and I'm not sure you
would even want to use such a path if it were real.

 thanks

-- 
===============================================
David Farmer               Email:[email-masked]
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================

---

From: john heasley <[email-masked]>
Date: Mon, 16 Sep 2019 16:03:09 +0000

Mon, Sep 16, 2019 at 10:18:09AM -0500, David Farmer:
> Furthermore, excessively long paths will use more memory

Not significantly.  It is not length that represents large consumption
by AS-PATHs, its variance of paths in the RIB.  ie: no sane
implementation copies the path to every RIB entry, its a pointer/reference.

1 transit 1 ibgp: 185637 BGP AS-PATH entries using 8.8M
2 transits: 338034 BGP AS-PATH entries using 16.5M
1 transit 47 peers: 299673 BGP AS-PATH entries using 15M

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 16 Sep 2019 18:17:12 +0200

> implementation copies the path to every RIB entry, its a pointer/reference.

An extra AS hop takes six bytes on the wire: two for the 16-bit AS path and four for the 32-bit AS path.

AS path prepending (all of it, not just the excessive stuff) increases the average AS path length by 0.4 - 0.5, that's about 15% (something like 3.5 -> 3.9 / 4.1 -> 4.6, depending on your vantage point).

Iljitsch

---

From: Jeffrey Haas <[email-masked]>
Date: Mon, 16 Sep 2019 12:42:38 -0400

Speaking as a vendor:

> On Sep 16, 2019, at 11:18 AM, David Farmer <[email-masked]> wrote:
> I'm aware of a Netflow tool that crashes when it receives BGP paths that are excessively long.  Furthermore, excessively long paths will use more memory which could cause stability issues in some situations.

There are a number of bugs in implementations over the life of BGP related to path length.  In particular, overflows either in the number of ASes in the segment (max 255), where a second segment needs to be created; overflows in the length of the path attribute and needing to switch the size of the length field (the extended bit).

This is the primary reason we've tended to see strong pushes for filter length - attempts to avoid exercising such bugs.  Ideally, implementations should have been upgraded to avoid them by this point.

I am not optimistic. :-)

With regard to memory use, it's not a real problem for implementations and is generally noise in the real world vs. the total number of paths in your router.

> 
> Having a sanity limit on path length isn't a bad idea. Personally, I think 20 a little on the short side, but 40 or 50 seems like a reasonable limit. Anything beyond that is most definitely excessive, and I'm not sure you would even want to use such a path if it were real.

Quoting a coworker: constants are almost always wrong. :-)

-- Jeff

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Mon, 16 Sep 2019 18:25:21 +0000

I agree with Jeff.
Not that Cisco has these bugs :)
Just kidding Jeff, I take your point.
Cisco once had a bug at 255. It's long been fixed.

I'll add that Netflow can use the AS-PATH. When that option is used,
the AS-PATH is downloaded to FIB, which is more constrained of memory than BGP is.
I don't know what kinds of bugs could result from long AS-PATHs in Netflow collectors.

Regards,
Jakob.

-----Original Message-----
From: GROW <[email-masked]> On Behalf Of Jeffrey Haas
Sent: Monday, September 16, 2019 9:43 AM
To: David Farmer <[email-masked]>
Cc: Iljitsch van Beijnum <iljitsch=[email-masked]>; [email-masked]
Subject: Re: [GROW] Limiting AS path length?

I am not optimistic. :-)

-- Jeff

_______________________________________________
GROW mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/grow

---

From: Jared Mauch <[email-masked]>
Date: Mon, 16 Sep 2019 14:28:07 -0400

Operator here.

We have limitations around 128 to avoid some vendor bugs. I suspect we could remove them but people with mental histories of pain always worry about that change. 

Sent from my iStarship

> On Sep 16, 2019, at 12:42 PM, Jeffrey Haas <[email-masked]> wrote:
> 
> Speaking as a vendor:
> 
>

---

From: Job Snijders <[email-masked]>
Date: Mon, 16 Sep 2019 21:37:10 +0300

Limiting the AS_PATH length - from an IETF RFC publication process in
context of providing operational guidance, probably shouldn’t be “limit the
path length to avoid vendor bugs”.

Instead, the guidance perhaps should be “please report and fix bugs”,
right? :-)

Kind regards,

Job

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 16 Sep 2019 21:54:36 +0200

On 16 Sep 2019, at 20:37, Job Snijders <[email-masked]> wrote:

> Limiting the AS_PATH length - from an IETF RFC publication process in context of providing operational guidance, probably shouldn’t be “limit the path length to avoid vendor bugs”. 

> Instead, the guidance perhaps should be “please report and fix bugs”, right? :-)

In theory, yes. In practice, how does that work? Suppose I prepend 250 times, and a few hops away, routers start to explode because they hit 255 or 256 ASes in the path? How do these people debug the issue? Once they know the problem, how do they install a temporary fix as they wait for a real fix?

Also, it's unlikely that any given bug will ever be completely eradicated from the internet even if fixed software is available. The real time global nature of BGP routing makes all of this much more complex.

And what if I make it 675 ASes instead and watch sparks fly as a few hops away routers hit the 4096-byte BGP message size?

Or I make it 700 ASes with only a 16-bit AS path or a truncated 32-bit AS path, so the first 32-bit router that tries to create a 700-hop 32-bit AS path exceeds 4096 bytes?

Now all of these problems can easily be avoided by a 120 or so AS limit. Currently, the longest path that I see at Routeviews is 45, with 40 prepends. (That same network also advertises about a /13 worth of space as 2200 individual /24s for good measure.) 120 won't be hitting any overflow bugs, but it's still rather excessive at three times the longest non-prepended path. Also, 40 prepends is not going to give you anything that 39 prepends, or 30 or 20, for that matter, won't accomplish.

I've seen max path length limits of 40 or even as low as 20 mentioned.

But before we decide _where_ to draw the line, we first have to decide if we think this particular line should be drawn at all. It looks to me like AS path length filtering is sufficiently common that trying to come up with a coordinated limit as a best practice would be worthwhile.

(BTW, as far as I'm concerned we deprecate the use of AS_SETs and multiple AS_PATHs in order to remove some unnecessary complexity from BGP.)

---

From: Claudio Jeker <[email-masked]>
Date: Mon, 16 Sep 2019 22:36:59 +0200

On Mon, Sep 16, 2019 at 09:54:36PM +0200, Iljitsch van Beijnum wrote:
> In theory, yes. In practice, how does that work? Suppose I prepend 250
> times, and a few hops away, routers start to explode because they hit
> 255 or 256 ASes in the path? How do these people debug the issue? Once
> they know the problem, how do they install a temporary fix as they wait
> for a real fix?
> 
> Also, it's unlikely that any given bug will ever be completely
> eradicated from the internet even if fixed software is available. The
> real time global nature of BGP routing makes all of this much more
> complex.

As one person writing BGP routing software I need to say that these
issues are on us to fix and people should complain loudly about them.
Writing a test with e.g. exabgp to exactly check this condition is not
hard and can be added to any regress test. So having such a bug in
production software is not acceptable.
Also not updating your infrastructure systems is not acceptable either
especially if you are providing transit to other customers that pay you.
 
> And what if I make it 675 ASes instead and watch sparks fly as a few
> hops away routers hit the 4096-byte BGP message size?
> 
> Or I make it 700 ASes with only a 16-bit AS path or a truncated 32-bit
> AS path, so the first 32-bit router that tries to create a 700-hop
> 32-bit AS path exceeds 4096 bytes?

You are applying a band-aid to a broken bone. There is many more ways you
can push an UPDATE to over 4096 bytes. Using AS path alone is probably the
least successful. There are many more transitive attributes that you can
use to bloat an update. So whatever limit you come up with will not
protect you from tripping over the message size limit.

It would be great if there is a standards document properly describing
what to do in such a case because this is one of the underspecified corner
cases in the current spec.
 
> Now all of these problems can easily be avoided by a 120 or so AS limit.
> Currently, the longest path that I see at Routeviews is 45, with 40
> prepends. (That same network also advertises about a /13 worth of space
> as 2200 individual /24s for good measure.) 120 won't be hitting any
> overflow bugs, but it's still rather excessive at three times the
> longest non-prepended path. Also, 40 prepends is not going to give you
> anything that 39 prepends, or 30 or 20, for that matter, won't
> accomplish.
> 
> But before we decide _where_ to draw the line, we first have to decide
> if we think this particular line should be drawn at all. It looks to me
> like AS path length filtering is sufficiently common that trying to come
> up with a coordinated limit as a best practice would be worthwhile.
> 
> (BTW, as far as I'm concerned we deprecate the use of AS_SETs and
> multiple AS_PATHs in order to remove some unnecessary complexity from
> BGP.)

It would be great if all the 2-byte AS complexity could be slowly removed.
Allow systems to no longer accept sessions without the 4-byte AS
capability and remove all the code to generate and merge AS4_PATH and
friends.

-- 
:wq Claudio

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 16 Sep 2019 23:43:06 +0300

Iljitsch van Beijnum wrote on 16/09/2019 22:54:
> In theory, yes. In practice, how does that work? Suppose I prepend
> 250 times, and a few hops away, routers start to explode because they
> hit 255 or 256 ASes in the path? How do these people debug the issue?
> Once they know the problem, how do they install a temporary fix as
> they wait for a real fix?
why do you assume someone else will care if you lose connectivity 
because you've done something obviously pointless?

Small prepends can influence traffic engineering.  Large prepends won't 
make any more difference.  Gigantic prepends cause operators to roll 
their eyes, if they bother to look for this sort of thing - which they 
rarely do.

If people want to shoot themselves in the foot, then they should be 
given plenty of leeway to do so.  There's no reason to draw a line at 
the ietf for this sort of thing.

Nick

---

From: Joe Provo <[email-masked]>
Date: Mon, 16 Sep 2019 16:54:40 -0400

On Mon, Sep 16, 2019 at 11:43:06PM +0300, Nick Hilliard wrote:
[snip]
> https://www.ietf.org/mailman/listinfo/grow

+100

I've had to filter dump prepends back in the day when 50+
would clobber a Certain vendor's platform. If some operations 
fora wish to publish BCPs regarding ridiculous prepends, that's 
a different kettle of fish. 

Cheers,

Joe

-- 
Posted from my personal account - see X-Disclaimer header.
Joe Provo / Gweep / Earthling

---

From: David Farmer <[email-masked]>
Date: Mon, 16 Sep 2019 18:35:56 -0500

On Mon, Sep 16, 2019 at 1:25 PM Jakob Heitz (jheitz) <[email-masked]>
wrote:
...

> I don't know what kinds of bugs could result from long AS-PATHs in Netflow

In the case I was referring to the netwflow collector also takes a BGP feed
from the router, to get BGP communities, full AS-Path, etc... It is was a
bug triggered buy long AS-Paths in this receive only BGP implementation
that caused us issues.

---

From: David Farmer <[email-masked]>
Date: Mon, 16 Sep 2019 18:46:32 -0500

...

> With regard to memory use, it's not a real problem for implementations and
> is generally noise in the real world vs. the total number of paths in your
> router.
>

Yes, for a good implementation with sufficient memory, excessive path
length is probably noise. However, for a router already on the edge,
excessively long path can easily push it over the edge, along with a myriad
of other things of course.

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Tue, 17 Sep 2019 14:42:55 +0000

+1

From: GROW <[email-masked]> On Behalf Of Job Snijders
Sent: Monday, September 16, 2019 2:37 PM
To: Jared Mauch <[email-masked]>
Cc: Iljitsch van Beijnum <iljitsch=[email-masked]>; [email-masked]
Subject: Re: [GROW] Limiting AS path length?

Kind regards,

Job

---

From: john heasley <[email-masked]>
Date: Tue, 17 Sep 2019 23:36:18 +0000

> cases in the current spec.

isnt this rfc7606

---

From: Claudio Jeker <[email-masked]>
Date: Wed, 18 Sep 2019 08:36:24 +0200

> > cases in the current spec.

I only see input related checks in rfc7606 and there is nothing describing
what to do when the total lenght of an UPDATE exceeds 4096 bytes. At least
I did not spot it.

draft-ietf-idr-bgp-extended-messages-36 has a paragraph covering how to
handle oversized messages:

   A BGP speaker that has advertised the BGP Extended Message capability
   to its peers, may receive an UPDATE from one of its peers that
   produces an ongoing announcement that is larger than 4,096 octets.
   When propagating that UPDATE onward to a neighbor which has not
   advertised the BGP Extended Message capability, the speaker SHOULD
   try to reduce the outgoing message size by removing attributes
   eligible under the "attribute discard" approach of [RFC7606].  If the
   message is still too big, then it must not be sent to the neighbor
   ([RFC4271], Section 9.2).  Additionally, if the NLRI was previously
   advertised to that peer, it must be withdrawn from service
   ([RFC4271], Section 9.1.3).

Hopefully implementations will pick up this approach independent from the BGP
Extended Message Capability.

-- 
:wq Claudio

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Thu, 19 Sep 2019 17:44:13 +0200

FYI:

RouteViews data reveals the following:

- the longest AS path currently seen is 45 hops, 5 "real" hops and 40 prepends
- the longest unprepended path is 14 hops
- 99.95% of prefixes have an AS path are shorter than 20 hops
- if we remove prepends, 99.95% of prefixes have an AS path are shorter than 10 hops

In theory, if two networks both have 45 hops, that could make the total 90 hops.

Also, if 14 hops all apply a completely reasonable three prepends, that's 56 hops total.

In practice, if networks make sure to not add additional prepends for paths that are already very long, and make sure they're not unusually far from the DFZ core of the internet, a 50 hop limit should be safe today.

---

From: William Herrin <[email-masked]>
Date: Thu, 19 Sep 2019 08:56:50 -0700

> Instead, the guidance perhaps should be “please report and fix bugs”,

Hi Job,

"Don't poke the bear," is generally good advice. When we have sufficient
reason to believe a particular behavior is error-prone, we should
discourage that behavior. Regardless of whose "fault" it is.

AND we should encourage folks to report and fix bugs.

Regards,
Bill Herrin

-- 
William Herrin
[email-masked]
https://bill.herrin.us/