Subject: Re: [GROW] I-D Action: draft-ietf-grow-as-path-prepending-00.txt

From: Jeffrey Haas <[email-masked]>
Date: Tue, 27 Oct 2020 16:48:41 -0400

I'm rather far behind in my mail, but hopefully current on this thread.  A
few comments on the adopted draft:

The thing I find most worthy in this document is the discussion about how
excessive prepending increases your vulnerability to route hijacking.
This issue will exist whenever you have a well connected peer that is
willing to forge their AS_PATH.  It's probably worth the general nod to
bgpsec because it implies how you don't get out of that problem without
locking down the contents of the as path.  The well connected peer issue is
inferred in 3.1.

While I agree with Jakob's comments that lead to the updated section 3.4
text, trying to normalize "avoid other people's bugs" is awkward at best.
3.5 similarly fits in that category.

Section 4 is a nice start to describing the solution space, but is
problematic on a few levels:
- Scoping communities from your directly attached ISP may not help you in
  some circumstances.  
- More specifics is a hard yank on traffic - if you're even permitted to
  originate them.  If you're not lucky, your prefix is already so long from
  being a small service provider that you can't make it longer.
- BGP origin code as a high order tie-breaker is a technique that works
  well, but only at places where as-paths are of equivalent length.  For
  many situations resulting in long paths, I'm not sure that helps.

Section 5's guidance of "no more than 5 prepends" is probably fine these
days, but that is largely a matter of the current diameter of the Internet.

Section 7 - Security Considerations - could likely use a paragraph
discussing how long prepending may make you more vulernable to route
hijacking.

----

Feedback on the contents done, a personal anecdote:

Just prior to 2000, I used to work for a small regional ISP.  That ISP was
multi-homed to a tier one and two tier two providers, each of which were
trying to fight their way into tier-one space.  Our motivations for our
multihoming were one part resiliency, one part cheapest bandwidth for the
dollar, and one part the fact that connectivity at the time to specific
sites meant that we had a strong reason to use a specific provider.

It was our experience at that time that it was necessary to use prepends in
the range of 3-5 for portions of our address space in order to try to
provide appropriate inbound load balancing.  The number was that high
because at the points we were trying to balance traffic for, the denseness
of the Internet topology meant that the unpadded routes were often getting
closely tied for path length.

Prepending wasn't a complete panacea.  We regularly experienced traffic
spikes after traffic rebalancing until we found a prepend length that was
stable enough.  I recognize at this point that we were suffering from the
effects of BGP Wedgies.  (See RFC 4264.)

I've been out of the operator game for a very long time.  We've generally
seen the diameter of the Internet decrease since I was in the business.  The
problems I was solving at the time still exist, but aren't quite as severe
in the well connected bits of the Internet.  I would not be shocked if some
of the far corners of the Internet still need this for similar motivations
than my employer had twenty years ago.

----

I will offer one final bit of feedback: Some customers simply filter very
long AS_PATHs.  The issues noted in this draft are effectively self
correcting in the presence of aggressive filter policies.

On Tue, Sep 08, 2020 at 05:16:30PM +0000, Michael McBride wrote:
> Hello wg,
> 
> We have addressed all comments so far. Thank you. We still need to expand the prepend alternatives section. Please give it another look and let's see what else should be included.
> 
> mike
> 
> -----Original Message-----
> From: GROW <[email-masked]> On Behalf Of [email-masked]
> Sent: Tuesday, September 08, 2020 9:58 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [GROW] I-D Action: draft-ietf-grow-as-path-prepending-00.txt
> 
> 
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
> This draft is a work item of the Global Routing Operations WG of the IETF.
> 
>         Title           : AS Path Prepending
>         Authors         : Mike McBride
>                           Doug Madory
>                           Jeff Tantsura
>                           Robert Raszuk
>                           Hongwei Li
> 	Filename        : draft-ietf-grow-as-path-prepending-00.txt
> 	Pages           : 10
> 	Date            : 2020-09-08

-- Jeff

---

From: Michael McBride <[email-masked]>
Date: Fri, 30 Oct 2020 19:01:19 +0000

Thank you Jeffrey. We just update the draft with your comments. Perhaps we can discuss a bit more on the upcoming grow wg call.

mike

-----Original Message-----
From: Jeffrey Haas <[email-masked]> 
Sent: Tuesday, October 27, 2020 1:49 PM
To: Michael McBride <[email-masked]>
Cc: [email-masked]
Subject: Re: [GROW] I-D Action: draft-ietf-grow-as-path-prepending-00.txt

I'm rather far behind in my mail, but hopefully current on this thread.  A few comments on the adopted draft:

The thing I find most worthy in this document is the discussion about how excessive prepending increases your vulnerability to route hijacking.
This issue will exist whenever you have a well connected peer that is willing to forge their AS_PATH.  It's probably worth the general nod to bgpsec because it implies how you don't get out of that problem without locking down the contents of the as path.  The well connected peer issue is inferred in 3.1.

While I agree with Jakob's comments that lead to the updated section 3.4 text, trying to normalize "avoid other people's bugs" is awkward at best.
3.5 similarly fits in that category.

Section 4 is a nice start to describing the solution space, but is problematic on a few levels:

Section 5's guidance of "no more than 5 prepends" is probably fine these days, but that is largely a matter of the current diameter of the Internet.

Section 7 - Security Considerations - could likely use a paragraph discussing how long prepending may make you more vulernable to route hijacking.

----

Just prior to 2000, I used to work for a small regional ISP.  That ISP was multi-homed to a tier one and two tier two providers, each of which were trying to fight their way into tier-one space.  Our motivations for our multihoming were one part resiliency, one part cheapest bandwidth for the dollar, and one part the fact that connectivity at the time to specific sites meant that we had a strong reason to use a specific provider.

It was our experience at that time that it was necessary to use prepends in the range of 3-5 for portions of our address space in order to try to provide appropriate inbound load balancing.  The number was that high because at the points we were trying to balance traffic for, the denseness of the Internet topology meant that the unpadded routes were often getting closely tied for path length.

Prepending wasn't a complete panacea.  We regularly experienced traffic spikes after traffic rebalancing until we found a prepend length that was stable enough.  I recognize at this point that we were suffering from the effects of BGP Wedgies.  (See RFC 4264.)

I've been out of the operator game for a very long time.  We've generally seen the diameter of the Internet decrease since I was in the business.  The problems I was solving at the time still exist, but aren't quite as severe in the well connected bits of the Internet.  I would not be shocked if some of the far corners of the Internet still need this for similar motivations than my employer had twenty years ago.

----

I will offer one final bit of feedback: Some customers simply filter very long AS_PATHs.  The issues noted in this draft are effectively self correcting in the presence of aggressive filter policies.

-- Jeff

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Sun, 1 Nov 2020 05:56:59 +0000

Michael,

I support the document.

I would not single out the incident of August in an RFC
or indeed any other incident. BCPs are not created out
of single incidents, but from a pattern emerging out of
multiple incidents. It is enough to say that a certain
scenario can occur and has occurred.

Pre-provision of AS-prepend and then removing a prepend
to the working ISP to prefer it over the failed ISP
is not a solution.

Suppose during normal operation you prepend to both of
your transit providers. Then one of your providers
can remove your prepends to gain an advantage over
the other provider.

Suppose the AS that wants to avoid the blackhole is not
the origin AS of the route. If that AS were to pre-provision
AS-prepends, then it would lose to other providers that don't.

I notice that section 3.4 is a copy of one of my emails to
the list. I did not intend for that to be placed into an RFC,
but as some background for how memory is actually used.
For the RFC, I would rephrase it as so:

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Long AS-paths cause an increase in memory usage by BGP
speakers. The memory usage is not so much a concern in the
control plane BGP implementations, but more so when AS-paths
are included in Netflow messages. Netflow is processed in
the forwarding plane, where memory is more expensive than
in the control plane.

A concern about an AS-path longer than 255 is the extra complexity
required to process it, because it needs to be encoded in
more than one AS_SEQUENCE in the AS_PATH BGP path attribute.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Having contributed actual text to the draft, I'm happy to
be co-author.

Regards,
Jakob.

-----Original Message-----
From: GROW <[email-masked]> On Behalf Of Michael McBride
Sent: Friday, October 30, 2020 12:01 PM
To: Jeffrey Haas <[email-masked]>
Cc: [email-masked]
Subject: Re: [GROW] I-D Action: draft-ietf-grow-as-path-prepending-00.txt

mike

-----Original Message-----
From: Jeffrey Haas <[email-masked]> 
Sent: Tuesday, October 27, 2020 1:49 PM
To: Michael McBride <[email-masked]>
Cc: [email-masked]
Subject: Re: [GROW] I-D Action: draft-ietf-grow-as-path-prepending-00.txt

----

----

-- Jeff

_______________________________________________
GROW mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/grow