Subject: Re: [rrg] BGP scaling limit

From: Enke Chen <[email-masked]>
Date: Sat, 31 Jan 2009 21:17:36 -0800

Hi, Danny:

Just saw this email as I am not on the IRTF mailing list:

Excerpt from the email:

------------------------------------------------------------------------------

I don't agree with this.  Offer me an alternative for 10 or
more interconnections between the largest networks today?
Offer me an alternative for how those paths and the current
inter-domain routing protocol amplify (not minimize) the number
of paths.
-----------------------------------------------------------

Here is what I (Enke) am thinking:

I do agree that the current operational model of doing shortest-exit
(and thus resulting in path multiplications) has scalability concerns,
especially with the increase in the number of peering locations between
providers.

So here is an alternative (operationally):

      accept MEDs from each other and perform longest-exit.

The MEDs could be set by the sender based on the location (community)
so that at most 2 or 3 peering locations would advertise the same
MED.

For example, for a particular prefix originated from New York:

        Peering Location         MED
        -----------------------------
         New York / DC           10

                  Chicago/ Denver                       20
                  SFO / LA                                 30
                  .....

The location/community based route-map should be uniform (i.e., 
identical) at the peering
locations.

This approach should reduce the number paths significantly, and yet 
maintain redundancy
and convergence properties in a network.

Any comments?

-- Enke

    * /From/: Danny McPherson <danny at arbor.net
      <mailto:[email-masked]>>
    * /To/: <tony.li at tony.li <mailto:[email-masked]>>
    * /Cc/: 'RRG' <rrg at irtf.org <mailto:[email-masked]>>
    * /Date/: Mon, 12 Jan 2009 14:01:24 -0700

------------------------------------------------------------------------

On Jan 12, 2009, at 10:34 AM, Tony Li wrote:

    As you point out, this is more to do with the specific protocol than
    it is to do with the overall routing architecture. As such, this is
    why it's

    really much more of the domain of IDR than RRG.

Well, so long as the folks designing the next generation
understand the problem and mitigate the constraints.  I'm
not totally sure the former is the case, which has effects
on the latter.

    BTW, AFAIK, route reflection _reduces_ the number of paths as
    compared to

    full-mesh IBGP.  Am I missing something?

Depends on the number of RRs, where the best path
is, network architecture, etc..  The growth curves
I've seen (wFrom rrg-bounces at irtf.org  Mon Jan 12 13:01:51 2009

On Jan 12, 2009, at 10:34 AM, Tony Li wrote:

    As you point out, this is more to do with the specific protocol than
    it is to do with the overall routing architecture. As such, this is
    why it's

    really much more of the domain of IDR than RRG.

Well, so long as the folks designing the next generation
understand the problem and mitigate the constraints.  I'm
not totally sure the former is the case, which has effects
on the latter.

    BTW, AFAIK, route reflection _reduces_ the number of paths as
    compared to

    full-mesh IBGP.  Am I missing something?

Depends on the number of RRs, where the best path
is, network architecture, etc..  The growth curves
I've seen (which I hhich I hope to be able to share soon)
illustrate that the number of iBGP paths is growing
at anywhere from 6-14x the rate of the DFZ, and
the reason is because of external factors, protocol
design issues, and implementations.

    Paths that are introduced for the sake of traffic engineering are a
    well

    understood and self-inflicted problem.

Really?  So a multi-homed AS is TE?  So networks that interconnect
in 10 places result in 10 paths for each prefix and that's self-
inflicted?  What's the alternative to avoiding this self-inflicted
pain?

     If those folks that introduced the
    extra paths took equivalent interest in limiting their dispersion and

    cooperated in filtering unnecessary long prefixes, this could
    readily be

    addressed.

No, I'm not talking about just extraneous or covered prefixes,
that's only a tiny part of the problem I'm referring to.
Denseness of external interconnection, unique attributes,
BGP as defined today AND TE have effects on these, I'd say the
latter the least.

     In short, this is not an issue with the fundamental

    architecture, it's about how the architecture is being used. Thus,
    this is

    largely an operational issue.

I don't agree with this.  Offer me an alternative for 10 or
more interconnections between the largest networks today?
Offer me an alternative for how those paths and the current
inter-domain routing protocol amplify (not minimize) the number
of paths.

    Further, if we did something to fundamentally remove the capability to

    traffic engineer in the future, I strongly suspect that that would
    be an

    architectural issue, most likely fatal.

Again, TE is only a tiny part of the problem I'm referring
to here - fully dependent on ones definition of TE, of course.

-danny

_______________________________________________
rrg mailing list
rrg at irtf.org
http://www.irtf.org/mailman/listinfo/rrg

ope to be able to share soon)
illustrate that the number of iBGP paths is growing
at anywhere from 6-14x the rate of the DFZ, and
the reason is because of external factors, protocol
design issues, and implementations.

    Paths that are introduced for the sake of traffic engineering are a
    well

    understood and self-inflicted problem.

Really?  So a multi-homed AS is TE?  So networks that interconnect
in 10 places result in 10 paths for each prefix and that's self-
inflicted?  What's the alternative to avoiding this self-inflicted
pain?

     If those folks that introduced the
    extra paths took equivalent interest in limiting their dispersion and

    cooperated in filtering unnecessary long prefixes, this could
    readily be

    addressed.

No, I'm not talking about just extraneous or covered prefixes,
that's only a tiny part of the problem I'm referring to.
Denseness of external interconnection, unique attributes,
BGP as defined today AND TE have effects on these, I'd say the
latter the least.

     In short, this is not an issue with the fundamental

    architecture, it's about how the architecture is being used. Thus,
    this is

    largely an operational issue.

I don't agree with this.  Offer me an alternative for 10 or
more interconnections between the largest networks today?
Offer me an alternative for how those paths and the current
inter-domain routing protocol amplify (not minimize) the number
of paths.

    Further, if we did something to fundamentally remove the capability to

    traffic engineer in the future, I strongly suspect that that would
    be an

    architectural issue, most likely fatal.

Again, TE is only a tiny part of the problem I'm referring
to here - fully dependent on ones definition of TE, of course.

-danny

_______________________________________________
rrg mailing list
rrg at irtf.org
http://www.irtf.org/mailman/listinfo/rrg

_______________________________________________
rrg mailing list
[email-masked]
http://www.irtf.org/mailman/listinfo/rrg

---

From: Enke Chen <[email-masked]>
Date: Sat, 31 Jan 2009 21:29:27 -0800

BTW, the MED setting could also be based on the normalized metric.  For 
example:

           IGP metric                     MED
          --------------------------------
             0-20                               10
            21-40                              20
            41-100                            30
            ...

-- Enke

---

From: "Tony Li" <[email-masked]>
Date: Thu, 5 Feb 2009 21:45:17 -0800

Gentlemen,

This discussion is strictly out of scope for RRG.  This is about BGP
internals, not about routing and addressing architectures.  Please take it
to IDR, where it would be most appropriate.

Regards,
Tony

|-----Original Message-----
|From: Enke Chen [mailto:[email-masked]] 
|Sent: Thursday, February 05, 2009 8:39 PM
|To: Danny McPherson
|Cc: [email-masked]; [email-masked]; Enke Chen
|Subject: Re: [rrg] BGP scaling limit
|
|Danny McPherson wrote:
|>
|> On Jan 31, 2009, at 10:17 PM, Enke Chen wrote:
|>>
|>> So here is an alternative (operationally):
|>>
|>>     accept MEDs from each other and perform longest-exit.
|>>
|>> The MEDs could be set by the sender based on the location 
|(community)
|>> so that at most 2 or 3 peering locations would advertise the same
|>> MED.
|>>
|>> For example, for a particular prefix originated from New York:
|>>
|>>       Peering Location         MED
|>>       -----------------------------
|>>        New York / DC           10
|>>
|>>                 Chicago/ Denver                       20
|>>                 SFO / LA                                 30
|>>                 .....
|>>
|>> The location/community based route-map should be uniform (i.e., 
|>> identical) at the peering
|>> locations.
|>>
|>> This approach should reduce the number paths significantly, and yet 
|>> maintain redundancy
|>> and convergence properties in a network.
|>
|> The increasing number of interconnections will still be
|> necessary, and the number of internally amplified routes
|> will grow accordingly.  With each BGP speaker or Route
|> Reflector set/cluster MEDs would possibly decrease the
|> number of routes contained in the core (as every router
|> wouldn't default to advertising the local paths as best)
|> but it only marginally improves the problem, which is still
|> highly topologically dependent and constrained by the
|> current BGP internal scaling model.
|
|Marginally???  The approach reduces the number of paths to be 
|a constant
|(chosen by a provider) from being proportional to the number of peering
|locations.
|
|>
|> Furthermore, MEDs trigger lots of other badness (e.g.,
|> oscillation), require inter-provider coordination and
|> normalization (e.g., with different IGPs and metric space,
|> internal metric allocation models and multipliers vary
|> considerably, etc..), can make peer gaming much simpler,
|> and are usually broken when considering routes originated
|> in multiple locations from an adjacent peer AS.
|
|The actual metrics/MED do not require coordination between providers.
|What is needed is for the providers to accept the metric/MED from
|each other.
|
|MED induced oscillation is a concern, but it is rare.
| 
|>
|> I.e., MEDs should turn closest-exit (hot potato) routing
|> into best-exit routing (cold potato/?), but in my experience
|> most often result in mashed potato routing (e.g., S 7.1.1
|> of RFC 4277).
|
|It is all about tradeoffs.  As you might remember,  InternetMCI used to
|accept MEDs from other providers and it worked just fine
|
|-- Enke
|.
|
|

---

From: Danny McPherson <[email-masked]>
Date: Thu, 5 Feb 2009 20:57:23 -0700

> For example, for a particular prefix originated from New York:

The increasing number of interconnections will still be
necessary, and the number of internally amplified routes
will grow accordingly.  With each BGP speaker or Route
Reflector set/cluster MEDs would possibly decrease the
number of routes contained in the core (as every router
wouldn't default to advertising the local paths as best)
but it only marginally improves the problem, which is still
highly topologically dependent and constrained by the
current BGP internal scaling model.

Furthermore, MEDs trigger lots of other badness (e.g.,
oscillation), require inter-provider coordination and
normalization (e.g., with different IGPs and metric space,
internal metric allocation models and multipliers vary
considerably, etc..), can make peer gaming much simpler,
and are usually broken when considering routes originated
in multiple locations from an adjacent peer AS.

I.e., MEDs should turn closest-exit (hot potato) routing
into best-exit routing (cold potato/?), but in my experience
most often result in mashed potato routing (e.g., S 7.1.1
of RFC 4277).

-danny

---

From: Enke Chen <[email-masked]>
Date: Thu, 05 Feb 2009 20:39:03 -0800

> current BGP internal scaling model.

Marginally???  The approach reduces the number of paths to be a constant
(chosen by a provider) from being proportional to the number of peering
locations.

> in multiple locations from an adjacent peer AS.

The actual metrics/MED do not require coordination between providers.
What is needed is for the providers to accept the metric/MED from
each other.

MED induced oscillation is a concern, but it is rare.
 
> most often result in mashed potato routing (e.g., S 7.1.1

It is all about tradeoffs.  As you might remember,  InternetMCI used to
accept MEDs from other providers and it worked just fine

-- Enke
.