Subject: Re: [GROW] Choice of Large vs. Extended Community for Route Leaks Solution

From: Jeffrey Haas <[email-masked]>
Date: Thu, 1 Apr 2021 09:33:10 -0400

[Note, commenting as an individual contributor...]

On Wed, Mar 31, 2021 at 03:10:08PM -0700, Brian Dickson wrote:
> On Wed, Mar 31, 2021 at 7:57 AM Sriram, Kotikalapudi (Fed) <
> [email-masked]> wrote:
> > We (authors of the WKLC draft) can continue working on creating an IANA
> > WKLC registry for the future but I think the route leaks solution draft can
> > switch to using Transitive Extended Community. Especially, if that could
> > help expedite the route leaks draft and its deployment? I would like to
> > seek advice regarding that (I'm cc'ing GROW also here).
> 
> 
> Sorry to argue in public, but I disagree very strongly on the second part
> here.
> 
> We would like to continue proceeding with use of a LC range for
> implementation, using a single (or small number) of Global Administrator
> values.
> 
> I think we should request that a small block of GAs surrounding the initial
> assignments be tentatively marked something like Reserved for IANA
> assignment.
> That is different from actually establishing a registry or assigning them
> specifically for WKLCs, but would be compatible with subsequent WKLC work.

The issue being faced here for WKLC is a few related items:
- There needs to be some number of global admin (AS) reserved for generic use.
- Since this was not done at the time the feature was shipped, there is no
  support in anyone's code to treat subsequently allocated ranges as
  "special".

The two larger criticisms of the WKLC draft to date is that it wanted a LOT
of the AS space to be used, and that its transitivity functionality would
not fit well into an incremental deployment of the feature.  If the
transitivity requirements are removed and the space is narrowed, that at
least moves it closer to existing community operational practices.  E.g. a
single AS allocated for the purpose that leaves two unstructured fields, or
a small number of ASes.

This then would require implementations that want to treat the new
well-known fields as "special" to have code to do so.  So, we're already
looking at some level of new code.

Absent that new code, what we have is generic policy, along with the
implementation's normal propagation behavior - which is behind a knob for
some implementations.

> The move to using LC values was precipitated by the observation that the
> path for getting attributes deployed would be very long, and that operators
> (actual network operators) are looking for a solution which can be deployed
> *now*.

If the state can fit into an extended community, that similarly can be done
NOW.  A significant portion of the extended community space is available on
first-come, first-serve basis.

https://www.iana.org/assignments/bgp-extended-communities/bgp-extended-communities.xhtml

But it doesn't change the above issues.

> Nothing has changed in this regard; the WKLC draft is IMHO still the right
> path, and only the size of the initial allocation is problematic.
> 
> Having 1-4 GA values (from the 32-bit range of potential values) is not
> burdensome IMNSHO, and is a lot less of a concern than the 1/64 (or 1/16)
> of the range of 32-bit ASNs originally requested/suggested.
> 
> If we can all agree that 1-4 GA values assigned for this is acceptable, I
> suggest a revised version of the draft and assessment of consensus on the
> revised draft for adoption and last call.
> 
> LC is the ONLY viable path, given the nebulous state of implementation and
> use for EC/WC or attributes. LC is already deployed, and assigning a few
> GAs by IDR is the only roadblock to the draft in GROW getting approved.

I question your conclusion.

The desire is to have an attribute that is used for route leak mitigation
purposes.

This thread illustrates that it's very easy for such a control community in
any flavor (regular, extended, large) to be stripped because the service
provider hasn't proactively acted to permit it.

Does this really solve the problem?

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 1 Apr 2021 11:11:43 -0400

On Wed, Mar 31, 2021 at 04:26:06PM -0700, Randy Bush wrote:
> > The point Jakob follows up with in this thread strongly suggests
> > communities are not a viable mechanism.
> 
> communities are rarely a viable mechanism.  too malleable, forgable,
> ...  once again, see our paper.

Randy adds elsewhere:

Florian Streibelt, Franziska Lichtblau, Robert Beverly, Anja Feldmann,
Cristel Pelsser, Georgios Smaragdakis, Randy Bush; BGP Communities: Even
more Worms in the Routing Can; ACM Internet Measurement Conference 2018
https://archive.psg.com/181101.imc-communities.pdf

-- Jeff

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Thu, 1 Apr 2021 20:13:33 +0000

Brian,

>We would like to continue proceeding with use of a LC range for implementation, using a single (or small number) of Global Administrator values.

I should have clarified. I am not opposed to staying on course with a WKLC based solution. I only thought transitivity readily came with Transitive Extended Community. Your proposal about one or a very small number of WKLC GA values seems fine and it is consistent with Sue’s advice.  Please see my next post (in this thread) which is about transitivity measurements of EC and LC that should be helpful in this discussion.

Sriram 
________________________________________
From: Brian Dickson <[email-masked]>
Sent: Wednesday, March 31, 2021 6:10 PM
To: Sriram, Kotikalapudi (Fed)
Cc: Susan Hares; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Choice of Large vs. Extended Community for Route Leaks Solution

On Wed, Mar 31, 2021 at 7:57 AM Sriram, Kotikalapudi (Fed) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Sue,

Thanks for the detailed thoughts you have shared on the IDR list about the WKLC draft and route leaks solution draft (while also responding to Brian’s post).

At one point in the past, the route leaks solution needed 8 bytes of user data space to accommodate two ASNs but then there was a design change (more than a year ago) and the current draft [1] requires only 4 bytes of user data space (one ASN). So, it seems possible to use a Transitive Extended Community instead of WKLC.

We (authors of the WKLC draft) can continue working on creating an IANA WKLC registry for the future but I think the route leaks solution draft can switch to using Transitive Extended Community. Especially, if that could help expedite the route leaks draft and its deployment? I would like to seek advice regarding that (I'm cc'ing GROW also here).

[Brian] Sorry to argue in public, but I disagree very strongly on the second part here.

[Brian] We would like to continue proceeding with use of a LC range for implementation, using a single (or small number) of Global Administrator values.

[Brian] I think we should request that a small block of GAs surrounding the initial assignments be tentatively marked something like Reserved for IANA assignment.
That is different from actually establishing a registry or assigning them specifically for WKLCs, but would be compatible with subsequent WKLC work.

[Brian] The move to using LC values was precipitated by the observation that the path for getting attributes deployed would be very long, and that operators (actual network operators) are looking for a solution which can be deployed *now*.

[Brian] Nothing has changed in this regard; the WKLC draft is IMHO still the right path, and only the size of the initial allocation is problematic.

[Brian] Having 1-4 GA values (from the 32-bit range of potential values) is not burdensome IMNSHO, and is a lot less of a concern than the 1/64 (or 1/16) of the range of 32-bit ASNs originally requested/suggested.

[Brian] If we can all agree that 1-4 GA values assigned for this is acceptable, I suggest a revised version of the draft and assessment of consensus on the revised draft for adoption and last call.

[Brian] LC is the ONLY viable path, given the nebulous state of implementation and use for EC/WC or attributes. LC is already deployed, and assigning a few GAs by IDR is the only roadblock to the draft in GROW getting approved.

Brian

---

From: Brian Dickson <[email-masked]>
Date: Thu, 1 Apr 2021 13:55:21 -0700

>   support in anyone's code to treat subsequently allocated ranges as

This is correct. However, I am making the claim that lack of special
handling in anyone's code is NOT a blocker to use by operators.
Specifically, the proposed route-leak-detection-mitigation LC range
(assuming a very small number of fixed GA values) would be able
to be used by operators for arbitrary LC matching, which would be
sufficient to implement the detection/mitigation mechanisms.

While code would facilitate automatically doing the mitigation stuff, it is
not the only way mitigation can be achieved.

Having GA values assigned for this is necessary. Having code is not
necessary, but would be desirable (and should be a long-term goal).

The code side of things is true regardless of mechanism used, BTW -- it
could be LC, EC, WC, new attribute, or anything like that.

> looking at some level of new code.

It is not necessary that this be done in code.

In fact, that is the precise motivation for using LC: no code is strictly
required.

Existing code for matching LC values is sufficient for operators to both
mark prefixes, and match LC values for implementing policy.

E.g. In vendor-agnostic pseudo-code:
IF(PEER_GROUP == MY_CUSTOMER && LC.GA == TBD1 && LC.VALUE1 == TBD2) THEN
DROP # Customer is sending us a marked prefix -- marked prefixes can only
be sent provider->customer
IF(PEER_GROUP == MY_PEER || PEER_GROUP == MY_TRANSIT) THEN ADD LC{LC.GA =
TBD1, LC.VALUE1 = TBD2, LC.VALUE2 = MY_ASN}

> implementation's normal propagation behavior - which is behind a knob for

This is correct.

> Does this really solve the problem?

It solves the problem on an expedited basis.

It is not the exclusive solution, which would be to augment this initial
solution with additional mechanisms (e.g. codify WKLCs generally and this
WKLC specifically; codify the behavior using a new optional transitive
attribute or possibly mandatory transitive attribute).

What is also true about this (proposed mechanism + WKLC) is that there is
an incentive for providers (i.e. ISPs) to ask or tell their customers to
turn the knob on.

Having customers turn the knob on provides a convenient way to detect and
block route leaks.
Observe that any route leak from a customer, if not blocked by other means
or any means (e.g. if the ISP does not filter based on prefix or as_path),
harms the ISP directly.
Turning the knob on is comparatively easy, as it is a session configuration
element, rather than a policy mechanism.
ISPs who want to confirm that their customers have enabled the knob, can
provide further instructions to set an ISP-specific LC on routes the
customer sends, which can be used to validate that LCs are being sent.
(Ideally that should be a "test and remove" step, or a "confirm and apply
LC-stripping on the specific LC" so as to avoid causing more garbage being
sent to the DFZ.

Additionally, this range of WKLCs should never be seen within the DFZ. They
are only set when sending prefixes to customers. Any observed instances of
the GA can and should be addressed via non-technical (out of band, human to
human) processes, to get the leak stopped.

This is one of the rare cases where the problem and (however suboptimal)
solution are not asymmetric. The benefit is seen by the party doing the
thing rather than having to have someone else do the thing to get the
benefit.

Brian

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Thu, 1 Apr 2021 21:06:03 +0000

There may be a knob that AS operators have for permitting transitivity, but we need to look at measurements to understand whether or not operators actually allow transitivity to EC and LC.

NIST BGP measurements (thanks to my colleague Lilia Hannachi) were shared on the GROW list in May 2020:
https://mailarchive.ietf.org/arch/msg/grow/JPD1-hhSvVXIZbUlNQ_1hmzD6IA/

A portion is copied below. The AS path length (# unique ASes) distributions for BGP updates with Communities (Regular, Large, and Extended) are shown here. It is evident that both LC and EC propagate multiple AS hops. Mass stripping of LC or EC at the first hop is not evident.  The peak happens at AS path length 4 or 5 and that is good. That is the behavior that is helpful for route leak solution. The solution can still function even if some ASes strip. We can do some more detailed studies if needed. 
 
*********************************************************************
RIPE-RIS: Community ANALYSIS (Collector : rrc03 From 2020-04-30 00:00 To 2020-04-30 00:55)
*********************************************************************
# Updates = 1075583 (Total)
# (Regular) COMMUNITY = 859239 (79.89%)
AS path length distribution =    1: 170 (0.02%)    2: 44803 (5.21%)    3: 141072 (16.42%)    4: 276271 (32.15%)    5: 238325 (27.74%)    6: 114158 (13.29%)    7: 31365 (3.65%)    8: 9018 (1.05%)    9: 2690 (0.31%)    10: 811 (0.09%)    11: 358 (0.04%)    12: 169 (0.02%)    13: 22 (0%)    14: 7 (0%)

# LARGE_COMMUNITY = 152818 (14.21%)
AS path length distribution =    2: 5655 (3.7%)    3: 17205 (11.26%)    4: 54372 (35.58%)    5: 45492 (29.77%)    6: 22065 (14.44%)    7: 6422 (4.2%)    8: 1068 (0.7%)    9: 397 (0.26%)    10: 71 (0.05%)    11: 35 (0.02%)    12: 26 (0.02%)    13: 6 (0%)    14: 4 (0%)

# EXTENDED COMMUNITIES = 44606 (4.15%)
AS path length distribution =    2: 2269 (5.09%)    3: 7435 (16.67%)    4: 17657 (39.58%)    5: 11600 (26.01%)    6: 3967 (8.89%)    7: 1221 (2.74%)    8: 371 (0.83%)    9: 57 (0.13%)    10: 19 (0.04%)    11: 8 (0.02%)    12: 1 (0%)    13: 1 (0%)
*********************************************************************

Sriram

---

From: Brian Dickson <[email-masked]>
Date: Thu, 1 Apr 2021 14:59:55 -0700

On Thu, Apr 1, 2021 at 2:06 PM Sriram, Kotikalapudi (Fed) <
[email-masked]> wrote:

> There may be a knob that AS operators have for permitting transitivity,
> but we need to look at measurements to understand whether or not operators
> actually allow transitivity to EC and LC.
>
> A portion is copied below. The AS path length (# unique ASes)
> distributions for BGP updates with Communities (Regular, Large, and
> Extended) are shown here. It is evident that both LC and EC propagate
> multiple AS hops. Mass stripping of LC or EC at the first hop is not
> evident.  The peak happens at AS path length 4 or 5 and that is good. That
> is the behavior that is helpful for route leak solution. The solution can
> still function even if some ASes strip. We can do some more detailed
> studies if needed.
>
> AS path length distribution =    1: 170 (0.02%)    2: 44803 (5.21%)    3:
> 141072 (16.42%)    4: 276271 (32.15%)    5: 238325 (27.74%)    6: 114158
> (13.29%)    7: 31365 (3.65%)    8: 9018 (1.05%)    9: 2690 (0.31%)    10:
> 811 (0.09%)    11: 358 (0.04%)    12: 169 (0.02%)    13: 22 (0%)    14: 7
> (0%)
>
> AS path length distribution =    2: 5655 (3.7%)    3: 17205 (11.26%)    4:
> 54372 (35.58%)    5: 45492 (29.77%)    6: 22065 (14.44%)    7: 6422 (4.2%)
>   8: 1068 (0.7%)    9: 397 (0.26%)    10: 71 (0.05%)    11: 35 (0.02%)
> 12: 26 (0.02%)    13: 6 (0%)    14: 4 (0%)
>
> AS path length distribution =    2: 2269 (5.09%)    3: 7435 (16.67%)    4:
> 17657 (39.58%)    5: 11600 (26.01%)    6: 3967 (8.89%)    7: 1221 (2.74%)
>   8: 371 (0.83%)    9: 57 (0.13%)    10: 19 (0.04%)    11: 8 (0.02%)    12:
> 1 (0%)    13: 1 (0%)

One main issue will be the use of LC vs WC vs regular communities, among
the largest networks (e.g. networks nominally at the tier-1 designation).
Going from e.g. a wikipedia page with list of ASNs, that would be all or
most of these ASNs:
7018, 1299, 1239, 3320, 6453, 6461, 3356, 3549, 2914, 3257, 3491, 701,
6762, 3491.

Can you get a top-level summary of LCs with GA values from that list, and
whatever the corresponding thing would be for EC or WC?
E.g. How many unique LC values are there that match 3320:* ? Maybe a table
with ASN vs common LC count would help illustrate this.

If the majority (or vast majority) are already using LCs, and many fewer
are using WC/EC, this definitely should tilt the argument towards LCs.

Any of these networks which is using LCs extensively will very likely to be
unwilling to use anything other than a WKLC (and this use of LCs was to a
significant degree the result of requests from operators that use LCs to do
it with LCs.)

Brian

---

From: Job Snijders <[email-masked]>
Date: Fri, 2 Apr 2021 00:34:10 +0200

On Wed, Mar 31, 2021 at 06:09:42PM -0400, Jeffrey Haas wrote:
> > There is already a separate draft in IDR that has passed WGLC, and it uses a new transitive BGP Path Attribute 'Only to Customer (OTC)':
> > https://tools.ietf.org/html/draft-ietf-idr-bgp-open-policy-15
> > We view that as a longer-term solution, while the EC/LC-based approach is meant to be deployed quickly.  
> 
> I've been caught napping on the changes in open policy and will have to go
> read this.

If we're talking about 'deploying quickly', something that can be done
on any of the BGP implementations in operation today (capable of
matching a single RFC 1997 BGP community), is to use the NOPEER community.

I really think NOPEER Community [RFC 3765] is suitable as synonymous to
"Only to Customer". It would change the draft from "route leak detection"
into "route leak prevention", but as bgp-open-policy-XX require new code
anyway, this new code might as well use on an existing attribute to
foster incremental deployment.

If future BGP-OPEN compliant BGP speakers end up tagging routes with
NOPEER when received from a "role: peer" BGP neighbor, and automatically
block outbound route announcements towards "role: transit" and "role: peer"
BGP neigbhor sessions, the effect of "Only to Customer" (role: customer)
is achieved.

A confusing aspect in this dialogue:
The meaning of the word 'peer' RFC 3765 is different than the meaning of
the word 'peer' in draft-ietf-idr-bgp-open-policy-15, the latter
document appears to use the word as a description of the positions a
network can have in the 'gao and rexford' model.

When reading draft-ietf-idr-bgp-open-policy-15 keep this mapping in mind:
  'role: peer'      == RFC 3765 'peer'
  'role: provider'  == RFC 3765 'peer'
  'role: rs'        == RFC 3765 'peer'

An implementation which is evaluating whether to export a given best
path to a neighbor which has role ('peer', 'provider', 'rs') as
established through draft-ietf-idr-bgp-open-policy; is RECOMMENDED to
not export routes which carry the NOPEER community.

Using NOPEER (instead of EC or LC) can be implemented on both bgp-open
capable implementations, but also through configuration in existing
deployments.

Reading RFC 3765 it seems appropriate to use this well-known community
'attribute' for this purpose. RFC 3765 is intended as a general purpose
route propagation restriction community. Bgp-open capable
implementations which use NOPEER in this way, are be compatible with
existing deployments of NOPEER.

Kind regards,

Job

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Fri, 2 Apr 2021 06:38:12 +0000

When the collector sees a route with AS-PATH length 5 with a community on it, that does not imply that the community traveled through 5 AS hops. The community could have been added at any of the ASes in the path. Where does the data show that any communities transited any AS boundaries?

Regards,
Jakob.

> On Apr 1, 2021, at 2:06 PM, Sriram, Kotikalapudi (Fed) <[email-masked]> wrote:
>

---

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Fri, 2 Apr 2021 16:51:34 +0000

Jakob,

Thanks. I was expecting this question. We will work on a recursive path analysis to answer it. Brian's suggestion also will be considered. The data crunching may take a week or so.

My hunch was that when the AS path length is 5 or 6 or more, the ASes at the far end (most recently added) are not likely the savvy ones to be doing EC/LC but more likely the earlier ASes added the EC/LC. We'll see what the more detailed data analysis reveals.   

Sriram 

________________________________________
From: Jakob Heitz (jheitz) <[email-masked]>
Sent: Friday, April 2, 2021 2:38 AM
To: Sriram, Kotikalapudi (Fed)
Cc: Jeffrey Haas; Susan Hares; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Brian Dickson
Subject: Re: Choice of Large vs. Extended Community for Route Leaks Solution

Regards,
Jakob.

> https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailarchive.ietf.org%2Farch%2Fmsg%2Fgrow%2FJPD1-hhSvVXIZbUlNQ_1hmzD6IA%2F&amp;data=04%7C01%7Ckotikalapudi.sriram%40nist.gov%7C7e0ddfc991ff4aa54f9308d8f5a1e8e7%7C2ab5d82fd8fa4797a93e054655c61dec%7C1%7C0%7C637529423059320205%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=WPgxMhjSfAx%2FueDd487sVRqVWCAYY%2FMIF51gzbf%2Bm7Q%3D&amp;reserved=0
>