Subject: Re: [Idr] I-D Action: draft-ietf-idr-deprecate-as-set-confed-set-02.txt

From: "Sriram, Kotikalapudi (Fed)" <[email-masked]>
Date: Sun, 10 Nov 2019 05:01:01 +0000

Alejandro,
Sander,

[Alejandro]
>>As a comment, we did a small research and 
>>these are our findings (hope the table shows ok):
>>.....snip.....

[Sander]
>Those numbers are very manageable. If we gather a couple 
>of volunteers we could personally contact each AS.

Thanks for the measurement numbers and your comments.

We had earlier reported similar numbers (totals across all RIR regions)
with many more details:
https://mailarchive.ietf.org/arch/msg/idr/tE8o0HZeLLUEulMkszVBlxqILeU

The link to our detailed data is here:
https://www.nist.gov/sites/default/files/documents/2019/10/23/detailed-as_set-analysis.txt 

Our numbers and yours are consistent, except that we also showed that
globally there only 21 unique routes (prefixes) with AS_SET that seem meaningful!
The rest of the routes with AS_SET (about 450) seem meaningless; details at the link above.

Even the 21 will likely do fine if they did not use AS_SET. They can deaggregate and 
provide the full AS path or they can aggregate and put in AGGREGATOR and
ATOMIC_AGGREGATE.  If you look at this part of our data:

*** When there is AGGREGATOR without AS_SET ***
	# Unique prefixes (with or without AS_SET) : 826535
	# Unique prefixes without AS_SET but with AGGREGATOR: 75698
	% Unique prefixes without AS_SET but with AGGREGATOR: 9.158%
	# Unique prefixes with ATOMIC_AGGREGATE: 47258 
	# Unique prefixes with AGGREGATOR and ATOMIC_AGGREGATE: 44971
	# Unique prefixes with AGGREGATOR and without ATOMIC_AGGREGATE: 31769
           (the last three lines added newly)

it seems clear that a very large number of routes/ASes aggregate without AS_SET
and very likely many of them face similar scenario as those with AS_SET
and yet they do not seem to encounter problems (despite not using AS_SET).
I think the main reason is that even if the aggregate is received
via another upstream provider and gets installed in as AS that contributed to the aggregate,
the AS really never uses that aggregate to route data because it has received 
the other components (more specifics) of the aggregate from the provider who aggregated.
As Jeff has pointed out, if a more specific gets cut off (due to outage), then looping
in the data plane can occur. 
This would typically last only for a short period until recovery happens.
Now a days, the recovery times are fast. And possibly looping of data for a brief period
is not much worse than the data not reaching its destination for that period.  

Sriram

---

From: Donatas Abraitis <[email-masked]>
Date: Sun, 10 Nov 2019 10:15:41 +0200

Hi folks!

we have a standing PR in FRRouting regarding this change too:
https://github.com/FRRouting/frr/pull/5305

On Sun, Nov 10, 2019 at 7:01 AM Sriram, Kotikalapudi (Fed)
<kotikalapudi.sriram=[email-masked]> wrote:
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

-- 
Donatas

---

From: Alejandro Acosta <[email-masked]>
Date: Sun, 10 Nov 2019 19:03:34 -0400

> https://www.nist.gov/sites/default/files/documents/2019/10/23/detailed-as_set-analysis.txt 

Thanks for this, happy to see our results are similar.

We also published something, a little bit longer that what I sent to the
list few days ago.

https://labs.lacnic.net/Una_mirada_al_uso_de_BGP_AS_SET_en_la_DFZ/

Regards,

P.S. Sorry, in Spanish :-{}

---

From: Jared Mauch <[email-masked]>
Date: Sun, 10 Nov 2019 19:09:21 -0500

> On Nov 10, 2019, at 12:01 AM, Sriram, Kotikalapudi (Fed) <kotikalapudi.sriram=[email-masked]> wrote:
> 
> Alejandro,
> Sander,
> 
> [Alejandro]

We seem to have some issues as $dayJob is listed in here.. I’m researching this now.

I was surprised to find some of our routes announced with this, eg:

23.52.172.0/23 2914 20940 20940 {21342}
2600:140b:f000::/48 2914 20940 20940 {21342}
2a02:26f0:e00::/48 2914 6762 20940 20940 {21342}

- Jared

---

From: Jeffrey Haas <[email-masked]>
Date: Sun, 10 Nov 2019 23:02:22 -0500

Alejandro,

> https://labs.lacnic.net/Una_mirada_al_uso_de_BGP_AS_SET_en_la_DFZ/

Sorry for not looking at this in more detail (my spanish is weak) but I find
a very puzzling item in your results: IPv6

In IPv6, we shouldn't have any motivation to do aggregation in the
traditional sense.  Have you done any analysis of the prefixes involved in
that IPv6 aggregation?

Your data also supports my observations about the oddities for private ASes
and thus likely bugs in some implementations vs. aggregation and private
ASes and remove-private.

-- Jeff

---

From: "Hannachi, Lilia (IntlAssoc)" <[email-masked]>
Date: Tue, 12 Nov 2019 21:41:03 +0000

Jeff,

> traditional sense.  Have you done any analysis of the prefixes involved in

Earlier we (NIST) had shared the analysis for IPv4.
Listed below is the AS_SET analysis for IPv6.
Please let me know if you have any further questions.

Lilia
--------------

AS_SET ANALYSIS (OREGON6 2019-10-03:00)
The Routeviews OREGON6 collector peers with (23) ASes

*********************************************************************
Summary (for IPv6)
*********************************************************************

        Total # Updates : 155659
        # Updates with AS_SET : 38
        Percentage of routes with AS_SET : 0.024%
        # Total # ASes (globally) : 18027
        # ASes that create updates with AS_SET : 27
        Percentage of ASes that create updates with AS_SET : 0.15%

        # Routes with AS_SET (after eliminating AS path redundancy): 38
        Explanation: These are routes with unique {prefix, AS_SET, 1st AS after AS_SET, AGGREGATOR} combinations.

Out of the 38 routes with AS_SETs:
        *** Routes with AS_SET that seem meaningless or malformed ***
        # Routes with only one AS in AS_SET : 26
        # Routes with Reserved ASN in AS_SET : 0
        # Routes with common AS in the AS_SEQUENCE and AS_SET : 4
        # Routes with repeated ASes in the AS_SET : 0
        # Routes that are /48 prefix (aggregate) announcements : 6
        Total # Routes that seem meaningless or malformed : 27

      Total # Routes that seem meaningful : 11

        Distribution of # unique ASes in the AS_SET : 1:26,2:7,3:1,4:3,8:1

        # Routes with AS_SET where AGGREGATOR does not match the right most AS in AS_SEQUENCE : 1

        # Routes with unique {prefix, AS_SET, AGGREGATOR} : 38
        # Routes with unique {prefix, AS_SET} : 38

*** When there is AGGREGATOR without AS_SET ***
        # Unique prefixes (with or without AS_SET) : 78625
        # Unique prefixes without AS_SET but with AGGREGATOR: 8651
        % Unique prefixes without AS_SET but with AGGREGATOR: 11.003%
        # Unique prefixes with ATOMIC_AGGREGATE: 3823
        # Unique prefixes with AGGREGATOR and ATOMIC_AGGREGATE: 3650
        # Unique prefixes with AGGREGATOR and without ATOMIC_AGGREGATE: 5039
        # Unique prefixes with AS_SET and ATOMIC_AGGREGATE: 2

*********************************************************************
List of ASes (AGGREGATOR) Ranked according to # Routes with AS_SET
*********************************************************************

        AS: 14080    # Routes with AS_SET: 4    Organization: Telmex Colombia S.A., CO
        AS: 271    # Routes with AS_SET: 3    Organization: BCNET-AS - BCnet, CA
        AS: 20940    # Routes with AS_SET: 3    Organization: AKAMAI-ASN1, US
        AS: 9443    # Routes with AS_SET: 2    Organization: INTERNETPRIMUS-AS-AP Primus Telecommunications, AU
        AS: 24158    # Routes with AS_SET: 2    Organization: TAIWANMOBILE-AS Taiwan Mobile Co., Ltd., TW
        AS: 45147    # Routes with AS_SET: 2    Organization: NAPINFO-TRANSIT-AS-AP PT. NAP Info Lintas Nusa, ID
        AS: 52320    # Routes with AS_SET: 2    Organization: GlobeNet Cabos Submarinos Colombia, S.A.S., CO
        AS: 22    # Routes with AS_SET: 1    Organization: DNIC-AS-00022 - Navy Network Information Center (NNIC), US
        AS: 8376    # Routes with AS_SET: 1    Organization: Jordan Data Communications, JO
        AS: 8697    # Routes with AS_SET: 1    Organization: JTC-AS8697 Jordan Telecom Group ( JTG ), JO
        AS: 47377    # Routes with AS_SET: 1    Organization: ORANGE_BELGIUM_SA KPN Belgium Business NV has been acquired by Mobistar, BE
        AS: 65401    # Routes with AS_SET: 1    Organization: -Private Use AS-, ZZ
        AS: 7160    # Routes with AS_SET: 1    Organization: NETDYNAMICS - Oracle Corporation, US
        AS: 6509    # Routes with AS_SET: 1    Organization: CANARIE-NTN - Canarie Inc, CA
        AS: 32440    # Routes with AS_SET: 1    Organization: LONI - Louisiana Board of Regents/Louisiana Optical Network Initiative (LONI), US
        AS: 6364    # Routes with AS_SET: 1    Organization: ATLANTIC-NET-1 - Atlantic.net, Inc., US
        AS: 27919    # Routes with AS_SET: 1    Organization: IXP ECUADOR, EC
        AS: 8664    # Routes with AS_SET: 1    Organization: ICM-PUB public network services, PL
        AS: 41943    # Routes with AS_SET: 1    Organization: VMI-KISTA, SE
        AS: 41426    # Routes with AS_SET: 1    Organization: SAUDIDATASTC-AS, SA
        AS: 203507    # Routes with AS_SET: 1    Organization: AVIRADE Kaplaneiweg 1, DE
        AS: 44814    # Routes with AS_SET: 1    Organization: BTEL-BG-AS, BG
        AS: 35548    # Routes with AS_SET: 1    Organization: SMARTTERRA-AS, DE
        AS: 62917    # Routes with AS_SET: 1    Organization: GIGABITCO - Gigabit Communications, US
        AS: 34056    # Routes with AS_SET: 1    Organization: KIEVNET KievNet ISP AS, UA
        AS: 9924    # Routes with AS_SET: 1    Organization: TFN-TW Taiwan Fixed Network, Telco and Network Service Provider., TW
        AS: 26281    # Routes with AS_SET: 1    Organization: RIML-CORP-AS-1 - BlackBerry Limited, CA

*********************************************************************
List of Routes with AS_SET that seem meaningless or malformed:
*********************************************************************

        2607:ffc0:1000::/36     209 6939 18705 26281 {13340}
        2607:f170:50::/44       57463 6939 2828 18672 6364 {6364}
        2001:6a0::/32   7018 6939 8664 {8890}
        2607:f8f0:210::/44      57866 1299 6939 271 271 271 271 {26415}
        2800:484::/32   57463 6939 3257 14080 {10620}
        2a01:4a40::/32  37100 44814 {60974}
        2402:7500:600::/40      57463 6939 3491 9924 24158 {131614}
        2800:480::/32   57463 6939 3257 14080 {10620}
        2a03:f600::/32  57463 6939 1299 3301 25146 41943 {34236}
        2a00:1688::/32  57866 9002 34056 {201948}
        2600:1408:9800::/48     209 6461 20940 20940 {16625}
        2a02:26f0:e00::/48      57463 6939 6762 20940 20940 {21342}
        2403:4800:9000::/36     57463 6939 4826 9443 {16625}
        2600:140b:f000::/48     7018 1299 4637 4637 4637 4637 20940 20940 {21342}
        2606:b400:8814::/48     57463 6939 3356 7160 {7160}
        2403:4800::/48  57463 6939 4826 9443 {38285}
        2a00:18d8::/32  57463 6939 5511 8697 {328126}
        2803:a8a0::/32  7660 2516 6762 27947 27919 {27919,266802}
        2800:485::/32   57463 6939 3257 14080 {10620}
        2402:4640::/33  22652 4788 38044 {23736}
        2402:7500:800::/40      57463 6939 3491 9924 24158 {131614}
        2a01:9700::/32  57463 6939 8697 8376 {8376}
        2a06:e380::/29  57866 1299 203507 203507 {206970}
        2a00:6480::/32  57463 6939 35548 {29018}
        2001:480:10::/48        57866 2914 668 22 {7224}
        2a02:2578::/32  3277 6939 1299 39386 41426 {51375}
        2a01:cc00::/32  57463 6939 5511 47377 {12493}

*********************************************************************
List of Routes with AS_SET that seem meaningful:
*********************************************************************

        2404:c000::/32  1403 6939 7473 45147 {16625,17727,55818,64313}
        2001:410::/32   3277 6939 6509 {271,7860,8111}
        2800:b40::/32   57463 52320 {6939,18678,262220,262589}
        2402:7500::/32  57463 6939 2914 9924 9924 9924 {24158,131614}
        2404:c000::/33  7018 1299 45147 {16625,55818}
        2607:f8f0::/32  57866 1299 577 271 271 271 271 {4476,16462,26415,393249}
        2604:82c0::/32  57463 6939 33132 62917 {32451,54426}
        2800:482::/32   40191 3257 14080 {10620,64250}
        2803:6700::/32  31019 43531 52320 {2,3257,6762,6939,18747,19429,262220,263210}
        2607:f390::/32  57463 6939 3356 32440 {2055,17244}
        2607:f8f0:600::/40      57866 1299 577 271 271 271 271 {4476,393249}

*********************************************************************
List of All Routes with AS_SET by AGGREGATOR
*********************************************************************

***********************
AGGREGATOR= 14080
***********************
        2800:482::/32   40191 3257 14080 {10620,64250}  AGGREGATOR=14080
        2800:484::/32   57463 6939 3257 14080 {10620}   AGGREGATOR=14080
        2800:480::/32   57463 6939 3257 14080 {10620}   AGGREGATOR=14080
        2800:485::/32   57463 6939 3257 14080 {10620}   AGGREGATOR=14080
***********************
AGGREGATOR= 271
***********************
        2607:f8f0:210::/44      57866 1299 6939 271 271 271 271 {26415} AGGREGATOR=271
        2607:f8f0:600::/40      57866 1299 577 271 271 271 271 {4476,393249}    AGGREGATOR=271
        2607:f8f0::/32  57866 1299 577 271 271 271 271 {4476,16462,26415,393249}        AGGREGATOR=271
***********************
AGGREGATOR= 20940
***********************
        2600:1408:9800::/48     209 6461 20940 20940 {16625}    AGGREGATOR=20940
        2600:140b:f000::/48     7018 1299 4637 4637 4637 4637 20940 20940 {21342}       AGGREGATOR=20940
        2a02:26f0:e00::/48      57463 6939 6762 20940 20940 {21342}     AGGREGATOR=20940
***********************
AGGREGATOR= 9443
***********************
        2403:4800::/48  57463 6939 4826 9443 {38285}    AGGREGATOR=9443
        2403:4800:9000::/36     57463 6939 4826 9443 {16625}    AGGREGATOR=9443
***********************
AGGREGATOR= 24158
***********************
        2402:7500:600::/40      57463 6939 3491 9924 24158 {131614}     AGGREGATOR=24158
        2402:7500:800::/40      57463 6939 3491 9924 24158 {131614}     AGGREGATOR=24158
***********************
AGGREGATOR= 45147
***********************
        2404:c000::/33  7018 1299 45147 {16625,55818}   AGGREGATOR=45147
        2404:c000::/32  1403 6939 7473 45147 {16625,17727,55818,64313}  AGGREGATOR=45147
***********************
AGGREGATOR= 52320
***********************
        2803:6700::/32  31019 43531 52320 {2,3257,6762,6939,18747,19429,262220,263210}  AGGREGATOR=52320
        2800:b40::/32   57463 52320 {6939,18678,262220,262589}  AGGREGATOR=52320
***********************
AGGREGATOR= 22
***********************
        2001:480:10::/48        57866 2914 668 22 {7224}        AGGREGATOR=22
***********************
AGGREGATOR= 8376
***********************
        2a01:9700::/32  57463 6939 8697 8376 {8376}     AGGREGATOR=8376
***********************
AGGREGATOR= 8697
***********************
        2a00:18d8::/32  57463 6939 5511 8697 {328126}   AGGREGATOR=8697
***********************
AGGREGATOR= 47377
***********************
        2a01:cc00::/32  57463 6939 5511 47377 {12493}   AGGREGATOR=47377
***********************
AGGREGATOR= 65401
***********************
        2402:4640::/33  22652 4788 38044 {23736}        AGGREGATOR=65401
***********************
AGGREGATOR= 7160
***********************
        2606:b400:8814::/48     57463 6939 3356 7160 {7160}     AGGREGATOR=7160
***********************
AGGREGATOR= 6509
***********************
        2001:410::/32   3277 6939 6509 {271,7860,8111}  AGGREGATOR=6509
***********************
AGGREGATOR= 32440
***********************
        2607:f390::/32  57463 6939 3356 32440 {2055,17244}      AGGREGATOR=32440
***********************
AGGREGATOR= 6364
***********************
        2607:f170:50::/44       57463 6939 2828 18672 6364 {6364}       AGGREGATOR=6364
***********************
AGGREGATOR= 27919
***********************
        2803:a8a0::/32  7660 2516 6762 27947 27919 {27919,266802}       AGGREGATOR=27919
***********************
AGGREGATOR= 8664
***********************
        2001:6a0::/32   7018 6939 8664 {8890}   AGGREGATOR=8664
***********************
AGGREGATOR= 41943
***********************
        2a03:f600::/32  57463 6939 1299 3301 25146 41943 {34236}        AGGREGATOR=41943
***********************
AGGREGATOR= 41426
***********************
        2a02:2578::/32  3277 6939 1299 39386 41426 {51375}      AGGREGATOR=41426
***********************
AGGREGATOR= 203507
***********************
        2a06:e380::/29  57866 1299 203507 203507 {206970}       AGGREGATOR=203507
***********************
AGGREGATOR= 44814
***********************
        2a01:4a40::/32  37100 44814 {60974}     AGGREGATOR=44814
***********************
AGGREGATOR= 35548
***********************
        2a00:6480::/32  57463 6939 35548 {29018}        AGGREGATOR=35548
***********************
AGGREGATOR= 62917
***********************
        2604:82c0::/32  57463 6939 33132 62917 {32451,54426}    AGGREGATOR=62917
***********************
AGGREGATOR= 34056
***********************
        2a00:1688::/32  57866 9002 34056 {201948}       AGGREGATOR=34056
***********************
AGGREGATOR= 9924
***********************
        2402:7500::/32  57463 6939 2914 9924 9924 9924 {24158,131614}   AGGREGATOR=9924
***********************
AGGREGATOR= 26281
***********************
        2607:ffc0:1000::/36     209 6939 18705 26281 {13340}    AGGREGATOR=26281