Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

From: "[email-masked]" <[email-masked]>
Date: Wed, 9 Feb 2022 18:13:03 +0800

This draft is trying to solve problems existing in networks today. For the solution, I think it is difficult for the receiving router to decide which PE should be blocked when its VRF routes reach a threshould. Why not block any PE sending routes to the overflowing VRF? 

Best Regards,
Zhenqiang Li

[email-masked]
 
From: Susan Hares
Date: 2022-02-04 20:39
To: [email-masked]
Subject: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)
Resending the adoption call – with the appropriate header
 
From: Idr [mailto:[email-masked]] On Behalf Of Susan Hares
Sent: Friday, February 4, 2022 7:33 AM
To: [email-masked]
Subject: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt : (2/4/2022 to 2/18/2022)
 
This begins a 2 week WG adoption call  for
draft-wang-idr-vpn-prefix-orf-00.txt
https://datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf.
 
In your discussion on adoption, please consider the following questions:
 
1) Does this draft describe problems existing in networks today?
 
2) Does the solution describe address these problems?
The solution has both a general solution and BGP encodings.
 
3) Would the solution be useful in networks today?
 
Thank you, Susan Hares

---

From: Aijun Wang <[email-masked]>
Date: Wed, 9 Feb 2022 18:31:34 +0800

Hi, Zhenqiang:
Thanks for your questions!
Actually, https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf#section-4.1.1 describes the general process for identifying the offending VPN routes and its associated RD, RT and original PE. There maybe other knobs on the router to accomplish this aim, depending on the implementation on the local router.

“  On PE1, each VRF has a set threshold, we assume it is 80% of Maximum
   Prefix of VRF.  When the number of VPN1 VRF routing entries reaches
   the threshold, PE1 will start monitoring the RD carried by the
   received VPN routing entries.  Once the number of VPN routing entries
   exceed the prefix limit, PE1 will calculate the RD, RT and its source
   PE received the most times during this period, the result is
   RD31(RT1) from PE3, which is associated with RT1.  Then, PE1 will
   locally discards the VPN routes carry RD31(RT1) which come from PE3
   in VRF1.”

The reason that we identify the specific offending PE, rather than blocking the VPN routes from all PEs, is that we want to control the communication influence as small as possible. This is the aim that we want to achieve as fine control. 

Aijun Wang
China Telecom

> 在 2022年2月9日，18:12，[email-masked] 写道：
> 
> 
> _______________________________________________
> Idr mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idr

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 9 Feb 2022 12:37:45 +0100

Hi Aijun,

> offending VPN routes and its associated RD, RT and original PE.

The entire notion of "offending VPN routes" is a sick concept.

First it leads to partially broken VPNs which is an extremely bad thing no
serious SP would endorse.

Second you are accepting those routes in the first place so failing your
accepting screening should be no excuse to then partially filter those.

Last all along the problem was described as one of the weak PEs not capable
of handling some VPN routes. This is not the same as "offending routes". It
is a problem that you have configured such PE to handle more then it is
capable of handling in the first place so this is really your design or
provisioning problem - not a VPN route distribution one.

Still I maintain my position and do not support this to be adopted as an
IDR WG document.

Thx,
R.

On Wed, Feb 9, 2022 at 11:31 AM Aijun Wang <[email-masked]>
wrote:

> Actually,
> https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf#section-4.1.1 describes
> the general process for identifying the offending VPN routes and its
> associated RD, RT and original PE. There maybe other knobs on the router to
> accomplish this aim, depending on the implementation on the local router.
>
> The reason that we identify the specific offending PE, rather than
> blocking the VPN routes from all PEs, is that we want to control the
> communication influence as small as possible. This is the aim that we want
> to achieve as fine control.
>
> Aijun Wang
> China Telecom
>
> This draft is trying to solve problems existing in networks today. For the
> solution, I think it is difficult for the receiving router to decide which
> PE should be blocked when its VRF routes reach a threshould. Why not block
> any PE sending routes to the overflowing VRF?
>
> ------------------------------
> Best Regards,
> Zhenqiang Li
>
> [email-masked]
>
>
> *From:* Susan Hares <[email-masked]>
> *Date:* 2022-02-04 20:39
> *To:* [email-masked]
> *Subject:* [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call
> (2/4/2022 to 2/18/2022)
>
> *From:* Idr [mailto:[email-masked]] *On Behalf Of *Susan Hares
> *Sent:* Friday, February 4, 2022 7:33 AM
> *To:* [email-masked]
> *Subject:* [Idr] draft-wang-idr-vpn-prefix-orf-00.txt : (2/4/2022 to
> 2/18/2022)
>
>
>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 9 Feb 2022 11:52:40 +0000

Comments In-Line..

Thanks,
              Jim Uttaro

From: Idr <[email-masked]> On Behalf Of [email-masked]
Sent: Wednesday, February 9, 2022 5:13 AM
To: Hares <[email-masked]>; [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

This draft is trying to solve problems existing in networks today.
[Jim U>] Have you canvassed operators? I would be interested in the "problems" that have been identified in networks today. Pls provide data and not conjecture.
For the solution, I think it is difficult for the receiving router to decide which PE should be blocked when its VRF routes reach a threshould. Why not block any PE sending routes to the overflowing VRF?

________________________________
Best Regards,
Zhenqiang Li

[email-masked]<mailto:[email-masked]>

From: Susan Hares<mailto:[email-masked]>
Date: 2022-02-04 20:39
To: [email-masked]<mailto:[email-masked]>
Subject: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)
Resending the adoption call - with the appropriate header

From: Idr [mailto:[email-masked]] On Behalf Of Susan Hares
Sent: Friday, February 4, 2022 7:33 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt : (2/4/2022 to 2/18/2022)

This begins a 2 week WG adoption call  for
draft-wang-idr-vpn-prefix-orf-00.txt
https://datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf__;!!BhdT!yPUYUuirQ1P6jY-6LfD7vJRLsQR3n8-cWzq-Id3eLmvKj3ISHGjnW-6e6po$>.

Thank you, Susan Hares

---

From: Aijun Wang <[email-masked]>
Date: Wed, 9 Feb 2022 20:07:10 +0800

Hi, Jim:
Currently, we depended only the “maximum prefixes limit” on the PE router, so when the VPN routes surpasses the threshold, the BGP sessions with the peer will be broken.
It will influence all the VRFs that shares within the BGP sessions.
Haven’t you encountered such problems in your network?
We are trying to find the better solution than the current tools.

Aijun Wang
China Telecom

> 在 2022年2月9日，19:54，UTTARO, JAMES <[email-masked]> 写道：
> 
> 
> Comments In-Line..
>  
> Thanks,
>               Jim Uttaro
>  
> [Jim U>] Have you canvassed operators? I would be interested in the “problems” that have been identified in networks today. Pls provide data and not conjecture.

---

From: Aijun Wang <[email-masked]>
Date: Wed, 9 Feb 2022 20:28:18 +0800

Hi, Robert:

Aijun Wang
China Telecom

> 在 2022年2月9日，19:38，Robert Raszuk <[email-masked]> 写道：
> 
> 
> Hi Aijun,
> 
> First it leads to partially broken VPNs which is an extremely bad thing no serious SP would endorse. 

[WAJ] These “offending VPN Routes” are not filtered silently.  As described in the document:
“When the VPN Prefix ORF mechanism is triggered, the device must send
   an alarm information to network operators.”

The main aim of this mechanism is to alleviate the influence in the offended PE as small as possible.
Once the this ORF mechanism be triggered, the operator will step in to find the reason and the corresponding solution.
If we don’t catch this up quickly, the VPN  services on this device will all be influenced.

> 
> Second you are accepting those routes in the first place so failing your accepting screening should be no excuse to then partially filter those. 
[WAJ] As explained to zhenqiang, we are trying to get the fine control effect for this mechanism. 
> 
> Last all along the problem was described as one of the weak PEs not capable of handling some VPN routes. This is not the same as "offending routes". It is a problem that you have configured such PE to handle more then it is capable of handling in the first place so this is really your design or provisioning problem - not a VPN route distribution one. 
[WAJ] The VPN Prefixes ORF mechanism is triggered independently based on each PE’s capabilities, then there is no the concept of “weak PE”. 
And the RR only filter the VPN Prefixes towards the signaled PE and will not influence other PEs that don’t send the VPN Prefixes ORF message.

> 
> Still I maintain my position and do not support this to be adopted as an IDR WG document. 
> 
> Thx,
> R.
> 
> 
> 
> 
> 
> 
> 
> 
>> On Wed, Feb 9, 2022 at 11:31 AM Aijun Wang <[email-masked]> wrote:
>> Hi, Zhenqiang:

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 9 Feb 2022 13:48:57 +0100

> [WAJ] The VPN Prefixes ORF mechanism is triggered independently based
> on each PE’s capabilities, then there is no the concept of “weak PE”.

Sure there is.

Definition:    *PE not capable of holding VPN prefixes of VPN routes it is
configured to serve is called a weak PE. *

Now maybe you could at least try to explain what does it mean that PE
capabilities do not allow to keep all routes which given PE needs ?
Is this control plane problem (not enough CPU and RAM) or a data plane
problem (forwarding memory is too small) ?

Since day one of first shipped PE the basic feature was to inbound drop
routes which do not contain at least one intersecting import RT. If you say
that this is too much of work to drop such routes on the PEs - push RTs to
RR running on x86 with RTC to only get what is needed or swap your PE
vendor.

You keep saying prefix limit will drop all VPN routes ... not really if you
configure it properly on the EBGP sessions from your customers. It will
only drop or issue warning when threshold is crossed from single CE to
single VRF of the PE accepting such routes.

I know now you are going to say - it is too much work to do that towards
all CEs. Well then do not take any routes from CEs but next hops and use
CSC. Or maybe accept that offering L3VPNs requires some OPEX, CAPEX and
know how ...

Cheers,
R,

On Wed, Feb 9, 2022 at 1:28 PM Aijun Wang <[email-masked]> wrote:

> Once the this ORF mechanism be triggered, the operator will step in to
> find the reason and the corresponding solution.
> Last all along the problem was described as one of the weak PEs not
> capable of handling some VPN routes. This is not the same as "offending
> routes". It is a problem that you have configured such PE to handle more
> then it is capable of handling in the first place so this is really your
> design or provisioning problem - not a VPN route distribution one.
>
> [WAJ] The VPN Prefixes ORF mechanism is triggered independently based on
> each PE’s capabilities, then there is no the concept of “weak PE”.
> And the RR only filter the VPN Prefixes towards the signaled PE and will
> not influence other PEs that don’t send the VPN Prefixes ORF message.
>
>
>> This draft is trying to solve problems existing in networks today. For
>> the solution, I think it is difficult for the receiving router to decide
>> which PE should be blocked when its VRF routes reach a threshould. Why not
>> block any PE sending routes to the overflowing VRF?
>>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 9 Feb 2022 13:33:29 +0000

Comments In-Line

Thanks,
              Jim Uttaro

From: Aijun Wang <[email-masked]>
Sent: Wednesday, February 9, 2022 7:07 AM
To: UTTARO, JAMES <[email-masked]>
Cc: [email-masked]; Hares <[email-masked]>; [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Hi, Jim:
Currently, we depended only the “maximum prefixes limit” on the PE router, so when the VPN routes surpasses the threshold, the BGP sessions with the peer will be broken.
[Jim U>] Not true. Additional routing state will be throttled. There is a notification setting and a threshold setting. The only time this was encountered was when a customer mistakenly re-distributed the GRT into their VPN. They were actually quite appreciative of our notification as it helped them quickly isolate the problem and fix it.
It will influence all the VRFs that shares within the BGP sessions.
[Jim U>] What does this mean? Is this specific to Option B?
Haven’t you encountered such problems in your network?
[Jim U>] No.
We are trying to find the better solution than the current tools.
[Jim U>] Not required.
Aijun Wang
China Telecom

在 2022年2月9日，19:54，UTTARO, JAMES <[email-masked]<mailto:[email-masked]>> 写道：

Comments In-Line..

Thanks,
              Jim Uttaro

From: Idr <[email-masked]<mailto:[email-masked]>> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, February 9, 2022 5:13 AM
To: Hares <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Thank you, Susan Hares
_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!BhdT!z587vpWgIAY6_KETP44Fo8e1ZASFQCIfEodjzi5G17QCO0dTApqL8DQmB0y2p3Q$>

---

From: Aijun Wang <[email-masked]>
Date: Wed, 9 Feb 2022 21:50:48 +0800

Hi, Jim:

Aijun Wang
China Telecom

> 在 2022年2月9日，21:33，UTTARO, JAMES <[email-masked]> 写道：
> 
> 
> Comments In-Line
>  
> Thanks,
>               Jim Uttaro
>  

[WAJ] There are two kinds actions(warning only or broken down) when the threshold is surpassed. 
If you take warning only, the situations will not be abated on the offended PE when compared with the VPN Prefix ORF solution.

> [Jim U>] What does this mean? Is this specific to Option B?

[WAJ] No, we are discussing the intra-AS scenario now. The BGP session between the PE and RR is shared by all the VRFs on this PE.

---

From: Aijun Wang <[email-masked]>
Date: Wed, 9 Feb 2022 21:59:26 +0800

Hi, Robert:

Aijun Wang
China Telecom

> 在 2022年2月9日，20:49，Robert Raszuk <[email-masked]> 写道：
> 
> 
> 
> Definition:    PE not capable of holding VPN prefixes of VPN routes it is configured to serve is called a weak PE. 
> 
> Now maybe you could at least try to explain what does it mean that PE capabilities do not allow to keep all routes which given PE needs ? 
> Is this control plane problem (not enough CPU and RAM) or a data plane problem (forwarding memory is too small) ? 

> 
> Since day one of first shipped PE the basic feature was to inbound drop routes which do not contain at least one intersecting import RT. If you say that this is too much of work to drop such routes on the PEs - push RTs to RR running on x86 with RTC to only get what is needed or swap your PE vendor. 

[WAJ] Jim has given one example in their network for the possible misbehavior on one PE. I think this may happened in other operator’s network. 
> 
> You keep saying prefix limit will drop all VPN routes ... not really if you configure it properly on the EBGP sessions from your customers. It will only drop or issue warning when threshold is crossed from single CE to single VRF of the PE accepting such routes.
[WAJ] There are two options when the threshold is exceeded. Please refer to my reply to Jim.

> 
> I know now you are going to say - it is too much work to do that towards all CEs. Well then do not take any routes from CEs but next hops and use CSC. Or maybe accept that offering L3VPNs requires some OPEX, CAPEX and know how ... 
[WAJ] It’s possible but the threaten still exists.

---

From: "[email-masked]" <[email-masked]>
Date: Wed, 9 Feb 2022 23:36:34 +0800

Comments in-line beginning with [LZQ]

Best Regards,
Zhenqiang Li
[email-masked]
 
From: UTTARO, JAMES
Date: 2022-02-09 21:33
To: Aijun Wang
CC: [email-masked]; Hares; [email-masked]
Subject: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)
Comments In-Line
 
Thanks,
              Jim Uttaro

Hi, Jim:
Currently, we depended only the “maximum prefixes limit” on the PE router, so when the VPN routes surpasses the threshold, the BGP sessions with the peer will be broken.
[Jim U>] Not true. Additional routing state will be throttled. There is a notification setting and a threshold setting. The only time this was encountered was when a customer mistakenly re-distributed the GRT into their VPN. They were actually quite appreciative of our notification as it helped them quickly isolate the problem and fix it.
[LZQ] Is what you said conjected? If not, I believe this is a real problem. And by the way, I am from China Mobile, an operator, I don't need to canvass other operators. 

 
Comments In-Line..
 
Thanks,
              Jim Uttaro

Best Regards,
Zhenqiang Li
 
[email-masked]
 
From: Susan Hares
Date: 2022-02-04 20:39
To: [email-masked]

 

 
 
Thank you, Susan Hares

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Wed, 9 Feb 2022 16:10:05 +0000

Comments In-Line

Thanks,
              Jim Uttaro

Here is the thread..

I understood your first statement as indicating more than just the China Mobile Network. If this work is directed towards an industry wide problem than you should canvass to see if other operators require this specification.  You may be experiencing this problem on your network due to the architecture/design of your network. Feel free to reach out to others who do not seem to be experiencing your networks problem to better understand why this draft is unnecessary..

From: [email-masked] <[email-masked]>
Sent: Wednesday, February 9, 2022 10:37 AM
To: UTTARO, JAMES <[email-masked]>; Aijun Wang <[email-masked]>
Cc: Susan Hares <[email-masked]>; [email-masked]
Subject: Re: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

________________________________
Best Regards,
Zhenqiang Li
[email-masked]<mailto:[email-masked]>

From: UTTARO, JAMES<mailto:[email-masked]>
Date: 2022-02-09 21:33
To: Aijun Wang<mailto:[email-masked]>
CC: [email-masked]<mailto:[email-masked]>; Hares<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)
Comments In-Line

Thanks,
              Jim Uttaro

From: Aijun Wang <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, February 9, 2022 7:07 AM
To: UTTARO, JAMES <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Hares <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Hi, Jim:

Comments In-Line..

Thanks,
              Jim Uttaro

Thank you, Susan Hares

---

From: "Aijun Wang" <[email-masked]>
Date: Thu, 10 Feb 2022 10:31:04 +0800

Hi, Jim:

Actually, draft https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-routes-control-analysis/ describes more possible scenarios for the potential VPN prefixes overflow.

With the trending/deployment of EVPN/SRv6 technologies, such problems will be emerged  inevitably because it is more easy to deploy the inter-AS VPN services.

Even for the intra-AS scenario, as that discussed in the current draft, it is possible for the misconfiguration, or misbehavior of PE; or the increase of VPN prefixes in one VRF crashes some PEs in the network and influences other existing VPN services on such PEs. 

VPN Prefix ORF mechanism gives us the capabilities to isolate the problem per VRF, not per BGP session, or per PE.

For standard activity, we should look forward to the trend, not only focus on the current status/experiences.

And, there are at least three operators that have widespread networks have interested in the solution, I think it will also benefit your service deployment in future. 

Best Regards

Aijun Wang

China Telecom

From: UTTARO, JAMES <[email-masked]> 
Sent: Thursday, February 10, 2022 12:10 AM
To: [email-masked]; Aijun Wang <[email-masked]>
Cc: Susan Hares <[email-masked]>; [email-masked]
Subject: RE: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Comments In-Line

Thanks,

              Jim Uttaro

Here is the thread..

 

From: [email-masked] <mailto:[email-masked]>  <[email-masked] <mailto:[email-masked]> > 
Sent: Wednesday, February 9, 2022 10:37 AM
To: UTTARO, JAMES <[email-masked] <mailto:[email-masked]> >; Aijun Wang <[email-masked] <mailto:[email-masked]> >
Cc: Susan Hares <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> 
Subject: Re: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

  _____  

Best Regards,

Zhenqiang Li

[email-masked] <mailto:[email-masked]> 

From: UTTARO, JAMES <mailto:[email-masked]> 

Date: 2022-02-09 21:33

To: Aijun Wang <mailto:[email-masked]> 

CC: [email-masked] <mailto:[email-masked]> ; Hares <mailto:[email-masked]> ; [email-masked] <mailto:[email-masked]> 

Comments In-Line

Thanks,

              Jim Uttaro

From: Aijun Wang <[email-masked] <mailto:[email-masked]> > 
Sent: Wednesday, February 9, 2022 7:07 AM
To: UTTARO, JAMES <[email-masked] <mailto:[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]> ; Hares <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> 
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Hi, Jim:

 

[Jim U>] No. 

[Jim U>] Not required.

Aijun Wang

China Telecom

在 2022年2月9日，19:54，UTTARO, JAMES <[email-masked] <mailto:[email-masked]> > 写道：

Comments In-Line..

Thanks,

              Jim Uttaro

From: Idr <[email-masked] <mailto:[email-masked]> > On Behalf Of [email-masked] <mailto:[email-masked]> 
Sent: Wednesday, February 9, 2022 5:13 AM
To: Hares <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> 
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

  _____  

Best Regards,

Zhenqiang Li

[email-masked] <mailto:[email-masked]> 

From: Susan Hares <mailto:[email-masked]> 

Date: 2022-02-04 20:39

To: [email-masked] <mailto:[email-masked]> 

From: Idr [mailto:[email-masked]] On Behalf Of Susan Hares
Sent: Friday, February 4, 2022 7:33 AM
To: [email-masked] <mailto:[email-masked]> 
Subject: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt : (2/4/2022 to 2/18/2022)

https://datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf__;!!BhdT!yPUYUuirQ1P6jY-6LfD7vJRLsQR3n8-cWzq-Id3eLmvKj3ISHGjnW-6e6po$> .

Thank you, Susan Hares

_______________________________________________
Idr mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/idr <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!BhdT!z587vpWgIAY6_KETP44Fo8e1ZASFQCIfEodjzi5G17QCO0dTApqL8DQmB0y2p3Q$>

---

From: Gyan Mishra <[email-masked]>
Date: Thu, 10 Feb 2022 00:34:47 -0500

Hi Jim

I don’t think there are many SRv6 deployments worldwide and I believe China
is one of the leaders on proliferation of IPV6 by operators  as well as now
SRv6.

Most operators around the world have large IPv4 backbones and it will take
time to migrate.

So as Aijun is stating as more operators deploy SRv6 / EVPN and have
inter-as EVPN and  with SR-TE it makes it so easy stretch EVPNs so you end
up with these cases.

The issue I believe is with  EVPN Type 2 route flood inter-as and not IP
VPN prefixes.

Aijun

Is that correct ?

Kind Regards

Gyan

On Wed, Feb 9, 2022 at 9:31 PM Aijun Wang <[email-masked]> wrote:

> Hi, Jim:
>
> Actually, draft
> https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-routes-control-analysis/
> describes more possible scenarios for the potential VPN prefixes overflow.
>
> With the trending/deployment of EVPN/SRv6 technologies, such problems will
> be emerged  inevitably because it is more easy to deploy the inter-AS VPN
> services.
>
> Even for the intra-AS scenario, as that discussed in the current draft, it
> is possible for the misconfiguration, or misbehavior of PE; or the increase
> of VPN prefixes in one VRF crashes some PEs in the network and influences
> other existing VPN services on such PEs.
>
>
>
> And, there are at least three operators that have widespread networks have
> interested in the solution, I think it will also benefit your service
> deployment in future.
>
>
>
> Best Regards
>
>
>
> Aijun Wang
>
> China Telecom
>
>
>
> *From:* UTTARO, JAMES <[email-masked]>
> *Sent:* Thursday, February 10, 2022 12:10 AM
> *To:* [email-masked]; Aijun Wang <[email-masked]>
> *Cc:* Susan Hares <[email-masked]>; [email-masked]
> *Subject:* RE: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption
> call (2/4/2022 to 2/18/2022)
>
>
>
> Comments In-Line
>
>
>
> Thanks,
>
>               Jim Uttaro
>
>
>
> Here is the thread..
>
>
>
> *[Jim U>] Have you canvassed operators? I would be interested in the
> “problems” that have been identified in networks today. Pls provide data
> and not conjecture.*
>
> *[LZQ] Is what you said conjected? If not, I believe this is a real
> problem. And by the way, I am from China Mobile, an operator, I don't need
> to canvass other operators. *
>
>
>
> I understood your first statement as indicating more than just the China
> Mobile Network. If this work is directed towards an industry wide problem
> than you should canvass to see if other operators require this
> specification.  You may be experiencing this problem on your network due to
> the architecture/design of your network. Feel free to reach out to others
> who do not seem to be experiencing your networks problem to better
> understand why this draft is unnecessary..
>
>
>
>
>
>
>
> *From:* [email-masked] <[email-masked]>
> *Sent:* Wednesday, February 9, 2022 10:37 AM
> *To:* UTTARO, JAMES <[email-masked]>; Aijun Wang <
> [email-masked]>
> *Cc:* Susan Hares <[email-masked]>; [email-masked]
> *Subject:* Re: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption
> call (2/4/2022 to 2/18/2022)
>
>
>
> *From:* UTTARO, JAMES <[email-masked]>
>
> *Date:* 2022-02-09 21:33
>
> *To:* Aijun Wang <[email-masked]>
>
> *CC:* [email-masked]; Hares <[email-masked]>; [email-masked]
>
> *Subject:* RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call
> (2/4/2022 to 2/18/2022)
>
> Comments In-Line
>
>
>
> Thanks,
>
>               Jim Uttaro
>
>
>
> *From:* Aijun Wang <[email-masked]>
> *Sent:* Wednesday, February 9, 2022 7:07 AM
> *To:* UTTARO, JAMES <[email-masked]>
> *Cc:* [email-masked]; Hares <[email-masked]>; [email-masked]
> *Subject:* Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call
> (2/4/2022 to 2/18/2022)
>
>
>
> Hi, Jim:
>
> Currently, we depended only the “maximum prefixes limit” on the PE
> router, so when the VPN routes surpasses the threshold, the BGP sessions
> with the peer will be broken.
>
> *[Jim U>] Not true. Additional routing state will be throttled. There is a
> notification setting and a threshold setting. The only time this was
> encountered was when a customer mistakenly re-distributed the GRT into
> their VPN. They were actually quite appreciative of our notification as it
> helped them quickly isolate the problem and fix it.*
>
> *[LZQ] Is what you said conjected? If not, I believe this is a real
> problem. And by the way, I am from China Mobile, an operator, I don't need
> to canvass other operators. *
>
>
>
>
>
> *[Jim U>] What does this mean? Is this specific to Option B?*
>
> *[Jim U>] No. *
> *[Jim U>] Not required.*
> *From:* Idr <[email-masked]> *On Behalf Of *
> *Sent:* Wednesday, February 9, 2022 5:13 AM
> *To:* Hares <[email-masked]>; [email-masked]
> *Subject:* Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call
> (2/4/2022 to 2/18/2022)
>
>
>
> *[Jim U>] Have you canvassed operators? I would be interested in the
> “problems” that have been identified in networks today. Pls provide data
> and not conjecture.*
>
> For the solution, I think it is difficult for the receiving router to
> decide which PE should be blocked when its VRF routes reach a threshould.
> Why not block any PE sending routes to the overflowing VRF?
>
>
> *From:* Idr [mailto:[email-masked] <[email-masked]>] *On
> Behalf Of *Susan Hares
> https://datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf__;!!BhdT!yPUYUuirQ1P6jY-6LfD7vJRLsQR3n8-cWzq-Id3eLmvKj3ISHGjnW-6e6po$>
> .
>
>
>
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!BhdT!z587vpWgIAY6_KETP44Fo8e1ZASFQCIfEodjzi5G17QCO0dTApqL8DQmB0y2p3Q$>
>
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: "Aijun Wang" <[email-masked]>
Date: Thu, 10 Feb 2022 15:18:59 +0800

Hi, Gyan:

Yes, the main aim is to assure the EVPN/SRv6 based services to be deployed within intra and inter-AS, but such mechanism can also be applied to the traditional MPLS VPN scenarios.

And, with the aid of such fine control mechanism, the operator has more confidences to adopt the new technologies.

Best Regards

Aijun Wang

China Telecom

From: Gyan Mishra <[email-masked]> 
Sent: Thursday, February 10, 2022 1:35 PM
To: Aijun Wang <[email-masked]>
Cc: Susan Hares <[email-masked]>; UTTARO, JAMES <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Hi Jim

I don’t think there are many SRv6 deployments worldwide and I believe China is one of the leaders on proliferation of IPV6 by operators  as well as now SRv6.

Most operators around the world have large IPv4 backbones and it will take time to migrate.

So as Aijun is stating as more operators deploy SRv6 / EVPN and have inter-as EVPN and  with SR-TE it makes it so easy stretch EVPNs so you end up with these cases.  

The issue I believe is with  EVPN Type 2 route flood inter-as and not IP VPN prefixes.

Aijun 

Is that correct ?

Kind Regards 

Gyan

On Wed, Feb 9, 2022 at 9:31 PM Aijun Wang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, Jim:

Best Regards

Aijun Wang

China Telecom

From: UTTARO, JAMES <[email-masked] <mailto:[email-masked]> > 
Sent: Thursday, February 10, 2022 12:10 AM
To: [email-masked] <mailto:[email-masked]> ; Aijun Wang <[email-masked] <mailto:[email-masked]> >
Cc: Susan Hares <[email-masked] <mailto:[email-masked]> >; [email-masked] <mailto:[email-masked]> 
Subject: RE: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Comments In-Line

Thanks,

              Jim Uttaro

Here is the thread..

 

  _____  

Best Regards,

Zhenqiang Li

Date: 2022-02-09 21:33

Comments In-Line

Thanks,

              Jim Uttaro

Hi, Jim:

 

[Jim U>] No. 

[Jim U>] Not required.

Aijun Wang

China Telecom

Comments In-Line..

Thanks,

              Jim Uttaro

  _____  

Best Regards,

Zhenqiang Li

Date: 2022-02-04 20:39

Thank you, Susan Hares

-- 

Gyan Mishra

Network Solutions Architect 

Email [email-masked] <mailto:[email-masked]> 

M 301 502-1347

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 10:50:26 +0100

Aijun & Gyan,

> it is possible for the misconfiguration, or misbehavior of PE; or the
increase of VPN
> prefixes in one VRF crashes some PEs in the network and influences other
existing
> VPN services on such PEs.

To mitigate those issues long time ago we have provided a local
configuration knob which limits number of prefixes which given VRF can
accept. *Note this is completely different knob from already discussed
prefix limit on the EBGP session towards CEs. *

syntax:

*maximum* *prefix* maximum [mid-threshold]

maximum

Maximum number of prefixes allowed in the VRF instance. Range is 32 to
2000000.

mid-threshold

(Optional) Integer specifying at what percentage of the maximum argument
value the software starts to generate a Simple Network Management Protocol
(SNMP) trap. Range is 1 to 100.

It works equally well on ingress or egress PEs on a per VRF basis. It
generates alarm and protects the infrastructure. When I was helping to
deploy L3VPNs to many global SPs number of them adopted the use of this
feature and to the best of my knowledge they keep using it.

I do not see what is missing to address your reported concerns.

Rgs,
Robert

---

From: Aijun Wang <[email-masked]>
Date: Thu, 10 Feb 2022 18:16:56 +0800

Hi, Robert:
This is the working base for the VPN Prefix ORF mechanism as that described in https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf#section-4.1.1.

We just push the extra VPN Prefixes further back to the peer, as the aim of the ORF mechanism, to relief the PE from processing the BGP UPDATE messages again. 

Aijun Wang
China Telecom

> 在 2022年2月10日，17:50，Robert Raszuk <[email-masked]> 写道：
> 
> 
> Aijun & Gyan,
> 
> > it is possible for the misconfiguration, or misbehavior of PE; or the increase of VPN 
> > prefixes in one VRF crashes some PEs in the network and influences other existing 
> To mitigate those issues long time ago we have provided a local configuration knob which limits number of prefixes which given VRF can accept. Note this is completely different knob from already discussed prefix limit on the EBGP session towards CEs. 
> 
> syntax:
> 
> maximum prefix maximum [mid-threshold]
> 
> maximum
> 
> Maximum number of prefixes allowed in the VRF instance. Range is 32 to 2000000.
> 
> mid-threshold
> 
> (Optional) Integer specifying at what percentage of the maximum argument value the software starts to generate a Simple Network Management Protocol (SNMP) trap. Range is 1 to 100.
> 
> 
> It works equally well on ingress or egress PEs on a per VRF basis. It generates alarm and protects the infrastructure. When I was helping to deploy L3VPNs to many global SPs number of them adopted the use of this feature and to the best of my knowledge they keep using it. 
> 
>> On Thu, Feb 10, 2022 at 3:31 AM Aijun Wang <[email-masked]> wrote:
>> Hi, Jim:
>>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 11:56:52 +0100

> We just push the extra VPN Prefixes further back to the peer, as the aim
of the ORF
> mechanism, to relief the PE from processing the BGP UPDATE messages
again.

The point is that you CAN NOT do that safely.

VPNv4/v6 routes come with a list of RTs. Based on those RTs routes are
imported to local VRFs. The import map further makes it even more non
deterministic as not only RT(s) but possibly many other match criteria is
used to make per prefix import decision.

So the fact that one VRF overflows on a PE because it has blind import RT
attached to that route does not mean that you can safely ask your upstream
RR or PEs to stop sending such route(s) to you. That has been explained in
detail when we last time spoke about this document, but it seems you are
not taking this into consideration.

And that is the main fundamental to the RFC4364 architecture issue I have
with the draft.

Thx,
R.

On Thu, Feb 10, 2022 at 11:16 AM Aijun Wang <[email-masked]>
wrote:

> Hi, Robert:
> This is the working base for the VPN Prefix ORF mechanism as that
> described in
> https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf#section-4.1.1
> .
>
> We just push the extra VPN Prefixes further back to the peer, as the aim
> of the ORF mechanism, to relief the PE from processing the BGP UPDATE
> messages again.
>
> Aijun Wang
> China Telecom
>
>> With the trending/deployment of EVPN/SRv6 technologies, such problems
>> will be emerged  inevitably because it is more easy to deploy the inter-AS
>> VPN services.
>>
>> Even for the intra-AS scenario, as that discussed in the current draft,
>> increase of VPN prefixes in one VRF crashes some PEs in the network and
>> influences other existing VPN services on such PEs.
>>
>>
>>
>> And, there are at least three operators that have widespread networks
>> have interested in the solution, I think it will also benefit your service
>> deployment in future.
>>
>>
>>
>> Best Regards
>>
>>
>>
>> Aijun Wang
>>
>> China Telecom
>>
>>
>>
>> *[Jim U>] Not true. Additional routing state will be throttled. There is
>> a notification setting and a threshold setting. The only time this was

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Thu, 10 Feb 2022 11:43:55 +0000

Gyan,

              So, the draft is targeted to a future where network services migrate to SRV6/EVPN, not a lack of functionality in the current set of tools for existing networks except in the case of China Mobile ( I would like to understand what is unique in this network that is driving this ask ). If that is the case then the draft should focus on the deficiencies of the current tool set for these FOUs. Not sure I get the statement in re RT-2 EVPN routes, is a distinction being made between these and let’s say RT-5?

Thanks,
              Jim Uttaro

Sent: Thursday, February 10, 2022 12:35 AM

Hi Jim

Aijun

Is that correct ?

Kind Regards

Gyan

On Wed, Feb 9, 2022 at 9:31 PM Aijun Wang <[email-masked]<mailto:[email-masked]>> wrote:
Hi, Jim:
Actually, draft https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-routes-control-analysis/<https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-wang-idr-vpn-routes-control-analysis/__;!!BhdT!2fA_b9k4tDNDtZyioqm3O8hinKjqjXnzYV1IMVwk7WZuJsfddfyj_cAYW_od73I$> describes more possible scenarios for the potential VPN prefixes overflow.
With the trending/deployment of EVPN/SRv6 technologies, such problems will be emerged  inevitably because it is more easy to deploy the inter-AS VPN services.
Even for the intra-AS scenario, as that discussed in the current draft, it is possible for the misconfiguration, or misbehavior of PE; or the increase of VPN prefixes in one VRF crashes some PEs in the network and influences other existing VPN services on such PEs.

Best Regards

Aijun Wang
China Telecom

From: UTTARO, JAMES <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, February 10, 2022 12:10 AM
To: [email-masked]<mailto:[email-masked]>; Aijun Wang <[email-masked]<mailto:[email-masked]>>
Cc: Susan Hares <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: RE: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Comments In-Line

Thanks,
              Jim Uttaro

Here is the thread..

From: [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, February 9, 2022 10:37 AM
To: UTTARO, JAMES <[email-masked]<mailto:[email-masked]>>; Aijun Wang <[email-masked]<mailto:[email-masked]>>
Cc: Susan Hares <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Thanks,
              Jim Uttaro

Hi, Jim:

Comments In-Line..

Thanks,
              Jim Uttaro

Thank you, Susan Hares
_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!BhdT!z587vpWgIAY6_KETP44Fo8e1ZASFQCIfEodjzi5G17QCO0dTApqL8DQmB0y2p3Q$>
_______________________________________________
Idr mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idr<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!BhdT!2fA_b9k4tDNDtZyioqm3O8hinKjqjXnzYV1IMVwk7WZuJsfddfyj_cAYXaxyV6I$>
--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!BhdT!2fA_b9k4tDNDtZyioqm3O8hinKjqjXnzYV1IMVwk7WZuJsfddfyj_cAYuAfegQY$>

Gyan Mishra

Email [email-masked]<mailto:[email-masked]>

M 301 502-1347

---

From: Aijun Wang <[email-masked]>
Date: Thu, 10 Feb 2022 19:44:21 +0800

Hi, Robert:
Route-Map is used to set the prefixes attributes on demand. The VPN Prefixes threshold should be checked before the route map action. Right?

Aijun Wang
China Telecom

> 在 2022年2月10日，18:57，Robert Raszuk <[email-masked]> 写道：
> 
> 
> 
> > We just push the extra VPN Prefixes further back to the peer, as the aim of the ORF 
> > mechanism, to relief the PE from processing the BGP UPDATE messages again. 
> 
> VPNv4/v6 routes come with a list of RTs. Based on those RTs routes are imported to local VRFs. The import map further makes it even more non deterministic as not only RT(s) but possibly many other match criteria is used to make per prefix import decision. 
> 
> So the fact that one VRF overflows on a PE because it has blind import RT attached to that route does not mean that you can safely ask your upstream RR or PEs to stop sending such route(s) to you. That has been explained in detail when we last time spoke about this document, but it seems you are not taking this into consideration. 
> 
> And that is the main fundamental to the RFC4364 architecture issue I have with the draft. 
> 
> Thx,
> R.
> 
> 
>> On Thu, Feb 10, 2022 at 11:16 AM Aijun Wang <[email-masked]> wrote:
>> Hi, Robert:

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 13:00:42 +0100

In the implementations I tested in the past max prefix per VRF was checked
post import or import map permit.

On Thu, Feb 10, 2022 at 12:44 PM Aijun Wang <[email-masked]>
wrote:

> Hi, Robert:
> Route-Map is used to set the prefixes attributes on demand. The VPN
> Prefixes threshold should be checked before the route map action. Right?
>
>
> Aijun Wang
> China Telecom
>
>>> VPN Prefix ORF mechanism gives us the capabilities to isolate the
>>> problem per VRF, not per BGP session, or per PE.
>>>
>>> *To:* [email-masked]; Aijun Wang <[email-masked]
>>> >
>>> *Subject:* RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption
>>> *Subject:* Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption
>>> *Subject:* Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption

---

From: Aijun Wang <[email-masked]>
Date: Thu, 10 Feb 2022 21:23:12 +0800

Hi, Robert:
Even as you described, current VPN Prefixes ORF message doesn’t give the wrong information to the peer, right?
The accepted routes must first pass the import route target filter. 
Although additional  route map criteria are not carried in the ORF messages, there will be no “wrong filter”. On the contrary, filter based only on the RD, RT and original PE information can further protect this VRF/PE, because the routes that match the imported RT but not the route map are not wanted by this VRF.
Is the logic Right?

Aijun Wang
China Telecom

> 在 2022年2月10日，20:00，Robert Raszuk <[email-masked]> 写道：
> 
> 
> 
> In the implementations I tested in the past max prefix per VRF was checked post import or import map permit.
> 
>> On Thu, Feb 10, 2022 at 12:44 PM Aijun Wang <[email-masked]> wrote:
>> Hi, Robert:

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 15:42:25 +0100

> wrong information to the peer, right?

Unfortunately this is wrong.

It signals too much (or too coarse) in the cases where VPN route carries
more then single RT and those RTs are used with or without extra policy to
get imported into more then one VRF.

That is the fundamental issue.

And this is not as much of an issue with your proposal, but with base L3VPN
specification. It allows flexibility and you are hit by it.

The only possible option if you want to continue would be to add a strong
restriction into your document (all scenarios) which says:

"VPN prefix ORF message MUST NOT be originated if there is any not affected
VRF which imports subject VPN prefix originated with given RD and contains
at least one intersecting RTs when compared against list of RD and RT(s)
carried by VPN routes signalled for filtering".

Thx,
R.

PS. I still think this draft is not needed, but I will not be raising
blocking red flag if you at least fix it in such a way that it is not
harmful to other VPNs. Currently it is.

> Although additional  route map criteria are not carried in the ORF
> messages, there will be no “wrong filter”. On the contrary, filter based
> only on the RD, RT and original PE information can further protect this
> VRF/PE, because the routes that match the imported RT but not the route map
> are not wanted by this VRF.
> Is the logic Right?
>
> Aijun Wang
> China Telecom
>
>>>> *Subject:* RE: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt-
>>>> Adoption call (2/4/2022 to 2/18/2022)
>>>>
>>>>
>>>>
>>>> Comments In-Line
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>               Jim Uttaro
>>>>
>>>>
>>>>
>>>> Here is the thread..
>>>>
>>>>
>>>>
>>>> I understood your first statement as indicating more than just the
>>>> China Mobile Network. If this work is directed towards an industry wide
>>>> problem than you should canvass to see if other operators require this
>>>> *Subject:* Re: RE: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt-
>>>> Adoption call (2/4/2022 to 2/18/2022)
>>>>
>>>>
>>>>
>>>> *[Jim U>] Not true. Additional routing state will be throttled. There
>>>> is a notification setting and a threshold setting. The only time this was

---

From: Gyan Mishra <[email-masked]>
Date: Thu, 10 Feb 2022 13:13:04 -0500

Jim

The draft focuses on both EVPN and IP VPN as in scope to the draft as Aijun
mentioned.

What I was mentioning was the scenario with China Telecom and the other two
providers related to their specific use case being EVPN related however
this solution applies to both EVPN and IP VPN.  I believe they may had IP
VPN issues as well as the issue as described below can be problematic and
impact any operator.  This is not an operations or design issue.

The solution provides granularity to filter on the source PE flooding
excessive routes which could be EVPN or IP VPN.  The PE does not have to be
a weak PE and could be any PE with plenty of memory and CPU as well.

The Crux of issue is the flood of unwanted routes by a rogue PE sourcing
the unwanted flooding of RT-2 or IP prefix flooding.

**  If the PE-CE maximum prefix threshold and per VPN maximum prefix
threshold gets hit due to the flood of routes now that impacts the
legitimate valid RT-2 or IP prefixes.  As a result all customers could be
impacted that are part of the VPN or MAC VRF. So that’s the major issue
this draft is solving in trying to clamp down and squash the offending
rogue PE**. I mentioned this before that if you set the maximum prefix high
or to the maximum you might as well not set it as it’s not helpful in the
prevention of flood of routes.   You can think of this like a DDOS attack
by a rogue PE impacting all customers that are part of the VPN importing
the RT.

The issue with weak PE or PE being overwhelmed with the flood of routes as
well is an issue and this draft addresses any of those possible scenarios.

I mentioned EVPN  RT-2 as you may have 1000s of MAC/IP routes depending on
how flat your subnet is per RT-5 so the numbers go up pretty quickly and
easy to flood.  Also with

The issue is worse inter-as options b as all RTs have to be accepted with
knob retain-rt-all on inter-as PE ASBR for option-b and RTs cannot be
filtered on RR for option-c same issue for both EVPN and IP VPN.

The draft discusses why existing mechanisms PE-CE maximum prefix and per
VPN maximum prefix does not help this issue.

Kind Regards

Gyan

On Thu, Feb 10, 2022 at 6:44 AM UTTARO, JAMES <[email-masked]> wrote:

> Gyan,
>
>
>
>               So, the draft is targeted to a future where network services
> migrate to SRV6/EVPN, not a lack of functionality in the current set of
> tools for existing networks except in the case of China Mobile ( I would
> like to understand what is unique in this network that is driving this ask
> ). If that is the case then the draft should focus on the deficiencies of
> the current tool set for these FOUs. Not sure I get the statement in re
> RT-2 EVPN routes, is a distinction being made between these and let’s say
> RT-5?
>
>
>
> Thanks,
>
>               Jim Uttaro
>
>
>
> *From:* Gyan Mishra <[email-masked]>
> *Sent:* Thursday, February 10, 2022 12:35 AM
> *Cc:* Susan Hares <[email-masked]>; UTTARO, JAMES <[email-masked]>;
> [email-masked]; [email-masked]
> I don’t think there are many SRv6 deployments worldwide and I believe
> China is one of the leaders on proliferation of IPV6 by operators  as well
> as now SRv6.
>
>
>
> <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-wang-idr-vpn-routes-control-analysis/__;!!BhdT!2fA_b9k4tDNDtZyioqm3O8hinKjqjXnzYV1IMVwk7WZuJsfddfyj_cAYW_od73I$>
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!BhdT!2fA_b9k4tDNDtZyioqm3O8hinKjqjXnzYV1IMVwk7WZuJsfddfyj_cAYXaxyV6I$>
>
> --
>
>
> <https://urldefense.com/v3/__http:/www.verizon.com/__;!!BhdT!2fA_b9k4tDNDtZyioqm3O8hinKjqjXnzYV1IMVwk7WZuJsfddfyj_cAYuAfegQY$>
>
> *Gyan Mishra*
>
> *Network Solutions Architect *
>
-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Thu, 10 Feb 2022 13:22:39 -0500

Verizon has had this issue with the rogue PE and the workaround is to set
the maximum prefix high as well as per VPN maximum prefix but then that
allows the flood to occur and defeats the purpose of maximum prefix to
manage resources.  So it’s an industry wide known problem but just had not
been brought to light until this draft.

Kind Regards

Gyan

On Thu, Feb 10, 2022 at 1:13 PM Gyan Mishra <[email-masked]> wrote:

> Jim
>
> The draft focuses on both EVPN and IP VPN as in scope to the draft as
> Aijun mentioned.
>
> What I was mentioning was the scenario with China Telecom and the other
> two providers related to their specific use case being EVPN related however
>>               So, the draft is targeted to a future where network
>> services migrate to SRV6/EVPN, not a lack of functionality in the current
>> set of tools for existing networks except in the case of China Mobile ( I
>> would like to understand what is unique in this network that is driving
>> this ask ). If that is the case then the draft should focus on the
>> deficiencies of the current tool set for these FOUs. Not sure I get the
>> statement in re RT-2 EVPN routes, is a distinction being made between these
>> and let’s say RT-5?
>>
>>
>>
>> Thanks,
>>
>>               Jim Uttaro
>>
>>
>>
>> will be emerged  inevitably because it is more easy to deploy the
>> inter-AS VPN services.
>>

*Gyan Mishra*

*M 301 502-1347*

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 20:37:30 +0100

Gyan,

> The Crux of issue is the flood of unwanted routes by a rogue PE sourcing
the
> unwanted flooding of RT-2 or IP prefix flooding.

If this statement is really true then crux of the issue has been
recognized, worked on and addressed in early 2000s.

Only routes which *are wanted* get propagated in a properly
designed network.

Hint: RFC4684

Now if you accept too many of VPN routes you can not treat them as
unwanted. It is like you would be throwing away passengers in the middle of
the flight just because plane is running out of fuel. You need to be
careful during boarding process to accept only as much as you can carry. As
simple as this.

If you however consider problem statement Aijun is describing it seems that
PE or RR which noticed some attack vector within one VPN could immediately
report this to the mgmt station which in turn should immediately shut down
the eBGP session from the CE or CEs which originate such attack. No
protocol extension needed at all and immediate action can be much more
effective then just pushing some ORF to RRs.

Thank you,
R.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Thu, 10 Feb 2022 20:26:50 +0000

Good problem.
Bad solution.

Consider an originator PE, PE1, an RR and two receiving PEs, PE2, PE3.

PE1----RR----PE2
        |
        +----PE3

PE1 sends the routes. PE2 hits a limit, but PE3 does not.
The RR I have drawn may be multiple RRs, a confederation or even a multi AS network.
PE2 should do some combination of the following:
- drop the routes from PE1,
- accept no more routes from PE1 (may cause loops)
- send a notification to PE1.
In particular, RR must not block routes from PE1, because PE3 still wants them.

An ORF does not satisfy the above.
I would be happy to support a new message in BGP (possibly attached to a REFRESH message) to achieve a good solution.

Regards,
Jakob.

From: Idr <[email-masked]> On Behalf Of Gyan Mishra
Sent: Thursday, February 10, 2022 10:23 AM
To: UTTARO, JAMES <[email-masked]>
Cc: [email-masked]; Susan Hares <[email-masked]>; [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Verizon has had this issue with the rogue PE and the workaround is to set the maximum prefix high as well as per VPN maximum prefix but then that allows the flood to occur and defeats the purpose of maximum prefix to manage resources.  So it’s an industry wide known problem but just had not been brought to light until this draft.

Kind Regards

Gyan

On Thu, Feb 10, 2022 at 1:13 PM Gyan Mishra <[email-masked]<mailto:[email-masked]>> wrote:
Jim

The draft focuses on both EVPN and IP VPN as in scope to the draft as Aijun mentioned.

What I was mentioning was the scenario with China Telecom and the other two providers related to their specific use case being EVPN related however this solution applies to both EVPN and IP VPN.  I believe they may had IP VPN issues as well as the issue as described below can be problematic and impact any operator.  This is not an operations or design issue.

The solution provides granularity to filter on the source PE flooding excessive routes which could be EVPN or IP VPN.  The PE does not have to be a weak PE and could be any PE with plenty of memory and CPU as well.

The Crux of issue is the flood of unwanted routes by a rogue PE sourcing the unwanted flooding of RT-2 or IP prefix flooding.

**  If the PE-CE maximum prefix threshold and per VPN maximum prefix threshold gets hit due to the flood of routes now that impacts the legitimate valid RT-2 or IP prefixes.  As a result all customers could be impacted that are part of the VPN or MAC VRF. So that’s the major issue this draft is solving in trying to clamp down and squash the offending rogue PE**. I mentioned this before that if you set the maximum prefix high or to the maximum you might as well not set it as it’s not helpful in the prevention of flood of routes.   You can think of this like a DDOS attack by a rogue PE impacting all customers that are part of the VPN importing the RT.

The issue with weak PE or PE being overwhelmed with the flood of routes as well is an issue and this draft addresses any of those possible scenarios.

I mentioned EVPN  RT-2 as you may have 1000s of MAC/IP routes depending on how flat your subnet is per RT-5 so the numbers go up pretty quickly and easy to flood.  Also with

The issue is worse inter-as options b as all RTs have to be accepted with knob retain-rt-all on inter-as PE ASBR for option-b and RTs cannot be filtered on RR for option-c same issue for both EVPN and IP VPN.

The draft discusses why existing mechanisms PE-CE maximum prefix and per VPN maximum prefix does not help this issue.

Kind Regards

Gyan

On Thu, Feb 10, 2022 at 6:44 AM UTTARO, JAMES <[email-masked]<mailto:[email-masked]>> wrote:
Gyan,

Thanks,
              Jim Uttaro

From: Gyan Mishra <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, February 10, 2022 12:35 AM
To: Aijun Wang <[email-masked]<mailto:[email-masked]>>
Cc: Susan Hares <[email-masked]<mailto:[email-masked]>>; UTTARO, JAMES <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Hi Jim

Aijun

Is that correct ?

Kind Regards

Gyan

Best Regards

Aijun Wang
China Telecom

Comments In-Line

Thanks,
              Jim Uttaro

Here is the thread..

Thanks,
              Jim Uttaro

Hi, Jim:

Comments In-Line..

Thanks,
              Jim Uttaro

Thank you, Susan Hares

Gyan Mishra

M 301 502-1347

--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<http://www.verizon.com/>

Gyan Mishra

M 301 502-1347

--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<http://www.verizon.com/>

Gyan Mishra

M 301 502-1347

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 21:39:38 +0100

Hey Jakob,

Happy Birthday !

> I would be happy to support a new message in BGP (possibly attached
> to a REFRESH message) to achieve a good solution.

When I wrote last paragraph I was thinking about the same - but stopped
upon two issues:

a) can not find any advantage on signalling this inband vs out of band (ie
via mgmt channel including NNI mgmt)

b) what is PE1 supposed to do if as you say the very same routes are
working fine on PE3 and just PE2 has issues ? PE1 can not shut the session
to CE, PE1 can not stop sending them to RR etc ... so what's the point ?

I was even crazy enough to think that maybe we could stuff Flowspec v2 with
such message, but again (a) won again.

Best,
R.

On Thu, Feb 10, 2022 at 9:27 PM Jakob Heitz (jheitz) <jheitz=
[email-masked]> wrote:

> I would be happy to support a new message in BGP (possibly attached to a
> REFRESH message) to achieve a good solution.
>
>
>
> Regards,
>
> Jakob.
>
>
>
> *From:* Idr <[email-masked]> *On Behalf Of * Gyan Mishra
> *Sent:* Thursday, February 10, 2022 10:23 AM
> *Cc:* [email-masked]; Susan Hares <[email-masked]>;
> *[Jim U>] Have you canvassed operators? I would be interested in the *
> *“problems**” that have been identified in networks today. Pls provide
> data and not conjecture.*
>
> *[Jim U>] Have you canvassed operators? I would be interested in the *
> *“problems**” that have been identified in networks today. Pls provide
> data and not conjecture.*
>

---

From: "UTTARO, JAMES" <[email-masked]>
Date: Thu, 10 Feb 2022 20:49:30 +0000

Gyan,

              Comments In-Line..

Thanks,
              Jim Uttaro

From: Gyan Mishra <[email-masked]>
Sent: Thursday, February 10, 2022 1:23 PM
To: UTTARO, JAMES <[email-masked]>
Cc: Aijun Wang <[email-masked]>; Susan Hares <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Verizon has had this issue with the rogue PE [Jim U>] Not sure what a rogue PE but I like the term 😉 and the workaround is to set the maximum prefix high as well as per VPN maximum prefix but then that allows the flood to occur[Jim U>]  How is that?? How big of a flood are you worried about?  and defeats the purpose of maximum prefix to manage resources[Jim U>] Works for me.  So it’s an industry wide known problem but just had not been brought to light until this draft.[Jim U>] I do not think so .Been running VPNV4 network with 10s of millions of routes with thousands of- highly and under-powered routers for 20+ yrs. The tools and the loading rules have worked well.

[Jim U>] IN terms of RT-2 I will use an analogy from Kompella. We currently have a loading rule that prohibits more than 50 MACs from being learned from the customer.. We are doing the same with EVPN. I would suggest looking at your loading rules.

Kind Regards

Gyan

Kind Regards

Gyan

Thanks,
              Jim Uttaro

Hi Jim

Aijun

Is that correct ?

Kind Regards

Gyan

Best Regards

Aijun Wang
China Telecom

Comments In-Line

Thanks,
              Jim Uttaro

Here is the thread..

Thanks,
              Jim Uttaro

Hi, Jim:

Comments In-Line..

Thanks,
              Jim Uttaro

Thank you, Susan Hares

Gyan Mishra

M 301 502-1347

--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!BhdT!3MfJ9MCG_xOTa5ToOaf5-ONhyMvRTGbuxYOrFr_yUvjZnf1Pyx8EJ02TMo9av5g$>

Gyan Mishra

M 301 502-1347

--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<https://urldefense.com/v3/__http:/www.verizon.com/__;!!BhdT!3MfJ9MCG_xOTa5ToOaf5-ONhyMvRTGbuxYOrFr_yUvjZnf1Pyx8EJ02TMo9av5g$>

Gyan Mishra

M 301 502-1347

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Thu, 10 Feb 2022 20:56:17 +0000

The reality here is that someone leaks the GRT into his VPN.
PE3 may prefer to filter that GRT and carry on anyway, because the real routes are important.
PE2 doesn't care and kicks him out.
Still, it's important that the PE1 operator knows.
If we think the existing network monitoring tools are enough, then I'm fine with it.

The other thing is that (contrary to the claims in the draft),
https://datatracker.ietf.org/doc/html/rfc5292 can do this ORF already.
RFC5292 says nothing about configuration of the feature.
Configuration of the feature is not an interoperability issue,
therefore it's not covered in an RFC.
This draft amounts to a feature request of a router vendor.
The IETF should not be used to submit feature requests.

Regards,
Jakob.

From: Robert Raszuk <[email-masked]>
Sent: Thursday, February 10, 2022 12:40 PM
To: Jakob Heitz (jheitz) <[email-masked]>
Cc: Gyan Mishra <[email-masked]>; UTTARO, JAMES <[email-masked]>; [email-masked]; [email-masked]; Susan Hares <[email-masked]>
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Hey Jakob,

Happy Birthday !

> to a REFRESH message) to achieve a good solution.

When I wrote last paragraph I was thinking about the same - but stopped upon two issues:

a) can not find any advantage on signalling this inband vs out of band (ie via mgmt channel including NNI mgmt)

b) what is PE1 supposed to do if as you say the very same routes are working fine on PE3 and just PE2 has issues ? PE1 can not shut the session to CE, PE1 can not stop sending them to RR etc ... so what's the point ?

I was even crazy enough to think that maybe we could stuff Flowspec v2 with such message, but again (a) won again.

Best,
R.

On Thu, Feb 10, 2022 at 9:27 PM Jakob Heitz (jheitz) <jheitz=[email-masked]<mailto:[email-masked]>> wrote:
Good problem.
Bad solution.

PE1----RR----PE2
        |
        +----PE3

Regards,
Jakob.

From: Idr <[email-masked]<mailto:[email-masked]>> On Behalf Of Gyan Mishra
Sent: Thursday, February 10, 2022 10:23 AM
To: UTTARO, JAMES <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Susan Hares <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Kind Regards

Gyan

Kind Regards

Gyan

Thanks,
              Jim Uttaro

Hi Jim

Aijun

Is that correct ?

Kind Regards

Gyan

Best Regards

Aijun Wang
China Telecom

Comments In-Line

Thanks,
              Jim Uttaro

Here is the thread..

Thanks,
              Jim Uttaro

Hi, Jim:

Comments In-Line..

Thanks,
              Jim Uttaro

Thank you, Susan Hares

Gyan Mishra

M 301 502-1347

--

Gyan Mishra

M 301 502-1347

--

Gyan Mishra

M 301 502-1347

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 10 Feb 2022 16:00:13 -0500

Basic prefix ORF was mentioned a while back.  It does RD-only fine.

The current draft-wang has optional TLVs that don't fit into that paradigm any longer.

We are starting to see some level of standardization of prefix-counters in telemetry land.  However, none of those provide RD-level granularity.

-- Jeff

> On Feb 10, 2022, at 3:56 PM, Jakob Heitz (jheitz) <jheitz=[email-masked]> wrote:
> 
> https://datatracker.ietf.org/doc/html/rfc5292 <https://datatracker.ietf.org/doc/html/rfc5292> can do this ORF already.
> On Thu, Feb 10, 2022 at 9:27 PM Jakob Heitz (jheitz) <jheitz=[email-masked] <mailto:[email-masked]>> wrote:
> Good problem.
> Bad solution.
>  
> From: Idr <[email-masked] <mailto:[email-masked]>> On Behalf Of Gyan Mishra
> To: UTTARO, JAMES <[email-masked] <mailto:[email-masked]>>
> Cc: [email-masked] <mailto:[email-masked]>; Susan Hares <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> On Thu, Feb 10, 2022 at 1:13 PM Gyan Mishra <[email-masked] <mailto:[email-masked]>> wrote:
> Jim
>  
> On Thu, Feb 10, 2022 at 6:44 AM UTTARO, JAMES <[email-masked] <mailto:[email-masked]>> wrote:
> Gyan,
>  
> From: Gyan Mishra <[email-masked] <mailto:[email-masked]>> 
> To: Aijun Wang <[email-masked] <mailto:[email-masked]>>
> Cc: Susan Hares <[email-masked] <mailto:[email-masked]>>; UTTARO, JAMES <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>
> On Wed, Feb 9, 2022 at 9:31 PM Aijun Wang <[email-masked] <mailto:[email-masked]>> wrote:
> Hi, Jim:
> Actually, draft https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-routes-control-analysis/ <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/html/draft-wang-idr-vpn-routes-control-analysis/__;!!BhdT!2fA_b9k4tDNDtZyioqm3O8hinKjqjXnzYV1IMVwk7WZuJsfddfyj_cAYW_od73I$> describes more possible scenarios for the potential VPN prefixes overflow.
> From: UTTARO, JAMES <[email-masked] <mailto:[email-masked]>> 
> To: [email-masked] <mailto:[email-masked]>; Aijun Wang <[email-masked] <mailto:[email-masked]>>
> Cc: Susan Hares <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> From: [email-masked] <mailto:[email-masked]> <[email-masked] <mailto:[email-masked]>> 
> To: UTTARO, JAMES <[email-masked] <mailto:[email-masked]>>; Aijun Wang <[email-masked] <mailto:[email-masked]>>
> Cc: Susan Hares <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> CC: [email-masked] <mailto:[email-masked]>; Hares <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>
> From: Aijun Wang <[email-masked] <mailto:[email-masked]>> 
> To: UTTARO, JAMES <[email-masked] <mailto:[email-masked]>>
> Cc: [email-masked] <mailto:[email-masked]>; Hares <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> 在 2022年2月9日，19:54，UTTARO, JAMES <[email-masked] <mailto:[email-masked]>> 写道：
> 
> Comments In-Line..
>  
> Thanks,
>               Jim Uttaro
>  
> From: Idr <[email-masked] <mailto:[email-masked]>> On Behalf Of [email-masked] <mailto:[email-masked]>
> To: Hares <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> From: Idr [mailto:[email-masked] <mailto:[email-masked]>] On Behalf Of Susan Hares
> https://datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf <https://urldefense.com/v3/__https:/datatracker.ietf.org/doc/draft-wang-idr-vpn-prefix-orf__;!!BhdT!yPUYUuirQ1P6jY-6LfD7vJRLsQR3n8-cWzq-Id3eLmvKj3ISHGjnW-6e6po$>.
>  
> https://www.ietf.org/mailman/listinfo/idr <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/idr__;!!BhdT!2fA_b9k4tDNDtZyioqm3O8hinKjqjXnzYV1IMVwk7WZuJsfddfyj_cAYXaxyV6I$>
> --
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>_______________________________________________
> Idr mailing list
> [email-masked]

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 22:26:10 +0100

Hey,

Prefix limit on PE-CE detects this, drops the eBGP session to CE - problem
solved.

Now if you are talking about SP attacking his own customers then max prefix
to VRF also protects from it.

> https://datatracker.ietf.org/doc/html/rfc5292 can do this ORF already.

Yes I and others have brought this point already.

They say in the draft:

   2) Address Prefix ORF

   Using Address Prefix ORF to filter VPN routes need to pre-
   configuration, but it is impossible to know which prefix may cause
   overflow in advance.

But they do not understand the notion of prefix :)

> The IETF should not be used to submit feature requests.

Yup modulo they are stiffing it more with RT, Source-PE etc ...

I think authors do not fully understand how ORF works. Moreover they want
it to magically loop free go hop by hop but stating Oh just think that RR
will build his own ORF.

I think we have a bug in the ORF RFC which would explicitly prohibit
building it from dynamically received policy. Time for a -bis there ...

Thx,
R,.

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 22:31:50 +0100

> We are starting to see some level of standardization of prefix-counters in
> telemetry land.  However, none of those provide RD-level granularity.
>

I would fully support such extension/addition to telemetry. Or use SNMP
trap or just syslog warning.

And we really do not need RD level .. we just need per vrf prefix count. RD
is overwritten anyway during import to a local RD so mapping of translation
of RT to src RD is needed anyway.

---

From: "Jakob Heitz (jheitz)" <[email-masked]>
Date: Thu, 10 Feb 2022 21:32:26 +0000

You can do loop free if you send the ORF only to where you received the associated routes.
However, when you no longer receive the routes (maybe because the ORF took effect??),
then you have to remove the ORF (to retain loop free). You get the next step.

Regards,
Jakob.

Sent: Thursday, February 10, 2022 1:26 PM

Hey,

Prefix limit on PE-CE detects this, drops the eBGP session to CE - problem solved.

Now if you are talking about SP attacking his own customers then max prefix to VRF also protects from it.

They say in the draft:

   2) Address Prefix ORF

   overflow in advance.

I think authors do not fully understand how ORF works. Moreover they want it to magically loop free go hop by hop but stating Oh just think that RR will build his own ORF.

I think we have a bug in the ORF RFC which would explicitly prohibit building it from dynamically received policy. Time for a -bis there ...

Thx,
R,.

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 10 Feb 2022 16:36:51 -0500

> On Feb 10, 2022, at 4:31 PM, Robert Raszuk <[email-masked]> wrote:
> 
>  
> I would fully support such extension/addition to telemetry. Or use SNMP trap or just syslog warning.  

While I'm fully capable of writing new MIBs and code, I hope to never do so again in my career. :-)

Yang modules is easy though.

> 
> And we really do not need RD level .. we just need per vrf prefix count. RD is overwritten anyway during import to a local RD so mapping of translation of RT to src RD is needed anyway. 

In general, we're lucky that the network instance model we're using in IETF/OC Yang means that state can be readily be per table/vrf quite easily.  That part is no longer the semantic leap.

The issue the authors of draft-wang still have - and one that needed discussion during our original very long thread - is what are you counting?  In their case, they wanted to count RD since that's the element they want to filter on.  This leads to an interesting disconnect that for implementations I'm familiar with that the VRF view of things, where they have the desired limits, don't necessarily have backpointers to the state that let them know what the contributing RD is for the VRF installed routes.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 22:37:04 +0100

Yeh sure, but the point is that such behaviour of dynamic ORF policy
transitiveness is not described in any document and it can not be just
swapped under the carpet as local implementation issue.

On Thu, Feb 10, 2022 at 10:32 PM Jakob Heitz (jheitz) <[email-masked]>
wrote:

> *From:* Robert Raszuk <[email-masked]>
> *Sent:* Thursday, February 10, 2022 1:26 PM
> *To:* Jakob Heitz (jheitz) <[email-masked]>
> *Cc:* Gyan Mishra <[email-masked]>; UTTARO, JAMES <[email-masked]>;
> [email-masked]; [email-masked]; Susan Hares <[email-masked]>

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 10 Feb 2022 22:42:00 +0100

>
> This leads to an interesting disconnect that for implementations I'm
> familiar with that the VRF view of things, where they have the desired
> limits, don't necessarily have backpointers to the state that let them know
> what the contributing RD is for the VRF installed routes.
>

That is critical to this entire idea.

As You may recall Jeff we discussed this.

The answer was that this is a local PE  implementation issue and the PE
will just "figure it out" :) While simple to say and maybe not that hard to
add a back pointer to src path there is per path cost associated with that.
Leave alone additional memory and cycles to keep per src RD import
counters.

---

From: Gyan Mishra <[email-masked]>
Date: Thu, 10 Feb 2022 17:14:59 -0500

Jim

Responses in-line

On Thu, Feb 10, 2022 at 3:50 PM UTTARO, JAMES <[email-masked]> wrote:

> *Sent:* Thursday, February 10, 2022 1:23 PM
> *Cc:* Aijun Wang <[email-masked]>; Susan Hares <[email-masked]>;
> Verizon has had this issue with the rogue PE *[Jim U>] *Not sure what a
> rogue PE but I like the term 😉
>
    Gyan> PE sourcing a flood of prefixes

and the workaround is to set the maximum prefix high as well as per VPN
> maximum prefix but then that allows the flood to occur*[Jim U>] * How is
> that?? How big of a flood are you worried about?
>
   Gyan> It had been a long while ago that we have have seen a flood of
routes but nothing that has caused us to worry about resources, and at the
time the solution was to ratchet up the maximum prefix - problem solved.
Overall it had not caused any problems increasing the maximum prefix.

> and defeats the purpose of maximum prefix to manage resources*[Jim U>] *Works
> for me.
>
Gyan> I have not encountered any issue in management of resources with
maximum prefix being increased to mitigate a flood. I cannot speak for the
other authors on what they have encountered.

> So it’s an industry wide known problem but just had not been brought to
> light until this draft.*[Jim U>] *I do not think so .Been running VPNV4
> network with 10s of millions of routes with thousands of- highly and
> under-powered routers for 20+ yrs. The tools and the loading rules have
> worked well.
>
>  Gyan>  Understood as I have as well had  no issues from the flood.  As
> far as industry wide problem let me restate as a from my experience known
> operations  issue moreso and not a problem and the easy fix has been to
> ratchet up the maximum prefix and we have not seen any issues from doing
> so.  I cannot speak for the other authors as I think the problem statement
> they are describing is slightly different.
>
> *[Jim U>] IN terms of RT-2 I will use an analogy from Kompella. We
> currently have a loading rule that prohibits more than 50 MACs from being
> learned from the customer.. We are doing the same with EVPN. I would
> suggest looking at your loading rules.*
>
> Gyan> Understood good point  for L2 VPNs. Agreed.
>
Aijun

Can you expand on the EVPN scenario?

> <https://urldefense.com/v3/__http:/www.verizon.com/__;!!BhdT!3MfJ9MCG_xOTa5ToOaf5-ONhyMvRTGbuxYOrFr_yUvjZnf1Pyx8EJ02TMo9av5g$>
>
> *Gyan Mishra*
>
> <https://urldefense.com/v3/__http:/www.verizon.com/__;!!BhdT!3MfJ9MCG_xOTa5ToOaf5-ONhyMvRTGbuxYOrFr_yUvjZnf1Pyx8EJ02TMo9av5g$>
>
> *Gyan Mishra*
>
-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Gyan Mishra <[email-masked]>
Date: Thu, 10 Feb 2022 19:26:55 -0500

Robert

> recognized, worked on and addressed in early 2000s.

    Gyan> Unwanted is bad choice of words -  as long as the route matches
the RT it’s imported wanted
- Agreed.  So it’s wanted routes explicitly imported that is an attack
vector from a source PE for the VPN exactly what Aijun is describing.

> Only routes which *are wanted* get propagated in a properly

    Gyan> Understood

> unwanted.

It is like you would be throwing away passengers in the middle of the
> flight just because plane is running out of fuel. You need to be careful
> during boarding process to accept only as much as you can carry. As simple
> as this.
>

   Gyan> Understood.  It’s wanted routes explicitly imported that is an an
attack vector on the VPN

>
> If you however consider problem statement Aijun is describing it seems
> that PE or RR which noticed some attack vector within one VPN
>

Gyan> That is the same problem statement I was describing

could immediately report this to the mgmt station which in turn should
> immediately shut down the eBGP session from the CE or CEs which originate
> such attack.
>

Gyan> The solution with using ORF  is an automated filtering capability
with RD granularity so we can detect the source PE where the attack vector
originated.

No protocol extension needed at all and immediate action can be much more
> effective then just pushing some ORF to RRs.

Gyan> no solution exists today to filter on RD automatically as all
filtering is based on RT or prefix.

>> The issue with weak PE or PE being overwhelmed with the flood of routes
>> as well is an issue and this draft addresses any of those possible
>> scenarios.
>>
>>
>> I mentioned EVPN  RT-2 as you may have 1000s of MAC/IP routes depending
>> on how flat your subnet is per RT-5 so the numbers go up pretty quickly and
>> easy to flood.  Also with
>>

*Gyan Mishra*

*M 301 502-1347*

---

From: "Aijun Wang" <[email-masked]>
Date: Fri, 11 Feb 2022 09:46:05 +0800

Hi, Robert:

Thanks for your suggestions!

Actually,  https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf#section-4.1.1 has described such scenario as the following sentences:

“On PE2, both VRF1 and VRF2 import VPN routes with RT1.  If PE2

   triggers VPN Prefix ORF mechanism when VRF1 overflows, VRF2 cannot

   receive VPN routes with RT1 as well.  PE2 should not trigger the VPN

   Prefix ORF message to RR(RD31, RT1, source PE is PE3) until all the

   VRFs that import RT1 overflow.”

Anyway, your recommended description is more generic, we can incorporate it later in the implementation consideration part.

Many thanks for your comments along the discussion, I think we have made one great step towards its adoption.

Best Regards

Aijun Wang

China Telecom

From: Robert Raszuk <[email-masked]> 
Sent: Thursday, February 10, 2022 10:42 PM
To: Aijun Wang <[email-masked]>
Cc: UTTARO, JAMES <[email-masked]>; [email-masked]; idr@ietf. org <[email-masked]>; Susan Hares <[email-masked]>
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

It signals too much (or too coarse) in the cases where VPN route carries more then single RT and those RTs are used with or without extra policy to get imported into more then one VRF. 

And this is not as much of an issue with your proposal, but with base L3VPN specification. It allows flexibility and you are hit by it. 

The only possible option if you want to continue would be to add a strong restriction into your document (all scenarios) which says: 

"VPN prefix ORF message MUST NOT be originated if there is any not affected VRF which imports subject VPN prefix originated with given RD and contains at least one intersecting RTs when compared against list of RD and RT(s) carried by VPN routes signalled for filtering". 

Thx,

R.

PS. I still think this draft is not needed, but I will not be raising blocking red flag if you at least fix it in such a way that it is not harmful to other VPNs. Currently it is. 

Is the logic Right?

Aijun Wang

China Telecom

在 2022年2月10日，20:00，Robert Raszuk <[email-masked] <mailto:[email-masked]> > 写道：

On Thu, Feb 10, 2022 at 12:44 PM Aijun Wang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, Robert:

Aijun Wang

China Telecom

在 2022年2月10日，18:57，Robert Raszuk <[email-masked] <mailto:[email-masked]> > 写道：

Thx,

R.

On Thu, Feb 10, 2022 at 11:16 AM Aijun Wang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, Robert:

Aijun Wang

China Telecom

在 2022年2月10日，17:50，Robert Raszuk <[email-masked] <mailto:[email-masked]> > 写道：

Aijun & Gyan,

syntax:

maximum

mid-threshold

Rgs,
Robert

On Thu, Feb 10, 2022 at 3:31 AM Aijun Wang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, Jim:

Best Regards

Aijun Wang

China Telecom

Comments In-Line

Thanks,

              Jim Uttaro

Here is the thread..

 

  _____  

Best Regards,

Zhenqiang Li

Date: 2022-02-09 21:33

Comments In-Line

Thanks,

              Jim Uttaro

Hi, Jim:

 

[Jim U>] No. 

[Jim U>] Not required.

Aijun Wang

China Telecom

Comments In-Line..

Thanks,

              Jim Uttaro

  _____  

Best Regards,

Zhenqiang Li

Date: 2022-02-04 20:39

Thank you, Susan Hares

---

From: "Wanghaibo (Rainsword)" <[email-masked]>
Date: Fri, 11 Feb 2022 03:37:46 +0000

Hi Robert,
       Although this is a local PE implementation problem, as far as I know, some vendors, such as Cisco and Huawei, can directly obtain RD information in VRF routes.

Regards,
Haibo
From: Idr [mailto:[email-masked]] On Behalf Of Robert Raszuk
Sent: Friday, February 11, 2022 5:42 AM
To: Jeffrey Haas <[email-masked]>
Cc: Jakob Heitz (jheitz) <jheitz=[email-masked]>; UTTARO, JAMES <[email-masked]>; [email-masked]; Sue Hares <[email-masked]>; [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

This leads to an interesting disconnect that for implementations I'm familiar with that the VRF view of things, where they have the desired limits, don't necessarily have backpointers to the state that let them know what the contributing RD is for the VRF installed routes.

The answer was that this is a local PE  implementation issue and the PE will just "figure it out" :) While simple to say and maybe not that hard to add a back pointer to src path there is per path cost associated with that. Leave alone additional memory and cycles to keep per src RD import counters.

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Feb 2022 10:06:55 +0100

Gyan,

RD is part of the prefix. It is not a standalone entity. Everywhere where
you can filter prefix you can also filter prefix /64 which is RD.

Thx,
R.

On Fri, Feb 11, 2022 at 1:27 AM Gyan Mishra <[email-masked]> wrote:

>> > The Crux of issue is the flood of unwanted routes by a rogue PE
>> sourcing the
>>> The issue is worse inter-as options b as all RTs have to be accepted
>>> with knob retain-rt-all on inter-as PE ASBR for option-b and RTs cannot be

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Feb 2022 10:09:16 +0100

>        Although this is a local PE implementation problem, as far as I
> know, some vendors, such as Cisco and Huawei, can directly obtain RD
> information in VRF routes.
>

That is not the point.

The point is that what is in a VRF is a local RD not src RD.

You need extra linkage now to remember the src RD when doing local import
== copy to a VRF.

Thx,
R.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 11 Feb 2022 07:27:55 -0500

Haibo,

>        Although this is a local PE implementation problem, as far as I know, some vendors, such as Cisco and Huawei, can directly obtain RD information in VRF routes.

For clarity, this is also possible in Juniper's implementation, but is not
currently a "natural" key.

The motivation for raising the point about telemetry for such a feature is
much like old SNMP semantics, if you force a key element in your telemetry,
you're putting specific types of burden on the implementation.

Abstractly you're now asking for not only per network-instance counters,
you're asking for a dynamically keyed for 8 bytes of arbitrary key
sub-counters.  It's not difficult code, but outside of the use case under
discussion, not something that makes a lot of sense in the L3VPN operational
architecture.

-- Jeff

---

From: "Wanghaibo (Rainsword)" <[email-masked]>
Date: Fri, 11 Feb 2022 13:43:40 +0000

Hi Jeff & Robert,

	I agree that to store the remote routes' RD we need extra linkage or other method. And it's not a natural key.
    However, RD information is useful for O&M. Eg, for the OptionB scenario, dual-homed PE may advertise same prefixes with different RD,
When they are imported to a specified VRF, the routes seems almost the same except the VPN label, which causes some difficulties for O&M person. If RD information is still stored in the route, O&M is more user-friendly.

Regards,
Haibo

-----Original Message-----
From: Jeffrey Haas [mailto:[email-masked]] 
Sent: Friday, February 11, 2022 8:28 PM
To: Wanghaibo (Rainsword) <[email-masked]>
Cc: Robert Raszuk <[email-masked]>; Jakob Heitz (jheitz) <jheitz=[email-masked]>; UTTARO, JAMES <[email-masked]>; [email-masked]; Sue Hares <[email-masked]>; [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Haibo,

>        Although this is a local PE implementation problem, as far as I know, some vendors, such as Cisco and Huawei, can directly obtain RD information in VRF routes.

For clarity, this is also possible in Juniper's implementation, but is not currently a "natural" key.

The motivation for raising the point about telemetry for such a feature is much like old SNMP semantics, if you force a key element in your telemetry, you're putting specific types of burden on the implementation.

Abstractly you're now asking for not only per network-instance counters, you're asking for a dynamically keyed for 8 bytes of arbitrary key sub-counters.  It's not difficult code, but outside of the use case under discussion, not something that makes a lot of sense in the L3VPN operational architecture.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Feb 2022 14:53:43 +0100

Haibo,

The issue is not that imported routes are stored without RDs. In most
implementations imported routes are still of the same VPNv4 trie.

The issue is that during import RD is overwritten to destination VRF or
VRFs (from the original one it arrives with on the PEs).

What you are asking is per path linkage of the imported path to the parent
path.

See operationally this is not needed as you can show command in the global
VPNv4 trie to display or count paths with just specific RT. So adding this
linkage between imported and original path is maybe not a rocket
science but opportunity for bugs and would need to be really agreed on that
it is both needed and it is a good idea.

I don't think we have reached that agreement yet.

Thx,
R.

On Fri, Feb 11, 2022 at 2:43 PM Wanghaibo (Rainsword) <
[email-masked]> wrote:

> Hi Jeff & Robert,
>
>         I agree that to store the remote routes' RD we need extra linkage
> or other method. And it's not a natural key.
>     However, RD information is useful for O&M. Eg, for the OptionB
> scenario, dual-homed PE may advertise same prefixes with different RD,
> When they are imported to a specified VRF, the routes seems almost the
> same except the VPN label, which causes some difficulties for O&M person.
> If RD information is still stored in the route, O&M is more user-friendly.
>
> Regards,
> Haibo
>
> Cc: Robert Raszuk <[email-masked]>; Jakob Heitz (jheitz) <jheitz=
> [email-masked]>; UTTARO, JAMES <[email-masked]>;
> [email-masked]; Sue Hares <[email-masked]>; [email-masked]
> discussion, not something that makes a lot of sense in the L3VPN
> operational architecture.
>
> -- Jeff
>
>

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 11 Feb 2022 11:52:40 -0500

Robert

As you mentioned in this thread about RD I am speaking of source RD,
however as you mentioned there is not a back pointer to the source RD as
once the VPNv4 prefix is imported the Source RD is overwritten by the Local
RD.

So then filtering is not possible on Source RD.

That can be a valuable feature for operators.

I know Cisco and Juniper have option to sort with a show command based on
any RD so I would think the back pointer must be there for the source RD
field but would have to be prior to the import.

Cisco XR
Pre import to VRF - Source RD intact
Show BGP vpnv4 unicast RD x.x.x.x:x.

Post import to VRF - Source RD is overwritten by local RD
Show BGP vpnv4 unicast VRF ABC

So the post import to VRF data structure seems to have  the back pointer to
the Any RD —> maybe it is possible to filter on any RD??

This is more of an implementation question but seems like the data is there
from what I can tell.

Kind Regards

Gyan

On Fri, Feb 11, 2022 at 4:07 AM Robert Raszuk <[email-masked]> wrote:

>>    Gyan> Understood.  It’s wanted routes explicitly imported that is an
>> an attack vector on the VPN
>>
>>>
>>>> sourcing the unwanted flooding of RT-2 or IP prefix flooding.
>>>>
>>>> The draft discusses why existing mechanisms PE-CE maximum prefix and
>>>> per VPN maximum prefix does not help this issue.
>>>>
>>>> Kind Regards
>>>>
>>>> Gyan
>>>>
>>>>> Even for the intra-AS scenario, as that discussed in the current
>>>>> draft, it is possible for the misconfiguration, or misbehavior of PE; or
>>>>> the increase of VPN prefixes in one VRF crashes some PEs in the network and

*Gyan Mishra*

*M 301 502-1347*

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 11 Feb 2022 18:04:06 +0100

> any RD so I would think the back pointer must be there

Not really - some implementations I have worked on in the past just store
all VPNv4/v6 routes pre and post import in the same data
structure (typically a trie).

So show command allows you to display routes with a given RD as it simply
walks the trie and picks those nets which have this RD prepended.

- - -

Frankly I am not quite clear if you zoom in what this draft is asking for.

The concern is that too much VPN routes got into a VRF. Well for it to get
into the VRF it needs first to be placed in the VPNv4 BGP table. I asked
the question if the concern is about control plane memory or CPU or rather
data plane memory - but got no answer.

If VRF has max prefix the issue described in the draft is mitigated. It it
does not have max prefix configured and we get much more then required
routes the entire PE may start malfunctioning or crash and if so nothing
including ORF will be send out of it.

The real enhancement which is needed is a proactive signalling - not to let
any issue to happen - not reactive ORF injected by potentially already
crashed PE.

Thx,
R.

On Fri, Feb 11, 2022 at 5:52 PM Gyan Mishra <[email-masked]> wrote:

> So the post import to VRF data structure seems to have  the back pointer
> to the Any RD —> maybe it is possible to filter on any RD??
>
> This is more of an implementation question but seems like the data is
> there from what I can tell.
>
> Kind Regards
>
> Gyan
>
>>>     Gyan> Unwanted is bad choice of words -  as long as the route
>>> matches the RT it’s imported wanted
>>> No protocol extension needed at all and immediate action can be much
>>>> more effective then just pushing some ORF to RRs.
>>>>
>>>
>>>>> What I was mentioning was the scenario with China Telecom and the
>>>>> other two providers related to their specific use case being EVPN related
>>>>> however this solution applies to both EVPN and IP VPN.  I believe they may
>>>>> had IP VPN issues as well as the issue as described below can be
>>>>> problematic and impact any operator.  This is not an operations or design
>>>>> issue.
>>>>>
>>>>> The issue with weak PE or PE being overwhelmed with the flood of
>>>>> routes as well is an issue and this draft addresses any of those possible
>>>>> scenarios.
>>>>>
>>>>>
>>>>> I mentioned EVPN  RT-2 as you may have 1000s of MAC/IP routes
>>>>> depending on how flat your subnet is per RT-5 so the numbers go up pretty
>>>>> quickly and easy to flood.  Also with
>>>>>

---

From: Gyan Mishra <[email-masked]>
Date: Wed, 16 Feb 2022 10:37:30 -0500

Hi Robert

Some answers in-line

Many Thanks

> structure (typically a trie).

    Gyan> Yep, Understood so overwritten and no way to preserve without a
back pointer to a new source RD field.

    Gyan> Understood

> data plane memory - but got no answer.

Gyan> The concern is Control plane memory being imported into the VRF as
well a pre import in the VPNv4 VPNv6 BGP table memory and CPU along with
data plane memory BGP messages queuing

> The real enhancement which is needed is a proactive signalling - not to
> let any issue to happen - not reactive ORF injected by potentially already
> crashed PE.
>

   Gyan> Section 4.1.1 describes the signaling used is based on VRF maximum
prefix set as the trigger to start the ORF process

https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf-01#section-4.1.1

   On PE1, each VRF has a set threshold, we assume it is 80% of Maximum
   in VRF1.

>>> RD is part of the prefix. It is not a standalone entity. Everywhere
>>> where you can filter prefix you can also filter prefix /64 which is RD.
>>>
>>> Thx,
>>> R.
>>>
>>>
>>>
>>>
>>>>>>> I don’t think there are many SRv6 deployments worldwide and I
>>>>>>> believe China is one of the leaders on proliferation of IPV6 by operators
>>>>>>>  as well as now SRv6.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> With the trending/deployment of EVPN/SRv6 technologies, such
>>>>>>> problems will be emerged  inevitably because it is more easy to
>>>>>>> deploy the inter-AS VPN services.
>>>>>>>
>>>>>>> And, there are at least three operators that have widespread
>>>>>>> networks have interested in the solution, I think it will also benefit your
>>>>>>> service deployment in future.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Best Regards
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Aijun Wang
>>>>>>>
>>>>>>> China Telecom
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *[Jim U>] Not true. Additional routing state will be throttled.
>>>>>>> There is a notification setting and a threshold setting. The only time this
>>>>>>> was encountered was when a customer mistakenly re-distributed the GRT into
>>>>>>> For the solution, I think it is difficult for the receiving router
>>>>>>> to decide which PE should be blocked when its VRF routes reach a
>>>>>>> threshould. Why not block any PE sending routes to the overflowing VRF?
>>>>>>>
>>>>>>>
>>>>>>> *Subject:* [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption
>>>>>>> *Subject:* [Idr] draft-wang-idr-vpn-prefix-orf-00.txt : (2/4/2022
>>>>>>> to 2/18/2022)
>>>>>>>
>>>>>>>
>>>>>>>

*Gyan Mishra*

*M 301 502-1347*

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 18 Feb 2022 09:30:45 -0500

Gyan,

There's two distinct discussion points here: Can you keep counters on routes per-RD+per-VRF, and having access to such a counter would you use it for an ORF.

I can tell you such a per-RD+per-VRF counter, although annoying to write, is something that could go into my implementation.  Implementors aren't fond of these types of counters because unlike the per-table counters, you're having to do filter operations and key lookups on route properties to maintain the counters rather than just keeping track of total tree contents.  

This is thus a general case of overhead.  Do you want to pay for the memory of the counter and the cost of indexing into the counter?

Picking one abstract model, for any trackable item you can have a counter built on PATRICIA for the key.  In this case, it's per-RD, per VRF.  If you implementation keeps per VRF resources, it's then the slightly cheaper just per-RD in that VRF. (Other people have different implementations.)

You're now paying for a PATRICIA lookup every time you add or delete a route from your main RIB entry.  You pay this tax during RIB maintenance operations.

It's perhaps little surprise that implementors of highly scalable implementations of BGP are wary of doing extra cost operations during RIB maintenance.

It's important to note the above is an example, and not necessarily the required implementation.  If you have belief that there's smaller numbers of RD in the system, you might choose a different lookup - say a hash table, balanced tree, etc.  The important detail is you're choosing to pay a per-RIB operation overhead for the counter.

Having chosen to pay the cost, is an ORF helpful?

Locally, having the counter, you can choose to also locally do discard.  The resource exhaustion issue is mitigated without an ORF and extra protocol machinery.

The remaining tax is the adjacent BGP router sending you extra routes you're going to throw away.

The unwanted routes do indeed "clog up" your peering session to the PE, but presumably even though excessive, they're probably stable.  Counters don't help you unless "too much" is also "stable enough to be too much".  After all, this feature isn't trying to fix route thrash.  Installing an ORF will actually generate more route updates because the adjacent router will need to send withdraws for all routes impacted by the ORF.

Having entered this fault condition where you're throwing stuff away, what is operationally expected?  Presumably with or without ORF, someone gets on the phone to tell the NOC to fix the route leak.

After the leak is fixed, you'd clear the block.  Whether the routes are recovered by withdrawing the ORF or doing a route refresh, you're still asking the adjacent router to do work.  Where is work being saved on that adjacent router?  An ORF is more filtering overhead for that router that's now per-peer rather than amortized over any form of its per-group work.

-- Jeff

> On Feb 11, 2022, at 11:52 AM, Gyan Mishra <[email-masked]> wrote:
> 
> Robert 
> 
> As you mentioned in this thread about RD I am speaking of source RD, however as you mentioned there is not a back pointer to the source RD as once the VPNv4 prefix is imported the Source RD is overwritten by the Local RD.
> 
> I know Cisco and Juniper have option to sort with a show command based on any RD so I would think the back pointer must be there for the source RD field but would have to be prior to the import. 
> 
> Cisco XR 
> So the post import to VRF data structure seems to have  the back pointer to the Any RD —> maybe it is possible to filter on any RD??
> 
> This is more of an implementation question but seems like the data is there from what I can tell.
> 
> Kind Regards 
> 
> Gyan
> 
> On Fri, Feb 11, 2022 at 4:07 AM Robert Raszuk <[email-masked] <mailto:[email-masked]>> wrote:
> Gyan,
> 
> > Gyan> no solution exists today to filter on RD automatically as all filtering is based on RT or prefix.
> 
> RD is part of the prefix. It is not a standalone entity. Everywhere where you can filter prefix you can also filter prefix /64 which is RD. 
> 
> Thx,
> R.
> 
> 
> 
> 
> On Fri, Feb 11, 2022 at 1:27 AM Gyan Mishra <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Robert 
> 
> On Thu, Feb 10, 2022 at 2:37 PM Robert Raszuk <[email-masked] <mailto:[email-masked]>> wrote:
> Gyan,
> 
> > The Crux of issue is the flood of unwanted routes by a rogue PE sourcing the 
> If this statement is really true then crux of the issue has been recognized, worked on and addressed in early 2000s. 
> 
>     Gyan> Unwanted is bad choice of words -  as long as the route matches the RT it’s imported wanted 
> - Agreed.  So it’s wanted routes explicitly imported that is an attack vector from a source PE for the VPN exactly what Aijun is describing.  
> 
> Only routes which are wanted get propagated in a properly designed network. 
> 
>     Gyan> Understood 
> 
> Hint: RFC4684
> 
> Now if you accept too many of VPN routes you can not treat them as unwanted.
> 
> It is like you would be throwing away passengers in the middle of the flight just because plane is running out of fuel. You need to be careful during boarding process to accept only as much as you can carry. As simple as this. 
> 
>    Gyan> Understood.  It’s wanted routes explicitly imported that is an an attack vector on the VPN
> 
> If you however consider problem statement Aijun is describing it seems that PE or RR which noticed some attack vector within one VPN 
> 
> could immediately report this to the mgmt station which in turn should immediately shut down the eBGP session from the CE or CEs which originate such attack. 
> 
> Gyan> The solution with using ORF  is an automated filtering capability with RD granularity so we can detect the source PE where the attack vector originated. 
> 
> No protocol extension needed at all and immediate action can be much more effective then just pushing some ORF to RRs. 
> 
> Gyan> no solution exists today to filter on RD automatically as all filtering is based on RT or prefix.
> 
> Thank you,
> R.
> 
> On Thu, Feb 10, 2022 at 7:13 PM Gyan Mishra <[email-masked] <mailto:[email-masked]>> wrote:
> Jim
> 
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>
> -- 
> https://www.ietf.org/mailman/listinfo/idr <https://www.ietf.org/mailman/listinfo/idr>

---

From: Aijun Wang <[email-masked]>
Date: Sat, 19 Feb 2022 07:30:22 +0800

Hi, Jeffrey:
I think you have rich experiences for the implementation of BGP features and there certainly exists some points to be overcome during the process. And any vendor can utilize their knobs to achieve this. Should we give such detail solutions spaces for the vendors’ decision and selection?
The main advantages of VPN Prefixes ORF mechanism is that it abates the side effect of overwhelming information, and also gives clues to operator to find the root causes of the abnormal results.

Aijun Wang
China Telecom

> 在 2022年2月18日，22:31，Jeffrey Haas <[email-masked]> 写道：
> 
> Gyan,
> 
>>>>>>> Email [email-masked]
>>>>>>> M 301 502-1347
>>>>>>> 
>>>>>>>  
>>>>>>> 
>>>>>> -- 
>>>>>> 
>>>>>> 
>>>>>> Gyan Mishra
>>>>>> Email [email-masked]
>>>>>> M 301 502-1347
>>>>>> 
>>>>>> 
>>>> Email [email-masked]
>>>> M 301 502-1347
>>>> 
>>>> 
>> -- 
>> 
>> 
>> Gyan Mishra
>> Email [email-masked]
>> M 301 502-1347
>> 
>>

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 19 Feb 2022 00:48:57 +0100

Hi Aijun,

I have an impression that we are going in circles. In the spirit to move
forward may I ask you precisely answer on behalf of all draft co-authors
the following questions - which still are not answered:

#1 - Is the concern of "overwhelmed PE" related to data plane memory or
control plane memory or both ?

#2 - If this is related to data plane memory what do you envision to be a
trigger for the action (ie. in this context to send RD ORF MSG out)

#3 - what do you expect to be the correct action where there is no single
RD src route significantly contributing to the overflow ?

#4 - iF you push ORF with filter you are required to get full route refresh
to remove unwanted RD NLRIs - do you have provision for this in the draft ?

#5 - if your concern is control plane memory overflow do you plan to
configure global or per vrf or none ingress max RD filter with counter ?

#6 - what happens if such counter #5 will be crossed when receiving RD with
from a peer who send just a few routes and elephants are already installed
?

#7 - if you do not have a plan to have any fixed max counter how are you
going to trigger generation of subject ORF messages ?

#8 - if #7 - how do you plan to recover if no such counters and due to
overflow PE crashes before generating any msg ?

#9 - and if there is counter what is the issue to drop excess (irrespective
if control plane wise or data plane wise) locally and generate alarm ?

I guess this should get us started to move this ping-pong thread on the
right realistic tracks,

Thx,
Robert

On Sat, Feb 19, 2022 at 12:30 AM Aijun Wang <[email-masked]>
wrote:

> Hi, Jeffrey:
> I think you have rich experiences for the implementation of BGP features
> and there certainly exists some points to be overcome during the process.
> And any vendor can utilize their knobs to achieve this. Should we give such
> detail solutions spaces for the vendors’ decision and selection?
> The main advantages of VPN Prefixes ORF mechanism is that it abates the
> side effect of overwhelming information, and also gives clues to operator
> to find the root causes of the abnormal results.
>
> Aijun Wang
> China Telecom
>
> There's two distinct discussion points here: Can you keep counters on
> routes per-RD+per-VRF, and having access to such a counter would you use it
> for an ORF.
>
> I can tell you such a per-RD+per-VRF counter, although annoying to write,
> is something that could go into my implementation.  Implementors aren't
> fond of these types of counters because unlike the per-table counters,
> you're having to do filter operations and key lookups on route properties
> to maintain the counters rather than just keeping track of total tree
> contents.
>
> This is thus a general case of overhead.  Do you want to pay for the
> memory of the counter and the cost of indexing into the counter?
>
> Picking one abstract model, for any trackable item you can have a counter
> built on PATRICIA for the key.  In this case, it's per-RD, per VRF.  If you
> implementation keeps per VRF resources, it's then the slightly cheaper just
> per-RD in that VRF. (Other people have different implementations.)
>
> You're now paying for a PATRICIA lookup every time you add or delete a
> route from your main RIB entry.  You pay this tax during RIB maintenance
> operations.
>
> It's perhaps little surprise that implementors of highly scalable
> implementations of BGP are wary of doing extra cost operations during RIB
> maintenance.
>
> It's important to note the above is an example, and not necessarily the
> required implementation.  If you have belief that there's smaller numbers
> of RD in the system, you might choose a different lookup - say a hash
> table, balanced tree, etc.  The important detail is you're choosing to pay
> a per-RIB operation overhead for the counter.
>
> Locally, having the counter, you can choose to also locally do discard.
> The resource exhaustion issue is mitigated without an ORF and extra
> protocol machinery.
>
> The unwanted routes do indeed "clog up" your peering session to the PE,
> but presumably even though excessive, they're probably stable.  Counters
> don't help you unless "too much" is also "stable enough to be too much".
> After all, this feature isn't trying to fix route thrash.  Installing an
> ORF will actually generate more route updates because the adjacent router
> will need to send withdraws for all routes impacted by the ORF.
>
> Having entered this fault condition where you're throwing stuff away, what
> is operationally expected?  Presumably with or without ORF, someone gets on
> the phone to tell the NOC to fix the route leak.
>
> After the leak is fixed, you'd clear the block.  Whether the routes are
> recovered by withdrawing the ORF or doing a route refresh, you're still
> asking the adjacent router to do work.  Where is work being saved on that
> adjacent router?  An ORF is more filtering overhead for that router that's
> now per-peer rather than amortized over any form of its per-group work.
>
> -- Jeff
>
>
>>>>>> *[Jim U>] Have you canvassed operators? I would be interested in
>>>>>> the “problems” that have been identified in networks today. Pls provide
>>>>>> *[Jim U>] Have you canvassed operators? I would be interested in
>>>>>> the “problems” that have been identified in networks today. Pls provide

---

From: Aijun Wang <[email-masked]>
Date: Sat, 19 Feb 2022 10:07:17 +0800

Hi, Robert:
Thanks for your detail considerations.
Please see my considerations/replies inline.

Aijun Wang
China Telecom

> 在 2022年2月19日，07:49，Robert Raszuk <[email-masked]> 写道：
> 
> 
> Hi Aijun,
> 
> I have an impression that we are going in circles. In the spirit to move forward may I ask you precisely answer on behalf of all draft co-authors the following questions - which still are not answered: 
> 
> #1 - Is the concern of "overwhelmed PE" related to data plane memory or control plane memory or both ? 
[WAJ] Mainly to relieve the pressure of control plane. Not only memory, but also the CPU utilization—-If the PE is busy dealing with the parse of overwhelming BGP updates, it will influence its capabilities to parse other normal BGP Updates, and the control plane pressure can certainly influence the data plane convergence speed/performance.
> 
> #2 - If this is related to data plane memory what do you envision to be a trigger for the action (ie. in this context to send RD ORF MSG out)
> 
> #3 - what do you expect to be the correct action where there is no single RD src route significantly contributing to the overflow ? 
[WAJ] In such situation, the decision algorithm should select the most recent ones. Other criteria may be added/considered, depending on the “ML” model that the PE adopts.
> 
> #4 - iF you push ORF with filter you are required to get full route refresh to remove unwanted RD NLRIs - do you have provision for this in the draft ?
[WAJ] Just try to follow the guidelines of RFC5291. If necessary, we can add more descriptions in the action of receiver side (https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf-01#section-5) later.
> 
> #5 - if your concern is control plane memory overflow do you plan to configure global or per vrf or none ingress max RD filter with counter ? 
[WAJ]The global prefixes limit should be configured to control the behavior of the shared BGP session. The per VRF prefixes limit should also be configured. This value is also the base for calculating the offending VPN Prefixes.
> 
> #6 - what happens if such counter #5 will be crossed when receiving RD with from a peer who send just a few routes and elephants are already installed ? 

[WAJ] Focus mainly the newly added routes. Then in such situations, the existing elephant routes/traffic shouldn’t be affected.

> 
> #7 - if you do not have a plan to have any fixed max counter how are you going to trigger generation of subject ORF messages ? 
[WAJ] Have the counter.
> 
> #8 - if #7 - how do you plan to recover if no such counters and due to overflow PE crashes before generating any msg ? 
> 
> #9 - and if there is counter what is the issue to drop excess (irrespective if control plane wise or data plane wise) locally and generate alarm ? 
[WAJ] We have discussed this point previously. I think it is same as the benefits of ORF mechanism, when comparing with the local throw away.
> 
> I guess this should get us started to move this ping-pong thread on the right realistic tracks, 
[WAJ] Will try to incorporate some of them in the “Implementation  Considerations” part, together with your previous suggestions.
Thanks for your comments/suggestions! Let’s move to such direction together.
> 
> Thx,
> Robert
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>> On Sat, Feb 19, 2022 at 12:30 AM Aijun Wang <[email-masked]> wrote:
>> Hi, Jeffrey:

---

From: Gyan Mishra <[email-masked]>
Date: Sat, 19 Feb 2022 02:18:22 -0500

Jeffrey

Responses In-line

Lots of good points ORF versus implementation of per RD+per VRF counter.

> routes per-RD+per-VRF, and having access to such a counter would you use it

Gyan> Good points agreed on if it’s possible to #1 keep the counters per
RD+per VRF #2 access the counters for ORF

> to maintain the counters rather than just keeping track of total tree

    Gyan> Understood.  So not just the table level but now route level key
lookup.  More granularity and complexity on route property keys such as for
storing now the local and source RD (back pointer) so the source RD is not
overwritten by local RD when the route is imported.

    Gyan> Understood

    Gyan> Understood

    Gyan> Understood

    Gyan> Understood

    Gyan> Understood

> Having chosen to pay the cost, is an ORF helpful?

    Gyan>

> The resource exhaustion issue is mitigated without an ORF and extra

    Gyan> I like it however as you mentioned there are  trade off with both
options

Gyan> Understood

  Gyan> Understood

> is operationally expected?
>

Gyan> Recovery of the overloaded node is more critical than dropped routes

Presumably with or without ORF, someone gets on the phone to tell the NOC
> to fix the route leak.

Gyan> Yep agreed

>
> After the leak is fixed, you'd clear the block.
>

    Gyan> Yes manual clear of block

Whether the routes are recovered by withdrawing the ORF or doing a route
> refresh, you're still asking the adjacent router to do work.  Where is work
> being saved on that adjacent router?
>

Gyan> Yea but that normal operation with ORF where adjacent router in this
case the RR sets adj-rib-out towards PE overflowing and recovers the PE
from the fault.

An ORF is more filtering overhead for that router that's now per-peer
> rather than amortized over any form of its per-group work.
>

Gyan>. That’s the trade off our ORF solution P2P filtering overhead where
the RD back pointer programming is possible as you  described but there is
definitely a cost as well as tax hit on the overall control plane
processing and performance hit.

*Gyan Mishra*

*M 301 502-1347*

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 19 Feb 2022 11:47:35 +0100

See inline ...

On Sat, Feb 19, 2022 at 3:07 AM Aijun Wang <[email-masked]>
wrote:

> [WAJ] Mainly to relieve the pressure of control plane. Not only memory,
> but also the CPU utilization—-If the PE is busy dealing with the parse of
> overwhelming BGP updates, it will influence its capabilities to parse other
> normal BGP Updates, and the control plane pressure can certainly influence
> the data plane convergence speed/performance.
>

[RR]  So now we are talking about two catastrofic events together - someone
purpusly or by accident injects lots of routes to a VPN and in addition he
makes sure they flap all the time - as if not the filtering is just one
time thing even if consuming some resources from PEs.

And what seems the worse part instead of detecting such thing at the src
you are asking all PE involved in this VPN to detect it, then run some form
of Machine Learning algorithm then push this info to RRs etc ...

Let's also not forget that dropping what is not needed has been day one key
behaviour in all implementations. So running BGP I/O and parsing a value
from the update to decide to take or drop the route has been very cheap
operation and most implementations handle it very well. Again let's keep in
mind that BGP does not retransmit periodically. It is done once and if
network is stable never again till you loose/reset the session.

> [WAJ] In such situation, the decision algorithm should select the most
> recent ones. Other criteria may be added/considered, depending on the “ML”
> model that the PE adopts.
>

[RR]  Very cool ! Indeed BGP FSM will be highly enriched by some ML !  But
perhaps it would be helpful to describe this first as a separate draft if
you care about consistent multivendor behaviour ?

> #4 - iF you push ORF with filter you are required to get full route
> refresh to remove unwanted RD NLRIs - do you have provision for this in the
> draft ?
>
> [WAJ] Just try to follow the guidelines of RFC5291. If necessary, we can
> add more descriptions in the action of receiver side (
> https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf-01#section-5)
> later.
>
>
> [WAJ]The global prefixes limit should be configured to control the
> behavior of the shared BGP session. The per VRF prefixes limit should also
> be configured. This value is also the base for calculating the offending
> VPN Prefixes.
>

[RR] Please observe that if you have per VRF prefix limit in the first
place the issue described in the draft will not occur. Reason being that
there will be no src VRF anywhere where you can originate an excessive
number of routes. And if routes are not in the src VRF they do not get
advertised over BGP to any other peer.

> [WAJ] Focus mainly the newly added routes. Then in such situations, the
> existing elephant routes/traffic shouldn’t be affected.
>

[RR]  Interesting ML ... I have been chatting with Jeff offline and
thought the complete opposite would make more sense.

Best,
R.

> #9 - and if there is counter what is the issue to drop excess
> (irrespective if control plane wise or data plane wise) locally and
> generate alarm ?
>
> [WAJ] We have discussed this point previously. I think it is same as the
> benefits of ORF mechanism, when comparing with the local throw away.
>
>
> [WAJ] Will try to incorporate some of them in the “Implementation
>  Considerations” part, together with your previous suggestions.
>> Having entered this fault condition where you're throwing stuff away,
>> what is operationally expected?  Presumably with or without ORF, someone
>> gets on the phone to tell the NOC to fix the route leak.
>>

---

From: Aijun Wang <[email-masked]>
Date: Sat, 19 Feb 2022 20:36:51 +0800

Hi, Robert:

Aijun Wang
China Telecom

> 在 2022年2月19日，18:47，Robert Raszuk <[email-masked]> 写道：
> 
> 
> See inline ...
> 
>> On Sat, Feb 19, 2022 at 3:07 AM Aijun Wang <[email-masked]> wrote:
>> Hi, Robert:
> [RR]  So now we are talking about two catastrofic events together - someone purpusly or by accident injects lots of routes to a VPN and in addition he makes sure they flap all the time - as if not the filtering is just one time thing even if consuming some resources from PEs.
> 
> And what seems the worse part instead of detecting such thing at the src you are asking all PE involved in this VPN to detect it, then run some form of Machine Learning algorithm then push this info to RRs etc ... 

[WAJ] If we think the “VPN Prefix ORF” is one protection mechanism for each PE, then it is may be more easily to be accepted.

> 
> Let's also not forget that dropping what is not needed has been day one key behaviour in all implementations. So running BGP I/O and parsing a value from the update to decide to take or drop the route has been very cheap operation and most implementations handle it very well. Again let's keep in mind that BGP does not retransmit periodically. It is done once and if network is stable never again till you loose/reset the session. 

[WAJ] Yes. We are trying to avoid the cathatrachic BGP updates. 

> [RR]  Very cool ! Indeed BGP FSM will be highly enriched by some ML !  But perhaps it would be helpful to describe this first as a separate draft if you care about consistent multivendor behaviour ? 

[WAJ] I think there maybe several factors to be considered to find the offending VPN routes. The final results is the local decision of each PE and needn’t be consistent across the network. Some guidelines can be added in the implementation considerations parts based on the futures test scenarios/results.

> [RR] Please observe that if you have per VRF prefix limit in the first place the issue described in the draft will not occur. Reason being that there will be no src VRF anywhere where you can originate an excessive number of routes. And if routes are not in the src VRF they do not get advertised over BGP to any other peer. 
[WAJ] I think this is the valid point. But you can’t depend solely the right behaviors of other peers. Some protection mechanism should be deployed in advance.
> [RR]  Interesting ML ... I have been chatting with Jeff offline and thought the complete opposite would make more sense. 
[WAJ] Currently we considered to find the offending routes from mainly the newly updated BGP routes when the threshold is passed.  Maybe this can be configured by the operator later. That is to say, the operator can give their preferences to influence the results of ML.

---

From: Jeffrey Haas <[email-masked]>
Date: Sat, 19 Feb 2022 15:21:55 -0500

> On Feb 19, 2022, at 2:18 AM, Gyan Mishra <[email-masked]> wrote:
> 
> Whether the routes are recovered by withdrawing the ORF or doing a route refresh, you're still asking the adjacent router to do work.  Where is work being saved on that adjacent router?  
> 
> Gyan> Yea but that normal operation with ORF where adjacent router in this case the RR sets adj-rib-out towards PE overflowing and recovers the PE from the fault.

... by sending more messages.

If the goal is to reduce load on the receiving PE in these situations in terms of messages, an ORF will do the opposite.

There seems to be a perverse assumption of spontaneous behavior because you've waved the ORF magic wand.

If your PE tripped past its prefix limit because there's currently a stream of 850K of leaked Internet coming in from one RD, and the transmitting device gets an ORF, it will take a while before that's processed just like any other configuration state change.  Perhaps you get the entire 850k routes before it's able to process the ORF.

And then you get 850k withdraws.

That you then later have to clear to get the N routes you were eventually asking for.

If you're lucky, the ORF may take effect sooner than that.  But there's zero guarantee.  This means that during the time between requesting the ORF to take effect and it having taken effect, you'll want to locally mitigate.  

And that's throwing out the routes locally anyway.

-- Jeff

---

From: Aijun Wang <[email-masked]>
Date: Sun, 20 Feb 2022 07:20:46 +0800

Hi, Jeffrey:
I think the concerns you raised is similar with the questions of what the convergence speed of BGP protocol in each device.
Actually there is possible the peer is slow to process the ORF message, although you tell it “IMMEDIATE” via the “when-to-refresh” field. But this is not the fault of protocol, it’s the fault of peering device.
And, in reality, in your situations, I think some of the 850K routes may leaked, but not possible all of them are passed before the ORF take effect. 

Aijun Wang
China Telecom

> 在 2022年2月20日，04:22，Jeffrey Haas <[email-masked]> 写道：
> 
> 
>

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 21 Feb 2022 22:42:23 -0500

Hi Jeffrey

On Sat, Feb 19, 2022 at 3:21 PM Jeffrey Haas <[email-masked]> wrote:

> If the goal is to reduce load on the receiving PE in these situations in
> terms of messages, an ORF will do the opposite.
>

    Gyan> It’s more of a timing related when the ORF is triggered.  Once
the PE hits the VPN prefix maximum threshold of 80%, the ORF is triggered
for the Source RD.  Using the example of a 850k internet table route leak.
Let’s say 50% of the routes were imported and the remaining 50% were
dropped from the VRF due to maximum prefix threshold, however the CPU and
memory resources are still climbing due to having to parse the BGP
updates.  Once the RD ORF is sent the RR installs the adj-rib-out towards
the PE and now all 850k routes are withdrawn, withdraw  messaging sent to
 the PE.  As soon as the PE has processed all 850k withdrawal messages,
memory and CPU are back down to normal levels post route leak.  During that
brief time interval you will see a blip on the CPU hit but then will be
completely recovered.  On the overloaded PE the goal is to get the BGP RIB
for the overflowing VRF back to normal and the only way to normalize the PE
is via signaling withdrawal of the leaked routes via RD ORF.   The problem
is completely mitigated for the overflowing PE once the withdraw messages
are all processed.  If nothing were done the PE VRF would remain in the
broken state unresolved.

> you've waved the ORF magic wand.

Gyan> We do acknowledge that there will be a CPU hit for the 850k leaked
withdrawal messages but that is only from time T1 when ORF is sent to when
the PE has processed all the withdrawal messages at time T2 at which time
the problem is 100% mitigated. So the CPU blip is only between time T1 and
T2.

>
> If your PE tripped past its prefix limit because there's currently a
> stream of 850K of leaked Internet coming in from one RD, and the
> transmitting device gets an ORF, it will take a while before that's
> processed just like any other configuration state change.  Perhaps you get
> the entire 850k routes before it's able to process the ORF.
>

Gyan> So the overflowing PE maybe delayed sending the ORF but as soon as
the RR gets the ORF the adj-rib-out which should be no different then a
prefix ORF which is immediate, however it may take time to send all the
850k withdrawal and it will take time for the overflowing PE to process all
the 850k withdrawals.  However once PE has processed all 850k withdrawals
the problem is 100% mitigated.  I don’t think we are adding fuel to the
fire by added messages for the 850k withdrawal message processing as once
processed the problem is resolved.  I agree that the overwhelmed PE may
take a while to process all the withdrawal messages maybe at a slower pace
as it is already running low on CPU resources, but by time T2 he will
eventually be fully recovered.  Problem solved.

So the trade off is do nothing and wait for the NOC to maybe reload the PE
or find some other intrusive method to recover from the problem state or
you use the VPN ORF automated method and recover the box.

> And then you get 850k withdraws.

Gyan> As soon as the RR gets the ORF it will install the adj-rib-out and
the 850k withdrawal will be sent to the PE to process the withdrawal.

> If you're lucky, the ORF may take effect sooner than that.  But there's
> zero guarantee.  This means that during the time between requesting the ORF
> to take effect and it having taken effect, you'll want to locally mitigate.
>
>

Gyan> The ORF being sent maybe slow by the PE but once finally sent, the RR
gets it and now the ball is rolling and the PE will get the 850k
 withdrawal messages and eventually the problem is resolved.

> And that's throwing out the routes locally anyway.

Gyan> The RD ORF is a method of locally mitigating no different then
applying a local filter to block the source RD route leak, but I agree you
avoid the 850k withdrawal message processing, however the local filter
 requires manual intervention.  I think it would be difficult to  automate
a local protection mechanism.

*Gyan Mishra*

*M 301 502-1347*

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Feb 2022 10:39:06 +0100

All,

> Once the PE hits the VPN prefix maximum threshold of 80%

Since authors confirmed that there is going to be VPN prefix max threshold
configured (as opposed to some local magic and watching depletion of local
PE's resources) then as I already stated there is *NO PROBLEM TO BE SOLVED*
.

Reason being is that VPN max prefix will also apply on the ingress PE and
therefore no excessive routes will be advertised in the first place
into the network.

Any decent SP (or for that matter even ISP) when you sign a VPN or IP
service will ask you to not only state in writing the number of expected
prefixes but in many cases will also ask to enumerate them.

So under assumption that there is VPN prefix max limit possibly with lower
warning threshold this draft has no use case.

- - -

One could potentially say that solutions like this may still be needed when
there is no max VPN prefix configured. Maybe this justification would work
but it requires solid description on what the trigger should be to
signal any additional filters.

Finally I think the approach taken by authors to drop randomly VPN prefixes
here and there is an operational nightmare and should be strongly
discouraged. Instead perhaps dropping all routes from a given VPN and
declaring failure to deliver service would be much better option allowing
to still deliver VPN for other tenants. And for that using RTC seems
trivial to implement in any network.

I do not support adoption of this draft in the current form.

Regards,
Robert

On Tue, Feb 22, 2022 at 4:42 AM Gyan Mishra <[email-masked]> wrote:

>> Gyan> Yea but that normal operation with ORF where adjacent router in
>> this case the RR sets adj-rib-out towards PE overflowing and recovers the
>> PE from the fault.
>>
>>
> Gyan> The ORF being sent maybe slow by the PE but once finally sent, the
> RR gets it and now the ball is rolling and the PE will get the 850k

---

From: Aijun Wang <[email-masked]>
Date: Tue, 22 Feb 2022 18:38:46 +0800

Hi, Robert:
It seems that we back to the origin again.
I have answered some issues that you raised. Let me answer them again.

Aijun Wang
China Telecom

> 在 2022年2月22日，17:40，Robert Raszuk <[email-masked]> 写道：
> 
> 
> All,
> 
> Since authors confirmed that there is going to be VPN prefix max threshold configured (as opposed to some local magic and watching depletion of local PE's resources) then as I already stated there is NO PROBLEM TO BE SOLVED. 
> 
> Reason being is that VPN max prefix will also apply on the ingress PE and therefore no excessive routes will be advertised in the first place into the network. 
> Any decent SP (or for that matter even ISP) when you sign a VPN or IP service will ask you to not only state in writing the number of expected prefixes but in many cases will also ask to enumerate them. 
> 
> So under assumption that there is VPN prefix max limit possibly with lower warning threshold this draft has no use case. 

[WAJ] As stated before, this is one kind of PROTECTION mechanism, you shouldn’t only depend on solely on the behavior of other end.

> 
> - - - 
> 
> One could potentially say that solutions like this may still be needed when there is no max VPN prefix configured. Maybe this justification would work but it requires solid description on what the trigger should be to signal any additional filters. 
[WAJ]There has been lot of discussions on this point. We have mentioned many times the bad behaved of the offending PE. It’s not the time to discuss it again.
> 
> Finally I think the approach taken by authors to drop randomly VPN prefixes here and there is an operational nightmare and should be strongly discouraged. Instead perhaps dropping all routes from a given VPN and declaring failure to deliver service would be much better option allowing to still deliver VPN for other tenants. And for that using RTC seems trivial to implement in any network. 

[WAJ] There is no randomly drop. The offending VPN prefixes is identified by the RD,RT or other source information.
We have discussed intensively that the RTC is not appropriate for the current scenario. 

>>>     Gyan> It’s more of a timing related when the ORF is triggered.  Once the PE hits the VPN prefix maximum threshold of 80%, the ORF is triggered for the Source RD.  Using the example of a 850k internet table route leak.  Let’s say 50% of the routes were imported and the remaining 50% were dropped from the VRF due to maximum prefix threshold, however the CPU and memory resources are still climbing due to having to parse the BGP updates.  Once the RD ORF is sent the RR installs the adj-rib-out towards the PE and now all 850k routes are withdrawn, withdraw  messaging sent to  the PE.  As soon as the PE has processed all 850k withdrawal messages, memory and CPU are back down to normal levels post route leak.  During that brief time interval you will see a blip on the CPU hit but then will be completely recovered.  On the overloaded PE the goal is to get the BGP RIB for the overflowing VRF back to normal and the only way to normalize the PE is via signaling withdrawal of the leaked routes via RD ORF.   The problem is completely mitigated for the overflowing PE once the withdraw messages are all processed.  If nothing were done the PE VRF would remain in the broken state unresolved.
>>> 
>> Gyan> We do acknowledge that there will be a CPU hit for the 850k leaked withdrawal messages but that is only from time T1 when ORF is sent to when the PE has processed all the withdrawal messages at time T2 at which time the problem is 100% mitigated. So the CPU blip is only between time T1 and T2.
>>> 
>> Gyan> So the overflowing PE maybe delayed sending the ORF but as soon as the RR gets the ORF the adj-rib-out which should be no different then a prefix ORF which is immediate, however it may take time to send all the 850k withdrawal and it will take time for the overflowing PE to process all the 850k withdrawals.  However once PE has processed all 850k withdrawals the problem is 100% mitigated.  I don’t think we are adding fuel to the fire by added messages for the 850k withdrawal message processing as once processed the problem is resolved.  I agree that the overwhelmed PE may take a while to process all the withdrawal messages maybe at a slower pace as it is already running low on CPU resources, but by time T2 he will eventually be fully recovered.  Problem solved.
>> 
>> So the trade off is do nothing and wait for the NOC to maybe reload the PE or find some other intrusive method to recover from the problem state or you use the VPN ORF automated method and recover the box.
>>> 
>> Gyan> As soon as the RR gets the ORF it will install the adj-rib-out and the 850k withdrawal will be sent to the PE to process the withdrawal.
>>> 
>> Gyan> The ORF being sent maybe slow by the PE but once finally sent, the RR gets it and now the ball is rolling and the PE will get the 850k  withdrawal messages and eventually the problem is resolved.
>>> 
>> Gyan> The RD ORF is a method of locally mitigating no different then applying a local filter to block the source RD route leak, but I agree you avoid the 850k withdrawal message processing, however the local filter  requires manual intervention.  I think it would be difficult to  automate a local protection mechanism.
>>> 
>>> -- Jeff
>>> 
>> -- 
>> 
>> 
>> Gyan Mishra

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 22 Feb 2022 07:43:32 -0500

Gyan,

> a local protection mechanism.

Are you making the assumption that your trigger mechanism couldn't simply
cause your implementation to discard routes?

If so... that's weird.

A prefix-limit can certainly just start discarding routes that are out of
bounds.  Consider Juniper's "drop-excess":

https://www.juniper.net/documentation/us/en/software/junos/bgp/topics/ref/statement/prefix-limit-edit-protocols-bgp.html

I typically do not recommend customers rely on this form of the
configuration.  It leads to less than deterministic behaviors and has the
issues of forwarding problems when more specific prefixes are being discarded
in the presence of less specific ones.

But much like the problem statement in your I-D, sometimes it's necessary.

-- Jeff

---

From: Aijun Wang <[email-masked]>
Date: Tue, 22 Feb 2022 21:14:39 +0800

Hi, Jeffrey:
I think there are some misunderstanding about the VPN Prefix ORF mechanism.

As stated in https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf-01#section-4.1.1, once the threshold is reached(for example, 80% of the max routes in one VRF), the PE will begin to monitor the upcoming VPN routes. 
And, as you assumed, if in such time, the 850k routes are coming to this PE from the RR.
The PE will trigger and send out the VPN Prefixes ORF mechanism upon it receives the 20% of the VRF routes limits. This value will be far less than your 850k.(We call this time is T1)
When the RR receives such messages, it will act immediately the filter.(We call this time is T2). The most part of the 850k routes will be filtered at the RR, thus release the PE to parse all of them.
The only passed routes are carried within the BGP Updates from T1 to T2. These are also the routes that should be withdrawn from the PE. Its value will also be far less than the 850k.
It won’t also require the local discard of 850k routes. 
Isn’t this the value of ORF mechanism?

We have stated several times that local discard doesn’t release the PE from processing the overwhelmed routes.

Aijun Wang
China Telecom

> 在 2022年2月22日，20:44，Jeffrey Haas <[email-masked]> 写道：
> 
> Gyan,
>

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Feb 2022 14:28:33 +0100

Aijun,

Consider a complete different solution.

In normal state push to RR from each PE (perhaps even using your VPN ORF
msg) information what is max number of routes you as the PE is willing to
accept with list of import RTs ?

That way RR when generating updates to PE can keep a counter and if this
counter reaches 80% generate a warning, but when it crosses 100% either
stops advertising anything with those RTs to PE or withdraws already
advertised routes with such RTs ?

That way PE is not overloaded ever. And the logic on RRs in fact is pretty
simple as this counter and per peer advertise flag is already present in
BGP in a number of implementations.

But It is clear you are closed on any alternative. You want to push yours
no matter what WG members are telling you. That's not good.

Thx,
R.

On Tue, Feb 22, 2022 at 2:14 PM Aijun Wang <[email-masked]>
wrote:

> As stated in
> https://datatracker.ietf.org/doc/html/draft-wang-idr-vpn-prefix-orf-01#section-4.1.1,
> once the threshold is reached(for example, 80% of the max routes in one
> VRF), the PE will begin to monitor the upcoming VPN routes.
> The PE will trigger and send out the VPN Prefixes ORF mechanism upon it
> receives the 20% of the VRF routes limits. This value will be far less than
> your 850k.(We call this time is T1)
> When the RR receives such messages, it will act immediately the filter.(We
> call this time is T2). The most part of the 850k routes will be filtered at
> the RR, thus release the PE to parse all of them.
> The only passed routes are carried within the BGP Updates from T1 to T2.
> These are also the routes that should be withdrawn from the PE. Its value
> will also be far less than the 850k.

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 22 Feb 2022 08:38:30 -0500

Aijun,

> I think there are some misunderstanding about the VPN Prefix ORF mechanism.

I have no such confusion.

I believe you have confused beliefs about how ORFs are implemented.

An ORF simply lets a piece of BGP policy be sent to a remote BGP Speaker.

This is only slightly different than if an operator provisioned an export
filter using configuration for that peer.  Such configuration state changes
are not usually immediate in their impact.  There will be lag between
reception of the ORF and its effect.

It is not possible for the sender of the ORF to know what the response
latency will be for processing the ORF.

If the originator of the ORF is about to suffer critical resource
exhaustion, it can't rely on this response latency.  It must locally
mitigate.

Mitigating RIB overload is a reasonable case.

Mitigating overhead from messaging load is a reasonable concern.  Your
vendors will have a good sense for such overhead because a common
performance test case for BGP Implementations is to receive routes at speed
and discard them as fast as possible.

In the presence of a reasonably stable Adj-Rib-Out from an upstream BGP
Speaker, there is a simple calculus for this feature:

- If the concern is minimizing RIB impact, local mitigation is a requirement
  due to response lag from ORF installation and processing.
- If the concern is minimizing messaging load, is the combined messaging
  load after the ORF has been sent and begins to be processed by the
  upstream BGP Speaker lower than if the upstream speaker simply completed
  sending the impactful routes and they were locally mitigated?
  + To say yes with confidence, you'd have to bound the time from ORF
    transmission to full processing.  I don't believe you can do this.

-- Jeff

---

From: Aijun Wang <[email-masked]>
Date: Tue, 22 Feb 2022 23:37:38 +0800

Hi, Robert: 
Thanks for your attractive proposal.
I think it is feasible, but it requires the RR doing the following things:
1: Records all the import RTs for each VRF on each PE.
2: Counter each VPN routes for each VRF on each PE based on the recording information in 1).
Compares with the current distributed solutions(each PE does itself calculation), your proposal is one kind of centralized solution.
And considering that not all of VRFs will be overflowed, the RR does much of useless work, right?
It seems the on demand work is more efficient?

And we are hearing carefully your every comment/suggestions. Some of them has been incorporated into the updated draft(for example, from the RD only filter, to RD+RT+Source PE information etc.)
We are open to any alternative solutions/proposals and I think they can be incorporated into the document even after its adoption if they can increase the solution efficiency.

Aijun Wang
China Telecom

> 在 2022年2月22日，21:28，Robert Raszuk <[email-masked]> 写道：
> 
> 
> Aijun,
> 
> In normal state push to RR from each PE (perhaps even using your VPN ORF msg) information what is max number of routes you as the PE is willing to accept with list of import RTs ? 
> 
> That way RR when generating updates to PE can keep a counter and if this counter reaches 80% generate a warning, but when it crosses 100% either stops advertising anything with those RTs to PE or withdraws already advertised routes with such RTs ? 
> 
> That way PE is not overloaded ever. And the logic on RRs in fact is pretty simple as this counter and per peer advertise flag is already present in BGP in a number of implementations. 
> 
> But It is clear you are closed on any alternative. You want to push yours no matter what WG members are telling you. That's not good. 
> 
> Thx,
> R.
> 
>> On Tue, Feb 22, 2022 at 2:14 PM Aijun Wang <[email-masked]> wrote:
>> Hi, Jeffrey:

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Feb 2022 16:47:24 +0100

Aijun,

RRs have no notion of VRFs. I have not suggested that RRs should develop
one.

The fundamental flaw in your proposal is VPN centric view and hence RD as
main part of the filter when you need to realize that  what constitutes L2
and L3 VPNs are elastic use of RTs.

Once you depart from notion of VPN to more basic RT based building block
you realize that such protection is not only much more granular but also
very scalable.

So in my suggested case RR or even remote PEs (we could define a way to
propagate this maybe in flowspec v2) all what is needed is to keep min max
route per RT counter when making decisions about issuing warning or
stopping to propage given routes around.

And nothing is centralized here. Every element of L2 or L3 VPNs is
centralized so are the RRs.

Such shift of direction is not to be done post WG adoption. This is
fundamental enough that new draft is needed and once adopted we could
discuss details and do a little fixes if WG members agree on the direction.

See WG adoption is probing for rough consensus to make sure what we adopt
at least moves into the right direction. It would be a failure to any WG to
adopt a document which fundamentally defines something which is hard to
implement or which goes into wrong direction.

Thx,
R.

On Tue, Feb 22, 2022 at 4:37 PM Aijun Wang <[email-masked]>
wrote:

> Compares with the current distributed solutions(each PE does itself
> calculation), your proposal is one kind of centralized solution.
> And we are hearing carefully your every comment/suggestions. Some of them
> has been incorporated into the updated draft(for example, from the RD only
> filter, to RD+RT+Source PE information etc.)
> We are open to any alternative solutions/proposals and I think they can be
> incorporated into the document even after its adoption if they can increase
> the solution efficiency.
>
> Aijun Wang
> China Telecom
>
>> When the RR receives such messages, it will act immediately the
>> filter.(We call this time is T2). The most part of the 850k routes will be
>> filtered at the RR, thus release the PE to parse all of them.

---

From: Aijun Wang <[email-masked]>
Date: Tue, 22 Feb 2022 23:58:40 +0800

Hi, Jeffrey:
There certainly will be some time from the ORF msg is sent out to it takes effect on the peer.
This is likely also the BGP convergence issues, as I said before.
The lag duration depends mainly on the efficiency of the RR.
And as I stated according to your 850k routes example, the routes leaked to the PE during this lag duration is just small part of the offending routes.
There may be some critical condition, for example, the RR can’t respond to the ORF message, or responses very slowly, then the effect of this mechanism is decreased. But I think this not the fault of this mechanism. We can also define some guidelines or requirements on the behavior of the VPN Prefixes ORF.

Aijun Wang
China Telecom

> 在 2022年2月22日，21:38，Jeffrey Haas <[email-masked]> 写道：
> 
> Aijun,
>

---

From: Aijun Wang <[email-masked]>
Date: Wed, 23 Feb 2022 00:05:38 +0800

Hi, Robert:
If you don’t counter the VPN routes on per VRF per PE base, how can you action on each VRF Prefix limit?
I know RR doesn’t keep the VRF states now, but your proposal is requiring it.

Aijun Wang
China Telecom

> 在 2022年2月22日，23:47，Robert Raszuk <[email-masked]> 写道：
> 
> 
> Aijun,
> 
> RRs have no notion of VRFs. I have not suggested that RRs should develop one. 
> 
> The fundamental flaw in your proposal is VPN centric view and hence RD as main part of the filter when you need to realize that  what constitutes L2 and L3 VPNs are elastic use of RTs. 
> 
> Once you depart from notion of VPN to more basic RT based building block you realize that such protection is not only much more granular but also very scalable. 
> 
> So in my suggested case RR or even remote PEs (we could define a way to propagate this maybe in flowspec v2) all what is needed is to keep min max route per RT counter when making decisions about issuing warning or stopping to propage given routes around. 
> 
> And nothing is centralized here. Every element of L2 or L3 VPNs is centralized so are the RRs. 
> 
> Such shift of direction is not to be done post WG adoption. This is fundamental enough that new draft is needed and once adopted we could discuss details and do a little fixes if WG members agree on the direction. 
> 
> See WG adoption is probing for rough consensus to make sure what we adopt at least moves into the right direction. It would be a failure to any WG to adopt a document which fundamentally defines something which is hard to implement or which goes into wrong direction. 
> 
> Thx,
> R.
> 
> 
> 
> 
> 
> 
>> On Tue, Feb 22, 2022 at 4:37 PM Aijun Wang <[email-masked]> wrote:
>> Hi, Robert:

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 22 Feb 2022 17:16:06 +0100

No.

VPN is an abstract name describing a collection of routes which meet
specific import RT policy. I am suggesting to go one level down and really
focus on how many routes we expect to get with each RT.

You know that based on SLAs signed with your customers, you know how many
routes your management subnets will inject, you know VPN topologies
resulting in very different RT marking when compared with full mesh etc ...

So it is RT which gives you most fine number of what is expected to be
received or to be send. If you focus on this you will see that even cases
you have excluded in the draft making your proposal very weak will
suddenly work fine and the ultimate goal will be accomplished.

Yes implementations will require to add per RT route limit - that can be
done per PE or globally. Such RT limit can be configured by mgmt station or
pushed via protocol extensions etc ... those are IMO good questions to be
discussed and maybe to be solved if you really see the need.

But the current proposal is not what you call "self network healing".

Thx,
R.

On Tue, Feb 22, 2022 at 5:05 PM Aijun Wang <[email-masked]>
wrote:

>> We are open to any alternative solutions/proposals and I think they can
>> be incorporated into the document even after its adoption if they can
>> increase the solution efficiency.
>>
>> Aijun Wang
>> China Telecom
>>
>> That way PE is not overloaded ever. And the logic on RRs in fact is
>> pretty simple as this counter and per peer advertise flag is already
>> present in BGP in a number of implementations.
>>

---

From: "Aijun Wang" <[email-masked]>
Date: Wed, 23 Feb 2022 09:29:01 +0800

Hi, Robert:

NO, per PE or globally “RT route limit” doesn’t work, or can’t get the same effect as the VPN Prefixes ORF mechanism.

And we can imagine your solution in one possible scenario:

Considering there are 1k PEs under one RR, and each PE is configured 1000 VRFs,

Then according to your proposal, the RR should record the RT import list for 1milliion VRFs, counter the VPN routes for each of them. 

And, in reality, there will be likely only one or two of the VRF has some issues to send the excessive VPN routes, then almost 99.9998% work on the RR is useless.

Then, I think your proposal is not the correct direction to follow.

Best Regards

Aijun Wang

China Telecom

From: Robert Raszuk <[email-masked]> 
Sent: Wednesday, February 23, 2022 12:16 AM
To: Aijun Wang <[email-masked]>
Cc: Jeffrey Haas <[email-masked]>; Gyan Mishra <[email-masked]>; idr@ietf. org <[email-masked]>; UTTARO, JAMES <[email-masked]>; Sue Hares <[email-masked]>; [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

No. 

VPN is an abstract name describing a collection of routes which meet specific import RT policy. I am suggesting to go one level down and really focus on how many routes we expect to get with each RT. 

You know that based on SLAs signed with your customers, you know how many routes your management subnets will inject, you know VPN topologies resulting in very different RT marking when compared with full mesh etc ... 

So it is RT which gives you most fine number of what is expected to be received or to be send. If you focus on this you will see that even cases you have excluded in the draft making your proposal very weak will suddenly work fine and the ultimate goal will be accomplished. 

Yes implementations will require to add per RT route limit - that can be done per PE or globally. Such RT limit can be configured by mgmt station or pushed via protocol extensions etc ... those are IMO good questions to be discussed and maybe to be solved if you really see the need. 

Thx,

R.

On Tue, Feb 22, 2022 at 5:05 PM Aijun Wang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, Robert:

Aijun Wang

China Telecom

在 2022年2月22日，23:47，Robert Raszuk <[email-masked] <mailto:[email-masked]> > 写道：

Aijun,

Thx,
R.

 

On Tue, Feb 22, 2022 at 4:37 PM Aijun Wang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, Robert: 

Aijun Wang

China Telecom

在 2022年2月22日，21:28，Robert Raszuk <[email-masked] <mailto:[email-masked]> > 写道：

Aijun,

Thx,
R.

On Tue, Feb 22, 2022 at 2:14 PM Aijun Wang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, Jeffrey:

 

Aijun Wang

China Telecom

在 2022年2月22日，20:44，Jeffrey Haas <[email-masked] <mailto:[email-masked]> > 写道：

Gyan,

 

If so... that's weird.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 23 Feb 2022 12:20:07 +0100

Aijun,

Sorry but your interpretation is not correct. There is no import nor VRFs
on the RRs.

All RRs need to do is to learn (dynamically or statically) how many routes
are expected to be received with given RT.

It can do it on ingress or on egress or both.

It is exactly like prefix-limit except the traditional prefix limit is per
session here would be per RT. And if you think about it can be applied also
on the sourcing PEs as opposed to RRs.

You keep saying the VRF limit is just one defence and you need another line
of defence. So the RT limit would be your another line of defence.

Rgs,
R.

On Wed, Feb 23, 2022 at 2:29 AM Aijun Wang <[email-masked]>
wrote:

> Hi, Robert:
>
> NO, per PE or globally “RT route limit” doesn’t work, or can’t get the
> same effect as the VPN Prefixes ORF mechanism.
>
>
>
> Then according to your proposal, the RR should record the RT import list
> for 1milliion VRFs, counter the VPN routes for each of them.
>
> And, in reality, there will be likely only one or two of the VRF has some
> issues to send the excessive VPN routes, then almost 99.9998% work on the
> RR is useless.
>
>
>
> *Sent:* Wednesday, February 23, 2022 12:16 AM
> *Cc:* Jeffrey Haas <[email-masked]>; Gyan Mishra <[email-masked]>;
> idr@ietf. org <[email-masked]>; UTTARO, JAMES <[email-masked]>; Sue Hares <
> [email-masked]>; [email-masked]

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 23 Feb 2022 18:44:54 +0000

Aijun Wang wrote on 18/02/2022 23:30:
> side effect of overwhelming information, and also gives clues to 
> operator to find the root causes of the abnormal results.

Can I make a suggestion for the security section? Consider a malicious 
actor connected to one of the PE devices in the networks in figures 1, 2 
or 3.  They can monitor how many routes come from each source.  If they 
have knowledge of the VPN infrastructure and know that the number of 
prefixes is close to the limit, they may be able to cause service 
disruption for other parties on the network by injecting enough routes 
to their PE to bring the VPN device which the largest number of prefixes 
over the limits specified in the draft for triggering prefixe removal, 
while ensuring that they aren't the party that injects the most routes.

In other words, there are situations where this proposal can be gamed by 
malicious actors to kick other parties off a VPN.

Nick

---

From: "Aijun Wang" <[email-masked]>
Date: Thu, 24 Feb 2022 10:22:47 +0800

Hi, Nick:
Let's state more clearly the reasonable trigger mechanism for VPN Prefix ORF message. 
We should know the initial aim is to position the abnormal behavior of the VRF on one PE.
Then when the VPN routes in one VRF on one PE exceeds the threshold, this PE can calculate which VRF(identified by the source RD) on which PE(identified by the source PE) contribute or led to the overflow(based on each VRF's prefix limit), then send the VPN Prefix ORF message to RR, let RR filter the BGP Updates to the offended ORF(identified by RT).
The information from other PEs, or from other VRFs on the offending PE, or to other VRFs on the offended PE will not be influenced.

Does the above explanations work for you(also for other experts that have similar concerns)? If so, we can add it into the document. 

And, to Gert:
In order to reduce the meaningless mail exchange, would you like to point out exactly what other technical concerns you have still? 
The mechanism may not be easily understood. But if the reader/experts follow the discussion closely before they make comments or jump into some conclusion directly, we can converge to the key points more quickly.

Then back to Nick's example:
If the malicious actor send the enough routes on its connected PE(offending PE) via one of the VRF(offending ORF) on this PE, and let the VRFs(offended VRFs) on other PEs(offended PEs) overflowed.
Then, we can say the offending VRF is in bad behavior, right?
Based on the above described behavior on the offended PE, such VRF/Source PE can be identified. Then the VPN Prefix ORF message is sent to the RR. RR will then filter the BGP updates to the offended VRF on the offended PE.
So, contrary to your statement, such mechanism filter exactly the source that injects the most routes. Right?

Best Regards

Aijun Wang
China Telecom

-----Original Message-----
From: Nick Hilliard <[email-masked]> 
Sent: Thursday, February 24, 2022 2:45 AM
To: Aijun Wang <[email-masked]>
Cc: [email-masked]
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Aijun Wang wrote on 18/02/2022 23:30:
> The main advantages of VPN Prefixes ORF mechanism is that it abates 
> the side effect of overwhelming information, and also gives clues to 

Can I make a suggestion for the security section? Consider a malicious actor connected to one of the PE devices in the networks in figures 1, 2 or 3.  They can monitor how many routes come from each source.  If they have knowledge of the VPN infrastructure and know that the number of prefixes is close to the limit, they may be able to cause service disruption for other parties on the network by injecting enough routes to their PE to bring the VPN device which the largest number of prefixes over the limits specified in the draft for triggering prefixe removal, while ensuring that they aren't the party that injects the most routes.

In other words, there are situations where this proposal can be gamed by malicious actors to kick other parties off a VPN.

Nick

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 24 Feb 2022 10:29:25 +0100

> to the offended ORF(identified by RT).

That is not what the draft says. The draft says that you always filter on
RD and in some cases on RD+RT.

So please do not change the facts as those facts are fundamental to
accepting or rejecting this draft.

Rgs,
R.

On Thu, Feb 24, 2022 at 3:22 AM Aijun Wang <[email-masked]>
wrote:

> Then when the VPN routes in one VRF on one PE exceeds the threshold, this
> PE can calculate which VRF(identified by the source RD) on which
> PE(identified by the source PE) contribute or led to the overflow(based on
> each VRF's prefix limit), then send the VPN Prefix ORF message to RR, let
> RR filter the BGP Updates to the offended ORF(identified by RT).
> The information from other PEs, or from other VRFs on the offending PE, or
> to other VRFs on the offended PE will not be influenced.
>
> Does the above explanations work for you(also for other experts that have
> similar concerns)? If so, we can add it into the document.
>
> And, to Gert:
> In order to reduce the meaningless mail exchange, would you like to point
> out exactly what other technical concerns you have still?
> The mechanism may not be easily understood. But if the reader/experts
> follow the discussion closely before they make comments or jump into some
> conclusion directly, we can converge to the key points more quickly.
>
> If the malicious actor send the enough routes on its connected
> PE(offending PE) via one of the VRF(offending ORF) on this PE, and let the
> VRFs(offended VRFs) on other PEs(offended PEs) overflowed.
> Based on the above described behavior on the offended PE, such VRF/Source
> PE can be identified. Then the VPN Prefix ORF message is sent to the RR. RR
> will then filter the BGP updates to the offended VRF on the offended PE.
> actor connected to one of the PE devices in the networks in figures 1, 2 or
> 3.  They can monitor how many routes come from each source.  If they have
> knowledge of the VPN infrastructure and know that the number of prefixes is
> close to the limit, they may be able to cause service disruption for other
> parties on the network by injecting enough routes to their PE to bring the
> VPN device which the largest number of prefixes over the limits specified
> in the draft for triggering prefixe removal, while ensuring that they
> aren't the party that injects the most routes.
>

---

From: Nick Hilliard <[email-masked]>
Date: Thu, 24 Feb 2022 11:45:23 +0000

> Then, we can say the offending VRF is in bad behavior, right?

No, a specific party connected to a VRF is the malicious actor. Please
remember that multiple parties may connect to a single VRF.  If there 
are other parties connected to the same VRF on the same PE, then the 
attack vector is substantially the same.

Aijun Wang wrote on 24/02/2022 02:22:
> Then when the VPN routes in one VRF on one PE exceeds the threshold,
> this PE can calculate which VRF(identified by the source RD) on which
> PE(identified by the source PE) contribute or led to the
> overflow(based on each VRF's prefix limit), then send the VPN Prefix
> ORF message to RR, let RR filter the BGP Updates to the offended
> ORF(identified by RT).

Yes, it could do this, or other things too, but these suggestions don't 
solve this particular style of problem.  They only handle some of the 
consequences.

I think you need to re-think the proposed algorithm completely in such a 
way that it has a deterministic outcome across entire routing domains 
and that it can't be abused by second-party actors. Then it needs to be 
specified much more clearly than it's specified at the moment, e.g. 
avoiding vague phrases like "PEs and RR should have some mechanisms to 
identify and control the advertisement of excessive VPN routes"

Nick

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 24 Feb 2022 10:31:11 -0500

On Thu, Feb 24, 2022 at 11:45:23AM +0000, Nick Hilliard wrote:
> remember that multiple parties may connect to a single VRF.  If
> there are other parties connected to the same VRF on the same PE,
> then the attack vector is substantially the same.

It might be helpful to build some less abstract examples.

Presume the VRF has statistics about its contributing routes, minimally
per-RD.  Let's say that there are N RDs in the VRF.

The challenge for drop strategies for oversubscribed resources is how you
decide what the drop criteria are.  One example strategy is fair allocation.

If there are N RD in the VRF, one strategy is each RD can keep 
prefix-limit/N routes each.  Effectively this is a per-RD prefix-limit.
This strategy only works well if each RD contributes a similar number of
routes.

When each RD may contribute different numbers of routes, 1/N of the resources
doesn't utilize the system efficiently.  What drop strategies can you use
for unequal distributions?

The proposal seems to simply choose the RD that has the highest number of
contributing routes.  When this is intended to mitigate a highly asymmetric
set of inputs, the choice seems easy.  RD-bad bursts to a number of
contributing routes significantly larger than the other contributors, like
receiving the full Internet table when we're expecting a small number of
routes.

Less extreme cases are not as clear.

Consider a router has a VRF-A that has an overall prefix-limit of 100
routes.  Consider that there are 4 contributing RDs.  A fair strategy would
be each RD can contribute 25 routes without penalty.

Setup:
RD-1, RD-2, RD-3: 10 routes
RD-4: 40 routes.

Utilization at start of scenario, 70%.  RD-4 is past the "fair" ratio, but
oversubscription is permitted.

Scenario 1: RD-4 increases to 80 routes.

In this circumstance, RD-4 is clearly "unfair" to RD-1, RD-2, RD-3.
Dropping RD-4 seems obvious.  This seems to match discussion from the
proposal authors to date.

Scenario 2: RD-1, RD-2, RD-3 increase to 25 routes each.
The load on the system is now at 155 routes.  RD-1, RD-2, RD-3 are still
using the theoretical fair share.  Do we drop RD-4?  I believe this is what
the proposal suggests.

Is this fair to RD-4?  They were previously permitted to burst past the fair
share and changed nothing.  RD-4 may have been operating stably at that
level for a while.

Scenario 3, different setup:
RD-1, RD-2: 26 routes
RD-3: 25 routes
RD-4: 10 routes

Utilization at start of scenario: 87%

We see that RD-1 and RD2 are past "fair" and RD-3 is at "fair".

RD-4 increases to 24 routes, less than "their fair share".  Utilization is
now at 101%.  Who do we drop?  RD-1?  RD-2?

The above scenario is an example of natural routing growth "squeeze" we tend
to see in the deployment of prefix limits.  If your threshold is too low,
natural growth or route churn within what should be acceptable tolerances
can lead to exceeding a limit.  For these reasons, implementations often
have "warn" limits and limits that have harder actions including dropping
peering sessions or discarding routes.

It should also be clear that the contrived examples above get much more
complex when we deal with the real world.  Whether route contribution per-RD
is uniform or highly asymmetrical depends on the deployment.  Sometimes the
contributor of the most distinct routes per-RD is the most important router;
e.g. the access router outside of the VPN.  This can lead to an outage for
the VPN.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Thu, 24 Feb 2022 16:51:56 +0100

Hi Jeff,

> The proposal seems to simply choose the RD that has the highest
> number of contributing routes.

That is not clearly stated.

Authors seem to also indicate to drop the last ones which exceeded the
limit in some of the messages irrespective to the percentage src RD
contribute to the threshold.

Moreover it also needs to be noticed that such drop (especially remote)
does not apply when given arriving route is imported to some other VRF and
there it does not cross the limit.

- - -

I think the first question to ask to the WG is if partial and random
connectivity better then no connectivity for a given VPN. IMO no
connectivity is much healthier.

If we converge on this then we could think about next steps. But then again
if we are taking about VPNs is this IDR topic or BESS topic ?

Thx,
R.

---

From: "Wei Wang" <[email-masked]>
Date: Fri, 25 Feb 2022 14:26:37 +0800

Hi Jeffrey,

&nbsp; &nbsp;Thanks for your valuable suggestions and scene refinement, we think the following principles can be applied to the drop strategies:

when a <RD, source PE&gt; tuple past the "fair" ratio but the prefix limit of VRF is not exceeded, PE should send warnings to the operator, and the drop strategies should not be triggered.

when a <RD, source PE&gt; tuple past the "fair" ratio&nbsp;and the prefix limit is exceeded and there is no other VRFs on offended PE need VPN routes with this RD, they should be dropped.
&nbsp; &nbsp;The above strategies is only an example, there may exist other algorithms to be explored further and should be researched in another document in details. The operators just need to make sure the algorithms in different devices consistent.&nbsp;

&nbsp; &nbsp;If you agree with the above strategies, we will update our draft accordingly to reflect them.&nbsp;
&nbsp; &nbsp;
&nbsp; &nbsp;For our considerations on the use of the principles in your mentioned scenarios, please see in-line replies with [WW].

Best Regards,
Wei
------------------&nbsp;Original&nbsp;------------------
From:                                                                                                                        "Jeffrey Haas"                                                                                    <[email-masked]&gt;;
Date:&nbsp;Thu, Feb 24, 2022 11:31 PM
To:&nbsp;"Nick Hilliard"<[email-masked]&gt;;
Cc:&nbsp;"idr"<[email-masked]&gt;;
Subject:&nbsp;Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

On Thu, Feb 24, 2022 at 11:45:23AM +0000, Nick Hilliard wrote:
&gt; Taking your last statement first:
&gt; 
&gt; &gt; Then, we can say the offending VRF is in bad behavior, right?
&gt; 
&gt; No, a specific party connected to a VRF is the malicious actor. Please
&gt; remember that multiple parties may connect to a single VRF.&nbsp; If
&gt; there are other parties connected to the same VRF on the same PE,
&gt; then the attack vector is substantially the same.

Presume the VRF has statistics about its contributing routes, minimally
per-RD.&nbsp; Let's say that there are N RDs in the VRF.

The challenge for drop strategies for oversubscribed resources is how you
decide what the drop criteria are.&nbsp; One example strategy is fair allocation.

If there are N RD in the VRF, one strategy is each RD can keep 
prefix-limit/N routes each.&nbsp; Effectively this is a per-RD prefix-limit.
This strategy only works well if each RD contributes a similar number of
routes.

When each RD may contribute different numbers of routes, 1/N of the resources
doesn't utilize the system efficiently.&nbsp; What drop strategies can you use
for unequal distributions?

The proposal seems to simply choose the RD that has the highest number of
contributing routes.&nbsp; When this is intended to mitigate a highly asymmetric
set of inputs, the choice seems easy.&nbsp; RD-bad bursts to a number of
contributing routes significantly larger than the other contributors, like
receiving the full Internet table when we're expecting a small number of
routes.

Consider a router has a VRF-A that has an overall prefix-limit of 100
routes.&nbsp; Consider that there are 4 contributing RDs.&nbsp; A fair strategy would
be each RD can contribute 25 routes without penalty.

Setup:

Utilization at start of scenario, 70%.&nbsp; RD-4 is past the "fair" ratio, but
oversubscription is permitted.

In this circumstance, RD-4 is clearly "unfair" to RD-1, RD-2, RD-3.
Dropping RD-4 seems obvious.&nbsp; This seems to match discussion from the
proposal authors to date.

Scenario 2: RD-1, RD-2, RD-3 increase to 25 routes each.
The load on the system is now at 155 routes.&nbsp; RD-1, RD-2, RD-3 are still
using the theoretical fair share.&nbsp; Do we drop RD-4?&nbsp; I believe this is what
the proposal suggests.

Is this fair to RD-4?&nbsp; They were previously permitted to burst past the fair
share and changed nothing.&nbsp; RD-4 may have been operating stably at that
level for a while.
[WW] After RD-4 past&nbsp;the "fair" ratio, router should send warnings to operators for further adjustment. If the prefix limit of VRF-A is exceeded and RD-4 still past the "fair", router will drop RD-4.

RD-4 increases to 24 routes, less than "their fair share".&nbsp; Utilization is
now at 101%.&nbsp; Who do we drop?&nbsp; RD-1?&nbsp; RD-2?
[WW] If RD-1 and RD-2 still past the "fair" when utilization is higher than 100%, router should drop them both.

The above scenario is an example of natural routing growth "squeeze" we tend
to see in the deployment of prefix limits.&nbsp; If your threshold is too low,
natural growth or route churn within what should be acceptable tolerances
can lead to exceeding a limit.&nbsp; For these reasons, implementations often
have "warn" limits and limits that have harder actions including dropping
peering sessions or discarding routes.

It should also be clear that the contrived examples above get much more
complex when we deal with the real world.&nbsp; Whether route contribution per-RD
is uniform or highly asymmetrical depends on the deployment.&nbsp; Sometimes the
contributor of the most distinct routes per-RD is the most important router;
e.g. the access router outside of the VPN.&nbsp; This can lead to an outage for
the VPN.
[WW] If we do nothing, the offending VPN routes may influence the communication among VPNs and the communication&nbsp;between PE and the router outside of the VPN. If we filter the route from the access router outside of the VPN, only the latter communication will be influenced.&nbsp;

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 25 Feb 2022 10:51:45 +0100

Hi Wei,

In respect to section 7.1 can you kindly clarify the logic on how
receiving PE will choose to fill "source PE" value when it is receiving
mixed routes both from local ASN as well as from option A or B or AB ?

Thx,
R.

On Fri, Feb 25, 2022 at 7:27 AM Wei Wang <[email-masked]> wrote:

> Hi Jeffrey,
>
>    Thanks for your valuable suggestions and scene refinement, we think the
> following principles can be applied to the drop strategies:
>
>    - when a <RD, source PE> tuple past the "fair" ratio but the prefix
>    limit of VRF is not exceeded, PE should send warnings to the operator, and
>    the drop strategies should not be triggered.
>    - when a <RD, source PE> tuple past the "fair" ratio and the prefix
>    limit is exceeded and there is no other VRFs on offended PE need VPN routes
>    with this RD, they should be dropped.
>
>    The above strategies is only an example, there may exist other
> algorithms to be explored further and should be researched in another
> document in details. The operators just need to make sure the algorithms in
> different devices consistent.
>    If you agree with the above strategies, we will update our draft
> accordingly to reflect them.
>
>    For our considerations on the use of the principles in your mentioned
> scenarios, please see in-line replies with [WW].
>
>
> Best Regards,
> Wei
> ------------------ Original ------------------
> *From:* "Jeffrey Haas" <[email-masked]>;
> *Date:* Thu, Feb 24, 2022 11:31 PM
> *To:* "Nick Hilliard"<[email-masked]>;
> *Cc:* "idr"<[email-masked]>;
> [WW] After RD-4 past the "fair" ratio, router should send warnings to
> operators for further adjustment. If the prefix limit of VRF-A is exceeded
> and RD-4 still past the "fair", router will drop RD-4.
>
> [WW] If we do nothing, the offending VPN routes may influence the
> communication among VPNs and the communication between PE and the router
> outside of the VPN. If we filter the route from the access router outside
> of the VPN, only the latter communication will be influenced.
>
> -- Jeff
>

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 25 Feb 2022 17:05:13 -0500

Wei,

On Fri, Feb 25, 2022 at 02:26:37PM +0800, Wei Wang wrote:
>    Thanks for your valuable suggestions and scene refinement, we think the following principles can be applied to the drop strategies:
> 
> when a <RD, source PE> tuple past the "fair" ratio but the prefix limit of VRF is not exceeded, PE should send warnings to the operator, and the drop strategies should not be triggered.
> 
> when a <RD, source PE> tuple past the "fair" ratio and the prefix limit is exceeded and there is no other VRFs on offended PE need VPN routes with this RD, they should be dropped.
>    The above strategies is only an example, there may exist other algorithms to be explored further and should be researched in another document in details. The operators just need to make sure the algorithms in different devices consistent. 
> 
>    If you agree with the above strategies, we will update our draft accordingly to reflect them. 

If this is your general intent for how the feature works, I think many of
the participants in the thread would appreciate such discussion becoming
part of your proposal.

Further discussion below:

>    For our considerations on the use of the principles in your mentioned scenarios, please see in-line replies with [WW].
> 
[...]

> RD-1, RD-2, RD-3: 10 routes

[...]

> [WW] After RD-4 past the "fair" ratio, router should send warnings to operators for further adjustment. If the prefix limit of VRF-A is exceeded and RD-4 still past the "fair", router will drop RD-4.

For this example, your intention is clear.

> [WW] If RD-1 and RD-2 still past the "fair" when utilization is higher than 100%, router should drop them both.

This is also clear.

> [WW] If we do nothing, the offending VPN routes may influence the communication among VPNs and the communication between PE and the router outside of the VPN. If we filter the route from the access router outside of the VPN, only the latter communication will be influenced. 

I understand your point, but believe we have different scenarios in mind
that impact the outcome.  That's fine.  I believe you understand my point
that dropping the largest may sometimes have negative impact to overall VPN
operations.  Your choice is that the negative impact of the excess routes is
more important.

-----

I want to add one final point to the scenarios I have been using for
discussion.  These examples all use a fixed number of RDs for the 1/N ratio.

For "fair" to be able to be determined, it would be necessary to know the
total number of expected RDs present at a given PE for a given VRF.  This is
effectively a form of per-RD-per-VRF prefix limit.

If you agree that this is per-RD-per-VRF prefix limit as configuration
state, I believe you now have a point of clarity you can add to your
proposal that defines its expected operation.  This permits more flexibility
than 1/N fairness where N is unknown.

-- Jeff

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 25 Feb 2022 23:42:50 +0100

Hi Jeff,

> proposal that defines its expected operation.
>

While I agree that in the context of this proposal per-RD-per-VRF *import*
prefix limit is a neat way to give a chance to ever implement this however
in the larger scope this introduces unbounded new operational complexity.

Today what is imported is all RT driven irrespective of src RD or RD
allocation schema.

Here you are proposing to introduce another fundamental to L2/L3 operation
dimension where some routes with a given RT would be import eligible and
some other routes also marked with identical RT would fail due to
associated other prefixes bearing the same RD.

RTs have been used as a functional marking. RD's sole purpose is to make
potentially overlapping prefixes within VPNs unique. So please could you
elaborate why would you not prefer to have per-RT-per-VRF *import* prefix
limit instead ?

Thx a lot,
R.

---

From: "Aijun Wang" <[email-masked]>
Date: Mon, 28 Feb 2022 09:24:19 +0800

Hi, Robert: 

Per-RD-per-VRF *import* prefix limit can identify/control the offending VPN prefixes.

Per-RT-per-VRF can’t achieve such goal.

And our consideration for your other question in another mail is the followings:

“In respect to section 7.1 can you kindly clarify the logic on how receiving PE will choose to fill "source PE" value when it is receiving mixed routes both from local ASN as well as from option A or B or AB ? ”

When the “Route Origin Community” is presented with the VPN prefixes routes, the “source PE” should be set the value of “Route Origin Community”.  
When it is not presented, the “source PE” should be set to the value of “NEXT_HOP”
 
And, To Jeffrey and all other IDR experts:
Thanks for your detail analysis and the discussions on this topic, we will update the draft in recent days to reflect the converged proposal.

Best Regards

Aijun Wang

China Telecom

From: [email-masked] <[email-masked]> On Behalf Of Robert Raszuk
Sent: Saturday, February 26, 2022 6:43 AM
To: Jeffrey Haas <[email-masked]>
Cc: idr <[email-masked]>
Subject: Re: [Idr] draft-wang-idr-vpn-prefix-orf-00.txt- Adoption call (2/4/2022 to 2/18/2022)

Hi Jeff, 

While I agree that in the context of this proposal per-RD-per-VRF *import* prefix limit is a neat way to give a chance to ever implement this however in the larger scope this introduces unbounded new operational complexity. 

Today what is imported is all RT driven irrespective of src RD or RD allocation schema. 

Here you are proposing to introduce another fundamental to L2/L3 operation dimension where some routes with a given RT would be import eligible and some other routes also marked with identical RT would fail due to associated other prefixes bearing the same RD. 

RTs have been used as a functional marking. RD's sole purpose is to make potentially overlapping prefixes within VPNs unique. So please could you elaborate why would you not prefer to have per-RT-per-VRF *import* prefix limit instead ? 

Thx a lot,

R.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 28 Feb 2022 12:02:00 +0100

Hi Aijun,

> Per-RT-per-VRF can’t achieve such goal.

It sure can. In fact it can in a much more granular fashion as RT is much
more granular then RD and since it is RT not RD which is used as key export
and import  parameter.

Operationally it would be magnitudes easier too. Some VPN may be attached
to 1000s PEs and use per vrf RD. Then on each PE under each VRF you would
need to enter 1000 lines listing each src RD and its limit. Sorry but I can
not endorse such proposal. Instead you could just provide single line
showing how many routes is expected per given import RT. Clean and easy.

And that should be done on export and not import too.

> When it is not presented, the “source PE” should be set to the value of “NEXT_HOP”

Then in option A, AB or B you have not accomplished anything what was
intended. And Route Origin is an optional extended community which
your inter-as peer may not set or strip when propagating VPN routes to
you.

> reflect the converged proposal.

I do not see that we have "converged" on anything here.

Rgs,
R.

On Mon, Feb 28, 2022 at 2:24 AM Aijun Wang <[email-masked]>
wrote:

> Hi, Robert:
>
>
>
> Per-RD-per-VRF **import** prefix limit can identify/control the offending
> VPN prefixes.
>
> “In respect to section 7.1 can you kindly clarify the logic on how
> mixed routes both from local ASN as well as from option A or B or AB ? ”
>
> *From:* [email-masked] <[email-masked]> *On Behalf Of *Robert
> Raszuk
> *Sent:* Saturday, February 26, 2022 6:43 AM
> *To:* Jeffrey Haas <[email-masked]>
> *Cc:* idr <[email-masked]>

---

From: Aijun Wang <[email-masked]>
Date: Mon, 28 Feb 2022 22:09:58 +0800

Hi, Robert:
We are talking the solution from the POV of the offended PE, but you are proposing the solution from the POV of offending PE.
And for the configuration on each VRF on each PE, there will be default QUOTA value for each VRF on the source PE. Only some special VRFs need to be explicitly configured on demand.
And for the value of “Source PE”, if the “Route Origin” is not attached, the offended PE can only signal to filter the specific  routes(identified by RD and RT)from the ASBR(for Option B). I think this is acceptable.

Aijun Wang
China Telecom

> 在 2022年2月28日，19:01，Robert Raszuk <[email-masked]> 写道：
> 
> 
> Hi Aijun, 
> 
> It sure can. In fact it can in a much more granular fashion as RT is much more granular then RD and since it is RT not RD which is used as key export and import  parameter. 
> 
> Operationally it would be magnitudes easier too. Some VPN may be attached to 1000s PEs and use per vrf RD. Then on each PE under each VRF you would need to enter 1000 lines listing each src RD and its limit. Sorry but I can not endorse such proposal. Instead you could just provide single line showing how many routes is expected per given import RT. Clean and easy. 
> 
> Then in option A, AB or B you have not accomplished anything what was intended. And Route Origin is an optional extended community which your inter-as peer may not set or strip when propagating VPN routes to you. 
> 
>> On Mon, Feb 28, 2022 at 2:24 AM Aijun Wang <[email-masked]> wrote:
>> Hi, Robert:
>> 
>>  
>>

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 28 Feb 2022 15:31:21 +0100

Hi,

We are talking the solution from the POV of the offended PE, but you are
> proposing the solution from the POV of offending PE.
>

Yes exactly. And my objective is to help you to solve the overall problem
you are trying to solve in the easiest and  operationally manageable way.
My objective is not to be stuck on one approach and only react on overflow.
My objective is to stop overflow before it even effects any PE.

> And for the configuration on each VRF on each PE, there will be default
> QUOTA value for each VRF on the source PE.
>

So tell me if you have such quota on the src PE how can any receiving PE
get more routes then this quota permits * number of src PEs ?

If you want to have two such quotas one for import and one for export I
would 100% support you in that.

So if PE under VRF X is attached to CE it is protected by max-prefix on the
PE-CE. Then it will be protected by maximum export as an extra safety
measure.

This is a local knob and no IETF draft is needed.

We could perhaps try to automate how you set this export limit
automatically and that would be fine too.

> Only some special VRFs need to be explicitly configured on demand.
>

I do not understand what you mean by the above sentence.

Thx,
R.

> And for the value of “Source PE”, if the “Route Origin” is not attached,
> the offended PE can only signal to filter the specific  routes(identified
> by RD and RT)from the ASBR(for Option B). I think this is acceptable.
>
> Aijun Wang
> China Telecom
>
>> Per-RD-per-VRF **import** prefix limit can identify/control the
>> offending VPN prefixes.
>>
>> While I agree that in the context of this proposal per-RD-per-VRF
>> *import* prefix limit is a neat way to give a chance to ever implement this
>> however in the larger scope this introduces unbounded new operational
>> complexity.
>>
>>
>>
>> Here you are proposing to introduce another fundamental to L2/L3
>> operation dimension where some routes with a given RT would be import
>> eligible and some other routes also marked with identical RT would fail due
>> to associated other prefixes bearing the same RD.
>>
>>
>>